---
jupyter:
  jupytext:
    notebook_metadata_filter: all,-language_info
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.2'
      jupytext_version: 1.13.7
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

# Where does Python look for modules?

See:

* [Python docs on
  sys.path](https://docs.python.org/3/library/sys.html#sys.path);
* [Python Module of the Week on: Import
  Path](https://pymotw.com/3/sys/imports.html#import-path).

Let’s say we have written a Python module and saved it as `a_module.py`, in
a directory called `code`.

We would normally do this with a text editor, but, for illustration, here we write out the module file using the Jupyter / IPython ``%%file`` magic command:

```{python}
# %%file code/a_module.py
""" This is a_module
"""

def a_func():
    return 99

print('Finished importing a_module.py')
```


We also have a script called `a_script.py` in a directory called `scripts`:


```{python}
# %%file scripts/a_script.py
""" This is a_script
"""

import a_module

print('Result of a_func is:', a_module.a_func())
```

At the moment, `a_script.py` will fail with:

```{python tags=c("raises-exception")}
run scripts/a_script.py
```

Above we ran the script within the Python process of the notebook, but we can also run the script in the terminal.  Here we are using the `%%bash` command at the top of the cell to run a terminal on Linux or Mac.  This may not work on Windows.

Notice that running the script this way gives the same error, for the same reason:

```{bash tags=c("raises-exception")}
python3 scripts/a_script.py
```

When Python hits the line `import a_module`, it tries to find a package or a
module called `a_module`.  A package is a directory containing modules, but we
will only consider modules for now.  A module is a file with a matching
extension, such as `.py`.  So, Python is looking for a file `a_module.py`, and
not finding it.

We will see the same effect at the interactive Python console, or in
Jupyter or IPython:

```{python tags=c("raises-exception")}
import a_module
```

## Python looks for modules in 'sys.path'

Python has a simple algorithm for finding a module with a given name, such as
`a_module`.  It looks for a file called `a_module.py` in the directories
listed in the variable `sys.path`.

```{python}
import sys

# Show sys.path
sys.path
```


The `a_module.py` file is in the `code` directory, and this directory is
not in the `sys.path` list.

`sys.path` is just a Python list, like any other:

```{python}
type(sys.path)
```

That means we can make the import work in our notebook, by appending the
`code` directory to the `sys.path` list:

```{python}
sys.path.append('code')

# Now the import will work
import a_module
```

There are various ways of making sure a directory is *always* on the Python
`sys.path` list when you run Python, including.

One of them is making the module part of an installable package, and install
it — see: [making a Python package](https://packaging.python.org/distributing)
— but we don't cover that here.


Now we have imported the module into this Python process, the import will work correctly in the script, executed within this Python process:

```{python}
run scripts/a_script.py
```

However, if we run the script in its own new terminal, we still get the error, because we aren't using the notebook Python process, and we therefore haven't successfully imported `a_module.py`:

```{bash tags=c("raises-exception")}
python3 scripts/a_script.py
```

As a crude solution to the problem above, you can do what we've done here, and
put the directory containing the module into the Python `sys.path` list, at
the top of the files that need it:

```{python}
# %%file scripts/a_script.py
""" This is a_script

We've made sure a_module is on the Python path this time.
"""

import sys
sys.path.append('code')

import a_module

print('Result of a_func is:', a_module.a_func())
```

Then:

```{bash}
python3 scripts/a_script.py
```

The simple `append` above will only work when running the script from a
directory containing the `code` subdirectory.  For example, here we are
running a few commands in the terminal, to show that the script fails if we
run it from another directory:

```{bash tags=c("raises-exception")}
mkdir another_dir
cd another_dir
# Run the script, but from the new directory.
python3 ../scripts/a_script.py
```

This is because the directory `code` that we specified is a relative path,
and therefore Python looks for the `code` directory in the current working
directory.

To make the hack work when running the code from any directory, you could use
some [path manipulation](path_manipulation.Rmd) on the [__file__ variable](dunders.Rmd):

```{python}
# %%file scripts/a_script.py
""" This is a_script

Another more general way of making sure the code directory is on the Python
path.
"""

from pathlib import Path

# Directory containing this script.
MY_DIRECTORY = Path(__file__).parent
# Code directory is in the directory above the one containing the script.
CODE_DIRECTORY = MY_DIRECTORY / '..' / 'code'
print('code directory is', str(CODE_DIRECTORY))

# Put this directory on the path.
# sys.path expects strings, not Path objects.
import sys
sys.path.append(str(CODE_DIRECTORY))

import a_module

print('Result of a_func is:', a_module.a_func())
```

Now the module import does work from this directory, or from `another_dir`

```{bash}
# Running from this directory
python3 scripts/a_script.py
```

```{bash}
# From another_directory
cd another_dir
python3 ../scripts/a_script.py
```
