Modules and Packages

By the end of the work on functions, your notebook had three of them: tokenise, count_words and top_words. They are all defined in that notebook, so you can use them whenever you need them there.

But what if you open another notebook? Maybe you want to count the words in a completely different set of reviews. Those functions will not be available there. You would have to copy all three definitions into the new notebook.

That is the same problem you had before you learned about functions: repeating code instead of reusing it. So how do you solve it?

A file of your own

You move the functions into a separate Python file. In Colab, open the file panel on the left side and create a new file called text_tools.py. Colab opens it in an editor right next to your notebook.

Copy the three functions into this file, and add a docstring at the top that explains what the file is for. Here is text_tools.py:

"""Clean and count the words in product reviews."""


def tokenise(review: str) -> list[str]:
    """Return the words of a review, lowercased, without . , ! or ?"""
    text = review.strip().lower()
    for mark in ".,!?":
        text = text.replace(mark, "")
    return text.split()


def count_words(tokens: list[str], stop_words: frozenset[str] = frozenset()) -> dict[str, int]:
    """Return how many times each token appears, leaving out the stop words."""
    counts: dict[str, int] = {}
    for token in tokens:
        if token not in stop_words:
            counts[token] = counts.get(token, 0) + 1
    return counts


def top_words(counts: dict[str, int], n: int = 5) -> list[tuple[str, int]]:
    """Return the n most common words as (word, count) pairs, most common first."""
    return sorted(counts.items(), key=lambda pair: pair[1], reverse=True)[:n]

Save the file.

Importing it

Now, back in the notebook, write import text_tools. That is all it takes. You can now call text_tools.tokenise on a review:

import text_tools
text_tools.tokenise("Too loud. Returned it, sadly!")

['too', 'loud', 'returned', 'it', 'sadly']

You get the same five words as before. But here is the interesting part: you never defined tokenise in this notebook. So what exactly did that import statement do? And how did Python know where to find your file?

When Python sees import text_tools, it looks for a module with that name. In this case, it finds text_tools.py.

Running the file

When Python imports that file for the first time, it executes the code inside it, from top to bottom. Your file contains three function definitions, so when Python executes those def statements, it creates three function objects, just as it did when you ran those cells in the notebook.

Python also creates a module object. That module object gives you access to the names defined in the file, and the name text_tools in your notebook now refers to it:

type(text_tools)

module

The names inside it are the three functions the file defined:

[name for name in dir(text_tools) if not name.startswith("_")]

['count_words', 'tokenise', 'top_words']

So that is what a moduleKey termA Python file that other code imports. is here: a Python file that you can import and use from somewhere else.

The dot

Remember how you accessed methods using a dot? You use the same dot notation with a module. text_tools.count_words gives you the count_words function inside the module, and text_tools.top_words gives you the other one.

Try them. Count the words in "Great kettle, great value!", then ask for the most common word:

counts = text_tools.count_words(text_tools.tokenise("Great kettle, great value!"))
text_tools.top_words(counts, 1)

[('great', 2)]

You get 'great', with a count of two. Exactly what you would expect.

The docstring you wrote at the top of the file belongs to the module too. If you call help(text_tools), it appears at the top of the help page. You can also read it directly:

text_tools.__doc__

'Clean and count the words in product reviews.'

So your functions are now reusable, and you have also made them easier to understand.

You do not always have to create modules yourself. Python already comes with a huge collection of them. Together, they are called the standard libraryKey termThe modules that come with Python: math, pathlib, csv, json, importlib..

math

One standard library module is math. Import it:

import math
math.sqrt(16)

4.0

Now the name math refers to the math module, so math.sqrt(16) gives 4.0.

So whether it is a module you created yourself or one that comes with Python, the basic idea is the same. You import it, and then use the dot to access what is inside.

Importing one name with from

There is more than one way to import something. So far, you have imported entire modules. But what if you only need one function? Suppose you only want sqrt from the math module. Instead of import math, you can write from math import sqrt:

from math import sqrt
sqrt(2)

1.4142135623730951

Now you call sqrt directly, without math in front of it, and you get approximately 1.4142. That is because Python has brought the name sqrt directly into your notebook's namespace.

A shorter name with as

Another variation you will see quite often is import math as m. This still imports the whole math module, but gives it a shorter name in your notebook:

import math as m
m.floor(7.8)

7

Same module, just a different name.

Which names an import gives you

So far you have seen modules you created yourself and modules that come with Python. There is a third category: modules and packages written by other developers. These are often called third-party packages, and they are one of the reasons Python is so useful. Instead of building everything from scratch, you can use code that other people have already written and published.

NumPy

Normally, you need to install these packages before you can import them. But Google Colab comes with quite a few popular packages already installed, and one of them is NumPy. Write import numpy as np. Here, np is just an alias for NumPy, and it is a very common convention: you will see it in almost every NumPy tutorial or notebook.

Suppose you have a list of your step counts for the week. You can pass that list to np.mean, and NumPy calculates the average for you:

import numpy as np
week = [8200, 10450, 6300, 12000, 9100, 7400, 11050]
np.mean(week)

np.float64(9214.285714285714)

The average is approximately 9214.2857. The notebook shows it wrapped in np.float64(...), which is NumPy's own float type. NumPy gets much more attention later in the course. For now, the important thing is that you can import a third-party package and use the functions it provides.

ModuleNotFoundError

What happens if you try to import something that is not installed? Try import emoji:

import emoji

ModuleNotFoundError: No module named 'emoji'

The package is not installed in this environment, so Python raises a ModuleNotFoundError, and the message says there is no module named emoji.

You can get the same error simply by misspelling a module name. Your file is called text_tools.py. What if you accidentally write import text_tool, without the "s"?

import text_tool

ModuleNotFoundError: No module named 'text_tool'

Again, a ModuleNotFoundError. So whenever you see this error, the first thing to check is the spelling. If the name is correct, check whether the package is installed and available in your current Python environment.

Where Python looks

That raises another question: how does Python actually find these modules? When you write an import statement, Python does not search randomly through every folder on your computer. It follows a search path, a list of locations Python checks when looking for modules. You can inspect it with sys.path, after import sys. The exact locations differ from one machine and environment to another.

In a typical notebook environment, one of those locations is the notebook's current working directory. That is how Python was able to find your text_tools.py file. The search path also includes locations for Python's standard library and for installed packages. Python checks the relevant locations in order and uses the first matching module it finds.

There is one more thing about modules that is especially useful when you start building your own Python projects. Every module has a special variable called __name__. That is two underscores before and after the word "name".

If you check __name__ directly in your notebook, you get '__main__':

__name__

'__main__'

But inside an imported module, __name__ normally contains that module's name. For text_tools, imported normally, it is 'text_tools':

text_tools.__name__

'text_tools'

A file that reports its name

Why does that matter? Here is an example. Create another file called shout.py. At the top, add a print call that says the file is running and shows the value of __name__. Then define a function called shout. At the bottom, add an if statement that checks whether __name__ is equal to "__main__", and if it is, calls the shout function. Here is shout.py:

"""A module that says how it was run."""
print("shout.py is running as", __name__)


def shout(text):
    return text.upper() + "!"


if __name__ == "__main__":
    print(shout("run directly"))

Running it directly

Now run this file directly. In a Colab cell you write !python shout.py; the exclamation mark tells Colab to run the line as a shell command. The command itself is:

python shout.py

shout.py is running as __main__
RUN DIRECTLY!

First, it prints that shout.py is running as __main__. Then it prints "RUN DIRECTLY!" in capital letters. So both parts ran.

Importing it

The main guard

This pattern is called the main guard. You will see it written as if __name__ == "__main__":, and it is very common in Python projects. It lets you put code inside a file that should run only when you execute that file directly. Maybe it is a quick demonstration, a test, or the starting point of a program. But when another file imports the module, that code does not run automatically.

The functions are still available, though:

shout.shout("imported")

'IMPORTED!'

So far you have worked with individual Python files. But imagine your project starts growing. You might have one module for cleaning text, another for counting words, and another for generating reports. Keeping everything in one folder can get messy.

So Python gives you a way to organize related modules into folders. That is where packages come in. A packageKey termA folder of modules, imported with dotted names: review_stats.text_tools. lets you group related modules together.

A folder with an __init__.py file

Create a folder called review_stats. Inside it, put your text_tools.py file, and also create an empty file called __init__.py. Now you have a folder containing two Python files:

ls review_stats

__init__.py
text_tools.py

Importing from a package

Here is how you import from it:

from review_stats.text_tools import tokenise
tokenise("Easy to clean!")

['easy', 'to', 'clean']

Notice the dot between review_stats and text_tools. It tells Python you are looking for the text_tools module inside the review_stats package, and from that module you want the tokenise function. So you can call tokenise directly, and you get 'easy', 'to' and 'clean'.

You can also import the module itself. With import review_stats.text_tools as tt, the name tt refers to your text_tools module. Call tt.top_words with a dictionary containing "quiet" with a count of two and "great" with a count of five, and ask for the top word:

import review_stats.text_tools as tt
tt.top_words({"quiet": 2, "great": 5}, 1)

[('great', 5)]

You get 'great', with a count of five.

What __init__.py is for

You might be wondering what the __init__.py file is for. When Python imports a regular package, it executes the package's __init__.py file as part of initializing that package. Here the file is empty, so there is nothing extra to execute.

Once you understand this folder structure, imports like numpy.random start making a lot more sense. The dot separates the parts of the import path: in that example, random is a submodule inside the NumPy package.

So packages are really just a way to organize related Python code and make it importable.

Three mistakes come up again and again when working with modules.

Editing a module after importing it

The first one is especially easy to make in notebooks. Imagine you have already imported your text_tools module. Then you open text_tools.py, add a new function called shout at the bottom, and save the file. These are the lines added to text_tools.py:

def shout(text):
    return text.upper()

Reloading

So how do you get the updated version? One option is to restart the Python session and import the module again. But there is another way. Python has a standard library module called importlib. Import it, and call importlib.reload(text_tools). This tells Python to reload your module, and now text_tools.shout works:

import importlib
importlib.reload(text_tools)
text_tools.shout("now it is there")

'NOW IT IS THERE'

So remember this when you are editing your own modules in Colab or Jupyter: saving the Python file does not automatically update an already imported module. You need to reload it or restart the session.

Wildcard imports

The second mistake is a wildcard import. You might come across code like this:

from math import *
sqrt(9)

3.0

That star means you are importing all the public names made available by the math module directly into your namespace. So you can call sqrt(9) without writing math in front of it, and you get 3.0.

But here is the problem. Imagine you are reading someone else's code, and you see sqrt(9). Where did sqrt come from? Was it imported from math? Was it defined somewhere else? With wildcard imports, that becomes harder to figure out. There is also a risk of imported names replacing names you have already defined.

That is why Python's style guide, PEP 8, recommends avoiding wildcard imports. It is usually much clearer to import the module, or to explicitly import the names you need.

A file named after a module

The third mistake is a little more subtle. Say you are writing your own Python file, and you decide to name it random.py. Seems harmless, right? But Python already has a standard library module called random.

Remember that Python checks the locations in its search path in order. In a typical project, the folder containing your own files is searched before the standard library. So when you write import random, Python might find your random.py instead of the standard library module you actually wanted, and the functions you expected from Python's random module are not there.

Here is a folder containing a random.py of your own:

def roll():
    return 4

From that folder, import random and call random.randint, a function the standard library's random module provides:

python -c "import random; random.randint(1, 6)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'random' has no attribute 'randint'

Python imported your file, which has no randint. This is called module shadowing. So avoid naming your own files after standard library modules or third-party packages you are using. Names like random.py, math.py or numpy.py can cause confusing import problems.

You started with three functions that were stuck inside one notebook. By moving them into a separate Python file, you made them reusable across notebooks, and that file became your module.

• Importing a module lets you access the names defined inside it, usually with dot notation.
• You can import an entire module, import specific names from it with from, or give it a shorter alias with as. The way you import something determines which names are available in your code.
• Python finds modules by checking the locations on its search path, sys.path, in order.
• __name__ is "__main__" when a file is run directly, and the main guard, if __name__ == "__main__":, lets you run certain code only in that case.
• A package groups modules in a folder with an __init__.py file.
• Three mistakes to watch for: forgetting to reload a module after editing it, using wildcard imports, and giving your own files names that conflict with existing modules.

Your challenge

Take the with_tax function from the work on arguments and scope:

1. Move it into a new Python file called money.py.
2. In your notebook, import it with from money import with_tax, and check that with_tax(12.50) gives you 13.5.
3. Once that is working, add a main guard to money.py. Inside it, print the result of with_tax(8.25).

The print inside the main guard should run when you execute money.py directly, with !python money.py, but not when you import it into your notebook. Give it a try.

Throughout, Colab has had Python and many popular packages installed for you. When you work on your own computer, you manage your Python environment and your project's dependencies yourself.