TDM 40100: Project 2 - Basic Python, Environment & Dependency Management, Documentation

In programming, dependency means there are external libraries and packages that are used for project features to work. There are several reasons why we need to manage dependencies. First is reproducibility. This means regardless of the way the code is ran, we obtain the same results every time. This also means if there is a problem and multiple people are looking at the project, the same issue will occur. The importance of reproducibility shows in all stages of development: you will make it easier for yourself if you can recreate the setting for all code to work, and for others if there are multiple people working on it.

The common issue that arises from no dependency management is version conflicts: two or more libraries used requires different versions than the one you installed because all of them are globally installed locally.

Thus, we can use virtual environments, to create a separate space for all of one project’s dependencies so that regardless of what we have in our system’s global installation or what other projects use, the one we are working on runs smoothly.

We will try creating a virtual environment using Conda. Open your terminal or command prompt and navigate to the directory where you want to set up your project. Then run the following command (replace env_proj with your preferred environment name if desired):

This will create a new conda environment called env_proj that explicitly installs the specified version of python into the environment. You can name it anything, but it should be easily recognizable and relevant.

Use below code to activate the environment.

Your conda environments are located in the directory ~/.conda/envs/ which we can check out by doing either of these below:

However, if we run (which you should always do if you are running into python environment issues):

We will still see this as our current python:

Our shell is still defaulting to run from the seminar python environment, which in most cases is likely what we want when working on Anvil. However, for this project we want to practice creating a new environment from scratch.

The reason it still defaults to run from the seminar environment is due to our shell’s PATH environment variable which you can see by running:

It is helpful to do some more research on how PATH works in your shell but for this project we can fix the issue by simply prepending our PATH with our new conda environment like so (change the name of the environment as necessary):

If you rerun the echo $PATH you should see that the path to your conda environment is at the front which means your shell will default to that python first.

Now when we run which python we should see:

Now we should be able to use the environment like normal on Anvil!

You may also know about pip which is another python package management tool in Python for libraries or modules that are not in the Python standard library; it searches through Python Package Index (PyPI), which is a public repository of open source packages that can be installed. Developers create setup containing metadata of the package and build distribution files, then upload to PyPI. When install command is ran by the user, pip locates the package in PyPI, resolve dependencies, and download wheels.

There is an analogous version of PyPI and set of commands for Conda which function about the same. Note that some packages or versions of packages may only be on one or the other, however all pip commands will work in a conda so you can install packages from PyPI into your conda environment. We will touch more on Conda later but for now we will go over the various pip commands:

There are multiple options available for most pip commands. We recommend you to check the documentation for full listings. (for example: pip.pypa.io/en/stable/cli/pip_install/ for pip install)

Don’t do it yet, but deactivating the conda virtual environment is very simple with:

We have one last step before being able to use this environment in our Jupyter Notebook - we need to make it a kernel!

For every other project we have been using the seminar kernel, which is a special form of a python environment that allows it run in a Jupyter Notebook.

Creating a kernel is actually very simple. Make sure you have your desired environment activated, then we can first install the ipykernel package and then install our environment as a kernel like so:

If you wait a minute or so, your brand new kernel should appear as an option in the menu in the top right where it normally says seminar!

Now, let’s open a Jupyter notebook using our brand new kernel and load the dataset in the environment we created.

If you try to import a library you did not install into your environment, we can see a similar error as above when trying to load up the dataset.

To fix this, head back into your command line and install pandas into your environment:

Now we should have pandas installed and you can import the dataset.

Refer to the deliverables below for what to do with the dataset.

pip list and pip freeze both output the installed Python packages. Try running both:

You’ll notice that the formatting are different. pip list will return a table format with separate columns for packages and current version. pip freeze returns a list in a requirement format (package-name==version), compatible with:

One can install all dependencies needed for a project using requirements.txt. On our end, we can create the file with:

Another way we can create a requirements file is through pipreqs.

And we can confirm it with:

Which should show something like:

Or this if it was not installed correctly:

If it all worked then we can run:

. is to indicate current directory. pipreqs will scan all .py files and get all import x and from x import y statements, find PyPI package name, and your installed version. Alphabetical ordering is used to write package==version into requirements.txt.

Depending on the situation, there are flags we can use with pipreqs, including

Again, we recommend you to check documentations to see all available options.

We created a conda environment in Question 1, but we have only covered pip so far.

Conda is another widely used environment management system, with the main difference from a pip virtual environment being that it works across multiple languages and is not limited to Python packages. It is also fast since we are installing precompiled binary packages on Conda; users have less worries when it comes to compiling extension modules of libraries written in different languages. Another difference is that while most environments are created and used at the scope of the project, Conda environments can be used across various projects and be activated from any location.

We already went over some of these in the first problem but we will do it again here:

Replace <env> with your environment name. You can also specify Python version by adding 'python=version' at the end.

Activate with:

Deactivate with:

To check what is installed in the conda environment, use the conda list command.

Note that you can check package list in a different Conda environment with

Remove existing environment with:

It will remove the environment, dependencies, and packages.

You can check list of other commands at: docs.conda.io/projects/conda/en/stable/commands/index.html

We can install with:

Again, we want our environments to be reproducible. Conda uses .yml file to do so, which is a text file using YAML formatting (very widely used for creating software project configuration files). Below command that outputs a file called environment.yml containing environment information regarding name, installed packages, versions, and other configuration details.

The file structure will look something like:

In the environment we created, let’s create some visualizations from our dataset (this should also be a refresher on basic python visualization and methods). Here is an example:

Above is a graphical representation of the annual USA carbon dioxide emission.

Read and understand what each columns represent here: github.com/owid/co2-data/blob/master/owid-co2-codebook.csv

We will be writing a function that outputs a graphical representation of CO2 emission by sector for a given year. If you look at the columns of the dataset, it provides information on the annual CO2 emission for each sources (coal, cement, gas, oil, and other industry causes).

Additionally, once you have your code written, you will write a docstring for this. Docstring is a way to document your code by describing usage and functionality using string literal. They appear first in a module or a function, and we declare them with three single or double quotes.

Documentation is important for several reasons. - Documentation explains not only about the code, but also gives insight into why we have it, functionality, and various specifics of the function or a class. This is helpful to both yourself as you code, and to others who might see your code, or work on a project with you to understand implementation details well. Efficient collaboration is one of the big part. - It supports debugging or problem solving by providing references and information about the program. It will help reduce time or minimize certain errors.

We will document using Google Style Docstrings. This formatting includes description, function arguments, returns, and raises. Read section 3.8 of in google.github.io/styleguide/pyguide.html.

Here is an example:

In this question, we are going to try out pdoc, a tool for automatically generating API documentation.

First, we’ll briefly take a look at python scripts since we will be using that for pdoc. Python script is a file with .py extension and it can be run in any way as long as Python is installed. Some ways they are ran include integrated development environment (IDE), text editors, command lines and terminal, interactive mode, or even from another script or web browsers. The given script is read by Python line by line.

We can run scripts in terminal by:

If using a text editor or an IDE, Python should be available (otherwise you can get an extension if text editor) and then can be ran.

Notebook content can be exported as a .py module with below command (nbconvert can convert notebook documents to other formats too):

From the py module, pdoc scans our python functions and classes and extract the docstring documentation. HTML page is created from that.

To use pdoc, run:

Creating an HTML file, we can run:

You should see a .html file created in under docs directory.

Here is a basic example. pdoc should output something like below for you:

There are customizations we can add to pdoc. For example, adding logo like below:

Sometimes, we need to make changes to our environment, for example adding more packages or needing a newer version of an existing package. For us, we just added a new pdoc install.

Here is how to make changes in a conda environment. Run below command after editing environment.yml as needed. It will update the existing environment by comparing packages and version in the file with the installed ones; if there are any packages that do not exist in the environment but listed in environment.yml, it will install them, and likewise if there are packages not used anymore that are present, it will remove them (specified with --prune).