Python Virtual Environments

The Iron Law of Python Management says we should always create a new virtual environment for each python project.

At the time of writing, PyPI (Python Package Index) has over 480,000 packages listed. PyPI does not guarantee that all the packages listed are compatible and can be installed together.

Virtual environments provide us a way to isolate different python packages from one another. Some reasons why you may want virtual environments:

Install packages with conflicting dependencies
Test how the code behaves with a different set of package versions
Share your coding environment with someone else
Different package requirements for different python projects

1 Built-in Python `venv` module

Python has shipped the venv module since Python 3.3. As long as you are using a supported version of Python, then you do not need to install any more tools to create a virtual environment.

Tip

In general, you do not want to install any packages in your base python environment. This can lead to confusion to where packages are installed as you switch around different virtual environments.

2 Create a `venv` Virtual Environment

Because of the Iron Law of Python Management, each project (i.e., folder) will have their own python venv.

2.1 Python Project Folder

First we need to make sure we are in the folder our python project exists in. You can confirm your location with:

pwd

The result should give you the path to your current Python project.

2.2 Python Version

pyenv allows us to install and use different versions of python. You can see all the versions you have installed with:

pyenv versions

You can also see what python you are using with:

pyenv which python

This will list the full path to the python binary that is currently being used. The currently activated python version will be used to create your venv. This is why you want to make sure the environment is clean (i.e., no extraneous python packages installed).

In this example, we will use Python 3.12.0.

pyenv shell 3.12.0

2.3 Create `venv`

The general command for creating a venv is:

python -m venv <PATH TO NEW VENV>

We will typically use .venv or venv for the venv name. So, the above command will look like this if we were to use venv as our venv name.

python -m venv venv

This will create a venv folder in your current directory that will store all the packages and dependencies we will install.

Note

There are pros and cons with how you name your venv.

If you use the .venv name with the . in the beginning, the venv folder that will be created will be hidden because of the leading . in the .venv name. Because it’s hidden, it may not always be readily apparent that the current project has a venv virtual environment.
If you use the venv name so the folder is not hidden, you will always see the venv folder in the project.

Whether or not you use venv or .venv, these names have become standardized where many other tools will know to look for these folder names when listing virtual environments.

You can also choose to name your venv the name of your project, e.g., python -m venv my_project. This will make it explicit and clear what environment you are working in if you are jumping between project folders with a common venv environment. However, because the my_project name is not standard and fairly arbitrary, not every tool will recognize you have a python venv virtual environment in your directory.

The venv folder also contains a link to the Python version that will be used. This is why it’s important to:

Use the correct python version before creating the venv
Make sure the original python environment does not have anything installed in it

The venv python module will install pip for us by default. You can also see from the tree diagram below, how all the different ways we can call python are linked together.

my_python_project % tree -L 4 .
.
└── venv
    ├── bin
    │   ├── Activate.ps1
    │   ├── activate
    │   ├── activate.csh
    │   ├── activate.fish
    │   ├── pip
    │   ├── pip3
    │   ├── pip3.12
    │   ├── python -> /Users/danielchen/.pyenv/versions/3.12.0/bin/python
    │   ├── python3 -> python
    │   └── python3.12 -> python
    ├── include
    │   └── python3.12
    ├── lib
    │   └── python3.12
    │       └── site-packages
    └── pyvenv.cfg

3 Activate

With our venv named venv created, we need to “activate” it.

MacOS / *nix / Git Bash
Windows

source venv/bin/activate

venv/bin/Activate.ps1

This will typically prepend the name of the virtual environment to your terminal prompt. Since we named our venv the name venv we should see venv in the beginning of our prompt.

Before activate:

After:

(venv) %

4 Install Packages

Now that we have our venv virtual environment activated, we are finally able to install packages. Let’s install the Python pandas library.

4.1 Starting with an Empty Environment

Before we do that, let’s prove that our environment is empty, and pandas is not installed.

We can see that pip freeze returns nothing. Meaning nothing is currently installed in this environment.

(venv) % pip freeze
(venv) %

If we load up the python interpreter, we also cannot import the pandas library.

(venv) % python
Python 3.12.0 (main, Oct  3 2023, 15:47:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>> exit()

(venv) %

5 Install Packages into Environment

Now we can use pip to install pandas

pip install pandas

This will install pandas along with any dependencies it needs.

(venv) % pip install pandas
Collecting pandas
  Obtaining dependency information for pandas from https://files.pythonhosted.org/packages/38/1b/e425daceff79695e67d115230bdeb57bbdd6cfff8c46d532e4e64d3dc966/pandas-2.1.1-cp312-cp312-macosx_11_0_arm64.whl.metadata
  Using cached pandas-2.1.1-cp312-cp312-macosx_11_0_arm64.whl.metadata (18 kB)
Collecting numpy>=1.26.0 (from pandas)
  Obtaining dependency information for numpy>=1.26.0 from https://files.pythonhosted.org/packages/7a/72/6d1cbdf0d770016bc9485f9ef02e73d5cb4cf3c726f8e120b860a403d307/numpy-1.26.0-cp312-cp312-macosx_11_0_arm64.whl.metadata
  Using cached numpy-1.26.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (53 kB)
Collecting python-dateutil>=2.8.2 (from pandas)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1 (from pandas)
  Obtaining dependency information for pytz>=2020.1 from https://files.pythonhosted.org/packages/32/4d/aaf7eff5deb402fd9a24a1449a8119f00d74ae9c2efa79f8ef9994261fc2/pytz-2023.3.post1-py2.py3-none-any.whl.metadata
  Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.1 (from pandas)
  Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Using cached pandas-2.1.1-cp312-cp312-macosx_11_0_arm64.whl (10.6 MB)
Using cached numpy-1.26.0-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB)
Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas
Successfully installed numpy-1.26.0 pandas-2.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 six-1.16.0 tzdata-2023.3

Now if we run pip freeze you will see pandas, the version installed, and all of its installed dependencies.

(venv) % pip freeze
numpy==1.26.0
pandas==2.1.1
python-dateutil==2.8.2
pytz==2023.3.post1
six==1.16.0
tzdata==2023.3

We can now import pandas without the ModuleNotFoundError.

(venv) % python
Python 3.12.0 (main, Oct  3 2023, 15:47:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>>

5.1 Install Packages as You Need

As long as you are in the correct venv, you can pip install any package you need from the Python Package Index (PyPI).

pip install matplotlib plotnine seaborn

6 Save Your Environment to `requirements.txt`

We’ve been using pip freeze to see what packages are installed in the current environment. We can actually save the contents of this output to a requirements.txt so the installed packages and versions are documented. We can then version control and/or give this file to other people for them to replicate our venv virtual environment.

pip freeze > requirements.txt

This will save a requirements.txt file into our current working directory.

7 Deactivate

Once you’re done with your current python venv, make sure you run deactivate so you do not pip install packages you do not need in your project.

deactivate

This will exit out of your venv, and you should see your prompt change back.

In the venv:

(venv) % deactivate

Deactivated:

my_python_project %

And we’re back to our original package environment

% python
Python 3.12.0 (main, Oct  3 2023, 15:47:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'

8 Install Packages from `requirements.txt`

If you or someone else wants to set up a new venv with a requirements.txt file, make sure you follow the steps on this page to create a new venv first.

Then, with the new venv activated, you can install all the packages and versions from the requirements.txt with:

pip install -r requirements.txt

This is one way reproducible programming environments are created in Python.

9 Conclusion

It’s important that every python project has its own virtual environment. Ideally it also has the environment saved into a requirements.txt file. This is the Iron Law of Python Management.

--- title: Python Virtual Environments --- The [Iron Law of Python Management](https://solutions.posit.co/write-code/minimum-viable-python/installing-packages/#the-iron-law-of-python-management) says we should _always_ create a new virtual environment for each python project. At the time of writing, PyPI (Python Package Index) has over 480,000 packages listed. PyPI does not guarantee that all the packages listed are compatible and can be installed together. Virtual environments provide us a way to isolate different python packages from one another. Some reasons why you may want virtual environments: 1. Install packages with conflicting dependencies 2. Test how the code behaves with a different set of package versions 3. Share your coding environment with someone else 4. Different package requirements for different python projects ## Built-in Python `venv` module Python has shipped the [`venv` module](https://docs.python.org/3/library/venv.html) since Python 3.3. As long as you are using a supported version of Python, then you do not need to install any more tools to create a virtual environment. ::: {.callout-tip} In general, you do not want to install any packages in your base python environment. This can lead to confusion to where packages are installed as you switch around different virtual environments. ::: ## Create a `venv` Virtual Environment Because of the [Iron Law of Python Management](https://solutions.posit.co/write-code/minimum-viable-python/installing-packages/#the-iron-law-of-python-management), each project (i.e., folder) will have their own python `venv`. ### Python Project Folder First we need to make sure we are in the folder our python project exists in. You can confirm your location with: ```sh pwd ``` The result should give you the path to your current Python project. ### Python Version `pyenv` allows us to install and use different versions of python. You can see all the versions you have installed with: ```sh pyenv versions ``` You can also see what python you are using with: ```sh pyenv which python ``` This will list the full path to the python binary that is currently being used. The currently activated python version will be used to create your `venv`. This is why you want to make sure the environment is clean (i.e., no extraneous python packages installed). In this example, we will use Python `3.12.0`. ```sh pyenv shell 3.12.0 ``` ### Create `venv` The general command for creating a `venv` is: ```sh python -m venv <PATH TO NEW VENV> ``` We will typically use `.venv` or `venv` for the venv name. So, the above command will look like this if we were to use `venv` as our `venv` name. ```sh python -m venv venv ``` This will create a `venv` folder in your current directory that will store all the packages and dependencies we will install. ::: {.callout-note} There are pros and cons with how you name your `venv`. - If you use the `.venv` name with the `.` in the beginning, the `venv` folder that will be created will be hidden because of the leading `.` in the `.venv` name. Because it's hidden, it may not always be readily apparent that the current project has a `venv` virtual environment. - If you use the `venv` name so the folder is not hidden, you will always see the `venv` folder in the project. Whether or not you use `venv` or `.venv`, these names have become standardized where many other tools will know to look for these folder names when listing virtual environments. - You can also choose to name your `venv` the name of your project, e.g., `python -m venv my_project`. This will make it explicit and clear what environment you are working in if you are jumping between project folders with a common `venv` environment. However, because the `my_project` name is not standard and fairly arbitrary, not every tool will recognize you have a python `venv` virtual environment in your directory. ::: The `venv` folder also contains a link to the Python version that will be used. This is why it's important to: 1. Use the correct python version before creating the `venv` 2. Make sure the original python environment does not have anything installed in it The `venv` python module will install `pip` for us by default. You can also see from the `tree` diagram below, how all the different ways we can call `python` are linked together. ```sh my_python_project % tree -L 4 . . └── venv ├── bin │ ├── Activate.ps1 │ ├── activate │ ├── activate.csh │ ├── activate.fish │ ├── pip │ ├── pip3 │ ├── pip3.12 │ ├── python -> /Users/danielchen/.pyenv/versions/3.12.0/bin/python │ ├── python3 -> python │ └── python3.12 -> python ├── include │ └── python3.12 ├── lib │ └── python3.12 │ └── site-packages └── pyvenv.cfg ``` ## Activate With our `venv` named `venv` created, we need to "activate" it. ::: {.panel-tabset} ## MacOS / *nix / Git Bash ```sh source venv/bin/activate ``` ## Windows ```powershell venv/bin/Activate.ps1 ``` ::: This will typically prepend the name of the virtual environment to your terminal prompt. Since we named our `venv` the name `venv` we should see `venv` in the beginning of our prompt. Before activate: ```sh % ``` After: ```sh (venv) % ``` ## Install Packages Now that we have our `venv` virtual environment activated, we are finally able to install packages. Let's install the Python `pandas` library. ### Starting with an Empty Environment Before we do that, let's prove that our environment is empty, and `pandas` is not installed. We can see that `pip freeze` returns nothing. Meaning nothing is currently installed in this environment. ```sh (venv) % pip freeze (venv) % ``` If we load up the `python` interpreter, we also cannot import the `pandas` library. ```sh (venv) % python Python 3.12.0 (main, Oct 3 2023, 15:47:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pandas Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'pandas' >>> exit() (venv) % ``` ## Install Packages into Environment Now we can use `pip` to install `pandas` ```sh pip install pandas ``` This will install `pandas` along with any dependencies it needs. ```{.sh} (venv) % pip install pandas Collecting pandas Obtaining dependency information for pandas from https://files.pythonhosted.org/packages/38/1b/e425daceff79695e67d115230bdeb57bbdd6cfff8c46d532e4e64d3dc966/pandas-2.1.1-cp312-cp312-macosx_11_0_arm64.whl.metadata Using cached pandas-2.1.1-cp312-cp312-macosx_11_0_arm64.whl.metadata (18 kB) Collecting numpy>=1.26.0 (from pandas) Obtaining dependency information for numpy>=1.26.0 from https://files.pythonhosted.org/packages/7a/72/6d1cbdf0d770016bc9485f9ef02e73d5cb4cf3c726f8e120b860a403d307/numpy-1.26.0-cp312-cp312-macosx_11_0_arm64.whl.metadata Using cached numpy-1.26.0-cp312-cp312-macosx_11_0_arm64.whl.metadata (53 kB) Collecting python-dateutil>=2.8.2 (from pandas) Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) Collecting pytz>=2020.1 (from pandas) Obtaining dependency information for pytz>=2020.1 from https://files.pythonhosted.org/packages/32/4d/aaf7eff5deb402fd9a24a1449a8119f00d74ae9c2efa79f8ef9994261fc2/pytz-2023.3.post1-py2.py3-none-any.whl.metadata Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB) Collecting tzdata>=2022.1 (from pandas) Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB) Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas) Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Using cached pandas-2.1.1-cp312-cp312-macosx_11_0_arm64.whl (10.6 MB) Using cached numpy-1.26.0-cp312-cp312-macosx_11_0_arm64.whl (13.7 MB) Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB) Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas Successfully installed numpy-1.26.0 pandas-2.1.1 python-dateutil-2.8.2 pytz-2023.3.post1 six-1.16.0 tzdata-2023.3 ``` Now if we run `pip freeze` you will see `pandas`, the version installed, and all of its installed dependencies. ```sh (venv) % pip freeze numpy==1.26.0 pandas==2.1.1 python-dateutil==2.8.2 pytz==2023.3.post1 six==1.16.0 tzdata==2023.3 ``` We can now `import` pandas without the `ModuleNotFoundError`. ```sh (venv) % python Python 3.12.0 (main, Oct 3 2023, 15:47:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pandas as pd >>> ``` ### Install Packages as You Need As long as you are in the correct `venv`, you can `pip install` any package you need from the [Python Package Index (PyPI)](https://pypi.org/). ```sh pip install matplotlib plotnine seaborn ``` ## Save Your Environment to `requirements.txt` We've been using `pip freeze` to see what packages are installed in the current environment. We can actually save the contents of this output to a `requirements.txt` so the installed packages and versions are documented. We can then version control and/or give this file to other people for them to replicate our `venv` virtual environment. ```sh pip freeze > requirements.txt ``` This will save a `requirements.txt` file into our current working directory. ## Deactivate Once you're done with your current python `venv`, make sure you run `deactivate` so you do not `pip install` packages you do not need in your project. ```sh deactivate ``` This will exit out of your `venv`, and you should see your prompt change back. In the `venv`: ```sh (venv) % deactivate ``` Deactivated: ```sh my_python_project % ``` And we're back to our original package environment ```sh % python Python 3.12.0 (main, Oct 3 2023, 15:47:53) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pandas Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'pandas' ``` ## Install Packages from `requirements.txt` If you or someone else wants to set up a new `venv` with a `requirements.txt` file, make sure you follow the steps on this page to create a new `venv` first. Then, with the new `venv` activated, you can install all the packages and versions from the `requirements.txt` with: ```sh pip install -r requirements.txt ``` This is one way reproducible programming environments are created in Python. ## Conclusion It's important that _every_ python project has its own virtual environment. Ideally it also has the environment saved into a `requirements.txt` file. This is the [Iron Law of Python Management](https://solutions.posit.co/write-code/minimum-viable-python/installing-packages/#the-iron-law-of-python-management).

1 Built-in Python venv module

2 Create a venv Virtual Environment

2.1 Python Project Folder

2.2 Python Version

2.3 Create venv

3 Activate

4 Install Packages

4.1 Starting with an Empty Environment

5 Install Packages into Environment

5.1 Install Packages as You Need

6 Save Your Environment to requirements.txt

7 Deactivate

8 Install Packages from requirements.txt

9 Conclusion

1 Built-in Python `venv` module

2 Create a `venv` Virtual Environment

2.3 Create `venv`

6 Save Your Environment to `requirements.txt`

8 Install Packages from `requirements.txt`