Isolating Project Dependencies With Virtualenv
Python includes a powerful packaging system to manage the module dependencies of your programs. You’ve probably used it to install third-party packages with the pip package manager command.
One confusing aspect of installing packages with pip is that it tries to install them into your global Python environment by default.
Sure, this makes any new packages you install available globally on your system, which is great for convenience. But it also quickly turns into a nightmare if you’re working with multiple projects that require different versions of the same package.
For example, what if one of your projects needs version 1.3 of a library while another project needs version 1.4 of the same library?
When you install packages globally there can be only one version of a Python library across all of your programs. This means you’ll quickly run into version conflicts—just like the Highlander did.
And it gets worse. You might also have different programs that need different versions of Python itself. For example, some programs might still run on Python 2 while most of your new development happens in Python 3. Or, what if one of your projects needs Python 3.3, while everything else runs on Python 3.6?
Besides that, installing Python packages globally can also incur a security risk. Modifying the global environment often requires you to run the pip install command with superuser (root/admin) credentials. Because pip downloads and executes code from the internet when you install a new package, this is generally not recommended. Hopefully the code is trustworthy, but who knows what it will really do?
Virtual Environments to the Rescue
The solution to these problems is to separate your Python environments with so-called virtual environments. They allow you to separate Python dependencies by project and give you the ability to select between different versions of the Python interpreter.
A virtual environment is an isolated Python environment. Physically, it lives inside a folder containing all the packages and other dependencies, like native-code libraries and the interpreter runtime, that a Python project needs. (Behind the scenes, those files might not be real copies but symbolic links to save memory.)
To demonstrate how virtual environments work, I’ll give you a quick walkthrough where we’ll set up a new environment (or virtualenv, as they’re called for short) and then install a third-party package into it.
Let’s first check where the global Python environment currently resides. On Linux or macOS, we can use the which command-line tool to look up the path to the pip package manager:
$ which pip3
/usr/local/bin/pip3
I usually put my virtual environments right into my project folders to keep them nice and separate. But you could also have a dedicated “python-environments” directory somewhere to hold all of your environments across projects. The choice is yours.
Let’s create a new Python virtual environment:
$ python3 -m venv ./venv
This will take a moment and will create a new venv folder in the current directory and seed it with a baseline Python 3 environment:
$ ls venv/
bin include lib pyvenv.cfg
If you check the active version of pip (with the which command), you’ll see it’s still pointing to the global environment, /usr/local/bin/pip3 in my case:
(venv) $ which pip3
/usr/local/bin/pip3
This means if you install packages now, they’d still end up in the global Python environment. Creating a virtual environment folder isn’t enough—you’ll need to explicitly activate the new virtual environment so that future runs of the pip command reference it:
$ source ./venv/bin/activate
(venv) $
Running the activate command configures your current shell session to use the Python and pip commands from the virtual environment instead.
Notice how this changed your shell prompt to include the name of the active virtual environment inside parentheses:(venv). Let’s check which pip executable is active now:
(venv) $ which pip3
/Users/dan/my-project/venv/bin/pip3
As you can see, running the pip3 command would now run the version from the virtual environment and not the global one. The same is true for the Python interpreter executable. Running python from the command-line would now also load the interpreter from the venv folder:
(venv) $ which python
/Users/dan/my-project/venv/bin/python
Note that this is still a blank slate, a completely clean Python environment. Running pip list will show an almost empty list of installed packages that only includes the baseline modules necessary to support pip itself:
(venv) $ pip list
pip (9.0.1)
setuptools (28.8.0)
Let’s go ahead and install a Python package into the virtual environment now. You’ll want to use the familiar pip install command for that:
(venv) $ pip install schedule
Collecting schedule
Downloading schedule-0.4.2-py2.py3-none-any.whl
Installing collected packages: schedule
Successfully installed schedule-0.4.2
You’ll notice two important changes here. First, you won’t need admin permissions to run this command any longer. And second, installing or updating a package with an active virtual environment means that all files will end up in a subfolder in the virtual environment’s directory.
Therefore, your project dependencies will be physically separated from all other Python environments on your system, including the global one. In effect, you get a clone of the Python runtime that’s dedicated to one project only.
By running pip list again, you can see that the schedule library was installed successfully into the new environment:
(venv) $ pip list
pip (9.0.1)
schedule (0.4.2)
setuptools (28.8.0)
If we spin up a Python interpreter session with the python command, or run a standalone .py file with it, it will use the Python interpreter and the dependencies installed into the virtual environment—as long as the environment is still active in the current shell session.
But how do you deactivate or “leave” a virtual environment again? Similar to the activate command, there’s a deactivate command that takes you back to the global environment:
(venv) $ deactivate
$ which pip3
/usr/local/bin
Using virtual environments will help keep your system uncluttered and your Python dependencies neatly organized. As a best practice, all of your Python projects should use virtual environments to keep their dependencies separate and to avoid version conflicts.
Understanding and using virtual environments also puts you on the right track to use more advanced dependency management methods like specifying project dependencies with requirements.txt files.
If you’re looking for a deep dive on this subject with additional productivity tips, be sure to check out my Managing Python Dependencies course available on dbader.org.
Key Takeaways
- Virtual environments keep your project dependencies separated. They help you avoid version conflicts between packages and different versions of the Python runtime.
- As a best practice, all of your Python projects should use virtual environments to store their dependencies. This will help avoid headaches.