Setting Python and its virtual environments right – using pyenv and virtualenv

TL;DR: Setup process

Python is one of the most used programming languages. It has a simple syntax which makes it a perfect choice for people to learn to program. It is also widely used in many areas of software development.

This is a brief article about why you should use virtual environments and a short introduction to pyenv that help you manage different python versions and environments. Apart from the installation instructions, which are in line with a macOS, the commands can be run on Linux distros with little to no changes.

So, is setting up python hard? No, it's quite easy. Now, what makes it something worth writing an article about is that most Operating Systems come with python installed and the fact that you should not use or mess with it.

For example, macOS is shipped with Python 2.

Importance of virtualenvs

To answer the question of why using the system installed python might create problems, let us understand something about the Python programming language itself.

All your python projects have their own set of dependencies and Python has its unique way of resolving these dependencies. Due to the decisions made about the storing and resolving of these dependencies, some interesting problems arise.

When you install a third-party package, it is stored in the location pointed by site.getsitepackages.

# On macOS
import site
print(site.getsitepackages())
[
  '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages',
  '/Library/Python/3.7/site-packages',
  '/AppleInternal/Library/Python/3.7/site-packages',
  '/AppleInternal/Tests/Python/3.7'
]

The new dependency would be stored in one of those locations.

Why does all this matter you ask? Well, because the interesting part is that all your python projects residing on the machine, will use the same directories to store and retrieve packages.

Say, you have a project called Bike and another project called Car. Assume that both the projects have the project Engine and their dependency. Further, both projects need a different version of the dependency-say Bike needs version 1 and Car needs version 2.

In the above scenario, Python would lift its hands as it cannot differentiate between versions in site-packages directory. So, both version 1 and version 2 would be stored at the same location, with the same name. Thus, both Car and Bike would have to use the same Engine, which is as bad as it sounds.

Going back to why you should not use system python-you risk breaking your system if you install a different version of a dependency.

So, what we need is an isolated environment for each of our project.

Enter scene, virtual environments

The concept behind virtual environments is simple-every project gets its own directories to store their dependencies.

There are a great set of tools that offer this functionality and here is a great StackOverflow answer that provides a comparison between them.

Using pyenv

pyenv is used to isolate Python versions. It allows us to easily download and switch between different versions of python.

Managing Python versions

On a mac, you can install pyenv like so...

brew install pyenv
# I use Homebrew for installing the tools needed for my development

pyenv will be added to your path after installation.

Once you are one with the installation add the following line to .bash_profile or .zshrc if you use zsh.

if command -v pyenv 1>/dev/null 2>&1; then eval "$(pyenv init -)"; fi
# Reload the file
source ~/.bash_profile # .zshrc for zsh

pyenv allows you to install any version of python available out there. You can check the list the available version by running the following command

pyenv install --list

Once you have found the version you want you can install it like so...

pyenv install p.q.r # replace p.q.r with the actual number

You can check the versions installed on your system by running the following command

pyenv versions
# A * indicates the currently activated version

You can switch your current terminal to another Python version with:

pyenv shell p.q.r

virtualenv

virtualenv is a tool that creates an isolated Python environment for each of your projects.

Instead of installing and using virtualenv directly, we'll use the dedicated pyenv plugin pyenv-virtualenv which will make things a bit easier. Install it via Homebrew.

brew install pyenv-virtualenv

After installation, add the following line to your .bash_profile or .zshrc.

if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi
# And reload it with:
source ~/.bash_profile # .zshrc for zsh

Now, let's say you have a project called amazing. You can set up a virtualenv for that project and the Python version it uses (replace p.q.r with the actual version you want):

pyenv virtualenv p.q.r amazing

See the list of virtualenvs you created with:

pyenv virtualenvs

To use your project's virtualenv, you need to activate it first (in every terminal where you are working on your project)

pyenv activate amazing
pyenv deactivate # to deactivate

Now when you install something, it will get installed in that virtualenv's folder, and not conflict with other projects.

A handy thing to remember is that you can set a default virtualenv for a project by running

pyenv local amazing

Next time you enter that project's directory, pyenv will automatically load the virtualenv for you.

Closing notes

I hope this article was useful to you in setting up your python environment and has conveyed why is it important to use virtualenv.

The python community has a lot of tools to help you get around these problems. As you progress, be sure to keep an eye out for different tools that are available out there and learn about them to use them towards a more productive you.