There are two philosophies when it comes to package installation, global first and local first. Global meaning all applications that rely on a certain package have access to the same copy of the library that was installed once. Local means that each project has its own folder of dependencies installed specifically for this project and each library is installed many times. NPM, for example, uses local first strategy. Pip in Python uses global first strategy by default.
Both approaches have their benefits and drawbacks. In this post I wanted to show how to use a Python library called virtualenv to created isolated (local) python environments.
As usual, it’s as simple as:
pip install virtualenv
Create a new virtual environment
Navigate to the folder where you want the project set up and run:
Normally, I just name my python folder as
venv so in my case I would run:
This will set up Python as well as pip that you can use locally.
Note that this command will install either Python 2 or Python 3 depending on your system set up. In my case it installs Python 3.
If I wanted to use Python 2, I can specify the path to Python version to use.
$ which python2 /usr/local/bin/python2 # <------- Use this $ virtualenv -p /usr/local/bin/python2 venv
For the purpose of this tutorial, let’s stick with default Python 3 installation.
Activating Virtual Environment
At this point we are still running Python in usual mode and running
pip install will install a package globally.
We need to “switch” into the virtual environment by running the activation command:
At this point your session will be updated to point to your local
pip from your
You can verify this by running:
(venv) $ which python /someproject/venv/bin/python (venv) $ which pip /someproject/venv/bin/pip
which will show that both Python and Pip are running locally.
Using the Virtual Environment
This is really it. You can now do your work in the virtual environment and anything installed with
pip install will show up in your local
For example let’s install Pandas.
(venv) $ pip install pandas # ... pip magic (venv) $ ls /someproject/venv/lib/python3.6/site-packages/ | grep pandas pandas pandas-0.23.4.dist-info
Deactivating the Virtual Environment
When you are done with local environment, simply run
deactivate in your shell to go back to the regular Python flow.
(venv) $ which python /someproject/venv/bin/python (venv) $ deactivate $ which python /usr/local/bin/python
Putting it all together
I have a repo where I show how to use Google Cloud API to convert speech to text. Let’s go ahead and clone it.
$ git clone email@example.com:akras14/speech-to-text.git $ cd speech-to-text/ $ ls . .gitignore requirements.txt transcripts .. README.md slow.py venv .git fast.py source $ cat .gitignore api-key.json parts/ venv
As you can see above, in my
.gitignore file I have an entry for my
venv folder, because I don’t want to check-in my local dependencies. Since there is no
venv folder in this cloned repo, so let’s create it and source it.
$ virtualenv venv $ source venv/bin/activate
At this point I am running local
pip. Let me go ahead and install all dependencies from
$ pip install -r requirements.txt
After install is done I will see all needed packages installed in my
(venv) $ cat requirements.txt google-api-python-client==1.6.4 httplib2==0.10.3 oauth2client==4.1.2 pyasn1==0.4.2 pyasn1-modules==0.2.1 rsa==3.4.2 six==1.11.0 SpeechRecognition==3.8.1 tqdm==4.19.5 uritemplate==3.0.0 (venv) $ ls venv/lib/python3.6/site-packages/ SpeechRecognition-3.8.1.dist-info oauth2client pyasn1_modules-0.2.1.dist-info tqdm __pycache__ oauth2client-4.1.2.dist-info rsa tqdm-4.19.5.dist-info apiclient pip rsa-3.4.2.dist-info uritemplate easy_install.py pip-18.0.dist-info setuptools uritemplate-3.0.0.dist-info google_api_python_client-1.6.4.dist-info pkg_resources setuptools-40.1.0.dist-info wheel googleapiclient pyasn1 six-1.11.0.dist-info wheel-0.31.1.dist-info httplib2 pyasn1-0.4.2.dist-info six.py httplib2-0.10.3.dist-info pyasn1_modules speech_recognition
As a side note,
requirements.txt is a
pip convention for storing dependencies. Kind of like
package.json file in NPM.
pip has a command called
freeze which will list all dependencies for the project.
$ pip freeze google-api-python-client==1.6.4 httplib2==0.10.3 oauth2client==4.1.2 pyasn1==0.4.2 pyasn1-modules==0.2.1 rsa==3.4.2 six==1.11.0 SpeechRecognition==3.8.1 tqdm==4.19.5 uritemplate==3.0.0
To record project dependancies we just have to redirect result of
pip freeze into a txt file.
pip freeze > requirements.txt
Checking this file in, will make sure that the next person using this project will have the same dependencies as we did.
In any case, now that our dependancies are installed, we are ready to run the repo. Once we are done, we can de-activate it or simply exit this shell session.
That’s really it. In true spirit of Python – using virtualenv is very simple and effective.