There are two philosophies when it comes to package installation, global first and local first. Global meaning all applications that rely on a certain package have access to the same copy of the library that was installed once. Local means that each project has its own folder of dependencies installed specifically for this project and each library is installed many times. NPM, for example, uses local first strategy. Pip in Python uses global first strategy by default.
Both approaches have their benefits and drawbacks. In this post I wanted to show how to use a Python library called virtualenv to created isolated (local) python environments.
Installing virtualenv
As usual, it’s as simple as:
pip install virtualenv
Create a new virtual environment
Navigate to the folder where you want the project set up and run:
virtualenv any_name_you_want
Normally, I just name my python folder as venv so in my case I would run:
virtualenv venv
This will set up Python as well as pip that you can use locally.
Note that this command will install either Python 2 or Python 3 depending on your system set up. In my case it installs Python 3.
If I wanted to use Python 2, I can specify the path to Python version to use.
$ which python2
/usr/local/bin/python2 # <------- Use this
$ virtualenv -p /usr/local/bin/python2 venv
For the purpose of this tutorial, let’s stick with default Python 3 installation.
Activating Virtual Environment
At this point we are still running Python in usual mode and running pip install will install a package globally.
We need to “switch” into the virtual environment by running the activation command:
source venv/bin/activate
At this point your session will be updated to point to your local python and pip from your venv folder.
You can verify this by running:
(venv) $ which python
/someproject/venv/bin/python
(venv) $ which pip
/someproject/venv/bin/pip
which will show that both Python and Pip are running locally.
Using the Virtual Environment
This is really it. You can now do your work in the virtual environment and anything installed with pip install will show up in your local venv folder.
For example let’s install Pandas.
(venv) $ pip install pandas
# ... pip magic
(venv) $ ls /someproject/venv/lib/python3.6/site-packages/ | grep pandas
pandas
pandas-0.23.4.dist-info
Deactivating the Virtual Environment
When you are done with local environment, simply run deactivate in your shell to go back to the regular Python flow.
(venv) $ which python
/someproject/venv/bin/python
(venv) $ deactivate
$ which python
/usr/local/bin/python
Putting it all together
I have a repo where I show how to use Google Cloud API to convert speech to text. Let’s go ahead and clone it.
$ git clone git@github.com:akras14/speech-to-text.git
$ cd speech-to-text/
$ ls
. .gitignore requirements.txt transcripts
.. README.md slow.py venv
.git fast.py source
$ cat .gitignore
api-key.json
parts/
venv
As you can see above, in my .gitignore file I have an entry for my venv folder, because I don’t want to check-in my local dependencies. Since there is no venv folder in this cloned repo, so let’s create it and source it.
$ virtualenv venv
$ source venv/bin/activate
At this point I am running local python and pip. Let me go ahead and install all dependencies from requirements.txt file.
$ pip install -r requirements.txt
After install is done I will see all needed packages installed in my /speech-to-text/venv/lib/python3.6/site-packages/ folder.
(venv) $ cat requirements.txt
google-api-python-client==1.6.4
httplib2==0.10.3
oauth2client==4.1.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
rsa==3.4.2
six==1.11.0
SpeechRecognition==3.8.1
tqdm==4.19.5
uritemplate==3.0.0
(venv) $ ls venv/lib/python3.6/site-packages/
SpeechRecognition-3.8.1.dist-info oauth2client pyasn1_modules-0.2.1.dist-info tqdm
__pycache__ oauth2client-4.1.2.dist-info rsa tqdm-4.19.5.dist-info
apiclient pip rsa-3.4.2.dist-info uritemplate
easy_install.py pip-18.0.dist-info setuptools uritemplate-3.0.0.dist-info
google_api_python_client-1.6.4.dist-info pkg_resources setuptools-40.1.0.dist-info wheel
googleapiclient pyasn1 six-1.11.0.dist-info wheel-0.31.1.dist-info
httplib2 pyasn1-0.4.2.dist-info six.py
httplib2-0.10.3.dist-info pyasn1_modules speech_recognition
As a side note, requirements.txt is a pip convention for storing dependencies. Kind of like package.json file in NPM.
pip has a command called freeze which will list all dependencies for the project.
$ pip freeze
google-api-python-client==1.6.4
httplib2==0.10.3
oauth2client==4.1.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
rsa==3.4.2
six==1.11.0
SpeechRecognition==3.8.1
tqdm==4.19.5
uritemplate==3.0.0
To record project dependancies we just have to redirect result of pip freeze into a txt file.
pip freeze > requirements.txt
Checking this file in, will make sure that the next person using this project will have the same dependencies as we did.
In any case, now that our dependancies are installed, we are ready to run the repo. Once we are done, we can de-activate it or simply exit this shell session.
deactivate
That’s really it. In true spirit of Python – using virtualenv is very simple and effective.
Thanks for this article. Small typo fix: “folder of depEndancies” instead of “depndancies”
Thank you! It’s fixed.