Python Virtual Environments — Isolating Your Projects

Python Virtual Environments — Isolating Your Projects

Every Python project has dependencies. Without isolation, installing package version 2.0 for one project can break another project that requires version 1.0. Virtual environments solve this by giving each project its own independent set of installed packages.

Learning Objectives

By the end of this tutorial, you will be able to:

1. Explain why virtual environments are essential for Python development
2. Create and activate virtual environments using venv and virtualenv
3. Manage conda environments for data science and multi-language projects
4. Export and install dependencies with pip freeze and requirements.txt
5. Use pyproject.toml for modern dependency specification
6. Manage projects with Poetry and Pipenv
7. Apply environment best practices for production and collaboration
8. Avoid common virtual environment mistakes

Why Virtual Environments

The Dependency Conflict Problem

Without virtual environments, every package you install goes into a shared global directory:

# Project A needs Django 4.2
pip install Django==4.2

# Project B needs Django 3.2 (legacy project)
pip install Django==3.2

# Now Project A is broken because Django was overwritten to 3.2

Virtual environments prevent this by creating isolated Python interpreters, each with its own packages directory.

Benefits of Virtual Environments

The venv Module

Python 3.3+ includes venv in the standard library. It is the recommended way to create virtual environments for most projects.

Creating a Virtual Environment

# Create a virtual environment named "venv"
python -m venv venv

# Common naming conventions
python -m venv .venv       # hidden directory (prefixed with dot)
python -m venv env         # another common name

This creates a directory containing:

• A copy of the Python interpreter
• A Lib/site-packages directory for installed packages
• A Scripts (Windows) or bin (macOS/Linux) directory with activation scripts
• A pyvenv.cfg configuration file

Activating the Environment

Windows (PowerShell):

.\venv\Scripts\Activate.ps1

Windows (Command Prompt):

.\venv\Scripts\activate.bat

macOS/Linux:

source venv/bin/activate

After activation, your prompt shows the environment name:

(venv) C:\Users\you\myproject>

Verify isolation:

(venv) $ which python
# Shows the venv path, not the global Python

Deactivating the Environment

deactivate

Installing Packages Inside the Environment

(venv) $ pip install requests
(venv) $ pip list
Package    Version
---------- -------
requests   2.31.0
urllib3    2.2.0

Removing a Virtual Environment

Simply delete the directory — no uninstallation needed:

deactivate
rmdir /s /q venv       # Windows
rm -rf venv            # macOS/Linux

virtualenv

virtualenv is a third-party tool that predates venv. It offers additional features and supports older Python versions.

pip install virtualenv
virtualenv venv
virtualenv --python=python3.11 venv  # specify version

Key Differences: venv vs virtualenv

Conda Environments

Conda manages non-Python dependencies (like CUDA and MKL) and supports multiple languages — popular in data science.

# Create an environment
conda create --name myproject python=3.11 numpy pandas

# Activate/deactivate
conda activate myproject
conda deactivate

# Export and import
conda env export > environment.yml
conda env create -f environment.yml

# Remove
conda env remove --name myproject

The environment.yml File

name: myproject
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.11
  - numpy=1.24.3
  - pandas=2.0.3
  - pip:
    - flask==2.3.3

Use conda for GPU support, non-Python libraries, or complex scientific stacks. Use venv for everything else.

pip freeze and requirements.txt

The standard way to document dependencies:

# Export all installed packages with versions
(venv) $ pip freeze > requirements.txt

# Install exact versions on another machine
(venv) $ pip install -r requirements.txt

Best Practices for requirements.txt

# Good — exact versions for reproducibility
Django==4.2.11
requests==2.31.0
numpy==1.24.3

# Bad — unpinned versions
Django
requests
numpy

Separate dev and production dependencies:

# requirements.txt (production)
Django==4.2.11
gunicorn==21.2.0

# requirements-dev.txt (development)
-r requirements.txt
pytest==8.1.1
black==24.3.0

pyproject.toml — The Modern Approach

PEP 518 introduced pyproject.toml as a standard way to define project metadata and build dependencies.

[build-system]
requires = ["setuptools>=68.0", "wheel"]
build-backend = "setuptools.backends._legacy:_Backend"

[project]
name = "myproject"
version = "0.1.0"
requires-python = ">=3.9"
dependencies = [
    "requests>=2.28,<3.0",
    "flask>=2.3",
]

[project.optional-dependencies]
dev = [
    "pytest>=8.0",
    "black>=24.0",
]

pip install -e .           # Install in editable mode
pip install -e ".[dev]"    # Install with dev dependencies

Poetry

Poetry is a dependency management and packaging tool that uses pyproject.toml exclusively.

pip install poetry
poetry new myproject       # Create new project
cd existing-project && poetry init  # Initialize existing

Managing Dependencies

poetry add requests
poetry add flask==2.3.3
poetry add --group dev pytest
poetry remove requests
poetry show --tree

The poetry.lock File

Poetry generates a poetry.lock file with exact resolved versions and hashes for reproducibility.

poetry install               # Install from lock file
poetry run python main.py    # Run in environment
poetry shell                 # Open shell in environment

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.31"
flask = "^2.3"

[tool.poetry.group.dev.dependencies]
pytest = "^8.1"
black = "^24.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Pipenv

Pipenv combines pip and virtualenv into a single tool, using Pipfile and Pipfile.lock.

pip install pipenv
pipenv --python 3.11        # Create environment
pipenv install flask        # Install package
pipenv install --dev pytest # Dev dependency
pipenv shell                # Activate shell
pipenv run python main.py   # Run command
pipenv lock                 # Lock dependencies
pipenv check                # Check for vulnerabilities

The Pipfile

[packages]
flask = "*"
requests = "==2.31.0"

[dev-packages]
pytest = "*"

[requires]
python_version = "3.11"

Environment Best Practices

Always Use Virtual Environments

# NEVER install packages globally for projects
pip install flask  # BAD

# ALWAYS create a virtual environment first
python -m venv venv
source venv/bin/activate
pip install flask  # GOOD

Add .gitignore Entries

venv/
.venv/
env/
__pycache__/
*.py[cod]

Document Your Environment

pip freeze > requirements.txt
git add requirements.txt
git commit -m "Update dependencies"

Use Consistent Python Versions

[project]
requires-python = ">=3.9,<4.0"

Separate Dev Dependencies

Keep your production environment minimal — only production packages in requirements.txt.

Common Mistakes

Mistake 1: Installing Packages Globally

# WRONG — installs into the system Python
pip install django

# RIGHT — always activate a virtual environment first
python -m venv venv
source venv/bin/activate
pip install django

Mistake 2: Forgetting to Activate Before Installing

# SOLUTION: Always check after activating
(venv) $ which python   # Should show venv path
(venv) $ pip list

Mistake 3: Committing the Virtual Environment to Git

# WRONG
git add venv/

# RIGHT
git add requirements.txt

Fix: Add venv/ to .gitignore immediately after creating it.

Mistake 4: Using Different Python Versions

# SOLUTION: Always verify after activating
(venv) $ python --version
Python 3.11.7

Mistake 5: Not Pinning Dependencies

# SOLUTION: Always pin exact versions
pip freeze > requirements.txt

Practice Exercises

Exercise 1: Create and Use a Virtual Environment

Task: Create a virtual environment, install Flask, verify isolation, and generate a requirements.txt.

python -m venv venv
source venv/bin/activate
pip install flask
pip list                        # Should show Flask
python -c "import flask; print(flask.__version__)"
deactivate
python -c "import flask"        # Should raise ImportError

Exercise 2: Reproduce an Environment

Task: Given a requirements.txt, create a fresh environment and install all dependencies.

python -m venv clean-env
source clean-env/bin/activate
pip install -r requirements.txt
pip list
diff <(pip freeze) requirements.txt

Exercise 3: Migrate to Poetry

Task: Convert a project using requirements.txt to use Poetry.

pip install poetry
cd myproject
poetry init
cat requirements.txt | while read line; do
    poetry add "$line"
done
cat pyproject.toml
poetry run python main.py
rm requirements.txt
git add pyproject.toml poetry.lock
git commit -m "Migrate from requirements.txt to Poetry"

Key Takeaways

• Virtual environments isolate project dependencies — each project gets its own packages without conflicts
• venv is built-in — use it for most Python 3.3+ projects
• virtualenv offers speed — faster creation and additional features
• conda handles non-Python deps — ideal for data science and GPU computing
• Always pip freeze > requirements.txt — pin exact versions for reproducibility
• pyproject.toml is the modern standard — use it for project metadata and dependencies
• Poetry and Pipenv provide complete workflows — lock files and dependency resolution in one tool
• Never install packages globally — always work inside a virtual environment
• Commit requirements.txt or pyproject.toml — never commit the virtual environment directory
• Separate dev and production dependencies — keep production environments minimal