Version Control with Git for Data Engineers

Why Git Matters for Data Engineers

Version control is fundamental for managing data pipeline code, configuration files, SQL transformations, and infrastructure-as-code. Git enables collaboration, rollback, and audit trails.

Theory: How Git Works Internally

Git stores data as a directed acyclic graph (DAG) of commits. Each commit points to its parent(s) and contains a snapshot of all tracked files.

• Blob: Stores file content (compressed).
• Tree: Directory structure mapping file names to blobs.
• Commit: Points to a tree, author, message, and parent commit(s).
• HEAD: A pointer to the current branch (which points to a commit).

Git Object Model

Git Basics

Essential Commands

# Initialize repository
git init

# Clone repository
git clone https://github.com/org/data-pipeline.git

# Check status
git status

# Add files to staging
git add file.py                    # Add specific file
git add .                          # Add all changes
git add -p                         # Interactive staging (patch mode)

# Commit
git commit -m "feat: add order processing pipeline"
git commit --amend                 # Amend last commit

# View history
git log --oneline -10              # Last 10 commits
git log --graph --oneline          # Visual branch history
git log --stat                     # Show changed files

# Diff
git diff                           # Unstaged changes
git diff --staged                  # Staged changes
git diff main..feature-branch      # Compare branches

# Remote operations
git remote add origin https://github.com/org/repo.git
git push origin main
git pull origin main
git fetch origin

Undoing Changes

# Discard unstaged changes
git checkout -- file.py
git restore file.py                # Git 2.23+

# Unstage a file
git reset HEAD file.py
git restore --staged file.py       # Git 2.23+

# Amend last commit (before push)
git commit --amend -m "new message"

# Revert a commit (creates new commit)
git revert abc123

# Reset to specific commit (DANGEROUS)
git reset --hard abc123            # Discard all changes
git reset --soft HEAD~1            # Keep changes, undo commit

Undo Operations Reference

Branching Strategies

Git Flow

Git Flow Branching Strategy:

Git Flow Workflow:

1. Create develop from main
2. Create feature/* branches from develop
3. Merge features back to develop
4. Create release/* from develop when ready
5. Merge release to main and tag
6. Create hotfix/* from main for critical fixes

# Create feature branch
git checkout -b feature/order-pipeline

# Work on feature
git add .
git commit -m "feat: implement order extraction"

# Push to remote
git push -u origin feature/order-pipeline

# Create pull request (via GitHub/GitLab CLI)
gh pr create --title "Add order pipeline" --body "Implements..."

# Merge to develop
git checkout develop
git merge feature/order-pipeline
git push origin develop

# Delete branch
git branch -d feature/order-pipeline
git push origin --delete feature/order-pipeline

Trunk-Based Development

Trunk-Based Development Workflow:

Trunk-Based Best Practices:

• Keep branches short-lived (hours to days)
• Use feature flags for incomplete features
• Automate testing and deployment
• Merge to main at least daily

Branching Strategy Comparison

Commit Message Conventions

Conventional Commits

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Types

Examples

# Simple commit
git commit -m "feat: add daily sales aggregation pipeline"

# With scope
git commit -m "feat(orders): implement CDC ingestion from PostgreSQL"

# With body
git commit -m "fix(transform): handle null values in amount column

Previously, null amounts would cause the aggregation to fail.
This fix adds a coalesce to default null amounts to 0.

Closes #123"

# Breaking change
git commit -m "feat!: change order schema to include customer segment

BREAKING CHANGE: order table now requires customer_segment column"

Pull Request Best Practices

PR Template

## Description
Brief description of changes

## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## Testing
- [ ] Unit tests added/updated
- [ ] Integration tests pass
- [ ] Manual testing completed

## Checklist
- [ ] Code follows style guidelines
- [ ] Self-review completed
- [ ] Documentation updated
- [ ] No new warnings

Review Checklist for Data Engineers

CI/CD Integration

GitHub Actions Workflow

# .github/workflows/data-pipeline.yml
name: Data Pipeline CI/CD

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install black flake8 mypy
      
      - name: Run black
        run: black --check .
      
      - name: Run flake8
        run: flake8 .
      
      - name: Run mypy
        run: mypy .

  test:
    runs-on: ubuntu-latest
    needs: lint
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Run tests
        run: pytest tests/ -v --cov=src --cov-report=xml
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3

  deploy:
    runs-on: ubuntu-latest
    needs: test
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Deploy to S3
        run: |
          aws s3 sync src/ s3://my-bucket/pipeline/
      
      - name: Notify deployment
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
            -d '{"text": "Pipeline deployed to production"}'

CI/CD Pipeline Flow for Data Engineering

Git Hooks

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-json
      - id: check-merge-conflict
      - id: detect-private-key
  
  - repo: https://github.com/psf/black
    rev: 24.1.0
    hooks:
      - id: black
  
  - repo: https://github.com/pycqa/flake8
    rev: 7.0.0
    hooks:
      - id: flake8
  
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.8.0
    hooks:
      - id: mypy

Managing Pipeline Code

Project Structure

data-pipeline/
+-- .github/
|   +-- workflows/
|       +-- ci.yml
|       +-- deploy.yml
+-- src/
|   +-- extractors/
|   |   +-- __init__.py
|   |   +-- postgres.py
|   |   +-- api.py
|   +-- transformers/
|   |   +-- __init__.py
|   |   +-- orders.py
|   |   +-- customers.py
|   +-- loaders/
|   |   +-- __init__.py
|   |   +-- warehouse.py
|   +-- utils/
|       +-- __init__.py
|       +-- config.py
+-- tests/
|   +-- unit/
|   +-- integration/
+-- sql/
|   +-- migrations/
|   +-- models/
+-- configs/
|   +-- dev.yaml
|   +-- staging.yaml
|   +-- prod.yaml
+-- docker/
|   +-- Dockerfile
+-- docs/
|   +-- architecture.md
+-- README.md
+-- requirements.txt
+-- pyproject.toml

dbt Project Structure

dbt_project/
+-- models/
|   +-- staging/
|   |   +-- stg_orders.sql
|   |   +-- stg_customers.sql
|   |   +-- stg_products.sql
|   +-- intermediate/
|   |   +-- int_orders_enriched.sql
|   |   +-- int_customer_metrics.sql
|   +-- marts/
|   |   +-- fct_orders.sql
|   |   +-- dim_customers.sql
|   |   +-- dim_products.sql
|   +-- sources/
|       +-- source.yml
+-- tests/
|   +-- assert_positive_amount.sql
|   +-- test_unique_orders.sql
+-- macros/
|   +-- generate_schema_name.sql
+-- snapshots/
|   +-- scd_customers.sql
+-- seeds/
|   +-- country_codes.csv
+-- dbt_project.yml
+-- profiles.yml

Monorepo vs Polyrepo

Comparison

Decision Framework

Best Practices for Data Engineering Repos

MathSummary Takeaways

1. Git is essential — master the core commands (add, commit, push, pull, branch, merge, rebase).
2. Use conventional commits — clear, consistent commit messages (feat:, fix:, docs:) improve changelogs and traceability.
3. Branch strategically — Git Flow for scheduled releases, trunk-based for continuous deployment.
4. PRs enable review — use templates and checklists for quality; require at least one reviewer.
5. CI/CD automates quality — lint, test, and deploy automatically on merge to main.
6. Git hooks prevent issues — catch formatting errors, secrets, and type issues before they reach the repository.
7. Structure matters — organize pipeline code with clear separation of extractors, transformers, and loaders.
8. Choose repo strategy wisely — monorepo for tight coupling and shared code, polyrepo for independent team ownership.

See Also

• What is Data Engineering — Introduction to data engineering
• Python for Data Engineers — Python libraries and patterns
• Docker for Data Engineers — Containerizing data pipelines
• Command Line & Shell Scripting — Bash fundamentals
• Cloud Platforms Overview — AWS, GCP, and Azure comparison

Practice Exercises

1. Git workflow: Set up a Git repository with Git Flow branching strategy. Create a feature branch, make changes, create a PR, and merge.

2. CI/CD pipeline: Create a GitHub Actions workflow that runs linting, tests, and deploys on merge to main.

3. Pre-commit hooks: Set up pre-commit hooks that run black, flake8, and check for secrets.

4. Code review: Review a teammate's PR using the data engineering checklist. Document your findings.

5. Repository structure: Design a project structure for a data pipeline that handles 3 different data sources and loads to a data warehouse.

Interview Q&A

Q1: Why is Git essential for data engineers?

Answer: Git provides version control for pipeline code, SQL transformations, infrastructure-as-code, and configuration files. It enables collaboration through branching and PRs, rollback capabilities for production issues, and audit trails for compliance. Data engineers use Git to track changes across the entire data stack—from Python ETL scripts to Terraform configs to dbt models.

Q2: Explain the difference between Git Flow and trunk-based development. When would you use each?

Answer: Git Flow uses permanent main and develop branches with temporary feature/*, release/*, and hotfix/* branches. It's ideal for projects with scheduled releases and versioned deployments. Trunk-based development uses short-lived feature branches that merge to main frequently (daily). It's better for CI/CD-heavy environments with continuous deployment. For streaming data pipelines, trunk-based is often preferred; for batch pipelines with release cycles, Git Flow works well.

Q3: How do you handle secrets and credentials in a data engineering Git repository?

Answer: Never commit secrets to Git. Use environment variables or secret managers (AWS Secrets Manager, HashiCorp Vault). Add .env files to .gitignore. Use GitHub Actions secrets for CI/CD. Implement pre-commit hooks with detect-private-key to catch accidental commits. For infrastructure-as-code, use encrypted variables or external secret stores.

Q4: What is the purpose of a .gitignore file in a data engineering project?

Answer: .gitignore prevents large data files (CSV, Parquet), environment configs, secrets, and build artifacts from being committed. For data projects, ignore data files, virtual environments, Docker volumes, compiled models, and local configuration. This keeps the repository clean and prevents accidental exposure of sensitive data.

Q5: How do you manage database schema migrations in Git?

Answer: Use migration files versioned in Git (e.g., Alembic for PostgreSQL, dbt for transformations). Each migration is a numbered SQL file tracked in Git. Run migrations as part of CI/CD pipeline. This provides audit trails, rollback capabilities, and ensures schema changes are reviewed and tested before production deployment.

Q6: Explain the concept of atomic commits and why they matter for data pipelines.

Answer: Atomic commits contain related changes that together form a complete, logical unit of work. For data pipelines, this means grouping related extractor, transformer, and loader changes in one commit. Benefits include easier code review, simpler rollback, cleaner git bisect for debugging, and better understanding of what changed and why.

Q7: How would you set up code review for a data pipeline PR?

Answer: Use a PR template with checklist: verify SQL optimization, check error handling, ensure logging is appropriate, confirm data quality checks are included, validate test coverage, check for hardcoded credentials, verify backward compatibility, and document rollback strategy. Require at least one reviewer with data engineering expertise. Run CI checks (linting, tests, integration tests) automatically.

Q8: What is the advantage of using conventional commits in data engineering?

Answer: Conventional commits (feat:, fix:, docs:, etc.) provide consistent commit messages that enable automated changelog generation, semantic versioning, and easier filtering in git log. For data teams, this makes it clear what type of change was made—whether it's a new pipeline feature, a bug fix in transformation logic, or documentation updates.

KnowledgeCheck