Version Control with Git for Data Engineers

Why Git Matters for Data Engineers

Version control is fundamental for managing data pipeline code, configuration files, SQL transformations, and infrastructure-as-code. Git enables collaboration, rollback, and audit trails.

┌─────────────────────────────────────────────────────────────┐
│                  GIT USE CASES IN DATA ENG                    │
├─────────────────────────────────────────────────────────────┤
│  Pipeline Code        │  Python scripts, DAGs               │
│  SQL Transformations  │  dbt models, views, functions       │
│  Infrastructure       │  Terraform, CloudFormation          │
│  Configuration        │  YAML, JSON, env files              │
│  Documentation        │  README, runbooks, data catalogs    │
│  Tests                │  Unit tests, integration tests      │
│  CI/CD                │  GitHub Actions, GitLab CI           │
└─────────────────────────────────────────────────────────────┘

Git Basics

Essential Commands

# Initialize repository
git init

# Clone repository
git clone https://github.com/org/data-pipeline.git

# Check status
git status

# Add files to staging
git add file.py                    # Add specific file
git add .                          # Add all changes
git add -p                         # Interactive staging (patch mode)

# Commit
git commit -m "feat: add order processing pipeline"
git commit --amend                 # Amend last commit

# View history
git log --oneline -10              # Last 10 commits
git log --graph --oneline          # Visual branch history
git log --stat                     # Show changed files

# Diff
git diff                           # Unstaged changes
git diff --staged                  # Staged changes
git diff main..feature-branch      # Compare branches

# Remote operations
git remote add origin https://github.com/org/repo.git
git push origin main
git pull origin main
git fetch origin

Undoing Changes

# Discard unstaged changes
git checkout -- file.py
git restore file.py                # Git 2.23+

# Unstage a file
git reset HEAD file.py
git restore --staged file.py       # Git 2.23+

# Amend last commit (before push)
git commit --amend -m "new message"

# Revert a commit (creates new commit)
git revert abc123

# Reset to specific commit (DANGEROUS)
git reset --hard abc123            # Discard all changes
git reset --soft HEAD~1            # Keep changes, undo commit

Branching Strategies

Git Flow

main ─────────●──────────────────●──────────────────●────────
              │                  │                  │
develop ──────●──●──●──●──●──●──●──●──●──●──●──●──●────
              │     │     │     │        │     │
feature/1 ────●──●──●     │     │        │     │
                          │     │        │     │
feature/2 ────────────────●──●──●        │     │
                                            │     │
hotfix/1 ──────────────────────────────────●──●──●

# Create feature branch
git checkout -b feature/order-pipeline

# Work on feature
git add .
git commit -m "feat: implement order extraction"

# Push to remote
git push -u origin feature/order-pipeline

# Create pull request (via GitHub/GitLab CLI)
gh pr create --title "Add order pipeline" --body "Implements..."

# Merge to develop
git checkout develop
git merge feature/order-pipeline
git push origin develop

# Delete branch
git branch -d feature/order-pipeline
git push origin --delete feature/order-pipeline

Trunk-Based Development

main ───●──────●──────●──────●──────●──────●──────●──────
        │      │      │      │      │      │      │
feat/a ──●──●──┤      │      │      │      │      │
              │      │      │      │      │      │
feat/b ───────●──●───┤      │      │      │      │
                        │      │      │      │      │
feat/c ────────────────●──●───┤      │      │      │
                              │      │      │      │
feat/d ───────────────────────●──●───┤      │      │
                                    │      │      │
feat/e ─────────────────────────────●──●───┤      │
                                          │      │
feat/f ───────────────────────────────────●──●───┤
                                                │
feat/g ─────────────────────────────────────────●

Commit Message Conventions

Conventional Commits

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Types

Type	Description	Example
feat	New feature	`feat(pipeline): add order extraction`
fix	Bug fix	`fix: correct date parsing in transform`
docs	Documentation	`docs: update README with setup instructions`
style	Formatting	`style: apply black formatting`
refactor	Code restructuring	`refactor: extract common utilities`
test	Adding tests	`test: add unit tests for transform`
chore	Maintenance	`chore: update dependencies`
perf	Performance	`perf: optimize bulk insert`
ci	CI/CD	`ci: add GitHub Actions workflow`
build	Build system	`build: add Dockerfile`

Examples

# Simple commit
git commit -m "feat: add daily sales aggregation pipeline"

# With scope
git commit -m "feat(orders): implement CDC ingestion from PostgreSQL"

# With body
git commit -m "fix(transform): handle null values in amount column

Previously, null amounts would cause the aggregation to fail.
This fix adds a coalesce to default null amounts to 0.

Closes #123"

# Breaking change
git commit -m "feat!: change order schema to include customer segment

BREAKING CHANGE: order table now requires customer_segment column"

Pull Request Best Practices

PR Template

## Description
Brief description of changes

## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## Testing
- [ ] Unit tests added/updated
- [ ] Integration tests pass
- [ ] Manual testing completed

## Checklist
- [ ] Code follows style guidelines
- [ ] Self-review completed
- [ ] Documentation updated
- [ ] No new warnings

Review Checklist for Data Engineers

┌─────────────────────────────────────────────────────────────┐
│              CODE REVIEW CHECKLIST                            │
├─────────────────────────────────────────────────────────────┤
│  □ Code follows project style guidelines                    │
│  □ SQL queries are optimized and indexed                    │
│  □ Error handling is comprehensive                          │
│  □ Logging is appropriate and informative                   │
│  □ Data quality checks are included                         │
│  □ Tests cover edge cases                                   │
│  □ Documentation is updated                                 │
│  □ No hardcoded credentials or secrets                      │
│  □ Backward compatibility maintained                        │
│  □ Performance implications considered                      │
│  □ Rollback strategy documented                             │
└─────────────────────────────────────────────────────────────┘

CI/CD Integration

GitHub Actions Workflow

# .github/workflows/data-pipeline.yml
name: Data Pipeline CI/CD

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install black flake8 mypy
      
      - name: Run black
        run: black --check .
      
      - name: Run flake8
        run: flake8 .
      
      - name: Run mypy
        run: mypy .

  test:
    runs-on: ubuntu-latest
    needs: lint
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Run tests
        run: pytest tests/ -v --cov=src --cov-report=xml
      
      - name: Upload coverage
        uses: codecov/codecov-action@v3

  deploy:
    runs-on: ubuntu-latest
    needs: test
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      
      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Deploy to S3
        run: |
          aws s3 sync src/ s3://my-bucket/pipeline/
      
      - name: Notify deployment
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
            -d '{"text": "Pipeline deployed to production"}'

Git Hooks

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-json
      - id: check-merge-conflict
      - id: detect-private-key
  
  - repo: https://github.com/psf/black
    rev: 24.1.0
    hooks:
      - id: black
  
  - repo: https://github.com/pycqa/flake8
    rev: 7.0.0
    hooks:
      - id: flake8
  
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.8.0
    hooks:
      - id: mypy

# Custom hook: check for secrets
# .git/hooks/pre-commit
#!/bin/bash
if git diff --cached --name-only | xargs grep -l "password\|secret\|api_key" 2>/dev/null; then
    echo "ERROR: Potential secrets detected in staged files!"
    exit 1
fi

Managing Pipeline Code

Project Structure

data-pipeline/
├── .github/
│   └── workflows/
│       ├── ci.yml
│       └── deploy.yml
├── src/
│   ├── extractors/
│   │   ├── __init__.py
│   │   ├── postgres.py
│   │   └── api.py
│   ├── transformers/
│   │   ├── __init__.py
│   │   ├── orders.py
│   │   └── customers.py
│   ├── loaders/
│   │   ├── __init__.py
│   │   └── warehouse.py
│   └── utils/
│       ├── __init__.py
│       └── config.py
├── tests/
│   ├── unit/
│   └── integration/
├── sql/
│   ├── migrations/
│   └── models/
├── configs/
│   ├── dev.yaml
│   ├── staging.yaml
│   └── prod.yaml
├── docker/
│   └── Dockerfile
├── docs/
│   └── architecture.md
├── README.md
├── requirements.txt
└── pyproject.toml

dbt Project Structure

dbt_project/
├── models/
│   ├── staging/
│   │   ├── stg_orders.sql
│   │   ├── stg_customers.sql
│   │   └── stg_products.sql
│   ├── intermediate/
│   │   ├── int_orders_enriched.sql
│   │   └── int_customer_metrics.sql
│   ├── marts/
│   │   ├── fct_orders.sql
│   │   ├── dim_customers.sql
│   │   └── dim_products.sql
│   └── sources/
│       └── source.yml
├── tests/
│   ├── assert_positive_amount.sql
│   └── test_unique_orders.sql
├── macros/
│   └── generate_schema_name.sql
├── snapshots/
│   └── scd_customers.sql
├── seeds/
│   └── country_codes.csv
├── dbt_project.yml
└── profiles.yml

Monorepo vs Polyrepo

Monorepo

organization/
├── data-pipeline/
├── data-warehouse/
├── ml-models/
├── dashboards/
├── shared-libs/
└── documentation/

Pros:

Atomic commits across dependencies
Easier refactoring
Single CI/CD pipeline
Shared code visibility

Cons:

Larger repository size
Complex CI/CD
Access control harder

Polyrepo

├── order-pipeline/        (repo)
├── customer-pipeline/     (repo)
├── warehouse-transforms/  (repo)
├── ml-features/           (repo)
└── shared-utils/          (repo)