CI/CD Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                    CI/CD PIPELINE ARCHITECTURE                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  DEVELOPMENT            BUILD                 DEPLOYMENT            │
│  ┌──────────┐         ┌──────────────┐      ┌──────────────┐      │
│  │ Git      │────────>│ Azure DevOps │─────>│ Dev          │      │
│  │ Repository│        │ Build Pipeline│     │ Environment  │      │
│  │ (ADF Git)│         │              │      │              │      │
│  └──────────┘         │ • Validate   │      └──────┬───────┘      │
│                       │ • Package    │             │                │
│                       │ • Test       │             ▼                │
│                       └──────────────┘      ┌──────────────┐      │
│                                             │ QA           │      │
│  ┌──────────────┐                          │ Environment  │      │
│  │ ARM/Bicep    │─────────────────────────>│              │      │
│  │ Templates    │                          └──────┬───────┘      │
│  └──────────────┘                                 │                │
│                                                   ▼                │
│  ┌──────────────┐                          ┌──────────────┐      │
│  │ Terraform    │─────────────────────────>│ Production   │      │
│  │ (Optional)   │                          │ Environment  │      │
│  └──────────────┘                          └──────────────┘      │
│                                                                     │
│  ADF GIT INTEGRATION:                                               │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │  Dev Branch ──> PR ──> Main Branch ──> Publish ──> Prod     │   │
│  │                                                               │   │
│  │  • Collaborative editing in ADF Studio                       │   │
│  │  • Branch-based development                                  │   │
│  │  • PR validation and code review                             │   │
│  │  • Automated publishing to collaboration mode                │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Bicep Template Example

// Data engineering infrastructure
param location string = resourceGroup().location
param environment string = 'prod'

// Storage Account
resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: 'stdatalake${environment}${location}'
  location: location
  kind: 'StorageV2'
  sku: { name: 'Standard_LRS' }
  properties: {
    isHnsEnabled: true
    supportsHttpsTrafficOnly: true
    minimumTlsVersion: 'TLS1_2'
    encryption: {
      services: { blob: { enabled: true } }
      keySource: 'Microsoft.Storage'
    }
  }
}

// Synapse Workspace
resource synapseWorkspace 'Microsoft.Synapse/workspaces@2023-05-01' = {
  name: 'syn-${environment}-workspace'
  location: location
  identity: { type: 'SystemAssigned' }
  properties: {
    defaultDataLakeStorage: {
      accountUrl: 'https://${storageAccount.name}.dfs.core.windows.net'
      filesystem: 'synapsefs'
    }
    sqlAdministratorLogin: 'sqladmin'
  }
}

// ADF with Git integration
resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' = {
  name: 'adf-${environment}'
  location: location
  identity: { type: 'SystemAssigned' }
  properties: {
    repoConfiguration: {
      accountName: 'your-ado-org'
      projectName: 'data-engineering'
      repositoryName: 'adf-repo'
      collaborationBranch: 'main'
      rootFolder: '/'
    }
  }
}

Azure DevOps Pipeline

# azure-pipelines.yml
trigger:
  branches:
    include:
      - main
      - feature/*

stages:
  - stage: Build
    jobs:
      - job: ValidateTemplates
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: AzureCLI@2
            inputs:
              azureSubscription: 'dataengineering-subscription'
              scriptType: 'bash'
              scriptLocation: 'inlineScript'
              inlineScript: |
                az deployment group validate \
                  --resource-group rg-dataengineering-dev \
                  --template-file infra/main.bicep \
                  --parameters environment=dev

  - stage: DeployDev
    dependsOn: Build
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployToDev
        environment: 'dev'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: 'dataengineering-subscription'
                    scriptType: 'bash'
                    scriptLocation: 'inlineScript'
                    inlineScript: |
                      az deployment group create \
                        --resource-group rg-dataengineering-dev \
                        --template-file infra/main.bicep \
                        --parameters environment=dev

  - stage: DeployProd
    dependsOn: DeployDev
    condition: succeeded()
    jobs:
      - deployment: DeployToProd
        environment: 'prod'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: 'dataengineering-subscription'
                    scriptType: 'bash'
                    scriptLocation: 'inlineScript'
                    inlineScript: |
                      az deployment group create \
                        --resource-group rg-dataengineering-prod \
                        --template-file infra/main.bicep \
                        --parameters environment=prod

Interview Questions

Q1: How do you handle parameterization across environments in ADF? A: Use ADF parameters, linked services with Key Vault references, and ARM template parameters. Store environment-specific values in Key Vault and reference them via expressions.

Q2: What is the difference between ADF collaboration mode and live mode? A: Collaboration mode (Git) enables collaborative development with branching and version control. Live mode is the published, running version. Changes in collaboration mode must be published to go live.

Q3: How do you implement blue-green deployments for data pipelines? A: Deploy new pipeline version alongside existing, validate with test data, switch traffic using deployment slots or feature flags, monitor for issues, and rollback if needed.