Azure DevOps: Pipelines, Repos & Artifacts for Data
CI/CD automation with Azure DevOps for data engineering workloads
DevOps for Data Engineering
YAML Pipeline Example
# azure-pipelines.yml
trigger:
branches:
include:
- main
- feature/*
variables:
- group: dataengineering-variables
stages:
- stage: Build
jobs:
- job: ValidateAndTest
pool:
vmImage: 'ubuntu-latest'
steps:
- task: UsePythonVersion@0
inputs:
versionSpec: '3.9'
- script: |
pip install pytest
pytest tests/ --junitxml=test-results.xml
displayName: 'Run Unit Tests'
- task: PublishTestResults@2
inputs:
testResultsFiles: 'test-results.xml'
- stage: DeployDev
dependsOn: Build
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
jobs:
- deployment: DeployToDev
environment: 'dev'
strategy:
runOnce:
deploy:
steps:
- task: AzureCLI@2
inputs:
azureSubscription: 'dataengineering-dev'
scriptType: 'bash'
inlineScript: |
az deployment group create \
--resource-group rg-dataengineering-dev \
--template-file infra/main.bicep \
--parameters environment=dev
- stage: DeployProd
dependsOn: DeployDev
condition: succeeded()
jobs:
- deployment: DeployToProd
environment: 'prod'
strategy:
runOnce:
deploy:
steps:
- task: AzureCLI@2
inputs:
azureSubscription: 'dataengineering-prod'
scriptType: 'bash'
inlineScript: |
az deployment group create \
--resource-group rg-dataengineering-prod \
--template-file infra/main.bicep \
--parameters environment=prod
Branch Strategy
âšī¸
Pro Tip: Use branch policies to require PR reviews and successful builds before merging to main. This ensures code quality and prevents broken deployments.
Interview Questions
Q1: How do you implement CI/CD for ADF with Azure DevOps? A: 1) Enable ADF Git integration, 2) Create build pipeline for validation, 3) Use ARM/Bicep templates for infrastructure, 4) Create release pipeline with environment promotion, 5) Implement approval gates for production.
Q2: What is the difference between build and release pipelines? A: Build pipelines compile, test, and package code. Release pipelines deploy artifacts to environments with approval gates and deployment strategies.
Q3: How do you handle rollback in Azure DevOps? A: Use deployment slots for blue-green deployments. Keep previous version artifacts. Implement automated rollback triggers on failure. Use infrastructure as code for environment recreation.