AWS CI/CD for Data Engineering Interview Q&A

Master CI/CD pipelines for data engineering on AWS covering CodePipeline, CodeBuild, CodeDeploy, and infrastructure as code patterns.

25 min readAdvanced

Why This Matters

CI/CD for data engineering is fundamentally different from application CI/CD. Data pipelines process terabytes of data, involve complex transformations, and require validation against data quality rules before deployment. Without proper CI/CD, organizations face broken pipelines in production, data quality regressions, and lengthy rollback cycles. Mastering AWS CI/CD services enables you to build automated, repeatable deployment processes that treat data pipelines as first-class software artifacts.

Real-World Project Structure

Architecture Diagram

data-pipeline-project/
├── infrastructure/
│   ├── cdk/
│   │   ├── lib/
│   │   │   ├── pipeline-stack.ts
│   │   │   ├── glue-stack.ts
│   │   │   ├── s3-stack.ts
│   │   │   └── monitoring-stack.ts
│   │   ├── bin/
│   │   │   └── app.ts
│   │   └── cdk.json
│   └── cloudformation/
│       ├── template.yaml
│       └── parameters.json
├── pipelines/
│   ├── glue-jobs/
│   │   ├── etl_transform.py
│   │   ├── data_quality.py
│   │   └── requirements.txt
│   ├── step-functions/
│   │   └── workflow.asl.json
│   └── lambda/
│       ├── orchestrator.py
│       └── validator.py
├── tests/
│   ├── unit/
│   │   ├── test_etl_transform.py
│   │   └── test_data_quality.py
│   ├── integration/
│   │   └── test_pipeline_e2e.py
│   └── data_quality/
│       ├── test_schema.py
│       └── test_completeness.py
├── buildspec.yml
├── appspec.yml
└── pipeline-config.yaml

CI/CD Architecture Diagram

Interview Questions & Answers

Q1: How do you design a CI/CD pipeline for AWS Glue ETL jobs?

Answer:

A production CI/CD pipeline for Glue jobs follows a multi-stage approach:

Source Stage:

Store Glue job scripts in CodeCommit or GitHub
Use branch protection rules requiring PR reviews
Tag releases with semantic versioning

Build Stage:

Run pylint, flake8, and mypy for code quality
Execute unit tests with pytest against job logic
Validate Glue job configurations against schemas

Test Stage:

Deploy to a dev Glue environment using CloudFormation
Run integration tests against sample data in S3
Validate output schemas and data quality metrics

Deploy Stage:

Use CloudFormation or CDK to update Glue job definitions
Deploy to staging, then promote to production after approval
Use deployment triggers for automatic rollback on failure

Architecture Diagram

Pipeline Flow:
CodeCommit -> CodeBuild (lint/test) -> CodeDeploy (staging)
  -> Manual Approval -> CodeDeploy (production)
  -> CloudWatch Alarms -> Auto Rollback on Failure

Q2: What is the difference between CodePipeline, CodeBuild, and CodeDeploy?

Answer:

Service	Purpose	Data Engineering Use
CodePipeline	Orchestrates the CI/CD workflow	Chains build, test, and deploy stages
CodeBuild	Compiles and tests code	Runs pytest, linting, builds Docker images
CodeDeploy	Deploys applications to targets	Deploys Glue jobs, Lambda functions, Step Functions

CodePipeline is the orchestrator. CodeBuild executes build tasks. CodeDeploy handles deployment to EC2, Lambda, ECS, or on-premises. For data engineering, CodePipeline orchestrates the entire workflow while CodeBuild handles testing and CodeDeploy manages Lambda and container deployments.

Q3: How do you implement infrastructure as code for data pipelines on AWS?

Answer:

Use AWS CDK or CloudFormation to define all pipeline resources:

# CDK Example for Glue Pipeline
from aws_cdk import (
    Stack,
    aws_glue as glue,
    aws_s3 as s3,
    aws_iam as iam,
)
from constructs import Construct

class DataPipelineStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        # S3 bucket for data
        data_bucket = s3.Bucket(self, "DataBucket",
            versioned=True,
            encryption=s3.BucketEncryption.S3_MANAGED,
            lifecycle_rules=[
                s3.LifecycleRule(
                    transitions=[
                        s3.Transition(
                            storage_class=s3.StorageClass.GLACIER,
                            transition_after=Duration.days(90)
                        )
                    ]
                )
            ]
        )

        # Glue job
        glue_job = glue.CfnJob(self, "ETLJob",
            name="daily-etl-pipeline",
            role=glue_role.role_arn,
            command=glue.CfnJob.CommandProperty(
                name="glueetl",
                script_location=f"s3://{data_bucket.bucket_name}/scripts/etl.py"
            ),
            default_arguments={
                "--job-language": "python",
                "--TempDir": f"s3://{data_bucket.bucket_name}/tmp/",
                "--output_path": f"s3://{data_bucket.bucket_name}/processed/"
            },
            glue_version="3.0",
            number_of_workers=10,
            worker_type="G.1X"
        )

Q4: How do you implement blue-green deployments for data pipelines?

Answer:

Blue-green deployments for data pipelines involve maintaining two identical environments:

Blue Environment (Current Production):

Active Glue jobs processing live data
S3 buckets with current data schema
Step Functions with current workflow

Green Environment (New Version):

Deployed with new code but not processing live data
Validated against test datasets
Ready to take over traffic

Deployment Process:

Deploy new version to green environment
Run validation tests on green with historical data
Compare output metrics between blue and green
Switch traffic to green using Route53 or Load Balancer
Keep blue as rollback target for 24-48 hours
Decommission blue after validation period

Q5: How do you handle secrets and credentials in CI/CD pipelines?

Answer:

Never store secrets in code or environment variables directly. Use AWS Secrets Manager or Parameter Store:

import boto3
import json

def get_pipeline_config():
    client = boto3.client('secretsmanager')

    try:
        response = client.get_secret_value(
            SecretId='data-pipeline/prod/credentials'
        )
        secrets = json.loads(response['SecretString'])

        return {
            'db_host': secrets['host'],
            'db_password': secrets['password'],
            'api_key': secrets['api_key']
        }
    except client.exceptions.ResourceNotFoundException:
        raise ValueError("Secret not found in Secrets Manager")
    except client.exceptions.AccessDeniedException:
        raise PermissionError("No access to Secrets Manager")

In buildspec.yml, reference secrets from Parameter Store:

env:
  parameter-store:
    DB_PASSWORD: "/data-pipeline/prod/db-password"
    API_KEY: "/data-pipeline/prod/api-key"

Q6: How do you implement rollback strategies for failed data pipeline deployments?

Answer:

Rollback strategies depend on the failure type:

Immediate Rollback (CodeDeploy):

CodeDeploy automatically rolls back if CloudWatch alarms trigger
Configure alarms for error rates, latency, and data quality metrics

Data Rollback:

Use S3 object versioning to restore previous data states
Maintain schema versions in Glue Data Catalog
Use point-in-time recovery for DynamoDB tables

Pipeline Rollback:

Revert CodeCommit to previous commit
Trigger CodePipeline with previous version
Use Step Functions execution history to replay

def rollback_pipeline(pipeline_name, target_version):
    client = boto3.client('codepipeline')

    try:
        # Get current execution
        response = client.get_pipeline_state(name=pipeline_name)

        # Stop current execution if in progress
        if response['stageStates'][0]['latestExecution']['status'] == 'InProgress':
            client.stop_pipeline_execution(
                name=pipeline_name,
                pipelineExecutionId=response['stageStates'][0]['latestExecution']['pipelineExecutionId']
            )

        # Trigger with previous version
        client.start_pipeline_execution(
            name=pipeline_name,
            source_revision_override={
                'actionName': 'Source',
                'revision': target_version
            }
        )
    except Exception as e:
        print(f"Rollback failed: {e}")
        raise

Q7: How do you monitor and alert on CI/CD pipeline failures?

Answer:

Implement multi-layer monitoring:

Pipeline Level:

CloudWatch Events for pipeline state changes
SNS notifications for failed executions
Slack integration via Lambda for real-time alerts

Data Quality Level:

Glue job metrics: runtime, rows processed, error counts
Custom CloudWatch metrics for data quality scores
Alarms on quality score drops below threshold

import boto3

def create_pipeline_alarms():
    cloudwatch = boto3.client('cloudwatch')

    # Alarm on pipeline failures
    cloudwatch.put_metric_alarm(
        AlarmName='DataPipelineFailure',
        MetricName='PipelineExecutionFailed',
        Namespace='AWS/CodePipeline',
        Statistic='Sum',
        Period=300,
        EvaluationPeriods=1,
        Threshold=1,
        ComparisonOperator='GreaterThanOrEqualToThreshold',
        AlarmActions=['arn:aws:sns:us-east-1:123456789:pipeline-alerts']
    )

    # Alarm on Glue job failures
    cloudwatch.put_metric_alarm(
        AlarmName='GlueJobFailure',
        MetricName='glue.job.run.failed',
        Namespace='AWS/Glue',
        Statistic='Sum',
        Period=300,
        EvaluationPeriods=1,
        Threshold=1,
        ComparisonOperator='GreaterThanOrEqualToThreshold',
        AlarmActions=['arn:aws:sns:us-east-1:123456789:pipeline-alerts']
    )

Q8: How do you implement testing strategies for data pipelines?

Answer:

Data pipeline testing requires multiple layers:

Unit Tests:

Test transformation logic in isolation
Mock S3 and Glue dependencies
Validate output schemas

Integration Tests:

Run against sample data in dev environment
Test end-to-end pipeline execution
Validate data quality metrics

Data Quality Tests:

Schema validation using Great Expectations
Completeness checks (no null values in required fields)
Uniqueness checks (primary keys)
Referential integrity across tables

import pytest
from unittest.mock import MagicMock, patch

class TestETLTransform:
    @patch('boto3.client')
    def test_transform_removes_duplicates(self, mock_boto3):
        # Arrange
        input_data = [
            {'id': 1, 'name': 'Alice'},
            {'id': 1, 'name': 'Alice'},  # duplicate
            {'id': 2, 'name': 'Bob'}
        ]

        # Act
        result = remove_duplicates(input_data)

        # Assert
        assert len(result) == 2
        assert result[0]['id'] == 1
        assert result[1]['id'] == 2

    def test_output_schema_matches_expected(self):
        # Arrange
        sample_output = get_sample_output()

        # Act
        schema_valid = validate_schema(sample_output)

        # Assert
        assert schema_valid is True
        assert 'id' in sample_output[0]
        assert 'name' in sample_output[0]
        assert 'timestamp' in sample_output[0]

Mathematical Formulas

Pipeline Execution Time:

Architecture Diagram

Total_Time = Build_Time + Test_Time + Deploy_Time + Validation_Time

Deployment Frequency:

Architecture Diagram

Deploy_Frequency = Successful_Deployments / Time_Period

Mean Time to Recovery (MTTR):

Architecture Diagram

MTTR = Total_Recovery_Time / Number_of_Incidents

Cost Per Deployment:

Architecture Diagram

Cost_Per_Deploy = (CodeBuild_ cost + CodePipeline_cost + Data_Transfer) / Deployments

Performance Considerations

Factor	Recommendation	Impact
Build parallelization	Run unit tests and linting in parallel	40% faster builds
Caching	Cache pip dependencies in S3	60% faster installs
Artifact compression	Compress build artifacts before S3 upload	50% less transfer time
Stage parallelization	Run independent stages simultaneously	30% pipeline speedup
Resource sizing	Use larger CodeBuild instances for big projects	2x faster builds
Test data management	Use subset of production data for tests	70% faster test runs

Security Considerations

Risk	Mitigation	Implementation
Secret exposure	Use Secrets Manager	Reference in buildspec.yml
Privilege escalation	Least-privilege IAM roles	Separate roles per pipeline stage
Code injection	Input validation	Sanitize all user inputs in build scripts
Supply chain attacks	Dependency scanning	Use CodeGuru and Safety
Artifact tampering	Code signing	Sign artifacts with KMS
Network exposure	VPC endpoints	Private connectivity for all services

Common Pitfalls

Pitfall	Problem	Solution
Hardcoded credentials	Security risk	Use Secrets Manager
No rollback strategy	Stuck in failed state	Implement automated rollback
Skipping tests	Broken deployments	Enforce test gates
No monitoring	Silent failures	Add CloudWatch alarms
Manual deployments	Human error	Automate everything
No environment parity	Works in dev, breaks in prod	Use identical IaC templates

AWS CI/CD for Data Engineering Interview Q&A

AWS CI/CD for Data Engineering Interview Q&A

Why This Matters

Real-World Project Structure

CI/CD Architecture Diagram

Interview Questions & Answers

Q1: How do you design a CI/CD pipeline for AWS Glue ETL jobs?

Q2: What is the difference between CodePipeline, CodeBuild, and CodeDeploy?

Q3: How do you implement infrastructure as code for data pipelines on AWS?

Q4: How do you implement blue-green deployments for data pipelines?

Q5: How do you handle secrets and credentials in CI/CD pipelines?

Q6: How do you implement rollback strategies for failed data pipeline deployments?

Q7: How do you monitor and alert on CI/CD pipeline failures?

Q8: How do you implement testing strategies for data pipelines?

Mathematical Formulas

Performance Considerations

Security Considerations

Common Pitfalls

Quiz

See Also

Premium Content