AWS Data CI/CD & DevOps

Master CI/CD for AWS data engineering including CodePipeline, CDK, testing strategies, deployment patterns, and MLOps integration.

18 min readAdvanced

Why This Matters

Data pipelines are fundamentally different from application code. They process massive volumes of data, integrate with multiple systems, and must maintain data quality guarantees. Without CI/CD, teams face manual deployment errors, inconsistent environments, slow release cycles, and difficulty rolling back failed changes.

CI/CD ensures that data pipelines are tested, validated, and deployed consistently across environments. Organizations with mature CI/CD practices deploy 10x more frequently with 50% fewer failures. The combination of CodePipeline, CodeBuild, and CloudFormation provides a robust platform for automating data infrastructure deployments.

Architecture Diagram

Why CI/CD Matters in Data Engineering

Core benefits:

Repeatability: Deploy the same configuration across dev, staging, and production
Auditability: Track every change with version control and deployment history
Speed: Reduce deployment time from days to minutes
Reliability: Catch errors before they reach production through automated testing
Scalability: Deploy changes across hundreds of pipelines simultaneously

Key Differences from Application CI/CD

Aspect	Application CI/CD	Data Pipeline CI/CD
Testing	Unit, integration, E2E	+ Data quality, schema validation
State	Stateless	Often stateful (backfills, idempotency)
Rollback	Simple revert	Data may need restoration
Environments	Isolated	May share data stores
Monitoring	Application metrics	+ Data freshness, SLA compliance

Production Code: CloudFormation Template

AWSTemplateFormatVersion: '2010-09-09'
Description: Data Pipeline CI/CD Infrastructure

Parameters:
  Environment:
    Type: String
    AllowedValues: [dev, staging, production]
  ProjectName:
    Type: String

Resources:
  # CodeCommit Repository
  SourceRepo:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: !Sub '${ProjectName}-etl-scripts'

  # CodeBuild Project
  BuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: !Sub '${ProjectName}-build'
      ServiceRole: !GetAtt BuildRole.Arn
      Artifacts:
        Type: CODEPIPELINE
      Environment:
        Type: LINUX_CONTAINER
        Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0
        ComputeType: BUILD_GENERAL1_MEDIUM
      Source:
        Type: CODEPIPELINE
        BuildSpec: buildspec.yaml

  # CodePipeline
  Pipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      Name: !Sub '${ProjectName}-pipeline'
      RoleArn: !GetAtt PipelineRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: SourceAction
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: '1'
              Configuration:
                RepositoryName: !Ref SourceRepo
                BranchName: main

        - Name: Build
          Actions:
            - Name: BuildAction
              ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: '1'
              Configuration:
                ProjectName: !Ref BuildProject

        - Name: Deploy
          Actions:
            - Name: DeployAction
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: '1'
              Configuration:
                ActionMode: CHANGE_SET_REPLACE
                StackName: !Sub '${ProjectName}-${AWS::StackName}'
                ChangeSetName: !Sub '${ProjectName}-changeset'
                TemplatePath: 'BuildOutput::template.yaml'
                Capabilities: CAPABILITY_IAM

  # Glue Job
  ETLJob:
    Type: AWS::Glue::Job
    Properties:
      Name: !Sub '${ProjectName}-etl-${Environment}'
      Role: !GetAtt GlueRole.Arn
      GlueVersion: '4.0'
      NumberOfWorkers: 10
      WorkerType: G.1X
      Command:
        Name: glueetl
        ScriptLocation: !Sub 's3://${ScriptsBucket}/etl/main.py'
      DefaultArguments:
        '--job-language': python
        '--TempDir': !Sub 's3://${DataBucket}/tmp/'
        '--enable-metrics': 'true'
        '--enable-continuous-cloudwatch-log': 'true'

  # IAM Roles
  GlueRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: glue.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole

Outputs:
  PipelineArn:
    Value: !GetAtt Pipeline.Arn
  RepositoryUrl:
    Value: !GetAtt SourceRepo.Arn

Production Code: AWS CDK

from aws_cdk import (
    Stack,
    aws_glue as glue,
    aws_s3 as s3,
    aws_codecommit as codecommit,
    aws_codebuild as codebuild,
    aws_codepipeline as codepipeline,
    aws_codepipeline_actions as actions,
    aws_iam as iam,
)
from constructs import Construct
import os

class DataPipelineCICDStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        # S3 Buckets
        scripts_bucket = s3.Bucket(self, "ScriptsBucket",
            versioned=True,
            encryption=s3.BucketEncryption.S3_MANAGED
        )

        data_bucket = s3.Bucket(self, "DataBucket",
            versioned=True,
            encryption=s3.BucketEncryption.KMS
        )

        # CodeCommit Repository
        repo = codecommit.Repository(self, "Repo",
            repository_name="data-pipeline-scripts"
        )

        # CodeBuild Project
        build_project = codebuild.PipelineProject(self, "Build",
            build_spec=codebuild.BuildSpec.from_source_filename("buildspec.yaml"),
            environment=codebuild.BuildEnvironment(
                build_image=codebuild.LinuxBuildImage.STANDARD_7_0,
                compute_type=codebuild.ComputeType.MEDIUM
            )
        )

        # CodePipeline
        source_output = codepipeline.Artifact("SourceOutput")
        build_output = codepipeline.Artifact("BuildOutput")

        pipeline = codepipeline.Pipeline(self, "Pipeline",
            pipeline_name="data-pipeline-cicd",
            stages=[
                codepipeline.StageProps(
                    stage_name="Source",
                    actions=[
                        actions.CodeStarConnectionsSourceAction(
                            action_name="GitHub_Source",
                            owner="myorg",
                            repo="data-pipeline",
                            branch="main",
                            output=source_output,
                            connection_arn="arn:aws:codestar-connections:us-east-1:ACCOUNT:connection/xxx"
                        )
                    ]
                ),
                codepipeline.StageProps(
                    stage_name="Build",
                    actions=[
                        actions.CodeBuildAction(
                            action_name="Build",
                            project=build_project,
                            input=source_output,
                            outputs=[build_output]
                        )
                    ]
                ),
                codepipeline.StageProps(
                    stage_name="Deploy",
                    actions=[
                        actions.CloudFormationCreateUpdateStackAction(
                            action_name="DeployInfrastructure",
                            template_path=build_output.at_path("template.yaml"),
                            stack_name="data-pipeline-stack",
                            capabilities=[cloudformation.Capabilities.CAPABILITY_IAM]
                        )
                    ]
                )
            ]
        )

        # Grant permissions
        repo.grant_read(build_project)

        # Glue Job
        glue.CfnJob(self, "ETLJob",
            name=f"{id}-etl-job",
            role=glue_role.role_arn,
            glue_version="4.0",
            number_of_workers=10,
            worker_type="G.1X",
            command=glue.CfnJob.JobCommandProperty(
                name="glueetl",
                script_location=f"s3://{scripts_bucket.bucket_name}/etl/main.py"
            ),
            default_arguments={
                "--job-language": "python",
                "--TempDir": f"s3://{data_bucket.bucket_name}/tmp/",
                "--enable-metrics": "true"
            }
        )

Production Code: Testing Framework

import pytest
import boto3
from unittest.mock import Mock, patch
from datetime import datetime

class TestETLTransformations:
    """Unit tests for ETL transformation logic."""

    def test_parse_raw_event(self):
        """Test event parsing from raw data."""
        raw_event = {
            'event_id': '12345',
            'timestamp': '2024-01-15T10:30:00Z',
            'user_id': 'user_789',
            'action': 'purchase',
            'amount': 99.99
        }

        parsed = parse_event(raw_event)

        assert parsed['event_id'] == '12345'
        assert parsed['amount'] == 99.99
        assert parsed['timestamp'] == datetime(2024, 1, 15, 10, 30, 0)

    def test_validate_record_valid(self):
        """Test valid record passes validation."""
        record = {
            'event_id': '12345',
            'amount': 100.00,
            'user_id': 'user_789'
        }

        assert validate_record(record) is True

    def test_validate_record_missing_field(self):
        """Test record with missing required field."""
        record = {
            'event_id': '12345',
            'amount': 100.00
        }

        assert validate_record(record) is False

    def test_validate_record_negative_amount(self):
        """Test record with negative amount."""
        record = {
            'event_id': '12345',
            'amount': -50.00,
            'user_id': 'user_789'
        }

        assert validate_record(record) is False

    def test_transform_aggregate(self):
        """Test aggregation transformation."""
        records = [
            {'user_id': 'user_1', 'amount': 100},
            {'user_id': 'user_1', 'amount': 50},
            {'user_id': 'user_2', 'amount': 200}
        ]

        result = aggregate_by_user(records)

        assert result['user_1'] == 150
        assert result['user_2'] == 200


class TestDataQuality:
    """Data quality validation tests."""

    def test_row_count_reconciliation(self):
        """Verify source and target row counts match."""
        source_count = get_source_count()
        target_count = get_target_count()

        assert source_count == target_count, (
            f"Row count mismatch: source={source_count}, target={target_count}"
        )

    def test_null_check(self):
        """Check for null values in required fields."""
        df = get_target_data()

        null_counts = df.isnull().sum()
        for column in ['event_id', 'user_id', 'amount']:
            assert null_counts[column] == 0, f"Found nulls in {column}"

    def test_schema_validation(self):
        """Validate target schema matches expected."""
        expected_schema = {
            'event_id': 'string',
            'user_id': 'string',
            'amount': 'float',
            'timestamp': 'timestamp'
        }

        actual_schema = get_target_schema()
        assert actual_schema == expected_schema


class TestInfrastructure:
    """Infrastructure tests using boto3."""

    def test_glue_job_exists(self):
        """Verify Glue job is configured correctly."""
        glue = boto3.client('glue')
        job = glue.get_job(Name='production-etl-job')

        assert job['Job']['GlueVersion'] == '4.0'
        assert job['Job']['NumberOfWorkers'] == 10

    def test_s3_bucket_encryption(self):
        """Verify S3 bucket has encryption enabled."""
        s3 = boto3.client('s3')
        encryption = s3.get_bucket_encryption(Bucket='my-data-lake')

        rules = encryption['ServerSideEncryptionConfiguration']['Rules']
        assert len(rules) > 0
        assert rules[0]['ApplyServerSideEncryptionByDefault']['SSEAlgorithm'] == 'aws:kms'

    def test_iam_role_permissions(self):
        """Verify IAM role has required permissions."""
        iam = boto3.client('iam')
        role = iam.get_role(RoleName='GlueETLRole')

        assert 'glue' in role['Role']['AssumeRolePolicyDocument'].lower()

Mathematical Formulas

Real-World Project Structure

Architecture Diagram

data-cicd-project/
├── infrastructure/
│   ├── cloudformation/
│   │   ├── pipeline.yaml             # CodePipeline setup
│   │   ├── build-project.yaml        # CodeBuild project
│   │   └── data-resources.yaml       # Glue, S3, IAM
│   ├── cdk/
│   │   ├── app.py                    # CDK app entry
│   │   ├── stacks/
│   │   │   ├── pipeline_stack.py     # CI/CD stack
│   │   │   └── data_stack.py         # Data resources
│   │   └── cdk.json
│   └── terraform/
│       ├── main.tf
│       ├── pipeline.tf
│       └── variables.tf
├── src/
│   ├── etl/
│   │   ├── main.py                   # Glue job script
│   │   ├── transformations.py        # Transformation logic
│   │   └── validations.py            # Data validations
│   └── utils/
│       ├── config.py                 # Configuration
│       └── logging.py                # Custom logging
├── tests/
│   ├── unit/
│   │   ├── test_transformations.py
│   │   └── test_validations.py
│   ├── integration/
│   │   ├── test_glue_job.py
│   │   └── test_data_quality.py
│   └── infrastructure/
│       └── test_cloudformation.py
├── buildspec.yaml                    # CodeBuild build spec
├── pytest.ini                        # Test configuration
└── .pre-commit-config.yaml           # Pre-commit hooks

Performance Considerations

Factor	Impact	Optimization
Build Time	Developer productivity	Cache dependencies, parallel builds
Test Execution	Pipeline speed	Run tests in parallel, use markers
CloudFormation Deploy	Deployment time	Use change sets, stack policies
CodeBuild Duration	CI/CD cost	Right-size build containers
Artifact Storage	Storage cost	Retain artifacts selectively

Security Considerations

Concern	Mitigation
Secrets in code	Use Secrets Manager, parameter store
Overly permissive roles	Implement least-privilege for pipeline roles
Unencrypted artifacts	Enable S3 encryption for artifact store
No audit trail	Enable CloudTrail for all pipeline actions
Cross-account access	Use assume roles with external IDs

Common Pitfalls

Pitfall	Consequence	Solution
Manual deployments	Human error, inconsistency	Automate all deployments
Skipping tests	Production failures	Enforce test gates
Shared environments	Test interference	Use isolated environments
Hardcoded config	Environment coupling	Use parameter store
No rollback plan	Extended downtime	Maintain rollback scripts
Ignoring data quality	Silent data corruption	Add data quality gates

Interview Questions & Answers

Q1: How would you implement CI/CD for an AWS Glue ETL pipeline?

Answer: Use CodeCommit for source control, CodeBuild for testing with unit tests and data quality checks, CodePipeline for orchestration, and CloudFormation for deploying Glue jobs. Include approval gates for production deployments and automated rollback capabilities. The pipeline should: (1) run unit tests on transformation logic, (2) validate SQL syntax, (3) run data quality checks with Great Expectations, (4) deploy to staging, (5) run integration tests, (6) require manual approval for production, (7) deploy to production with automated rollback on failure.

Q2: What is the difference between blue/green and canary deployments for data pipelines?

Answer: Blue/green maintains two complete environments and switches traffic instantly. Canary gradually shifts traffic from old to new, allowing monitoring at each stage. For data pipelines, canary is often preferred because it allows validation of data quality before full commitment. However, blue/green provides instant rollback capability. For critical pipelines, use blue/green; for non-critical, use canary. Data pipelines require careful handling of state -- ensure backfills work correctly in both environments.

Q3: How do you test data transformations in a CI/CD pipeline?

Answer: Use a layered approach: unit tests for individual transformation functions (fast, isolated), integration tests with sample data in CloudFormation-provisioned environments (realistic), data quality tests with Great Expectations (output validation), and end-to-end tests on a subset of production data (comprehensive). Include schema validation, row count reconciliation, and business rule validation. Mock external dependencies for unit tests; use real services for integration tests.

Q4: What role does CloudFormation play in data pipeline CI/CD?

Answer: CloudFormation provides Infrastructure as Code, ensuring consistent environments across dev, staging, and production. It manages Glue jobs, IAM roles, S3 buckets, and networking. Version-controlled templates enable reproducible deployments and easy rollback. Use change sets to preview changes before deployment. Stack policies protect critical resources. Nested stacks organize complex architectures. Parameterize templates for environment-specific values.

Q5: How do you handle secrets in data pipeline deployments?

Answer: Use AWS Secrets Manager or Parameter Store to store database credentials, API keys, and other sensitive data. Reference these in CloudFormation templates using dynamic references ({{resolve:secretsmanager:secret-name}}). Never commit secrets to source control. Implement secret rotation for database credentials. Use IAM roles instead of access keys for service-to-service authentication. Audit secret access via CloudTrail. For CI/CD pipelines, use OIDC tokens instead of long-lived credentials.

Q6: Describe a monitoring strategy for data pipeline CI/CD.

Answer: Monitor pipeline execution time, deployment frequency, failure rates, and rollback frequency using CloudWatch. Track data quality metrics, resource utilization, and cost. Set up alarms for anomalies and use X-Ray for distributed tracing across pipeline components. Create dashboards for: (1) pipeline health (success rate, duration), (2) deployment metrics (frequency, lead time), (3) quality metrics (test pass rate, data quality scores), (4) cost metrics (cost per deployment). Review metrics weekly and optimize quarterly.

Q7: How do you implement MLOps in an AWS data pipeline?

Answer: Use SageMaker Pipelines for ML workflow orchestration, CodePipeline for CI/CD automation, SageMaker Model Registry for model versioning, and SageMaker Endpoints for deployment. Include data validation, model validation, and A/B testing stages. Implement feature stores for feature versioning. Use model monitoring to detect drift. Automate retraining triggers based on performance degradation. Include rollback capabilities for model deployments.

Q8: What are the key metrics for measuring CI/CD effectiveness?

Answer: Lead time for changes (commit to production), deployment frequency (how often you deploy), mean time to recovery (MTTR), change failure rate, and pipeline execution time. Also track data quality metrics, cost efficiency, and environment consistency across deployments. Benchmark against industry standards: elite performers deploy multiple times per day with less than 15% change failure rate and MTTR under 1 hour. Use these metrics to drive continuous improvement.

AWS Data CI/CD: CodePipeline, CDK & Testing

AWS Data CI/CD & DevOps

Why This Matters

Architecture Diagram

Why CI/CD Matters in Data Engineering

Key Differences from Application CI/CD

Production Code: CloudFormation Template

Production Code: AWS CDK

Production Code: Testing Framework

Mathematical Formulas

Real-World Project Structure

Performance Considerations

Security Considerations

Common Pitfalls

Interview Questions & Answers

Q1: How would you implement CI/CD for an AWS Glue ETL pipeline?

Q2: What is the difference between blue/green and canary deployments for data pipelines?

Q3: How do you test data transformations in a CI/CD pipeline?

Q4: What role does CloudFormation play in data pipeline CI/CD?

Q5: How do you handle secrets in data pipeline deployments?

Q6: Describe a monitoring strategy for data pipeline CI/CD.

Q7: How do you implement MLOps in an AWS data pipeline?

Q8: What are the key metrics for measuring CI/CD effectiveness?

QuizBox

See Also

Need Expert AWS Data Engineering Help?