AWS Config Compliance for Data Engineers

Master AWS Config for data engineering compliance including resource recording, conformance packs, custom rules, automated remediation, and governance patterns for data infrastructure.

18 min readIntermediate

Why This Matters

AWS Config provides continuous compliance monitoring for your cloud infrastructure. For data engineers, Config ensures that your data resources (S3 buckets, Redshift clusters, Glue jobs, RDS databases) remain compliant with security policies over time. When a teammate inadvertently opens an S3 bucket to public access or disables encryption on a Redshift cluster, Config detects the change and triggers remediation. This proactive governance prevents data breaches, ensures compliance with regulations, and maintains the security posture of your data platform.

AWS Config Architecture

Config Rules for Data Engineering

Managed Rules for Data Resources

Rule	Resource	Compliance Check
s3-bucket-public-read-prohibited	S3	Blocks public read access
s3-bucket-public-write-prohibited	S3	Blocks public write access
s3-bucket-ssl-requests-only	S3	Enforces HTTPS
s3-bucket-server-side-encryption-enabled	S3	Requires encryption
redshift-cluster-kms-enabled	Redshift	Requires KMS encryption
rds-storage-encrypted	RDS	Requires storage encryption
iam-user-no-policies-check	IAM	Users have no direct policies
iam-group-has-users-check	IAM	Groups have assigned users
restricted-ssh	Security Groups	No open SSH access
vpc-flow-logs-enabled	VPC	Flow logs are active

Custom Config Rule for Data Engineering

import json
import boto3

def lambda_handler(event, context):
    """Custom Config rule: Check if Glue jobs have encrypted connections."""
    config = boto3.client('config')
    glue = boto3.client('glue')
    
    invoking_event = json.loads(event['invokingEvent'])
    configuration_item = invoking_event['configurationItem']
    
    if configuration_item['resourceType'] != 'AWS::Glue::Job':
        return
    
    job_name = configuration_item['resourceName']
    
    try:
        response = glue.get_job(JobName=job_name)
        job = response['Job']
        
        # Check if job has encrypted connections
        connections = job.get('Connections', {}).get('Connections', [])
        has_encrypted = False
        
        for conn_name in connections:
            conn = glue.get_connection(ConnectionName=conn_name)
            conn_props = conn['Connection']['ConnectionProperties']
            if conn_props.get('ENCRYPTED_KMS_KEY_ID'):
                has_encrypted = True
                break
        
        compliance_status = 'COMPLIANT' if has_encrypted else 'NON_COMPLIANT'
        
        config.put_evaluations(
            Evaluations=[
                {
                    'ComplianceResourceType': configuration_item['resourceType'],
                    'ComplianceResourceId': configuration_item['resourceId'],
                    'ComplianceType': compliance_status,
                    'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
                }
            ],
            ResultToken=event['resultToken']
        )
        
    except Exception as e:
        config.put_evaluations(
            Evaluations=[
                {
                    'ComplianceResourceType': configuration_item['resourceType'],
                    'ComplianceResourceId': configuration_item['resourceId'],
                    'ComplianceType': 'NON_COMPLIANT',
                    'Annotation': f'Error checking compliance: {str(e)}',
                    'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
                }
            ],
            ResultToken=event['resultToken']
        )

Conformance Packs for Data Engineering

AWS Managed Conformance Packs

Pack	Purpose	Rules Count
Operational-Best-Practices-for-S3	S3 security and management	20+
Operational-Best-Practices-for-Redshift	Redshift best practices	15+
Operational-Best-Practices-for-IAM	IAM governance	30+
AWS-Control-Tower	Multi-account governance	50+
NIST-800-53-rev4	NIST compliance	80+
PCI-DSS	Payment card compliance	60+

Deploying a Conformance Pack

import boto3

config = boto3.client('config')

# Deploy S3 best practices conformance pack
config.put_conformance_pack(
    ConformancePackName='DataLake-S3-BestPractices',
    TemplateS3Uri='s3://aws-configconforms-templates/Operational-Best-Practices-for-S3.yaml',
    ConformancePackInputParameters=[
        {
            'ParameterKey': 'S3BucketPublicAccessCheck',
            'ParameterValue': 'true'
        },
        {
            'ParameterKey': 'S3BucketSSLRequestsOnly',
            'ParameterValue': 'true'
        }
    ]
)

Custom Conformance Pack Template

AWSTemplateFormatVersion: '2010-09-09'
Description: Custom conformance pack for data engineering resources
Resources:
  S3BucketEncryptionCheck:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: data-lake-bucket-encryption
      Description: Checks if S3 buckets have server-side encryption enabled
      Source:
        Owner: CUSTOM_LAMBDA
        SourceIdentifier: arn:aws:lambda:us-east-1:123456789012:function:s3-encryption-check
        SourceDetails:
          - EventSource: aws.config
            MessageType: ConfigurationItemChangeNotification
      Scope:
        ComplianceResourceTypes:
          - AWS::S3::Bucket
  
  RedshiftClusterEncryption:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: data-warehouse-encryption
      Description: Checks if Redshift clusters have KMS encryption enabled
      Source:
        Owner: AWS
        SourceIdentifier: REDSHIFT_CLUSTER_KMS_ENABLED
  
  GlueJobEncryption:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: etl-job-encryption
      Description: Checks if Glue jobs use encrypted connections
      Source:
        Owner: CUSTOM_LAMBDA
        SourceIdentifier: arn:aws:lambda:us-east-1:123456789012:function:glue-encryption-check

Automated Remediation

SSM Remediation Configuration

import boto3

config = boto3.client('config')

# Create remediation action for non-encrypted S3 buckets
config.put_remediation_configurations(
    RemediationConfigurations=[
        {
            'ConfigRuleName': 's3-bucket-server-side-encryption-enabled',
            'TargetType': 'SSM_DOCUMENT',
            'TargetId': 'AWS-EnableS3BucketEncryption',
            'TargetVersion': '1',
            'Parameters': {
                'BucketName': {
                    'Value': 'RESOURCE_ID',
                    'ResourceValue': 'RESOURCE_ID'
                },
                'SSEAlgorithm': {
                    'Value': 'aws:kms'
                }
            },
            'Automatic': True,
            'MaximumAutomaticAttempts': 3,
            'RetryAttemptSecondsSeconds': 60
        }
    ]
)

Lambda Remediation Function

import json
import boto3

def lambda_handler(event, context):
    """Auto-remediate public S3 buckets by enabling block public access."""
    config_client = boto3.client('config')
    s3_client = boto3.client('s3')
    
    invoking_event = json.loads(event['invokingEvent'])
    configuration_item = invoking_event['configurationItem']
    
    if configuration_item['resourceType'] != 'AWS::S3::Bucket':
        return
    
    bucket_name = configuration_item['resourceName']
    
    try:
        # Enable S3 Block Public Access
        s3_client.put_public_access_block(
            Bucket=bucket_name,
            PublicAccessBlockConfiguration={
                'BlockPublicAcls': True,
                'IgnorePublicAcls': True,
                'BlockPublicPolicy': True,
                'RestrictPublicBuckets': True
            }
        )
        
        compliance_type = 'COMPLIANT'
        annotation = f'Public access blocked for bucket {bucket_name}'
        
    except Exception as e:
        compliance_type = 'NON_COMPLIANT'
        annotation = f'Failed to remediate: {str(e)}'
    
    config_client.put_evaluations(
        Evaluations=[
            {
                'ComplianceResourceType': configuration_item['resourceType'],
                'ComplianceResourceId': configuration_item['resourceId'],
                'ComplianceType': compliance_type,
                'Annotation': annotation,
                'OrderingTimestamp': configuration_item['configurationItemCaptureTime']
            }
        ],
        ResultToken=event['resultToken']
    )

Real-World Project Structure

Architecture Diagram

config-compliance-infra/
├── terraform/
│   ├── config/
│   │   ├── recorder.tf
│   │   ├── rules/
│   │   │   ├── s3-rules.tf
│   │   │   ├── redshift-rules.tf
│   │   │   ├── glue-rules.tf
│   │   │   └── iam-rules.tf
│   │   ├── conformance-packs.tf
│   │   └── remediation.tf
│   ├── lambda/
│   │   ├── s3-encryption-check/
│   │   ├── glue-encryption-check/
│   │   └── auto-remediation/
│   ├── ssm/
│   │   └── remediation-documents.tf
│   └── sns/
│       └── compliance-alerts.tf
├── lambda/
│   ├── custom-rules/
│   │   ├── s3_encryption_check.py
│   │   ├── glue_connection_check.py
│   │   └── redshift_audit_check.py
│   └── remediation/
│       ├── enable_s3_encryption.py
│       ├── block_public_access.py
│       └── enable_redshift_encryption.py
├── conformance-packs/
│   ├── data-engineering-best-practices.yaml
│   └── data-lake-security.yaml
└── dashboards/
    ├── compliance-overview.json
    └── remediation-history.json

Performance Considerations

Factor	Impact	Optimization
Recording Frequency	Cost vs timeliness	Use continuous recording for critical resources
Rule Evaluation	Processing time	Use periodic for non-critical rules
Conformance Pack Scope	Rule count vs coverage	Focus on data-relevant resource types
Remediation Speed	Time to compliance	Use automatic remediation for low-risk fixes
S3 Export	Query performance	Partition exported data by date
Custom Rule Lambda	Cold start latency	Keep functions warm for time-sensitive rules

Security Considerations

Control	Implementation	Purpose
Config Recorder Permissions	IAM role with read-only access	Minimal required permissions
Remediation Role	Separate IAM role for auto-fix	Isolate remediation permissions
S3 Bucket Security	Encryption, versioning, access logs	Protect configuration history
SNS Topic Encryption	KMS encryption for notifications	Protect alert content
Cross-Account Access	Organization-level Config	Centralized governance
Retention Policies	Configurable history retention	Balance compliance vs cost
Logging	CloudTrail for Config API calls	Audit Config changes

Interview Questions & Answers

Q1: What is the difference between AWS Config and CloudTrail?

Answer: AWS Config tracks the configuration state of resources over time (is the S3 bucket encrypted? is the security group open?). It answers "what does my resource look like?" and "how has it changed?". CloudTrail tracks API calls (who created the bucket? when was the security group modified?). It answers "who did what?" and "when did they do it?". Config is for compliance monitoring; CloudTrail is for audit logging. They complement each other: Config tells you a resource is non-compliant, CloudTrail tells you who made it non-compliant.

Q2: How do you use Config to ensure data lake security compliance?

Answer: Implementation approach: (1) Enable Config recording for S3, Redshift, IAM, VPC, and Glue resources; (2) Deploy conformance packs for S3 best practices (encryption, public access, SSL); (3) Custom rules for data-specific requirements (Glue job encryption, Redshift audit logging); (4) Automatic remediation for common issues (enable encryption, block public access); (5) Compliance dashboards for executive visibility; (6) Integration with Security Hub for centralized security findings; (7) Regular compliance reviews using Config queries. This ensures your data infrastructure remains compliant as it evolves.

Q3: What are conformance packs and when should you use them?

Answer: Conformance packs are collections of Config rules that implement a specific compliance standard or best practice. They are deployed as CloudFormation templates and provide a compliance score. Use them when: (1) You need to meet a specific compliance framework (PCI DSS, HIPAA, NIST); (2) You want to implement multiple related rules consistently; (3) You need a quick way to assess compliance across many resource types; (4) You want AWS-managed rule sets that are maintained and updated. Example: deploy the S3 best practices pack to ensure all data lake buckets are encrypted, private, and logged.

Q4: How does automatic remediation work in AWS Config?

Answer: Automatic remediation uses AWS Systems Manager Automation documents to fix non-compliant resources. When Config detects a violation, it triggers an SSM Automation runbook that executes the remediation steps. Configuration: (1) Create or select an SSM document with the remediation logic; (2) Associate it with a Config rule; (3) Set parameters (e.g., which encryption algorithm to use); (4) Enable automatic execution with retry logic. Examples: enable S3 bucket encryption, block public access, add missing tags, enable VPC flow logs. For custom logic, use Lambda-based remediation.

Q5: How do you handle Config across multiple AWS accounts?

Answer: Multi-account Config strategy: (1) Organizational Config rule - deploy rules from the management account that automatically apply to all member accounts; (2) Conformance packs - deploy pack templates from the management account; (3) Aggregator - use Config aggregator to view compliance across all accounts in a single dashboard; (4) Central S3 bucket - store all Config logs in a central security account; (5) Cross-account IAM roles - allow the management account to deploy rules in member accounts; (6) Tagging strategy - use tags to identify account ownership for compliance reporting.

Q6: What is the cost model for AWS Config?

Answer: Config pricing: (1) Configuration items - 0.001 per rule evaluation per month (first 1,000 free); (3) Conformance packs - same as individual rules; (4) S3 export - standard S3 pricing for stored configuration history; (5) Remediations - SSM Automation pricing for runbook executions. For a typical data platform with 100 resources and 30 rules, monthly cost is approximately $5-15. Config is very cost-effective for the compliance value it provides.

Q7: How do you query Config data for compliance reporting?

Answer: Query options: (1) Config console - built-in compliance dashboard with filtering; (2) AWS Config API - use GetComplianceDetailsByConfigRule for programmatic access; (3) Athena queries - export Config data to S3 and query with SQL; (4) AWS Config aggregator - cross-account compliance queries; (5) Security Hub - consolidated compliance findings from Config and other services; (6) QuickSight dashboards - visualize compliance trends over time. For detailed analysis, export Config snapshots to S3 and use Athena to find non-compliant resources, track compliance trends, and generate audit reports.

Q8: How do you implement a Config rule for Glue job compliance?

Answer: Glue compliance monitoring: (1) Managed rule: glue-job-bookmarks-enabled to ensure job bookmarks are configured; (2) Custom rule: Lambda function that checks Glue job connections for encryption, verifies IAM role permissions, and validates job parameters; (3) Conformance pack: group Glue-related rules together; (4) Remediation: Lambda function to update job configurations when non-compliant; (5) Monitoring: CloudWatch metrics on rule evaluation results. Key checks: encrypted connections, VPC configurations for network isolation, appropriate IAM roles, and CloudWatch logging enabled.

Common Pitfalls

Pitfall	Problem	Solution
No Config recording	Cannot track resource changes	Enable recorder for all data-relevant resources
Ignoring custom rules	Miss data-specific compliance needs	Create Lambda rules for Glue, Redshift specifics
No remediation	Violations persist	Enable auto-remediation for common issues
Too many rules	Alert fatigue, high cost	Focus on critical data resource types
No aggregator view	Can't see cross-account compliance	Use Config aggregator for multi-account
Missing S3 export	Cannot do historical analysis	Export Config snapshots to S3
No tagging	Can't attribute compliance	Enforce tags on all Config resources
Ignoring periodic rules	Wasted evaluations	Use periodic for non-critical, continuous for critical

AWS Config Compliance for Data Engineers

AWS Config Compliance for Data Engineers

Why This Matters

AWS Config Architecture

Config Rules for Data Engineering

Managed Rules for Data Resources

Custom Config Rule for Data Engineering

Conformance Packs for Data Engineering

AWS Managed Conformance Packs

Deploying a Conformance Pack

Custom Conformance Pack Template

Automated Remediation

SSM Remediation Configuration

Lambda Remediation Function

Real-World Project Structure

Performance Considerations

Security Considerations

Interview Questions & Answers

Q1: What is the difference between AWS Config and CloudTrail?

Q2: How do you use Config to ensure data lake security compliance?

Q3: What are conformance packs and when should you use them?

Q4: How does automatic remediation work in AWS Config?

Q5: How do you handle Config across multiple AWS accounts?

Q6: What is the cost model for AWS Config?

Q7: How do you query Config data for compliance reporting?

Q8: How do you implement a Config rule for Glue job compliance?

Common Pitfalls

Knowledge Check

See Also

Premium Content