šŸŽ‰ 75% of content is free forever — Unlock Premium from $10/mo →
CW
šŸ’¼ Servicesā„¹ļø Aboutāœ‰ļø ContactView Pricing Plansfrom $10

AWS Data Compliance for Data Engineers

AWS Data EngineeringRegulatory Compliance & Data Privacy⭐ Premium

Advertisement

AWS Data Compliance for Data Engineers

Master regulatory compliance on AWS — GDPR, HIPAA, SOC2, and compliance automation frameworks for production data pipelines.

21 min readAdvanced

Why This Matters

Data compliance is not optional — it is a legal and operational imperative. GDPR fines can reach 4% of annual global turnover, HIPAA violations cost up to 4.45M. For data engineers on AWS, compliance shapes every architecture decision: encryption at rest and in transit, access controls, data retention policies, audit logging, and the shared responsibility model. Interviewers expect you to explain not just what compliance requires, but how to implement it technically on AWS.

Compliance Architecture Overview

AWS Compliance ArchitectureAWS Responsibility (OF the Cloud)Physical SecurityHypervisorGlobal NetworkCompliance Certs: SOC2, ISO 27001, PCI DSS, HIPAA, FedRAMPCustomer Responsibility (IN the Cloud)Data ClassIAM PoliciesOS PatchingEncryption | Access Control | Audit Logging | Network ConfigCompliance FrameworksGDPREU Data ProtectionHIPAAHealth InformationSOC 2Security ControlsPCI DSSPayment CardsCCPACalifornia PrivacyFedRAMPFederal DataAWS Compliance ToolsAWS ConfigConfig Rules + ComplianceSecurity HubCentral Security DashboardCloudTrailAPI Audit LoggingMaciePII DiscoveryKMS / CloudHSMEncryption KeysCompliance as Code: CloudFormation Guard + Config Rules + Lambda Remediation

Real-World Project Structure

Architecture Diagram
aws-compliance-platform/
ā”œā”€ā”€ infrastructure/
│   ā”œā”€ā”€ terraform/
│   │   ā”œā”€ā”€ kms.tf                 # Customer-managed encryption keys
│   │   ā”œā”€ā”€ iam.tf                 # Least-privilege policies + SCPs
│   │   ā”œā”€ā”€ config.tf              # AWS Config rules
│   │   ā”œā”€ā”€ securityhub.tf         # Security Hub aggregation
│   │   ā”œā”€ā”€ cloudtrail.tf          # API audit logging
│   │   └── macie.tf               # PII discovery
│   └── cloudformation/
│       └── compliance-guard/
│           ā”œā”€ā”€ hipaa-rules.guard
│           ā”œā”€ā”€ gdpr-rules.guard
│           └── soc2-rules.guard
ā”œā”€ā”€ policies/
│   ā”œā”€ā”€ iam/
│   │   ā”œā”€ā”€ data-engineer-policy.json
│   │   ā”œā”€ā”€ analyst-policy.json
│   │   └── admin-policy.json
│   ā”œā”€ā”€ scp/
│   │   ā”œā”€ā”€ region-restriction.json
│   └── resource-policy/
│       └── s3-bucket-policy.json
ā”œā”€ā”€ scripts/
│   ā”œā”€ā”€ compliance_check.py        # Automated compliance scanning
│   ā”œā”€ā”€ pii_detector.py            # Macie integration
│   ā”œā”€ā”€ data_masking.py            # PII masking for non-prod
│   └── retention_enforcer.py      # Automated data lifecycle
ā”œā”€ā”€ monitoring/
│   ā”œā”€ā”€ cloudwatch_alarms.tf
│   ā”œā”€ā”€ config_rules/
│   │   ā”œā”€ā”€ encrypted-volumes.json
│   │   ā”œā”€ā”€ s3-public-access.json
│   │   └── rds-encryption.json
│   └── dashboards/
│       └── compliance-overview.json
└── tests/
    ā”œā”€ā”€ compliance_validation.py
    └── security_scans.py

GDPR Implementation on AWS

import boto3
import hashlib
import json
from datetime import datetime, timedelta

class GDPRComplianceManager:
    """Implement GDPR compliance controls on AWS."""

    def __init__(self):
        self.s3 = boto3.client('s3')
        self.dynamodb = boto3.resource('dynamodb')
        self.kms = boto3.client('kms')

    def pseudonymize_data(self, record, key_id):
        """Replace direct identifiers with pseudonymous tokens (GDPR Article 4(5))."""
        sensitive_fields = ['email', 'phone', 'ssn', 'ip_address', 'date_of_birth']
        pseudonymized = record.copy()

        for field in sensitive_fields:
            if field in pseudonymized:
                hash_value = hashlib.sha256(
                    f"{pseudonymized[field]},{key_id}".encode()
                ).hexdigest()[:16]
                pseudonymized[f'{field}_hash'] = hash_value
                del pseudonymized[field]

        pseudonymized['_pseudonymized_at'] = datetime.utcnow().isoformat()
        pseudonymized['_retention_days'] = 730  # 2 years default
        return pseudonymized

    def enforce_retention_policy(self, bucket_name, retention_days):
        """Automatically delete data exceeding retention period (GDPR Article 5(1)(e))."""
        cutoff_date = datetime.utcnow() - timedelta(days=retention_days)
        deleted_count = 0

        paginator = self.s3.get_paginator('list_objects_v2')
        for page in paginator.paginate(Bucket=bucket_name):
            for obj in page.get('Contents', []):
                if obj['LastModified'].replace(tzinfo=None) < cutoff_date:
                    self.s3.delete_object(Bucket=bucket_name, Key=obj['Key'])
                    deleted_count += 1

        return {'deleted': deleted_count, 'cutoff': cutoff_date.isoformat()}

    def handle_deletion_request(self, user_id, data_stores):
        """Process GDPR right to erasure request (Article 17)."""
        deletion_log = {
            'user_id': user_id,
            'requested_at': datetime.utcnow().isoformat(),
            'deletions': []
        }

        for store_name in data_stores:
            table = self.dynamodb.Table(store_name)
            response = table.delete_item(
                Key={'user_id': user_id},
                ReturnValues='ALL_OLD'
            )
            if 'Attributes' in response:
                deletion_log['deletions'].append({
                    'store': store_name,
                    'deleted_at': datetime.utcnow().isoformat()
                })

        # Log deletion for audit trail
        self._log_audit_event('GDPR_DELETION', deletion_log)
        return deletion_log

    def export_user_data(self, user_id, data_stores):
        """Export all user data in machine-readable format (Article 20 - Portability)."""
        export_data = {
            'user_id': user_id,
            'export_date': datetime.utcnow().isoformat(),
            'format': 'JSON',
            'data': {}
        }

        for store_name in data_stores:
            table = self.dynamodb.Table(store_name)
            response = table.query(
                KeyConditionExpression=boto3.dynamodb.conditions.Key('user_id').eq(user_id)
            )
            export_data['data'][store_name] = response['Items']

        return export_data

    def _log_audit_event(self, event_type, details):
        """Log compliance events to CloudWatch."""
        cloudwatch = boto3.client('cloudwatch')
        cloudwatch.put_metric_data(
            Namespace='Compliance/GDPR',
            MetricData=[{
                'MetricName': event_type,
                'Value': 1,
                'Unit': 'Count',
                'Dimensions': [
                    {'Name': 'Environment', 'Value': 'Production'}
                ]
            }]
        )

HIPAA Compliance on AWS

import boto3
import json

class HIPAAComplianceManager:
    """Implement HIPAA compliance controls on AWS."""

    def __init__(self):
        self.s3 = boto3.client('s3')
        self.kms = boto3.client('kms')

    def create_hipaa_bucket(self, bucket_name):
        """Create S3 bucket with HIPAA-compliant settings."""
        # Enable versioning
        self.s3.put_bucket_versioning(
            Bucket=bucket_name,
            VersioningConfiguration={'Status': 'Enabled'}
        )

        # Default encryption with KMS
        self.s3.put_bucket_encryption(
            Bucket=bucket_name,
            ServerSideEncryptionConfiguration={
                'Rules': [{
                    'ApplyServerSideEncryptionByDefault': {
                        'SSEAlgorithm': 'aws:kms',
                        'KMSMasterKeyID': 'alias/hipaa-key'
                    },
                    'BucketKeyEnabled': True
                }]
            }
        )

        # Block all public access
        self.s3.put_public_access_block(
            Bucket=bucket_name,
            PublicAccessBlockConfiguration={
                'BlockPublicAcls': True,
                'IgnorePublicAcls': True,
                'BlockPublicPolicy': True,
                'RestrictPublicBuckets': True
            }
        )

        # Enable access logging
        self.s3.put_bucket_logging(
            Bucket=bucket_name,
            BucketLoggingStatus={
                'LoggingEnabled': {
                    'TargetBucket': 'hipaa-access-logs',
                    'TargetPrefix': f'{bucket_name}/'
                }
            }
        )

        # Object Lock for WORM compliance
        self.s3.put_object_lock_configuration(
            Bucket=bucket_name,
            ObjectLockConfiguration={
                'ObjectLockEnabled': 'Enabled',
                'Rule': {
                    'DefaultRetention': {
                        'Mode': 'GOVERNANCE',
                        'Days': 2555  # ~7 years for HIPAA
                    }
                }
            }
        )

        return {'status': 'created', 'bucket': bucket_name, 'compliance': 'HIPAA'}

Compliance Automation Pipeline

import boto3
import json
from datetime import datetime

class ComplianceAutomationPipeline:
    """Automated compliance monitoring and enforcement."""

    def __init__(self, region='us-east-1'):
        self.config = boto3.client('config', region_name=region)
        self.securityhub = boto3.client('securityhub', region_name=region)
        self.sns = boto3.client('sns', region_name=region)

    def create_config_rules(self):
        """Create AWS Config rules for compliance enforcement."""
        rules = [
            {'ConfigRuleName': 'encrypted-volumes', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'ENCRYPTED_VOLUMES'}},
            {'ConfigRuleName': 's3-bucket-public-read-prohibited', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'S3_BUCKET_PUBLIC_READ_PROHIBITED'}},
            {'ConfigRuleName': 'rds-storage-encrypted', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'RDS_STORAGE_ENCRYPTED'}},
            {'ConfigRuleName': 'kms-cmk-not-scheduled-for-deletion', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'KMS_CMK_NOT_SCHEDULED_FOR_DELETION'}},
            {'ConfigRuleName': 'iam-user-unused-credentials-check', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'IAM_USER_UNUSED_CREDENTIALS_CHECK'}},
            {'ConfigRuleName': 'cloudtrail-enabled', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'CLOUD_TRAIL_ENABLED'}},
        ]

        for rule in rules:
            self.config.put_config_rule(ConfigRule=rule)
        return rules

    def check_compliance(self):
        """Evaluate current compliance posture."""
        paginator = self.config.get_paginator('describe_compliance_by_config_rule')
        non_compliant = []

        for page in paginator.paginate():
            for rule in page['ComplianceByConfigRules']:
                if rule['Compliance']['ComplianceType'] == 'NON_COMPLIANT':
                    non_compliant.append({
                        'rule': rule['ConfigRuleName'],
                        'status': 'NON_COMPLIANT'
                    })

        return {
            'timestamp': datetime.utcnow().isoformat(),
            'non_compliant_count': len(non_compliant),
            'non_compliant_rules': non_compliant
        }

    def send_compliance_alert(self, findings):
        """Send SNS notification for compliance violations."""
        if findings['non_compliant_count'] > 0:
            message = f"Compliance Alert: {findings['non_compliant_count']} violations\n\n"
            for finding in findings['non_compliant_rules']:
                message += f"- {finding['rule']}: {finding['status']}\n"

            self.sns.publish(
                TopicArn='arn:aws:sns:us-east-1:123456789012:compliance-alerts',
                Subject='AWS Compliance Violation Alert',
                Message=message
            )
            return True
        return False

    def generate_report(self):
        """Generate compliance report for audit."""
        return {
            'report_date': datetime.utcnow().isoformat(),
            'frameworks': ['HIPAA', 'GDPR', 'SOC2'],
            'compliance_status': self.check_compliance(),
            'remediation_required': True
        }

Data Classification and Handling

ClassificationDescriptionHandling RequirementsAWS Implementation
PublicNon-sensitive dataStandard controlsDefault S3 settings
InternalBusiness dataAuthenticated accessIAM policies, VPC
ConfidentialSensitive dataEncryption + access loggingKMS encryption, CloudTrail
RestrictedPII/PHI/PCIStrong encryption + strict IAMCloudHSM, strict IAM, Macie

Mathematical Formulas

Performance Considerations

MetricTargetImpact
Config Rule Evaluation< 1 hourNear real-time compliance
Macie Scan FrequencyWeeklyPII discovery latency
CloudTrail Log Delivery< 15 minAudit trail timeliness
Encryption Latency< 5 msApplication performance
Compliance ReportDailyAudit readiness
Remediation Time< 24 hoursRisk reduction
Backup Retention30-365 daysCost vs compliance
Access ReviewQuarterlyLeast-privilege maintenance

Security Considerations

LayerControlImplementation
Encryption at RestSSE-KMSCustomer-managed keys for all data
Encryption in TransitTLS 1.2+Enforced on all connections
Access ControlIAM + SCPsLeast-privilege + organization restrictions
Audit LoggingCloudTrail + ConfigAll API calls and config changes logged
Data DiscoveryMacieAutomated PII/PHI detection
Key ManagementKMS + CloudHSMFIPS 140-2 Level 3 for HIPAA
Network SecurityVPC + PrivateLinkNo public internet for sensitive data
MonitoringSecurity HubCentralized security findings

Interview Questions & Answers

Q1: What is the difference between GDPR and HIPAA?

Answer: GDPR is a broad data protection regulation for EU residents covering all personal data. HIPAA specifically protects health information (PHI) in the United States. Key differences: Scope (GDPR covers all personal data; HIPAA only covers PHI), Geography (GDPR is EU-focused; HIPAA is US-focused), Consent (GDPR requires explicit consent; HIPAA allows treatment-based processing), Fines (GDPR up to 4% revenue; HIPAA up to $1.5M per category per year), and Breach Notification (GDPR requires 72-hour notification; HIPAA requires 60-day notification).

Q2: How do you implement encryption at rest and in transit for HIPAA compliance?

Answer: At Rest: use S3 SSE-KMS with customer-managed keys, enable RDS encryption with KMS keys, use encrypted EBS volumes, consider CloudHSM for FIPS 140-2 Level 3. In Transit: enforce TLS 1.2+ using security policies, use ACM certificates for ALB/NLB, implement HTTPS-only policies on S3 buckets, enable RDS SSL connections. The key is ensuring encryption is enforced by policy, not just available as an option.

Q3: What is the Shared Responsibility Model for compliance?

Answer: AWS is responsible for security OF the cloud (physical data center security, hypervisor, global network, hardware lifecycle). The customer is responsible for security IN the cloud (data classification, IAM, OS patching, application-level security, network configuration, compliance monitoring). Understanding this distinction is critical — many compliance failures occur because customers assume AWS handles something that is actually their responsibility.

Q4: How do you handle a GDPR deletion request across multiple data stores?

Answer: 1) Identify all data stores containing the user's data (DynamoDB, S3, RDS, Elasticsearch, backups), 2) Implement a deletion orchestrator (Step Functions) to coordinate deletion, 3) Delete in reverse dependency order (applications first, then data stores), 4) Handle backups — either implement deletion or ensure unique encryption keys that can be destroyed, 5) Log the deletion request and completions for audit trail, 6) Verify deletion across all systems, 7) Ensure third-party processors also delete the data.

Q5: What is AWS Config and how does it help with compliance?

Answer: AWS Config continuously monitors and records AWS resource configurations, enabling automation of compliance evaluation. Key features: configuration history for all resources, compliance rules (e.g., "all EBS volumes must be encrypted"), automated remediation using Systems Manager, conformance packs for common frameworks (HIPAA, PCI DSS), and integration with Security Hub and CloudTrail. Config is the foundation of compliance-as-code on AWS.

Q6: How do you implement data masking for non-production environments?

Answer: Use AWS Glue for ETL-based masking in data pipelines. Implement format-preserving encryption for realistic test data. Use Amazon Macie to discover PII before masking. Store masking rules in Systems Manager Parameter Store. Key techniques: hash sensitive fields (one-way), redact with placeholders, partial masking (show first/last characters), and synthetic data generation for testing. Always ensure masked data cannot be reverse-engineered.

Q7: What are the key SOC 2 trust service criteria?

Answer: Five criteria: 1) Security (Common Criteria) — protection against unauthorized access, 2) Availability — system availability as committed, 3) Processing Integrity — system processing is complete, accurate, and timely, 4) Confidentiality — confidential information is protected as committed, 5) Privacy — personal information is collected, used, retained, and disclosed appropriately. Key controls include logical access, change management, incident response, risk management, and encryption.

Q8: How do you handle cross-border data transfers under GDPR?

Answer: 1) Identify all cross-border data transfers, 2) Use EU regions for processing and storing EU data (eu-west-1, eu-central-1), 3) Implement Standard Contractual Clauses (SCCs) via AWS Data Processing Addendum, 4) Encrypt data in transit and at rest with customer-managed keys, 5) Apply data minimization — transfer only necessary data, 6) Document all processing activities and transfer mechanisms, 7) Conduct transfer impact assessments, 8) Implement technical measures (encryption) as supplementary safeguards.

Common Pitfalls

PitfallConsequenceSolution
Hardcoded credentialsSecurity breach, compliance violationUse Secrets Manager + IAM roles
No encryption at restHIPAA/GDPR non-complianceSSE-KMS on all data stores
Missing audit logsCannot prove complianceEnable CloudTrail on all accounts
Over-privileged IAMExcessive access riskLeast-privilege + regular access reviews
No data classificationCannot enforce handling policiesImplement classification framework
Skipping retention policiesStorage bloat + regulatory riskAutomated lifecycle policies
No breach response planExtended notification delaysDocument and test IR procedures
Ignoring third-party riskSupply chain compliance gapsVendor assessment + DPAs

QuizBox

See Also

šŸ”’

Premium Content

AWS Data Compliance for Data Engineers

You've previewed the first section. Unlock this full lesson and 900+ advanced tutorials with a Premium plan.

šŸŽÆEnd-to-end Projects
šŸ’¼Interview Prep
šŸ“œCertificates
šŸ¤Community Access

Already a member? Log in

Advertisement