Why This Matters
Data compliance is not optional ā it is a legal and operational imperative. GDPR fines can reach 4% of annual global turnover, HIPAA violations cost up to 4.45M. For data engineers on AWS, compliance shapes every architecture decision: encryption at rest and in transit, access controls, data retention policies, audit logging, and the shared responsibility model. Interviewers expect you to explain not just what compliance requires, but how to implement it technically on AWS.
Compliance Architecture Overview
Real-World Project Structure
aws-compliance-platform/
āāā infrastructure/
ā āāā terraform/
ā ā āāā kms.tf # Customer-managed encryption keys
ā ā āāā iam.tf # Least-privilege policies + SCPs
ā ā āāā config.tf # AWS Config rules
ā ā āāā securityhub.tf # Security Hub aggregation
ā ā āāā cloudtrail.tf # API audit logging
ā ā āāā macie.tf # PII discovery
ā āāā cloudformation/
ā āāā compliance-guard/
ā āāā hipaa-rules.guard
ā āāā gdpr-rules.guard
ā āāā soc2-rules.guard
āāā policies/
ā āāā iam/
ā ā āāā data-engineer-policy.json
ā ā āāā analyst-policy.json
ā ā āāā admin-policy.json
ā āāā scp/
ā ā āāā region-restriction.json
ā āāā resource-policy/
ā āāā s3-bucket-policy.json
āāā scripts/
ā āāā compliance_check.py # Automated compliance scanning
ā āāā pii_detector.py # Macie integration
ā āāā data_masking.py # PII masking for non-prod
ā āāā retention_enforcer.py # Automated data lifecycle
āāā monitoring/
ā āāā cloudwatch_alarms.tf
ā āāā config_rules/
ā ā āāā encrypted-volumes.json
ā ā āāā s3-public-access.json
ā ā āāā rds-encryption.json
ā āāā dashboards/
ā āāā compliance-overview.json
āāā tests/
āāā compliance_validation.py
āāā security_scans.py
GDPR Implementation on AWS
import boto3
import hashlib
import json
from datetime import datetime, timedelta
class GDPRComplianceManager:
"""Implement GDPR compliance controls on AWS."""
def __init__(self):
self.s3 = boto3.client('s3')
self.dynamodb = boto3.resource('dynamodb')
self.kms = boto3.client('kms')
def pseudonymize_data(self, record, key_id):
"""Replace direct identifiers with pseudonymous tokens (GDPR Article 4(5))."""
sensitive_fields = ['email', 'phone', 'ssn', 'ip_address', 'date_of_birth']
pseudonymized = record.copy()
for field in sensitive_fields:
if field in pseudonymized:
hash_value = hashlib.sha256(
f"{pseudonymized[field]},{key_id}".encode()
).hexdigest()[:16]
pseudonymized[f'{field}_hash'] = hash_value
del pseudonymized[field]
pseudonymized['_pseudonymized_at'] = datetime.utcnow().isoformat()
pseudonymized['_retention_days'] = 730 # 2 years default
return pseudonymized
def enforce_retention_policy(self, bucket_name, retention_days):
"""Automatically delete data exceeding retention period (GDPR Article 5(1)(e))."""
cutoff_date = datetime.utcnow() - timedelta(days=retention_days)
deleted_count = 0
paginator = self.s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket=bucket_name):
for obj in page.get('Contents', []):
if obj['LastModified'].replace(tzinfo=None) < cutoff_date:
self.s3.delete_object(Bucket=bucket_name, Key=obj['Key'])
deleted_count += 1
return {'deleted': deleted_count, 'cutoff': cutoff_date.isoformat()}
def handle_deletion_request(self, user_id, data_stores):
"""Process GDPR right to erasure request (Article 17)."""
deletion_log = {
'user_id': user_id,
'requested_at': datetime.utcnow().isoformat(),
'deletions': []
}
for store_name in data_stores:
table = self.dynamodb.Table(store_name)
response = table.delete_item(
Key={'user_id': user_id},
ReturnValues='ALL_OLD'
)
if 'Attributes' in response:
deletion_log['deletions'].append({
'store': store_name,
'deleted_at': datetime.utcnow().isoformat()
})
# Log deletion for audit trail
self._log_audit_event('GDPR_DELETION', deletion_log)
return deletion_log
def export_user_data(self, user_id, data_stores):
"""Export all user data in machine-readable format (Article 20 - Portability)."""
export_data = {
'user_id': user_id,
'export_date': datetime.utcnow().isoformat(),
'format': 'JSON',
'data': {}
}
for store_name in data_stores:
table = self.dynamodb.Table(store_name)
response = table.query(
KeyConditionExpression=boto3.dynamodb.conditions.Key('user_id').eq(user_id)
)
export_data['data'][store_name] = response['Items']
return export_data
def _log_audit_event(self, event_type, details):
"""Log compliance events to CloudWatch."""
cloudwatch = boto3.client('cloudwatch')
cloudwatch.put_metric_data(
Namespace='Compliance/GDPR',
MetricData=[{
'MetricName': event_type,
'Value': 1,
'Unit': 'Count',
'Dimensions': [
{'Name': 'Environment', 'Value': 'Production'}
]
}]
)
HIPAA Compliance on AWS
import boto3
import json
class HIPAAComplianceManager:
"""Implement HIPAA compliance controls on AWS."""
def __init__(self):
self.s3 = boto3.client('s3')
self.kms = boto3.client('kms')
def create_hipaa_bucket(self, bucket_name):
"""Create S3 bucket with HIPAA-compliant settings."""
# Enable versioning
self.s3.put_bucket_versioning(
Bucket=bucket_name,
VersioningConfiguration={'Status': 'Enabled'}
)
# Default encryption with KMS
self.s3.put_bucket_encryption(
Bucket=bucket_name,
ServerSideEncryptionConfiguration={
'Rules': [{
'ApplyServerSideEncryptionByDefault': {
'SSEAlgorithm': 'aws:kms',
'KMSMasterKeyID': 'alias/hipaa-key'
},
'BucketKeyEnabled': True
}]
}
)
# Block all public access
self.s3.put_public_access_block(
Bucket=bucket_name,
PublicAccessBlockConfiguration={
'BlockPublicAcls': True,
'IgnorePublicAcls': True,
'BlockPublicPolicy': True,
'RestrictPublicBuckets': True
}
)
# Enable access logging
self.s3.put_bucket_logging(
Bucket=bucket_name,
BucketLoggingStatus={
'LoggingEnabled': {
'TargetBucket': 'hipaa-access-logs',
'TargetPrefix': f'{bucket_name}/'
}
}
)
# Object Lock for WORM compliance
self.s3.put_object_lock_configuration(
Bucket=bucket_name,
ObjectLockConfiguration={
'ObjectLockEnabled': 'Enabled',
'Rule': {
'DefaultRetention': {
'Mode': 'GOVERNANCE',
'Days': 2555 # ~7 years for HIPAA
}
}
}
)
return {'status': 'created', 'bucket': bucket_name, 'compliance': 'HIPAA'}
Compliance Automation Pipeline
import boto3
import json
from datetime import datetime
class ComplianceAutomationPipeline:
"""Automated compliance monitoring and enforcement."""
def __init__(self, region='us-east-1'):
self.config = boto3.client('config', region_name=region)
self.securityhub = boto3.client('securityhub', region_name=region)
self.sns = boto3.client('sns', region_name=region)
def create_config_rules(self):
"""Create AWS Config rules for compliance enforcement."""
rules = [
{'ConfigRuleName': 'encrypted-volumes', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'ENCRYPTED_VOLUMES'}},
{'ConfigRuleName': 's3-bucket-public-read-prohibited', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'S3_BUCKET_PUBLIC_READ_PROHIBITED'}},
{'ConfigRuleName': 'rds-storage-encrypted', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'RDS_STORAGE_ENCRYPTED'}},
{'ConfigRuleName': 'kms-cmk-not-scheduled-for-deletion', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'KMS_CMK_NOT_SCHEDULED_FOR_DELETION'}},
{'ConfigRuleName': 'iam-user-unused-credentials-check', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'IAM_USER_UNUSED_CREDENTIALS_CHECK'}},
{'ConfigRuleName': 'cloudtrail-enabled', 'Source': {'Owner': 'AWS', 'SourceIdentifier': 'CLOUD_TRAIL_ENABLED'}},
]
for rule in rules:
self.config.put_config_rule(ConfigRule=rule)
return rules
def check_compliance(self):
"""Evaluate current compliance posture."""
paginator = self.config.get_paginator('describe_compliance_by_config_rule')
non_compliant = []
for page in paginator.paginate():
for rule in page['ComplianceByConfigRules']:
if rule['Compliance']['ComplianceType'] == 'NON_COMPLIANT':
non_compliant.append({
'rule': rule['ConfigRuleName'],
'status': 'NON_COMPLIANT'
})
return {
'timestamp': datetime.utcnow().isoformat(),
'non_compliant_count': len(non_compliant),
'non_compliant_rules': non_compliant
}
def send_compliance_alert(self, findings):
"""Send SNS notification for compliance violations."""
if findings['non_compliant_count'] > 0:
message = f"Compliance Alert: {findings['non_compliant_count']} violations\n\n"
for finding in findings['non_compliant_rules']:
message += f"- {finding['rule']}: {finding['status']}\n"
self.sns.publish(
TopicArn='arn:aws:sns:us-east-1:123456789012:compliance-alerts',
Subject='AWS Compliance Violation Alert',
Message=message
)
return True
return False
def generate_report(self):
"""Generate compliance report for audit."""
return {
'report_date': datetime.utcnow().isoformat(),
'frameworks': ['HIPAA', 'GDPR', 'SOC2'],
'compliance_status': self.check_compliance(),
'remediation_required': True
}
Data Classification and Handling
| Classification | Description | Handling Requirements | AWS Implementation |
|---|---|---|---|
| Public | Non-sensitive data | Standard controls | Default S3 settings |
| Internal | Business data | Authenticated access | IAM policies, VPC |
| Confidential | Sensitive data | Encryption + access logging | KMS encryption, CloudTrail |
| Restricted | PII/PHI/PCI | Strong encryption + strict IAM | CloudHSM, strict IAM, Macie |
Mathematical Formulas
Performance Considerations
| Metric | Target | Impact |
|---|---|---|
| Config Rule Evaluation | < 1 hour | Near real-time compliance |
| Macie Scan Frequency | Weekly | PII discovery latency |
| CloudTrail Log Delivery | < 15 min | Audit trail timeliness |
| Encryption Latency | < 5 ms | Application performance |
| Compliance Report | Daily | Audit readiness |
| Remediation Time | < 24 hours | Risk reduction |
| Backup Retention | 30-365 days | Cost vs compliance |
| Access Review | Quarterly | Least-privilege maintenance |
Security Considerations
| Layer | Control | Implementation |
|---|---|---|
| Encryption at Rest | SSE-KMS | Customer-managed keys for all data |
| Encryption in Transit | TLS 1.2+ | Enforced on all connections |
| Access Control | IAM + SCPs | Least-privilege + organization restrictions |
| Audit Logging | CloudTrail + Config | All API calls and config changes logged |
| Data Discovery | Macie | Automated PII/PHI detection |
| Key Management | KMS + CloudHSM | FIPS 140-2 Level 3 for HIPAA |
| Network Security | VPC + PrivateLink | No public internet for sensitive data |
| Monitoring | Security Hub | Centralized security findings |
Interview Questions & Answers
Q1: What is the difference between GDPR and HIPAA?
Answer: GDPR is a broad data protection regulation for EU residents covering all personal data. HIPAA specifically protects health information (PHI) in the United States. Key differences: Scope (GDPR covers all personal data; HIPAA only covers PHI), Geography (GDPR is EU-focused; HIPAA is US-focused), Consent (GDPR requires explicit consent; HIPAA allows treatment-based processing), Fines (GDPR up to 4% revenue; HIPAA up to $1.5M per category per year), and Breach Notification (GDPR requires 72-hour notification; HIPAA requires 60-day notification).
Q2: How do you implement encryption at rest and in transit for HIPAA compliance?
Answer: At Rest: use S3 SSE-KMS with customer-managed keys, enable RDS encryption with KMS keys, use encrypted EBS volumes, consider CloudHSM for FIPS 140-2 Level 3. In Transit: enforce TLS 1.2+ using security policies, use ACM certificates for ALB/NLB, implement HTTPS-only policies on S3 buckets, enable RDS SSL connections. The key is ensuring encryption is enforced by policy, not just available as an option.
Q3: What is the Shared Responsibility Model for compliance?
Answer: AWS is responsible for security OF the cloud (physical data center security, hypervisor, global network, hardware lifecycle). The customer is responsible for security IN the cloud (data classification, IAM, OS patching, application-level security, network configuration, compliance monitoring). Understanding this distinction is critical ā many compliance failures occur because customers assume AWS handles something that is actually their responsibility.
Q4: How do you handle a GDPR deletion request across multiple data stores?
Answer: 1) Identify all data stores containing the user's data (DynamoDB, S3, RDS, Elasticsearch, backups), 2) Implement a deletion orchestrator (Step Functions) to coordinate deletion, 3) Delete in reverse dependency order (applications first, then data stores), 4) Handle backups ā either implement deletion or ensure unique encryption keys that can be destroyed, 5) Log the deletion request and completions for audit trail, 6) Verify deletion across all systems, 7) Ensure third-party processors also delete the data.
Q5: What is AWS Config and how does it help with compliance?
Answer: AWS Config continuously monitors and records AWS resource configurations, enabling automation of compliance evaluation. Key features: configuration history for all resources, compliance rules (e.g., "all EBS volumes must be encrypted"), automated remediation using Systems Manager, conformance packs for common frameworks (HIPAA, PCI DSS), and integration with Security Hub and CloudTrail. Config is the foundation of compliance-as-code on AWS.
Q6: How do you implement data masking for non-production environments?
Answer: Use AWS Glue for ETL-based masking in data pipelines. Implement format-preserving encryption for realistic test data. Use Amazon Macie to discover PII before masking. Store masking rules in Systems Manager Parameter Store. Key techniques: hash sensitive fields (one-way), redact with placeholders, partial masking (show first/last characters), and synthetic data generation for testing. Always ensure masked data cannot be reverse-engineered.
Q7: What are the key SOC 2 trust service criteria?
Answer: Five criteria: 1) Security (Common Criteria) ā protection against unauthorized access, 2) Availability ā system availability as committed, 3) Processing Integrity ā system processing is complete, accurate, and timely, 4) Confidentiality ā confidential information is protected as committed, 5) Privacy ā personal information is collected, used, retained, and disclosed appropriately. Key controls include logical access, change management, incident response, risk management, and encryption.
Q8: How do you handle cross-border data transfers under GDPR?
Answer: 1) Identify all cross-border data transfers, 2) Use EU regions for processing and storing EU data (eu-west-1, eu-central-1), 3) Implement Standard Contractual Clauses (SCCs) via AWS Data Processing Addendum, 4) Encrypt data in transit and at rest with customer-managed keys, 5) Apply data minimization ā transfer only necessary data, 6) Document all processing activities and transfer mechanisms, 7) Conduct transfer impact assessments, 8) Implement technical measures (encryption) as supplementary safeguards.
Common Pitfalls
| Pitfall | Consequence | Solution |
|---|---|---|
| Hardcoded credentials | Security breach, compliance violation | Use Secrets Manager + IAM roles |
| No encryption at rest | HIPAA/GDPR non-compliance | SSE-KMS on all data stores |
| Missing audit logs | Cannot prove compliance | Enable CloudTrail on all accounts |
| Over-privileged IAM | Excessive access risk | Least-privilege + regular access reviews |
| No data classification | Cannot enforce handling policies | Implement classification framework |
| Skipping retention policies | Storage bloat + regulatory risk | Automated lifecycle policies |
| No breach response plan | Extended notification delays | Document and test IR procedures |
| Ignoring third-party risk | Supply chain compliance gaps | Vendor assessment + DPAs |