Why This Matters
Backup and disaster recovery (DR) is not optional for production data pipelines ā it is a regulatory requirement and a business continuity imperative. Understanding the interplay between RTO (Recovery Time Objective), RPO (Recovery Point Objective), and cost is fundamental to designing resilient architectures on AWS. Interviewers expect you to explain not just what each DR strategy does, but when to choose one over another, how to test recovery procedures, and how to balance cost against risk.
DR Strategy Architecture
Real-World Project Structure
aws-dr-platform/
āāā infrastructure/
ā āāā terraform/
ā ā āāā primary-region/
ā ā ā āāā s3.tf # Versioned buckets + CRR
ā ā ā āāā rds.tf # Multi-AZ + cross-region replica
ā ā ā āāā dynamodb.tf # Global tables
ā ā ā āāā backup.tf # AWS Backup plans
ā ā ā āāā route53.tf # Health checks + failover
ā ā āāā dr-region/
ā ā āāā s3.tf # Replica bucket
ā ā āāā rds.tf # Read replica
ā ā āāā compute.tf # Pilot light EC2
ā ā āāā route53.tf # Failover routing
ā āāā cloudformation/
ā āāā backup-plan.yaml
ā āāā dr-stack.yaml
āāā scripts/
ā āāā backup_monitor.py # Backup status monitoring
ā āāā failover_test.py # Automated DR testing
ā āāā restore_validation.py # Restore integrity checks
ā āāā cost_optimization.py # Lifecycle policy management
āāā monitoring/
ā āāā cloudwatch_alarms.tf # Backup failure alerts
ā āāā dashboards/
ā āāā dr-overview.json # CloudWatch dashboard
āāā runbooks/
ā āāā dr-procedures.md
ā āāā failover-checklist.md
ā āāā rollback-plan.md
āāā tests/
āāā backup_restore_test.py
āāā dr_drill_automation.py
S3 Replication and Versioning
import boto3
import json
import logging
from botocore.exceptions import ClientError
logger = logging.getLogger(__name__)
class S3ReplicationManager:
"""Configure and manage S3 Cross-Region Replication."""
def __init__(self, region='us-east-1'):
self.s3 = boto3.client('s3', region_name=region)
self.iam = boto3.client('iam')
def enable_versioning(self, bucket_name):
"""Enable versioning on an S3 bucket."""
try:
self.s3.put_bucket_versioning(
Bucket=bucket_name,
VersioningConfiguration={'Status': 'Enabled'}
)
logger.info(f"Versioning enabled on {bucket_name}")
except ClientError as e:
logger.error(f"Failed to enable versioning: {e}")
raise
def create_replication_role(self, source_bucket, replica_bucket):
"""Create IAM role for S3 replication."""
trust_policy = {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "s3.amazonaws.com"},
"Action": "sts:AssumeRole"
}]
}
try:
role = self.iam.create_role(
RoleName='S3ReplicationRole',
AssumeRolePolicyDocument=json.dumps(trust_policy),
Description='Role for S3 cross-region replication'
)
policy_document = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:GetReplicationConfiguration", "s3:ListBucket"],
"Resource": f"arn:aws:s3:::{source_bucket}"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObjectVersionForReplication",
"s3:GetObjectVersionAcl",
"s3:GetObjectVersionTagging"
],
"Resource": f"arn:aws:s3:::{source_bucket}/*"
},
{
"Effect": "Allow",
"Action": ["s3:ReplicateObject", "s3:ReplicateDelete", "s3:ReplicateTags"],
"Resource": f"arn:aws:s3:::{replica_bucket}/*"
}
]
}
self.iam.put_role_policy(
RoleName='S3ReplicationRole',
PolicyName='S3ReplicationPolicy',
PolicyDocument=json.dumps(policy_document)
)
logger.info(f"Replication role created: {role['Role']['Arn']}")
return role['Role']['Arn']
except ClientError as e:
logger.error(f"Failed to create replication role: {e}")
raise
def configure_replication(self, source_bucket, replica_bucket, role_arn):
"""Configure S3 Cross-Region Replication."""
try:
self.s3.put_bucket_replication(
Bucket=source_bucket,
ReplicationConfiguration={
'Role': role_arn,
'Rules': [{
'ID': 'EntireBucket',
'Status': 'Enabled',
'Filter': {'Prefix': ''},
'Destination': {
'Bucket': f'arn:aws:s3:::{replica_bucket}',
'StorageClass': 'STANDARD_IA'
}
}]
}
)
logger.info(f"CRR configured: {source_bucket} -> {replica_bucket}")
except ClientError as e:
logger.error(f"Failed to configure replication: {e}")
raise
RDS Backup and Recovery
import boto3
import time
import logging
logger = logging.getLogger(__name__)
class RDSBackupManager:
"""Manage RDS automated backups and manual snapshots."""
def __init__(self, region='us-east-1'):
self.rds = boto3.client('rds', region_name=region)
def create_snapshot(self, db_instance_id, snapshot_id):
"""Create a manual RDS snapshot with error handling."""
try:
snapshot = self.rds.create_db_snapshot(
DBSnapshotIdentifier=snapshot_id,
DBInstanceIdentifier=db_instance_id,
Tags=[
{'Key': 'Purpose', 'Value': 'DisasterRecovery'},
{'Key': 'Environment', 'Value': 'Production'}
]
)
# Wait for completion
waiter = self.rds.get_waiter('db_snapshot_completed')
waiter.wait(
DBSnapshotIdentifier=snapshot_id,
WaiterConfig={'Delay': 30, 'MaxAttempts': 120}
)
logger.info(f"Snapshot created: {snapshot_id}")
return snapshot['DBSnapshot']['DBSnapshotArn']
except Exception as e:
logger.error(f"Failed to create snapshot: {e}")
raise
def copy_snapshot_to_dr_region(self, snapshot_arn, target_snapshot_id, source_region='us-east-1', target_region='us-west-2'):
"""Copy snapshot to DR region."""
try:
target_rds = boto3.client('rds', region_name=target_region)
copy_result = target_rds.copy_db_snapshot(
SourceDBSnapshotIdentifier=snapshot_arn,
TargetDBSnapshotIdentifier=target_snapshot_id,
SourceRegion=source_region,
KmsKeyId='alias/dr-key'
)
logger.info(f"Snapshot copied to {target_region}: {target_snapshot_id}")
return copy_result['DBSnapshot']['DBSnapshotArn']
except Exception as e:
logger.error(f"Failed to copy snapshot: {e}")
raise
def create_cross_region_replica(self, source_db_id, replica_db_id, target_region='us-west-2'):
"""Create cross-region read replica for DR."""
try:
target_rds = boto3.client('rds', region_name=target_region)
replica = target_rds.create_db_instance_read_replica(
DBInstanceIdentifier=replica_db_id,
SourceDBInstanceIdentifier=f'arn:aws:rds:us-east-1:123456789012:db:{source_db_id}',
SourceRegion='us-east-1',
DBInstanceClass='db.r5.large',
StorageType='gp3',
Iops=3000,
StorageEncrypted=True,
KmsKeyId='alias/dr-key',
BackupRetentionPeriod=7,
MultiAZ=True,
PubliclyAccessible=False,
Tags=[
{'Key': 'Purpose', 'Value': 'DisasterRecovery'},
{'Key': 'Environment', 'Value': 'Production'}
]
)
logger.info(f"Cross-region replica created: {replica_db_id}")
return replica['DBInstance']['DBInstanceArn']
except Exception as e:
logger.error(f"Failed to create replica: {e}")
raise
DynamoDB Backup Strategies
import boto3
import logging
logger = logging.getLogger(__name__)
class DynamoDBBackupManager:
"""Manage DynamoDB backups and global tables."""
def __init__(self, region='us-east-1'):
self.dynamodb = boto3.client('dynamodb', region_name=region)
def enable_pitr(self, table_name):
"""Enable Point-in-Time Recovery for DynamoDB."""
try:
self.dynamodb.update_continuous_backups(
TableName=table_name,
PointInTimeRecoverySpecification={
'PointInTimeRecoveryEnabled': True
}
)
logger.info(f"PITR enabled for {table_name}")
except Exception as e:
logger.error(f"Failed to enable PITR: {e}")
raise
def restore_to_point_in_time(self, table_name, target_name, restore_datetime):
"""Restore DynamoDB table to a specific point in time."""
try:
self.dynamodb.restore_table_from_point_in_time(
TargetTableName=target_name,
SourceTableName=table_name,
UseLatestRestorableTime=False,
RestoreDateTime=restore_datetime
)
logger.info(f"Restored {table_name} to {target_name} at {restore_datetime}")
except Exception as e:
logger.error(f"Failed to restore table: {e}")
raise
def create_global_table(self, table_config):
"""Create DynamoDB Global Table for multi-region DR."""
try:
self.dynamodb.create_global_table(
GlobalTableName=table_config['name'],
BillingMode=table_config.get('billing_mode', 'PAY_PER_REQUEST'),
AttributeDefinitions=table_config['attributes'],
KeySchema=table_config['key_schema'],
Replicas=table_config['replicas'],
StreamSpecification={
'StreamEnabled': True,
'StreamViewType': 'NEW_AND_OLD_IMAGES'
}
)
logger.info(f"Global table created: {table_config['name']}")
except Exception as e:
logger.error(f"Failed to create global table: {e}")
raise
Mathematical Formulas
Performance Considerations
| Metric | Target | Impact |
|---|---|---|
| RTO | < 15 min | Automated failover required |
| RPO | < 5 min | Continuous replication needed |
| Backup Window | Off-peak hours | Minimize production impact |
| Snapshot Retention | 30-365 days | Compliance + cost balance |
| Replication Lag | < 1 min | Near real-time consistency |
| Cross-Region Transfer | Minimize | Cost optimization |
| Restore Testing | Monthly | Validate recovery procedures |
| Backup Encryption | Always | Compliance requirement |
Security Considerations
| Layer | Control | Implementation |
|---|---|---|
| Backup Encryption | SSE-KMS | Customer-managed keys |
| Vault Lock | WORM compliance | Backup Vault Lock for regulatory |
| Access Control | IAM Policies | Restrict backup/restore permissions |
| Cross-Account | Resource Policies | Share backups across accounts |
| Audit | CloudTrail | All backup operations logged |
| Retention Enforcement | Lifecycle Policies | Automated cleanup + compliance |
| Versioning | S3 Versioning | Prevent accidental deletion |
| Deletion Protection | RDS + DynamoDB | Prevent accidental drops |
Interview Questions & Answers
Q1: Explain the difference between RTO and RPO with real-world examples.
Answer: RTO (Recovery Time Objective) is the maximum acceptable downtime after a failure. For example, an e-commerce platform might have an RTO of 15 minutes during peak hours ā meaning the system must be back online within 15 minutes. RPO (Recovery Point Objective) is the maximum acceptable data loss measured in time. That same platform might have an RPO of 5 minutes, meaning they can afford to lose up to 5 minutes of transactions. Lower values require more sophisticated (and expensive) solutions like continuous replication and automated failover.
Q2: When would you choose Pilot Light over Warm Standby for DR?
Answer: Choose Pilot Light when you have moderate RTO requirements (10-60 minutes), budget constraints exist but some DR investment is needed, core infrastructure (databases) must be continuously available, and you can tolerate scaling up time for compute resources. Choose Warm Standby when RTO must be under 10 minutes, user-facing applications require immediate availability, and budget allows for running scaled-down infrastructure with pre-configured networking and load balancing.
Q3: How does S3 versioning support disaster recovery?
Answer: S3 versioning enables recovery from accidental deletions and overwrites by maintaining multiple versions of each object. DR benefits include: accidental deletion recovery (previous versions remain accessible), overwrite recovery (restore to any previous state), audit trail (complete history of object changes), cross-region replication requirement (versioning is mandatory for CRR), and compliance (meets data retention requirements).
Q4: Describe your approach to implementing a comprehensive backup strategy for a data lake on AWS.
Answer: A comprehensive strategy includes four layers: 1) S3 Data Lake: versioning, CRR to secondary region, Object Lock for compliance, lifecycle policies for cost optimization, 2) ETL/Processing: backup Glue job definitions, version control ETL scripts in CodeCommit, snapshot EMR configurations, 3) Databases: RDS automated backups with 30-day retention, cross-region read replicas for Aurora, DynamoDB PITR, 4) Configuration: AWS Config rules, CloudFormation templates in version control, IAM policies backed up.
Q5: How do you test and validate disaster recovery procedures?
Answer: DR testing follows a structured approach: 1) Tabletop exercises with stakeholders to walk through scenarios, 2) Component testing ā test backup restoration for each service, validate integrity, measure actual recovery times, 3) Full DR drills ā simulate regional outage, execute complete failover, measure RTO/RPO achievement, document lessons learned. Automate testing with scripts that verify backups exist, restore to DR region, validate data integrity, and measure actual recovery times against targets.
Q6: Explain AWS Backup vault lock and its importance for compliance.
Answer: AWS Backup vault lock prevents backup deletion or modification, providing immutable backups for compliance requirements like SEC 17a-4, CFTC, and FINRA. Key features: WORM compliance (Write Once, Read Many), retention enforcement (cannot reduce retention period), legal hold (prevents deletion regardless of retention), and audit trail (CloudTrail logging of all access). The lock takes 72 hours to fully activate, providing a grace period for configuration changes.
Q7: How do you handle backup costs while maintaining compliance?
Answer: Cost optimization requires a tiered approach: 1) Storage tiering ā hot storage (S3 Standard) for recent backups, warm (S3 IA) for 30-90 days, cold (Glacier) for 90-365 days, deep archive for long-term compliance, 2) Retention optimization ā implement lifecycle policies, delete redundant backups, consolidate similar backups, 3) Monitoring ā CloudWatch alarms on backup cost increases, regular cost reviews, right-sizing backup resources.
Q8: Describe the trade-offs between different RDS backup approaches.
Answer: Automated backups provide 15-60 min RTO, 5 min RPO, low cost, and low complexity ā best for most production workloads. Manual snapshots provide 30-120 min RTO, until-snapshot RPO, low cost ā best for pre-migration backups. Cross-region replicas provide 5-15 min RTO, near real-time RPO, medium cost ā best for regional DR. Multi-AZ provides automatic failover, zero RPO, high cost ā best for high availability. Multi-region clusters provide near-zero RTO/RPO, very high cost ā best for mission-critical zero-downtime requirements.
Common Pitfalls
| Pitfall | Consequence | Solution |
|---|---|---|
| No backup testing | Untested recovery = failed recovery | Monthly DR drills with validation |
| Single-region backups | Regional outage = total loss | CRR to secondary region |
| No versioning | Accidental deletion unrecoverable | Enable versioning on all buckets |
| Missing encryption | Compliance violations | SSE-KMS on all backups |
| No retention policies | Unbounded storage costs | Lifecycle policies for tiering |
| Ignoring RPO/RTO | Misaligned expectations | Document and test SLAs |
| No rollback plan | Extended downtime on failure | Prepare and test rollback procedures |
| Skipping validation | Corrupted backups undetected | Verify integrity after each backup |