📦 Amazon S3 Deep Dive

Master S3 storage classes, lifecycle policies, versioning, replication, and S3 as the foundation for data lakes.

Module: AWS Data Engineering • Topic 4 of 65 • Premium Content

S3 Architecture Overview

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    AMAZON S3 ARCHITECTURE                                     │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    S3 BUCKET: data-lake-production                   │    │
│  │                                                                     │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │                    BUCKET LEVEL CONFIGURATION                 │  │    │
│  │  │                                                               │  │    │
│  │  │  • Versioning: Enabled                                        │  │    │
│  │  │  • Encryption: AES-256 / KMS                                 │  │    │
│  │  │  • Access Logging: Enabled                                    │  │    │
│  │  │  • Lifecycle Rules: Configured                                │  │    │
│  │  │  • Replication: Cross-Region Enabled                          │  │    │
│  │  │  • Object Lock: Governance Mode                               │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  │                                                                     │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │                    PREFIX HIERARCHY                            │  │    │
│  │  │                                                               │  │    │
│  │  │  data-lake-production/                                        │  │    │
│  │  │  ├── raw/                 (Incoming data)                     │  │    │
│  │  │  │   ├── landing/         (Temporary staging)                 │  │    │
│  │  │  │   ├── source_a/        (Source system A)                   │  │    │
│  │  │  │   └── source_b/        (Source system B)                   │  │    │
│  │  │  ├── processed/           (Cleaned data)                      │  │    │
│  │  │  │   ├── silver/          (Validated)                         │  │    │
│  │  │  │   └── gold/            (Business-ready)                    │  │    │
│  │  │  ├── curated/             (Aggregated)                        │  │    │
│  │  │  │   ├── daily/           (Daily aggregates)                  │  │    │
│  │  │  │   ├── weekly/          (Weekly reports)                    │  │    │
│  │  │  │   └── monthly/         (Monthly rollups)                   │  │    │
│  │  │  └── archive/             (Historical)                        │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    STORAGE CLASSES                                   │    │
│  │                                                                     │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌───────────┐ │    │
│  │  │  Standard   │  │  IA         │  │  One Zone-IA│  │  Glacier  │ │    │
│  │  │             │  │             │  │             │  │  Instant  │ │    │
│  │  │  $0.023/GB  │  │  $0.0125/GB │  │  $0.01/GB   │  │  $0.004/GB│ │    │
│  │  │  ms access  │  │  min 30 days│  │  min 30 days│  │  ms access│ │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └───────────┘ │    │
│  │                                                                     │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                │    │
│  │  │  Glacier    │  │  Glacier    │  │  Deep       │                │    │
│  │  │  Flexible   │  │  Deep       │  │  Archive    │                │    │
│  │  │  Retrieval  │  │  Archive    │  │  (DynamoDB) │                │    │
│  │  │  $0.0036/GB │  │  $0.00099/GB│  │  $0.00099/GB│                │    │
│  │  │  1-5 hrs    │  │  12 hrs     │  │  12 hrs     │                │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘                │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Storage Classes Comparison

Storage Class	Durability	Availability	Min Storage	Retrieval	Use Case
S3 Standard	11 9's	99.99%	None	ms	Frequently accessed
S3 Intelligent-Tiering	11 9's	99.9%	None	ms	Unknown access patterns
S3 Standard-IA	11 9's	99.9%	30 days	ms	Infrequent access
S3 One Zone-IA	11 9's	99.5%	30 days	ms	Recreatable, infrequent
S3 Glacier Instant	11 90's	99.9%	90 days	ms	Archive, fast retrieval
S3 Glacier Flexible	11 9's	99.99%	90 days	min-hours	Archive, flexible
S3 Glacier Deep Archive	11 9's	99.99%	180 days	12 hrs	Long-term archive

S3 Lifecycle Policies

Data Lake Lifecycle Configuration

{
  "Rules": [
    {
      "ID": "RawDataLifecycle",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "raw/"
      },
      "Transitions": [
        {
          "Days": 0,
          "StorageClass": "STANDARD"
        },
        {
          "Days": 90,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 180,
          "StorageClass": "GLACIER"
        },
        {
          "Days": 365,
          "StorageClass": "DEEP_ARCHIVE"
        }
      ],
      "Expiration": {
        "Days": 2555
      }
    },
    {
      "ID": "ProcessedDataLifecycle",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "processed/"
      },
      "Transitions": [
        {
          "Days": 0,
          "StorageClass": "STANDARD"
        },
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER_INSTANT_RETRIEVAL"
        },
        {
          "Days": 365,
          "StorageClass": "GLACIER"
        }
      ]
    },
    {
      "ID": "CuratedDataLifecycle",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "curated/"
      },
      "Transitions": [
        {
          "Days": 0,
          "StorageClass": "STANDARD"
        },
        {
          "Days": 365,
          "StorageClass": "STANDARD_IA"
        }
      ]
    },
    {
      "ID": "CleanupMultipartUploads",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    },
    {
      "ID": "CleanupExpiredMarkers",
      "Status": "Enabled",
      "Filter": {},
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    }
  ]
}

ℹ️

Pro Tip: Use S3 Intelligent-Tiering for data with unknown or changing access patterns. It automatically moves objects between tiers based on usage, saving up to 70% on storage costs.

S3 Versioning and Replication

Versioning Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    S3 VERSIONING & REPLICATION                                │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  SOURCE BUCKET (us-east-1)                                          │    │
│  │                                                                     │    │
│  │  Object: data/sales/2024-01-15.parquet                             │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  Version ID    │  Size    │  Storage Class  │  Last Modified  │  │    │
│  │  ├────────────────┼──────────┼─────────────────┼─────────────────┤  │    │
│  │  │  v1 (current)  │  2.5 GB  │  STANDARD        │  2024-01-15    │  │    │
│  │  │  v2            │  2.3 GB  │  STANDARD_IA     │  2024-01-14    │  │    │
│  │  │  v3            │  2.1 GB  │  GLACIER         │  2024-01-13    │  │    │
│  │  │  v4 (delete)   │  0       │  -               │  2024-01-12    │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│                    ┌───────────┴───────────┐                               │
│                    ▼                       ▼                               │
│  ┌──────────────────────────┐  ┌──────────────────────────┐               │
│  │  CROSS-REGION REPLICATION │  │  SAME-REGION REPLICATION │               │
│  │  (CRR)                    │  │  (SRR)                   │               │
│  │                           │  │                          │               │
│  │  Source: us-east-1        │  │  Source: us-east-1       │               │
│  │  Dest: us-west-2          │  │  Dest: us-east-1         │               │
│  │                           │  │  (Different account)     │               │
│  │  • Async replication      │  │                          │               │
│  │  • ~15 min latency        │  │  • Async replication     │               │
│  │  • Versioning required    │  │  • ~15 min latency       │               │
│  │  • Encryption required    │  │  • Versioning required   │               │
│  └──────────────────────────┘  └──────────────────────────┘               │
└─────────────────────────────────────────────────────────────────────────────┘

Replication Configuration

{
  "Role": "arn:aws:iam::123456789012:role/S3ReplicationRole",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {
        "Prefix": "curated/"
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::data-lake-disaster-recovery",
        "StorageClass": "STANDARD_IA",
        "ReplicationTime": {
          "Status": "Enabled",
          "Time": {
            "Minutes": 15
          }
        },
        "EncryptionConfiguration": {
          "ReplicaKmsKeyId": "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"
        }
      },
      "DeleteMarkerReplication": {
        "Status": "Enabled"
      }
    }
  ]
}

S3 Encryption Options

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    S3 ENCRYPTION OPTIONS                                      │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  SERVER-SIDE ENCRYPTION (SSE)                                        │    │
│  │                                                                     │    │
│  │  ┌─────────────────────┐  ┌─────────────────────┐                 │    │
│  │  │  SSE-S3             │  │  SSE-KMS             │                 │    │
│  │  │                     │  │                     │                 │    │
│  │  │  • AWS managed keys │  │  • Customer managed  │                 │    │
│  │  │  • AES-256          │  │  • AWS managed keys  │                 │    │
│  │  │  • No audit trail   │  │  • Audit trail       │                 │    │
│  │  │  • Free             │  │  • $1/key/month      │                 │    │
│  │  └─────────────────────┘  └─────────────────────┘                 │    │
│  │                                                                     │    │
│  │  ┌─────────────────────┐                                           │    │
│  │  │  SSE-C              │                                           │    │
│  │  │                     │                                           │    │
│  │  │  • Customer provided│                                           │    │
│  │  │  • Must manage keys │                                           │    │
│  │  │  • No AWS key mgmt  │                                           │    │
│  │  └─────────────────────┘                                           │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  CLIENT-SIDE ENCRYPTION                                              │    │
│  │                                                                     │    │
│  │  • Encrypt before upload                                            │    │
│  │  • Manage your own keys                                              │    │
│  │  • Full control over encryption                                     │    │
│  │  • More complex implementation                                      │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Default Encryption Configuration

{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
      },
      "BucketKeyEnabled": true
    }
  ]
}

⚠️

Security Warning: Always enable default encryption for data lakes. Use SSE-KMS for audit trails and compliance. Bucket keys reduce KMS costs by 99% for large buckets.

S3 Performance Optimization

Multipart Upload

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client('s3')

# Configure multipart upload
config = TransferConfig(
    multipart_threshold=1024 * 1024 * 100,  # 100 MB
    max_concurrency=10,
    multipart_chunksize=1024 * 1024 * 100,  # 100 MB
    use_threads=True
)

# Upload with multipart
s3.upload_file(
    'large_file.parquet',
    'data-lake-bucket',
    'raw/source/file.parquet',
    Config=config
)

# For very large files (>5GB), use custom multipart
def multipart_upload(bucket, key, file_path, part_size=100*1024*1024):
    import os
    
    # Create multipart upload
    response = s3.create_multipart_upload(Bucket=bucket, Key=key)
    upload_id = response['UploadId']
    
    parts = []
    file_size = os.path.getsize(file_path)
    
    with open(file_path, 'rb') as f:
        part_number = 1
        while True:
            data = f.read(part_size)
            if not data:
                break
            
            response = s3.upload_part(
                Bucket=bucket,
                Key=key,
                PartNumber=part_number,
                UploadId=upload_id,
                Body=data
            )
            
            parts.append({
                'PartNumber': part_number,
                'ETag': response['ETag']
            })
            part_number += 1
    
    # Complete multipart upload
    s3.complete_multipart_upload(
        Bucket=bucket,
        Key=key,
        UploadId=upload_id,
        MultipartUpload={'Parts': parts}
    )

S3 Transfer Acceleration

# Enable Transfer Acceleration
aws s3api put-bucket-accelerate-configuration \
  --bucket data-lake-production \
  --accelerate-configuration Status=Enabled

# Use accelerated endpoint
aws s3 cp large_file.parquet s3://data-lake-production/raw/ \
  --endpoint-url https://data-lake-production.s3-accelerate.amazonaws.com

S3 for Data Lake Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    S3 DATA LAKE ARCHITECTURE                                  │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  INGESTION LAYER                                                     │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌───────────┐ │    │
│  │  │ Kinesis     │  │ DMS         │  │ Glue Crawler│  │ API GW    │ │    │
│  │  │ Firehose    │  │ (CDC)       │  │ (Batch)     │  │ (REST)    │ │    │
│  │  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └─────┬─────┘ │    │
│  │         │                │                │               │        │    │
│  └─────────┼────────────────┼────────────────┼───────────────┼────────┘    │
│            │                │                │               │              │
│            ▼                ▼                ▼               ▼              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  RAW ZONE (Bronze)                                                   │    │
│  │  s3://data-lake-raw/                                                 │    │
│  │  ├── landing/              (Temporary staging)                       │    │
│  │  │   └── {date}/           (Daily partitions)                       │    │
│  │  ├── source_system_a/      (Source A data)                          │    │
│  │  │   ├── {yyyy}/{mm}/{dd}/ (Date-partitioned)                       │    │
│  │  │   └── _metadata.json    (Schema info)                            │    │
│  │  └── source_system_b/      (Source B data)                          │    │
│  │                                                                     │    │
│  │  Format: Raw JSON/CSV/Parquet                                        │    │
│  │  Retention: 90 days standard, then Glacier                           │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│                                ▼                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  PROCESSING LAYER (AWS Glue)                                         │    │
│  │                                                                     │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  Glue Crawler → Glue Catalog → Glue ETL Job                   │  │    │
│  │  │                                                               │  │    │
│  │  │  1. Discover schema (Crawler)                                 │  │    │
│  │  │  2. Register in catalog                                        │  │    │
│  │  │  3. Transform data (ETL job)                                  │  │    │
│  │  │  4. Write to processed zone                                   │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│                                ▼                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  PROCESSED ZONE (Silver/Gold)                                        │    │
│  │  s3://data-lake-processed/                                          │    │
│  │  ├── silver/               (Cleaned, validated)                     │    │
│  │  │   ├── transactions/     (Parquet, partitioned by date)          │    │
│  │  │   ├── customers/        (Delta Lake format)                      │    │
│  │  │   └── products/         (Optimized Parquet)                      │    │
│  │  └── gold/                 (Business-ready)                         │    │
│  │      ├── daily_metrics/    (Aggregated daily)                       │    │
│  │      ├── customer_360/     (Customer view)                          │    │
│  │      └── financial_reports/ (Financial data)                        │    │
│  │                                                                     │    │
│  │  Format: Parquet/Delta Lake/ORC                                      │    │
│  │  Retention: 2 years standard, then IA                                │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│              ┌─────────────────┼─────────────────┐                         │
│              ▼                 ▼                 ▼                         │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐                  │
│  │  Athena       │  │  Redshift     │  │  QuickSight   │                  │
│  │  (Ad-hoc)     │  │  (Warehouse)  │  │  (BI/Dash)    │                  │
│  └───────────────┘  └───────────────┘  └───────────────┘                  │
└─────────────────────────────────────────────────────────────────────────────┘

S3 Access Patterns for Data Engineering

S3 Select (Filter at Storage Layer)

import boto3

s3 = boto3.client('s3')

# Use S3 Select to filter data without downloading entire object
response = s3.select_object_content(
    Bucket='data-lake-processed',
    Key='silver/transactions/2024/01/15.parquet',
    Expression="SELECT * FROM s3object WHERE amount > 1000",
    ExpressionType='SQL',
    InputSerialization={
        'Parquet': {}
    },
    OutputSerialization={
        'JSON': {}
    }
)

# Process filtered results
for event in response['Payload']:
    if 'Records' in event:
        records = event['Records']['Payload'].decode('utf-8')
        print(records)

S3 Inventory

import boto3

s3 = boto3.client('s3')

# Create inventory configuration
s3.put_bucket_inventory_configuration(
    Bucket='data-lake-production',
    Id='daily-inventory',
    InventoryConfiguration={
        'Destination': {
            'S3BucketDestination': {
                'Format': 'Parquet',
                'Bucket': 'arn:aws:s3:::data-lake-inventory',
                'Prefix': 'inventory'
            }
        },
        'IsEnabled': True,
        'Id': 'daily-inventory',
        'IncludedObjectVersions': 'Current',
        'Schedule': {
            'Frequency': 'Daily'
        },
        'OptionalFields': [
            'Size',
            'LastModifiedDate',
            'StorageClass',
            'ETag',
            'IsMultipartUploaded',
            'ReplicationStatus',
            'EncryptionStatus',
            'IntelligentTieringAccessTier',
            'BucketKeyStatus',
            'ChecksumAlgorithm'
        ]
    }
)

Interview Questions & Answers

Q1: What is the difference between S3 Standard and S3 Intelligent-Tiering?

Answer:

S3 Standard: Highest availability (99.99%), highest cost, for frequently accessed data
S3 Intelligent-Tiering: Same durability, automatic tiering based on access patterns, no retrieval fees

Use S3 Intelligent-Tiering when you don't know the access pattern or it changes frequently. It can save up to 70% compared to Standard.

Q2: How do you handle S3 eventual consistency?

Answer: S3 provides strong read-after-write consistency for new objects. For overwrites and deletes:

Read-after-write: Immediately consistent
Read-after-delete: Immediately consistent (if you read a delete marker)
List operations: Eventually consistent

Best practice: Use versioning and DynamoDB for metadata to handle consistency requirements.

Q3: What are the best practices for S3 performance?

Answer:

Multipart Upload: For files >100MB
Parallel Requests: Use prefix-level parallelism (up to 5,500 GET/s per prefix)
Transfer Acceleration: For cross-region transfers
S3 Select: Filter at storage layer
Requester Pays: For large-scale analytics
S3 Batch Operations: For bulk operations

Q4: How do you implement S3 lifecycle policies for data lakes?

Answer:

Raw Zone: Standard → IA (90 days) → Glacier (180 days) → Deep Archive (365 days)
Processed Zone: Standard → IA (30 days) → Glacier Instant (90 days)
Curated Zone: Standard → IA (365 days)
Archive Zone: Glacier Deep Archive immediately

Always set expiration rules for temporary data (landing zones).

Q5: What is S3 Replication Time Control (RTC)?

Answer: S3 RTC provides a durability SLA of 99.99% for cross-region replication within 15 minutes. It's useful for:

Disaster recovery requirements
Compliance requirements
Low-latency access in multiple regions

Cost: $0.02 per GB replicated.

Cost Considerations

Component	Cost	Optimization
S3 Standard	$0.023/GB/month	Use lifecycle policies
S3 Standard-IA	$0.0125/GB/month	Min 30 days storage
S3 Glacier Instant	$0.004/GB/month	Min 90 days storage
S3 Glacier Flexible	$0.0036/GB/month	Min 90 days, retrieval fees
S3 Deep Archive	$0.00099/GB/month	Min 180 days, 12hr retrieval
Data Transfer	$0.09/GB outbound	Use VPC endpoints
PUT/COPY/POST	$0.005/1000 requests	Batch operations
GET/SELECT	$0.0004/1000 requests	Use S3 Select

⚠️

Cost Warning: S3 costs can add up quickly with large data lakes. Always implement lifecycle policies and use S3 Cost Analysis to identify optimization opportunities.

Summary

S3 is the foundation of data lakes on AWS. Key takeaways:

Storage Classes: Choose based on access patterns and retention requirements
Lifecycle Policies: Automate data movement between tiers
Versioning: Protect against accidental deletes and overwrites
Replication: Cross-region for DR, same-region for compliance
Encryption: Always enable, use SSE-KMS for audit trails
Performance: Multipart upload, parallel requests, S3 Select
Data Lake: Use prefix hierarchy for organization and access control

Amazon S3 Deep Dive for Data Engineers

📦 Amazon S3 Deep Dive

S3 Architecture Overview

Storage Classes Comparison

S3 Lifecycle Policies

Data Lake Lifecycle Configuration

S3 Versioning and Replication

Versioning Architecture

Replication Configuration

S3 Encryption Options

Default Encryption Configuration

S3 Performance Optimization

Multipart Upload

S3 Transfer Acceleration

S3 for Data Lake Architecture

S3 Access Patterns for Data Engineering

S3 Select (Filter at Storage Layer)

S3 Inventory

Interview Questions & Answers

Q1: What is the difference between S3 Standard and S3 Intelligent-Tiering?

Q2: How do you handle S3 eventual consistency?

Q3: What are the best practices for S3 performance?

Q4: How do you implement S3 lifecycle policies for data lakes?

Q5: What is S3 Replication Time Control (RTC)?

Cost Considerations

Summary