Why This Matters

Amazon DynamoDB is a fully managed, serverless NoSQL database that delivers single-digit millisecond performance at any scale. For data engineers, DynamoDB is critical because it provides the foundation for real-time data pipelines through DynamoDB Streams, enabling change data capture (CDC) patterns that feed analytics systems, data lakes, and downstream services.

What is Amazon DynamoDB?

Amazon DynamoDB is a fully managed, serverless NoSQL database that delivers single-digit millisecond performance at any scale. It supports both document and key-value data models and is designed to run high-throughput, low-latency applications.

Core Concepts

DynamoDB Data Model

DynamoDB Streams

DynamoDB Streams capture item-level modifications and make them available for processing.

Stream Configuration

import boto3

dynamodb = boto3.client('dynamodb')

# Enable DynamoDB Streams
dynamodb.update_table(
    TableName='orders',
    StreamSpecification={
        'StreamEnabled': True,
        'StreamViewType': 'NEW_AND_OLD_IMAGES'
    }
)

Stream View Types

Stream Processing with Lambda

import json
import boto3

dynamodb = boto3.resource('dynamodb')

def lambda_handler(event, context):
    for record in event['Records']:
        event_name = record['eventName']
        
        if event_name == 'INSERT':
            new_image = record['dynamodb']['NewImage']
            process_new_item(new_image)
        elif event_name == 'MODIFY':
            old_image = record['dynamodb']['OldImage']
            new_image = record['dynamodb']['NewImage']
            process_update(old_image, new_image)
        elif event_name == 'REMOVE':
            old_image = record['dynamodb']['OldImage']
            process_deletion(old_image)
    
    return {'statusCode': 200}

def process_new_item(new_image):
    table = dynamodb.Table('analytics')
    table.put_item(Item=new_image)
    print(f"Processed new item: {new_image['id']['S']}")

DynamoDB Design Patterns

Single-Table Design

Single-table design stores multiple entity types in one table using composite primary keys.

# Single-table design for e-commerce
# Partition Key: ENTITY#<type>#<id>
# Sort Key: METADATA | ORDER#<timestamp> | PRODUCT#<id>

items = [
    {
        'PK': 'ENTITY#CUSTOMER#12345',
        'SK': 'METADATA',
        'name': 'John Doe',
        'email': 'john@example.com'
    },
    {
        'PK': 'ENTITY#CUSTOMER#12345',
        'SK': 'ORDER#2026-01-15T10:30:00',
        'order_id': 'ORD-001',
        'total': 250.00
    },
    {
        'PK': 'ENTITY#PRODUCT#67890',
        'SK': 'METADATA',
        'name': 'Laptop',
        'price': 1299.99
    }
]

Access Patterns

Real-World Project Structure

Production DynamoDB Pipeline

dynamodb-pipeline/
├── tables/
│   ├── orders-table.json
│   └── customers-table.json
├── streams/
│   ├── stream-processor/
│   │   ├── lambda_function.py
│   │   └── requirements.txt
│   └── cdc-enricher/
│       └── lambda_function.py
├── analytics/
│   ├── athena-queries/
│   │   └── orders-analysis.sql
│   └── quicksight/
│       └── dashboard.json
├── monitoring/
│   ├── cloudwatch-alarms.json
│   └── dashboards.json
└── scripts/
    ├── backup-table.sh
    └── restore-table.sh

Production Python Code with Error Handling

import boto3
import json
import logging
from datetime import datetime

logger = logging.getLogger()
logger.setLevel(logging.INFO)

dynamodb = boto3.resource('dynamodb')
s3 = boto3.client('s3')

def lambda_handler(event, context):
    processed_count = 0
    error_count = 0
    
    for record in event['Records']:
        try:
            event_name = record['eventName']
            event_id = record['eventID']
            
            if event_name == 'INSERT':
                new_image = deserialize(record['dynamodb']['NewImage'])
                process_insert(new_image)
            elif event_name == 'MODIFY':
                old_image = deserialize(record['dynamodb']['OldImage'])
                new_image = deserialize(record['dynamodb']['NewImage'])
                process_modify(old_image, new_image)
            elif event_name == 'REMOVE':
                old_image = deserialize(record['dynamodb']['OldImage'])
                process_remove(old_image)
            
            processed_count += 1
            
        except Exception as e:
            error_count += 1
            logger.error(f"Error processing record {event_id}: {str(e)}")
    
    logger.info(json.dumps({
        'event': 'stream_processing_complete',
        'processed': processed_count,
        'errors': error_count
    }))
    
    return {
        'statusCode': 200,
        'body': {
            'processed': processed_count,
            'errors': error_count
        }
    }

def deserialize(image):
    """Convert DynamoDB JSON format to regular Python dict."""
    deserializer = boto3.dynamodb.types.TypeDeserializer()
    return {k: deserializer.deserialize(v) for k, v in image.items()}

#!/bin/bash
# Backup DynamoDB table to S3
set -euo pipefail

TABLE_NAME="orders"
BACKUP_S3="s3://dynamodb-backups/${TABLE_NAME}/$(date +%Y-%m-%d)"

echo "Starting backup of table: ${TABLE_NAME}"

aws dynamodb export-table-to-point-in-time \
    --table-arn "arn:aws:dynamodb:us-east-1:123456789012:table/${TABLE_NAME}" \
    --s3-bucket "dynamodb-backups" \
    --s3-prefix "${TABLE_NAME}/$(date +%Y-%m-%d)" \
    --export-format DYNAMODB_JSON

echo "Backup initiated. Check S3 for completion status."

Mathematical Formations

DynamoDB Capacity Calculation

Read Capacity Units (RCU):
  1 RCU = 1 strongly consistent read per second for items up to 4 KB
  1 RCU = 2 eventually consistent reads per second for items up to 4 KB

Write Capacity Units (WCU):
  1 WCU = 1 write per second for items up to 1 KB

Example:
  100 items/second × 2 KB average item size
  Read: 100 × 2 KB / 4 KB = 50 RCU
  Write: 100 × 2 KB / 1 KB = 200 WCU

Cost Calculation

On-Demand Mode:
  Read: $0.25 per million RCU
  Write: $1.25 per million WCU
  Storage: $0.25 per GB-month

Provisioned Mode:
  Read: $0.00013 per RCU-hour
  Write: $0.00065 per WCU-hour
  Storage: $0.25 per GB-month

Example (On-Demand):
  100 reads/sec + 50 writes/sec
  = (100 × 3,600,000 / 1M × $0.25) + (50 × 3,600,000 / 1M × $1.25)
  = $90 + $225
  = $315/month (excluding storage)

Performance Considerations

Security Considerations

Interview Questions & Answers

Q1: What is the difference between Query and Scan operations in DynamoDB?

Answer: Query retrieves items using the primary key (Partition Key + Sort Key). It's efficient and reads only the items that match. Scan reads every item in the table and is expensive. Use Query when you know the Partition Key and optionally the Sort Key range. Use Scan only when you must examine every item. Always add a FilterExpression to Scans to reduce consumed capacity.

Q2: How do you design DynamoDB tables for data engineering workloads?

Answer: Key design principles: (1) Single-table design for multiple access patterns, (2) Use composite primary keys (PK + SK) for efficient queries, (3) Design based on access patterns, not data relationships, (4) Use GSIs for alternate access patterns, (5) Keep items under 400 KB, (6) Use TTL for time-series data, (7) Enable DynamoDB Streams for CDC.

Q3: What are the costs associated with DynamoDB Streams?

Answer: DynamoDB Streams is priced per shard (2 MB/sec read, 1 MB/sec write). Each shard is $0.02 per hour. The first 2.5 million stream read requests per month are free. Lambda invocations from streams have their own pricing. In practice, streams are cost-effective for most workloads, but you should monitor shard usage and consider batching to reduce Lambda invocations.

Q4: How do you handle hot partitions in DynamoDB?

Answer: Hot partitions occur when one partition receives disproportionate traffic. Solutions: (1) Use distribute-like-counters for sequential access patterns, (2) Add random suffixes to partition keys, (3) Use adaptive capacity to handle bursts, (4) Consider DAX for read-heavy patterns, (5) Monitor CloudWatch metrics for consumed capacity.

Q5: What is single-table design and when should you use it?

Answer: Single-table design stores multiple entity types in one table using composite primary keys. Use it when: (1) You need to access related data efficiently, (2) You want to reduce the number of tables, (3) You need transactional guarantees across entities, (4) Your access patterns are well-defined. Avoid it when: (1) Entities are unrelated, (2) Access patterns change frequently, (3) You need complex queries across all items.

Q6: How do you implement CDC with DynamoDB Streams?

Answer: CDC implementation: (1) Enable DynamoDB Streams with NEW_AND_OLD_IMAGES, (2) Create Lambda function to process stream records, (3) Batch records for efficient processing, (4) Use DynamoDB checkpointing to track processed records, (5) Route to Kinesis Firehose for S3 delivery, (6) Handle errors with DLQ, (7) Monitor with CloudWatch metrics.

Q7: When would you choose DynamoDB over RDS for a data engineering workload?

Answer: Choose DynamoDB when: (1) You need single-digit millisecond latency, (2) Your data is key-value or document-based, (3) You need automatic scaling without configuration, (4) You want serverless operations, (5) Your access patterns are predictable. Choose RDS when: (1) You need complex SQL queries, (2) You need joins across multiple tables, (3) You have relational data with foreign keys, (4) You need ACID transactions across multiple tables.

Q8: How do you optimize DynamoDB costs for data engineering pipelines?

Answer: Cost optimization: (1) Use provisioned capacity with auto-scaling for predictable workloads, (2) Use on-demand for unpredictable traffic, (3) Enable TTL to automatically delete old data, (4) Use DAX to reduce read capacity consumption, (5) Batch writes to reduce WCU usage, (6) Monitor CloudWatch metrics to right-size capacity, (7) Use DynamoDB backups instead of snapshots for cost savings.

Common Pitfalls

QuizBox

See Also

• AWS Lambda
• AWS Kinesis
• AWS Glue