π DynamoDB Streams
Master DynamoDB Streams CDC patterns and real-time replication.
Module: AWS Data Engineering β’ Topic 48 of 65 β’ Premium Content
DynamoDB Streams Architecture
Architecture Diagram
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DYNAMODB STREAMS ARCHITECTURE β
β β
β DynamoDB Table β Streams β Lambda β Downstream Services β
β β
β Stream Records: β
β β’ INSERT: New item created β
β β’ MODIFY: Item updated β
β β’ REMOVE: Item deleted β
β β
β Features: β
β β’ 24-hour retention β
β β’ Ordered per partition key β
β β’ At-least-once delivery β
β β’ 1 MB payload limit β
β β
β Use Cases: β
β β’ Real-time analytics β
β β’ Cross-region replication β
β β’ Materialized views β
β β’ Event-driven architectures β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Lambda Trigger Example
import json
from decimal import Decimal
def lambda_handler(event, context):
for record in event['Records']:
event_name = record['eventName']
if event_name == 'INSERT':
new_image = record['dynamodb']['NewImage']
process_insert(new_image)
elif event_name == 'MODIFY':
old_image = record['dynamodb'].get('OldImage', {})
new_image = record['dynamodb']['NewImage']
process_modify(old_image, new_image)
elif event_name == 'REMOVE':
old_image = record['dynamodb']['OldImage']
process_remove(old_image)
def deserialize(image):
return {k: list(v.values())[0] for k, v in image.items()}
Interview Q&A
Q1: What are DynamoDB Streams?
Answer: A log of item-level modifications (INSERT, MODIFY, REMOVE) with 24-hour retention. Lambda or Kinesis can consume records.
Q2: How do you enable Streams?
Answer: Enable at table creation or update. Choose key-based or new/old image view.
Q3: What is the difference between Streams and Kinesis?
Answer: Streams: per-table, 24hr retention, limited throughput. Kinesis: independent, configurable retention, higher throughput.
Summary
- Events: INSERT, MODIFY, REMOVE item-level changes
- Retention: 24 hours
- Ordering: Per partition key
- Integration: Lambda triggers, Kinesis consumption
- Use Cases: CDC, real-time analytics, cross-region replication