AWS Streaming Pipeline Patterns for Data Engineers

Streaming vs Batch Processing

Understanding when to use streaming versus batch processing is fundamental to designing effective data architectures.

Key Differences

When to Choose Streaming

• Real-time fraud detection requires immediate response
• IoT sensor monitoring generates continuous data
• Live application metrics need instant visibility
• User activity tracking benefits from low latency
• Stock price updates demand real-time processing

When to Choose Batch

• Historical analysis over large datasets
• ML model training on complete datasets
• Regulatory reporting with fixed schedules
• Cost-sensitive workloads with flexible timing

Kinesis-based Streaming Pipeline

Amazon Kinesis provides a fully managed streaming platform ideal for ingesting and processing real-time data at scale.

Core Components

Kinesis Data Streams (KDS) captures and stores data records from multiple sources with configurable retention (24 hours default, up to 365 days).

Kinesis Data Firehose automatically scales and delivers streaming data to destinations including S3, Redshift, Elasticsearch, and Splunk.

Pipeline Architecture

Pricing Model

MSK-based Streaming Pipeline

Amazon Managed Streaming for Apache Kafka (MSK) provides a fully managed Kafka service with automatic provisioning, patching, and backups.

Why Choose MSK Over Kinesis

• Familiar Kafka ecosystem - existing Kafka applications work with minimal changes
• Rich connector ecosystem - hundreds of pre-built connectors
• Consumer groups - native support for Kafka consumer groups
• Message replay - unlimited replay within retention period
• Partitioning - flexible partition strategies

MSK Pipeline Architecture

MSK Cluster Sizing Guide

Lambda Processing Patterns

AWS Lambda provides serverless compute for stream processing with automatic scaling and pay-per-invocation pricing.

Processing Pattern Comparison

Error Handling Strategy

Record Processing Flow:
1. Lambda receives batch of records
2. Process each record independently
3. On error: retry with exponential backoff
4. After max retries: send to DLQ
5. Bisect batch to isolate bad records
6. Monitor ErrorBacklogs metric

Windowing and Aggregation

Windowing enables processing of unbounded streams by grouping records into finite time-based or count-based segments.

Windowing Types

Kinesis Analytics Windowing

Kinesis Data Analytics supports SQL-based windowing for streaming aggregation:

-- Tumbling window example
SELECT 
  product_id,
  TUMBLE_START(rowtime, INTERVAL '1' HOUR) AS window_start,
  TUMBLE_END(rowtime, INTERVAL '1' HOUR) AS window_end,
  SUM(quantity) AS total_quantity,
  SUM(amount) AS total_sales
FROM sales_stream
GROUP BY product_id, TUMBLE(rowtime, INTERVAL '1' HOUR);

-- Sliding window example
SELECT 
  user_id,
  HOP_START(rowtime, INTERVAL '5' MINUTE, INTERVAL '1' HOUR) AS window_start,
  COUNT(*) AS action_count
FROM user_events
GROUP BY user_id, HOP(rowtime, INTERVAL '5' MINUTE, INTERVAL '1' HOUR);

Streaming Analytics

Real-time analytics enable organizations to derive insights from data as it arrives, enabling immediate action on business events.

Real-time Analytics Architecture

Analytics Service Comparison

Interview Q&A

Q1: What is the difference between Kinesis Data Streams and Kinesis Data Firehose?

Answer:

Kinesis Data Streams (KDS) provides:

• Real-time data streaming with sub-second latency
• Manual shard management and scaling
• Data retention up to 365 days
• Consumer-managed processing logic
• Exactly-once processing semantics possible
• Best for custom processing requirements

Kinesis Data Firehose provides:

• Managed delivery service with automatic scaling
• Batch delivery to destinations (S3, Redshift, Elasticsearch)
• Data transformation via Lambda (optional)
• Automatic compression and encryption
• Built-in error handling and retry logic
• Best for loading data into storage/analytics services

When to use each:

• Use KDS when you need real-time processing with custom logic
• Use Firehose when you need simple delivery to a destination
• Use both together: KDS for processing, Firehose for delivery

Q2: How would you design a streaming pipeline to handle exactly-once processing?

Answer:

Achieving exactly-once processing requires coordination across multiple components:

1. Producer Side:

• Use KPL with aggregation for deduplication
• Implement idempotent writes with unique record IDs
• Use Kafka transactions with MSK

2. Processing Side:

• Implement checkpointing with DynamoDB table
• Use sequence numbers for deduplication
• Store processed records with version tracking

3. Consumer Side:

• Maintain processed record IDs for 24+ hours
• Use DynamoDB conditional writes for deduplication
• Implement idempotent downstream operations

Architecture Example:

Producer → KDS → Lambda (checkpoint + dedup) → S3/DynamoDB
                     ↓
              DynamoDB (checkpoint table)
              [ShardID, SequenceNumber, Timestamp]

Key Points:

• Checkpointing ensures no data loss on failure
• Deduplication prevents duplicate processing
• Idempotent operations allow safe retries

Q3: Explain the Lambda scaling behavior with Kinesis streams.

Answer:

Lambda scales Kinesis consumers based on shard count and parallelization factor:

Scaling Formula:

Max concurrent = Number of Shards × ParallelizationFactor

Default Behavior:

• 1 Lambda invocation per shard
• Each invocation processes one batch at a time
• Scaling happens every minute

Example:

• 10 shards = 10 concurrent Lambda functions (default)
• With ParallelizationFactor = 5: up to 50 concurrent

Configuration Options:

Best Practices:

• Increase BatchSize for throughput, decrease for latency
• Use Provisioned Concurrency for predictable workloads
• Monitor ConcurrentExecutions metric
• Set account-level concurrency limits

Q4: What are the common pitfalls in streaming pipeline design?

Answer:

1. Ordering Issues:

• Problem: Records arrive out of order
• Solution: Use partition keys, maintain sequence numbers
• Kinesis: Same partition key = same shard = ordered

2. Late Data:

• Problem: Events arrive after window closes
• Solution: Use allowed lateness, watermarking
• Kinesis Analytics: WATERMARK clause for tolerance

3. Backpressure:

• Problem: Consumer can't keep up with producer
• Solution: Scale shards, implement DLQ, throttle producers
• Monitor IteratorAge metric

4. Error Handling:

• Problem: Failed records block processing
• Solution: DLQ, retry with backoff, batch splitting
• Use BisectBatchOnFunctionError

5. Cost Overruns:

• Problem: Unexpected scaling or data volume
• Solution: Set alarms, use on-demand capacity
• Monitor PutRecords.SuccessfulRecords

6. State Management:

• Problem: In-memory state lost on failure
• Solution: External state store (DynamoDB, Redis)
• Use KCL or Flink for managed state

Q5: How do you monitor and troubleshoot streaming pipelines?

Answer:

Key Metrics to Monitor:

Troubleshooting Steps:

1. High Iterator Age:

   • Increase shard count
   • Check Lambda concurrency limits
   • Verify IAM permissions

2. Lambda Errors:

   • Check CloudWatch logs
   • Verify data format
   • Test with sample data

3. Firehose Delivery Failures:

   • Check destination permissions
   • Verify transformation Lambda
   • Check compression format

4. MSK Issues:

   • Check broker health
   • Verify security groups
   • Monitor disk usage

Tools:

• CloudWatch dashboards
• X-Ray tracing
• CloudTrail for API calls
• MSK/Lambda logs

Q6: Compare MSK and Kinesis for a new streaming project.

Answer:

Choose Kinesis when:

• Team is new to streaming
• Simpler operational requirements needed
• Tight AWS integration required
• Cost predictability important
• Serverless preferred

Choose MSK when:

• Existing Kafka expertise in team
• Need Kafka ecosystem connectors
• Multi-cloud portability required
• Complex event processing needed
• Consumer group management required

Cost Comparison (100 GB/day):

Migration Path:

1. Start with Kinesis for simplicity
2. Migrate to MSK if Kafka features needed
3. Use MSK Connect for connector ecosystem
4. Consider serverless MSK for variable workloads

Decision Matrix:

Q7: Describe the dead-letter queue pattern for stream processing.

Answer:

A Dead-Letter Queue (DLQ) captures records that fail processing after maximum retry attempts, preventing pipeline blockage while preserving failed records for investigation.

Implementation Pattern:

Stream → Lambda → Success: Process & Checkpoint
                    ↓ (after N retries)
                 Failure: Send to DLQ
                    ↓
              DLQ (SQS/SNS)
                    ↓
              Error Handler Lambda
                    ↓
              Log/Alert/Reprocess

Lambda Configuration:

{
  "DestinationConfig": {
    "OnFailure": {
      "Destination": "arn:aws:sqs:us-east-1:123456789:stream-dlq"
    }
  },
  "MaximumRetryAttempts": 3,
  "BisectBatchOnFunctionError": true,
  "FunctionResponseTypes": ["ReportBatchItemFailures"]
}

DLQ Processing Strategy:

1. Monitor DLQ queue depth
2. Trigger Lambda on DLQ messages
3. Log error details to CloudWatch
4. Send alerts for critical failures
5. Provide reprocessing capability

Benefits:

• Pipeline resilience without data loss
• Error isolation and investigation
• Audit trail for compliance
• Flexible retry strategies

Q8: How would you handle schema evolution in streaming pipelines?

Answer:

Schema evolution requires careful planning to avoid breaking changes:

Strategy 1: Schema Registry

• Use AWS Glue Schema Registry
• Avro/JSON Schema validation
• Backward/forward compatibility
• Version management

Strategy 2: Defensive Schema Design

• All fields optional (except key)
• Use default values
• Avoid removing fields
• Version your schemas

Strategy 3: Schema-on-Read

• Store raw data in S3
• Apply schema during query
• Use Glue Crawlers for discovery

Implementation with Glue Schema Registry:

from aws_schema_registry import SchemaRegistryClient
from aws_schema_registry.avro import AvroSchema

client = SchemaRegistryClient()
schema = AvroSchema.parse(open('schema.avsc').read())

# Register schema
schema_id = client.register_schema('my-stream', schema)

# Validate records
client.validate('my-stream', record, schema_id)

Best Practices:

• Use Avro for complex schemas
• Test compatibility before deployment
• Version schemas in Git
• Document schema changes
• Monitor for deserialization errors

Q9: Explain the KCL (Kinesis Client Library) and its role.

Answer:

The Kinesis Client Library (KCL) simplifies building consumer applications that handle:

Key Functions:

1. Shard Discovery - Automatically discovers and tracks shard assignments
2. Load Balancing - Distributes shards across workers
3. Checkpointing - Manages processing progress in DynamoDB
4. Fault Tolerance - Rebalances after worker failures
5. Fan-out - Supports enhanced fan-out consumers

KCL Architecture:

Kinesis Stream → KCL Worker → Process Records
                    ↓
              DynamoDB (checkpoints)
                    ↓
              Application Logic

Code Example (Python):

from amazon_kclpy import kcl

class RecordProcessor(kcl.RecordProcessorBase):
    def process_records(self, records):
        for record in records:
            data = json.loads(record.data.decode('utf-8'))
            self.process(data)
        self.checkpoint(records[-1])

KCL vs Lambda:

When to Use:

• Complex processing logic
• Stateful operations
• Large-scale consumers
• Custom shard management

Q10: Design a streaming pipeline for real-time IoT data processing.

Answer:

Architecture Components:

1. Data Ingestion Layer:

IoT Devices → MQTT → AWS IoT Core → Rules Engine → Kinesis Data Streams

2. Processing Layer:

• Lambda for simple transforms and filtering
• Kinesis Analytics for windowed aggregations
• Flink for complex event processing

3. Storage Layer:

• S3 for raw and processed data (Data Lake)
• DynamoDB for real-time lookups
• Timestream for time-series metrics

4. Analytics Layer:

• QuickSight for dashboards
• Grafana for operational metrics
• Custom API for applications

Pipeline Diagram:

Configuration:

• Kinesis shards: 50 (1M records/min)
• Lambda memory: 512MB
• Kinesis Analytics: 5 KUs
• S3 lifecycle: IA after 30 days

Cost Estimate:

• IoT Core: ~$500/month
• Kinesis: ~$400/month
• Lambda: ~$200/month
• S3: ~$100/month
• Total: ~$1,200/month

Monitoring:

• IoT Rules delivery failures
• Kinesis IteratorAge
• Lambda errors and throttles
• S3 put latency

This architecture handles 1M messages/minute with sub-second latency and 99.99% availability.

Summary

This topic covered the key concepts of AWS data engineering. Review the architecture diagrams, practice the interview questions, and understand the trade-offs between different service options.

Next Steps

Continue to the next topic to build on your AWS data engineering knowledge.

Knowledge Check

See Also

• AWS Kinesis
• AWS MSK
• Real-time Analytics