AWS Batch Processing for Data Engineers
AWS Batch Processing
Scheduled ETL Jobs, AWS Batch, and Batch Data Architectures
What is Batch Processing?
Batch processing is a data processing paradigm where jobs are collected, grouped, and executed as a single unit of work at scheduled intervals. Unlike stream processing that handles data in real-time, batch processing accumulates data over a period (hourly, daily, weekly) and processes it all at once.
đ¯
Interview Pro Tip: This concept is frequently asked in data engineering interviews. Be ready to explain the "why" behind it, not just the "what." Connect it to real-world scenarios and trade-offs.
Core Characteristics
| Feature | Description |
|---|---|
| Latency | Minutes to hours (acceptable delay) |
| Data Volume | Large datasets processed together |
| Scheduling | Time-based triggers (cron, EventBridge rules) |
| Idempotency | Jobs can be safely retried without side effects |
| Cost Efficiency | Leverages spot instances, reserved capacity |
Common Batch Use Cases
- Daily ETL pipelines: Extract from databases, transform, load into data warehouses
- Weekly aggregations: Roll up transaction data into summary tables
- Monthly financial reports: Aggregate revenue, calculate metrics
- Data warehouse refreshes: Full or incremental loads into Redshift/BigQuery
- ML model training: Periodic retraining on accumulated data
- Log archival: Compress and archive logs to S3 Glacier
đ
Deep Dive: Batch Processing Patterns
AWS Batch provides managed batch computing. Understanding job queues, compute environments, and retry strategies is essential. Learn more in our Batch vs Streaming guide and Pipeline Monitoring for observability.
SVG: Batch Processing Lifecycle
AWS Batch
AWS Batch is a fully managed service that enables developers to run batch computing workloads on AWS. It dynamically provisions optimal compute resources based on the volume and specific requirements of the submitted jobs.
Key Components
â ī¸
Common Interview Mistake: Don't just list features. Explain WHY each feature matters for data engineering and when you'd choose one option over another.
1. Job Definitions
A job definition specifies how a job should be run. It acts as a blueprint for your batch jobs.
2. Job Queues
Job queues receive submitted jobs and hold them until compute resources are available.
| Queue Type | Description |
|---|---|
| FIRST_CANCELLED | Oldest submitted job cancelled when capacity removed |
| FAIR_SHARE | Resources distributed equally across fair share groups |
| fifo | Jobs processed in FIFO order |
3. Compute Environments
Compute environments define the compute resources used to run jobs.
Managed Compute Environment:
resource "aws_batch_compute_environment" "managed" {
compute_environment_name = "batch-managed-env"
type = "MANAGED"
compute_resources {
type = "EC2"
min_vcpus = 0
max_vcpus = 256
instance_type = [
"optimal",
"c5.large",
"c5.xlarge",
"m5.large"
]
subnets = [aws_subnet.private.id]
security_groups = [aws_security_group.batch.id]
instance_role = aws_iam_instance_profile.batch.arn
allocation_strategy = "SPOT_CAPACITY_OPTIMIZED"
}
}
Fargate Compute Environment:
resource "aws_batch_compute_environment" "fargate" {
compute_environment_name = "batch-fargate"
type = "MANAGED"
compute_resources {
type = "FARGATE"
max_vcpus = 50
subnets = [aws_subnet.private.id]
security_groups = [aws_security_group.batch.id]
}
}
AWS Batch Job States
Architecture Diagram
SUBMITTED â PENDING â RUNNABLE â STARTED â SUCCEEDED
â
FAILED
| State | Description |
|---|---|
SUBMITTED | Job received by AWS Batch |
PENDING | Infrastructure being prepared |
RUNNABLE | Container created, waiting to run |
STARTED | Container executing |
SUCCEEDED | Job completed successfully |
FAILED | Job encountered an error |
SVG: AWS Batch Architecture
Scheduled Batch Pipelines
Scheduled batch pipelines use Amazon EventBridge to trigger AWS Step Functions workflows at defined intervals. This creates reliable, observable batch processing pipelines.
Pipeline Architecture Pattern
Step Functions Workflow Definition
{
"Comment": "Daily ETL Pipeline",
"StartAt": "ValidateInput",
"States": {
"ValidateInput": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456:function:validate-input",
"Next": "CheckDataFreshness"
},
"CheckDataFreshness": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.dataIsFresh",
"BooleanEquals": true,
"Next": "RunETLJob"
}
],
"Default": "NotifyDataStale"
},
"RunETLJob": {
"Type": "Task",
"Resource": "arn:aws:batch:us-east-1:123456:job-definition/etl-job",
"TimeoutSeconds": 7200,
"Retry": [
{
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 300,
"MaxAttempts": 3,
"BackoffRate": 2
}
],
"Next": "ValidateOutput"
},
"ValidateOutput": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456:function:validate-output",
"Next": "NotifySuccess"
},
"NotifySuccess": {
"Type": "Task",
"Resource": "arn:aws:sns:us-east-1:123456:etl-notifications",
"Parameters": {
"Message": "Daily ETL completed successfully",
"TopicArn": "arn:aws:sns:us-east-1:123456:etl-notifications"
},
"End": true
},
"NotifyDataStale": {
"Type": "Task",
"Resource": "arn:aws:sns:us-east-1:123456:etl-notifications",
"Parameters": {
"Message": "Data freshness check failed - pipeline halted",
"TopicArn": "arn:aws:sns:us-east-1:123456:etl-notifications"
},
"End": true
}
}
}
EventBridge Schedule Configuration
resource "aws_cloudwatch_event_rule" "daily_etl" {
name = "daily-etl-trigger"
description = "Triggers daily ETL pipeline at 2 AM UTC"
schedule_expression = "cron(0 2 * * ? *)"
}
resource "aws_cloudwatch_event_target" "step_functions" {
rule = aws_cloudwatch_event_rule.daily_etl.name
arn = aws_sfn_state_machine.etl_pipeline.arn
role_arn = aws_iam_role.eventbridge_stepfunctions.arn
}
SVG: Scheduled Pipeline Architecture
Batch vs Streaming Comparison
| Aspect | Batch Processing | Stream Processing |
|---|---|---|
| Latency | Minutes to hours | Milliseconds to seconds |
| Data Scope | Finite, bounded datasets | Unbounded, continuous data |
| Cost | Lower (spot instances, reserved) | Higher (always-on resources) |
| Complexity | Simpler error handling | Complex state management |
| Throughput | High (processed in bulk) | Moderate (per-record) |
| Use Cases | ETL, reports, ML training | Real-time dashboards, fraud detection |
| Tools | AWS Batch, Glue, EMR | Kinesis, MSK, Flink |
| Scheduling | Cron-based triggers | Continuous ingestion |
When to Choose Batch
- Processing can tolerate latency (minutes to hours)
- Data arrives in predictable volumes
- Complex transformations requiring full dataset visibility
- Cost optimization is critical
- Results feed into downstream systems that update periodically
When to Choose Streaming
- Real-time alerting or dashboards required
- Data must be processed as it arrives
- Event-driven architectures
- Low-latency user experiences needed
SVG: Batch vs Streaming Diagram
Batch Processing Patterns
Pattern 1: Daily ETL Pipeline
The most common batch pattern - extract data from sources, transform it, and load into a target system.
import boto3
import json
from datetime import datetime, timedelta
def lambda_handler(event, context):
"""
Daily ETL Orchestrator
Triggered by EventBridge at 2 AM UTC
"""
s3 = boto3.client('s3')
batch = boto3.client('batch')
# Calculate processing window
yesterday = datetime.utcnow() - timedelta(days=1)
processing_date = yesterday.strftime('%Y-%m-%d')
# Submit ETL job
job = batch.submit_job(
jobName=f'daily-etl-{processing_date}',
jobQueue='etl-production-queue',
jobDefinition='etl-job-definition',
containerOverrides={
'environment': [
{'name': 'PROCESSING_DATE', 'value': processing_date}, {'name': 'INPUT_PREFIX', 'value': f'raw/{processing_date}/'}, {'name': 'OUTPUT_PREFIX', 'value': f'processed/{processing_date}/'}
]
},
retryStrategy={
'attempts': 3
},
timeout={
'attemptDurationSeconds': 7200
}
)
return {
'statusCode': 200,
'jobId': job['jobId'],
'processingDate': processing_date
}
Pattern 2: Weekly Aggregation
Aggregate daily data into weekly summary tables for reporting.
Weekly Aggregation SQL Pattern:
-- Weekly sales aggregation
INSERT INTO weekly_sales_summary
WITH daily_sales AS (
SELECT
DATE_TRUNC('day', transaction_date) as sale_date,
product_category,
region,
SUM(amount) as daily_revenue,
COUNT(*) as daily_transactions
FROM raw_transactions
WHERE transaction_date BETWEEN
DATE_TRUNC('week', CURRENT_DATE - INTERVAL '7 days')
AND DATE_TRUNC('week', CURRENT_DATE) - INTERVAL '1 day'
GROUP BY 1, 2, 3
)
SELECT
DATE_TRUNC('week', sale_date) as week_start,
product_category,
region,
SUM(daily_revenue) as weekly_revenue,
SUM(daily_transactions) as weekly_transactions,
AVG(daily_revenue) as avg_daily_revenue,
MAX(daily_revenue) as peak_daily_revenue,
CURRENT_TIMESTAMP as aggregated_at
FROM daily_sales
GROUP BY 1, 2, 3;
Pattern 3: Monthly Report Generation
Generate comprehensive monthly reports with multiple data sources.
Pattern 4: Incremental Processing
Process only new or changed data since the last run.
def get_incremental_boundary(table_name):
"""
Query the bookmark table to find last processed timestamp
"""
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('etl_bookmarks')
response = table.get_item(
Key={'table_name': table_name}
)
if 'Item' in response:
return response['Item']['last_processed_timestamp']
else:
# First run - go back 7 days
return (datetime.utcnow() - timedelta(days=7)).isoformat()
def update_bookmark(table_name, timestamp):
"""
Update the bookmark after successful processing
"""
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('etl_bookmarks')
table.put_item(
Item={
'table_name': table_name,
'last_processed_timestamp': timestamp,
'updated_at': datetime.utcnow().isoformat()
}
)
SVG: Batch Processing Patterns Overview
AWS Batch Cost Optimization
Spot Instance Strategy
resource "aws_batch_compute_environment" "spot_optimized" {
compute_environment_name = "spot-batch-env"
type = "MANAGED"
compute_resources {
type = "EC2"
min_vcpus = 0
max_vcpus = 512
allocation_strategy = "SPOT_CAPACITY_OPTIMIZED"
instance_type = [
"c5.large",
"c5.xlarge",
"c5.2xlarge",
"m5.large",
"m5.xlarge"
]
spot_iam_fleet_role = aws_iam_role.spot_fleet.arn
}
service_role = aws_iam_role.batch_service.arn
}
Cost Comparison
| Instance Type | On-Demand (per hr) | Spot (per hr) | Savings |
|---|---|---|---|
| c5.large | 0.032 | 62% | |
| c5.xlarge | 0.064 | 62% | |
| m5.large | 0.038 | 60% | |
| m5.xlarge | 0.077 | 60% |
Right-Sizing Recommendations
def analyze_job_resources(job_name, days=30):
"""
Analyze historical job runs to recommend optimal resource allocation
"""
cloudwatch = boto3.client('cloudwatch')
# Get average CPU utilization
cpu_response = cloudwatch.get_metric_statistics(
Namespace='AWS/Batch',
MetricName='CPUUtilization',
Dimensions=[
{'Name': 'JobQueue', 'Value': f'{job_name}-queue'},
],
StartTime=datetime.utcnow() - timedelta(days=days),
EndTime=datetime.utcnow(),
Period=86400,
Statistics=['Average', 'Maximum']
)
avg_cpu = sum([d['Average'] for d in cpu_response['Datapoints']]) / len(cpu_response['Datapoints'])
max_cpu = max([d['Maximum'] for d in cpu_response['Datapoints']])
# Recommendation logic
if avg_cpu < 30:
recommendation = "Consider reducing vCPUs by 50%"
elif avg_cpu > 80:
recommendation = "Consider increasing vCPUs by 25%"
else:
recommendation = "Current sizing is appropriate"
return {
'job_name': job_name,
'avg_cpu_utilization': avg_cpu,
'max_cpu_utilization': max_cpu,
'recommendation': recommendation
}
Interview Q&A
Q1: What is the difference between AWS Batch and AWS Glue?
Answer:
| Aspect | AWS Batch | AWS Glue |
|---|---|---|
| Purpose | General-purpose batch computing | Serverless ETL and data integration |
| Workloads | Any batch workload (ML training, simulations, ETL) | ETL specifically (extract, transform, load) |
| Compute | EC2, Fargate, Spot | Fully managed Spark servers |
| Orchestration | Step Functions integration | Built-in job bookmarks |
| Data Catalog | Not included | Built-in Glue Data Catalog |
| Cost Model | Per compute second | Per Data Processing Unit (DPU) hour |
When to use Batch: Custom batch processing, ML training, non-ETL workloads requiring specific runtime environments.
When to use Glue: Standard ETL jobs, data integration, crawlers for schema discovery, Spark-based transformations.
Q2: How do you handle failures in AWS Batch jobs?
Answer:
AWS Batch provides multiple failure handling mechanisms:
# Job Definition with retry and timeout
job_definition = {
'jobDefinitionName': 'resilient-etl-job',
'containerProperties': {
'image': 'my-etl:latest',
'vcpus': 4,
'memory': 8192
},
'retryStrategy': {
'attempts': 3,
'evaluateOnExit': [
{
'onStatusReason': 'Host EC2*',
'action': 'RETRY'
}, {
'onReason': 'peak*',
'action': 'RETRY'
}, {
'onExitCode': '1',
'action': 'EXIT'
}
]
},
'timeout': {
'attemptDurationSeconds': 3600
}
}
Key strategies:
- Retry policies: Automatic retries with exponential backoff
- Timeout: Prevent hung jobs from consuming resources
- Dead-letter queues: Capture failed jobs for investigation
- Step Functions: Orchestrate retries with human approval steps
- Idempotent design: Safe to rerun without side effects
Q3: How do you optimize costs for AWS Batch workloads?
Answer:
- Spot Instances: Use
SPOT_CAPACITY_OPTIMIZEDallocation strategy for 60-90% savings - Right-sizing: Monitor CPU/memory utilization and adjust instance types
- Auto-scaling: Set appropriate min/max vCPUs to handle variable workloads
- Job consolidation: Batch multiple small jobs into larger ones
- Fargate for sporadic workloads: No idle capacity costs
- Reserved capacity: For predictable baseline workloads
# Cost-optimized compute environment
resource "aws_batch_compute_environment" "cost_optimized" {
compute_environment_name = "cost-optimized"
type = "MANAGED"
compute_resources {
type = "EC2"
min_vcpus = 0
max_vcpus = 256
allocation_strategy = "SPOT_CAPACITY_OPTIMIZED"
instance_type = ["optimal"]
}
}
Q4: Explain the AWS Batch job lifecycle states.
Answer:
Architecture Diagram
SUBMITTED â PENDING â RUNNABLE â STARTED â SUCCEEDED
â
FAILED
â
(retry attempts)
â
FAILED (final)
- SUBMITTED: Job received by AWS Batch, awaiting scheduling
- PENDING: Infrastructure being provisioned
- RUNNABLE: Container created, waiting to be started
- STARTED: Container is executing
- SUCCEEDED: Job completed with exit code 0
- FAILED: Job encountered error (non-zero exit code or timeout)
Common causes for stuck states:
PENDING: Insufficient capacity, compute environment scaled to 0RUNNABLE: Container image pull issuesSTARTED: Application errors, OOM kills
Q5: How would you implement idempotent batch jobs?
Answer:
Idempotency ensures jobs can be safely retried without producing duplicate results.
import hashlib
from datetime import datetime
def process_daily_data(processing_date, run_id):
"""
Idempotent batch job implementation
"""
# 1. Check if already processed
if is_already_processed(processing_date, run_id):
return {'status': 'skipped', 'reason': 'already processed'}
# 2. Process with deterministic output
input_data = extract_data(processing_date)
transformed_data = transform_data(input_data)
# 3. Write with conditional expression
write_to_s3(
bucket='processed-data',
key=f'daily/{processing_date}/data.parquet',
data=transformed_data,
metadata={'run_id': run_id, 'processed_at': datetime.utcnow().isoformat()}
)
# 4. Mark as processed (only after successful write)
mark_as_processed(processing_date, run_id)
return {'status': 'success', 'records_processed': len(transformed_data)}
def is_already_processed(processing_date, run_id):
"""Check DynamoDB for existing processing record"""
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('processing_bookmarks')
response = table.get_item(
Key={
'processing_date': processing_date,
'job_name': 'daily-etl'
}
)
if 'Item' in response:
return response['Item']['run_id'] == run_id
return False
Q6: What are the best practices for scheduling batch jobs on AWS?
Answer:
- Use EventBridge rules for cron-based scheduling with built-in retry
- Implement circuit breakers to pause scheduling during known outages
- Add monitoring with CloudWatch alarms on job failures
- Use Step Functions for complex multi-step workflows
- Implement job dependencies with Step Functions or custom logic
# EventBridge schedule with error handling
resource "aws_cloudwatch_event_rule" "daily_etl" {
name = "daily-etl-schedule"
schedule_expression = "cron(0 2 * * ? *)"
state = "ENABLED"
}
resource "aws_cloudwatch_event_target" "step_functions" {
rule = aws_cloudwatch_event_rule.daily_etl.name
arn = aws_sfn_state_machine.etl_pipeline.arn
role_arn = aws_iam_role.eventbridge_invoke.arn
input = jsonencode({
'pipeline' = 'daily-etl'
'trigger' = 'scheduled'
})
}
# CloudWatch alarm for job failures
resource "aws_cloudwatch_metric_alarm" "etl_failure" {
alarm_name = "daily-etl-failures"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
metric_name = "FailedJobs"
namespace = "AWS/Batch"
period = 3600
statistic = "Sum"
threshold = 0
alarm_description = "Daily ETL job failed"
alarm_actions = [aws_sns_topic.alerts.arn]
dimensions = {
JobQueue = "etl-production-queue"
}
}
Q7: How do you handle data dependencies between batch jobs?
Answer:
For dependent job chains, use Step Functions to orchestrate execution order:
{
"Comment": "Job Chain with Dependencies",
"StartAt": "Extract",
"States": {
"Extract": {
"Type": "Task",
"Resource": "arn:aws:batch:job/extract-job",
"Next": "Transform"
},
"Transform": {
"Type": "Task",
"Resource": "arn:aws:batch:job/transform-job",
"Next": "Load"
},
"Load": {
"Type": "Task",
"Resource": "arn:aws:batch:job/load-job",
"Next": "Validate"
},
"Validate": {
"Type": "Task",
"Resource": "arn:aws:lambda:validate-output",
"End": true
}
}
}
For parallel jobs with shared dependencies, use Map state in Step Functions:
{
"Type": "Map",
"ItemsPath": "$.regions",
"MaxConcurrency": 5,
"Iterator": {
"StartAt": "ProcessRegion",
"States": {
"ProcessRegion": {
"Type": "Task",
"Resource": "arn:aws:batch:job/region-processor",
"End": true
}
}
}
}
Q8: What metrics should you monitor for batch processing systems?
Answer:
| Metric | Source | Alarm Threshold |
|---|---|---|
| Job Success Rate | CloudWatch | < 95% |
| Job Duration | CloudWatch | > SLA threshold |
| Queue Depth | CloudWatch | > 100 pending jobs |
| CPU Utilization | CloudWatch | > 80% sustained |
| Memory Utilization | CloudWatch | > 85% sustained |
| Spot Instance Interruptions | EC2 Events | Any interruption |
| Failed Job Count | CloudWatch | > 0 |
# CloudWatch dashboard for batch monitoring
def create_batch_dashboard():
cloudwatch = boto3.client('cloudwatch')
dashboard_body = {
'widgets': [
{
'type': 'metric',
'properties': {
'metrics': [
['AWS/Batch', 'SubmittedJobs', 'JobQueue', 'production'],
['AWS/Batch', 'RunningJobs', 'JobQueue', 'production'],
['AWS/Batch', 'FailedJobs', 'JobQueue', 'production']
],
'period': 300,
'stat': 'Sum',
'region': 'us-east-1',
'title': 'Job Status'
}
}, {
'type': 'metric',
'properties': {
'metrics': [
['AWS/Batch', 'CPUUtilization', 'JobQueue', 'production'],
['AWS/Batch', 'MemoryUtilization', 'JobQueue', 'production']
],
'period': 300,
'stat': 'Average',
'region': 'us-east-1',
'title': 'Resource Utilization'
}
}
]
}
cloudwatch.put_dashboard(
DashboardName='BatchProcessing',
DashboardBody=json.dumps(dashboard_body)
)
Q9: How do you implement retry logic with exponential backoff for AWS Batch?
Answer:
AWS Batch supports automatic retries via job definition configuration:
{
"retryStrategy": {
"attempts": 3,
"evaluateOnExit": [
{
"onStatusReason": "Host EC2*",
"action": "RETRY"
}, {
"onReason": "Task * timed out",
"action": "RETRY"
}, {
"onExitCode": "0",
"action": "EXIT"
}, {
"onExitCode": "!0",
"action": "EXIT"
}
]
}
}
For custom backoff logic, use Step Functions:
{
"Retry": [
{
"ErrorEquals": ["States.TaskFailed"],
"IntervalSeconds": 60,
"MaxAttempts": 3,
"BackoffRate": 2
}
]
}
This creates retry intervals: 60s â 120s â 240s.
Q10: Describe a production batch processing architecture you would design.
Answer:
Architecture Components:
-
Ingestion Layer
- S3 buckets with lifecycle policies
- Kinesis Data Firehose for streaming ingestion to S3
-
Processing Layer
- AWS Batch with mixed compute (Fargate for small jobs, EC2 Spot for large)
- Step Functions for orchestration
- EventBridge for scheduling
-
Storage Layer
- S3 Data Lake (Bronze/Silver/Gold zones)
- Redshift for analytics
- DynamoDB for bookmarks and metadata
-
Monitoring Layer
- CloudWatch dashboards and alarms
- SNS notifications for failures
- X-Ray for tracing
-
Governance
- AWS Lake Formation for access control
- Glue Data Catalog for metadata
- CloudTrail for auditing
Key Design Decisions:
- Idempotency: Every job uses processing date + run ID for deduplication
- Partitioning: Data partitioned by date for efficient incremental processing
- Cost Optimization: 80% Spot instances with Fargate fallback
- Error Handling: Dead-letter queues with automated remediation
- SLA Management: Step Functions with timeout and escalation paths
Summary
Batch processing on AWS provides a robust, scalable, and cost-effective approach to handling large-scale data processing workloads. By leveraging AWS Batch, Step Functions, and EventBridge, you can build reliable pipelines that process data on schedule while optimizing costs through Spot instances and right-sizing.
Key Takeaways
- AWS Batch manages compute provisioning automatically with support for EC2, Fargate, and Spot
- Step Functions provide workflow orchestration with built-in error handling and retry logic
- EventBridge enables flexible scheduling with cron expressions and event-driven triggers
- Idempotency and bookmarking are essential for reliable batch processing
- Cost optimization through Spot instances can reduce compute costs by 60-90%
- Monitoring with CloudWatch ensures visibility into job performance and failures