Why This Matters

AWS Lambda is a serverless, event-driven compute service that lets you run code without provisioning or managing servers. For data engineers, Lambda is the glue that connects services, transforms data in real-time, triggers downstream processes, and eliminates the operational overhead of managing EC2 instances for short-lived tasks.

What is AWS Lambda?

AWS Lambda is a serverless, event-driven compute service that lets you run code without provisioning or managing servers. You pay only for the compute time you consume — there is no charge when your code is not running.

Core Characteristics

When Lambda Fits Data Engineering

• Event-driven ETL: React to new files landing in S3 or new records in Kinesis/DynamoDB Streams
• Micro-batch processing: Trigger small, frequent transformations without maintaining a cluster
• Orchestration glue: Connect Step Functions, Glue, Redshift, and other services
• Real-time alerting: Process CloudWatch alarms or custom metrics
• Data validation and enrichment: Validate incoming data, enrich with lookups, and route to destinations

Lambda Triggers for Data Engineering

Trigger Details

S3 Event Notifications

• Triggered on object create, delete, or restore events
• Supports filtering by prefix and suffix (e.g., only .csv files in raw/ prefix)
• Ideal for landing zone processing

DynamoDB Streams

• Captures item-level modifications (create, update, delete)
• Processing window: up to 5 minutes of retention
• Use cases: CDC processing, real-time analytics

Kinesis Data Streams

• Lambda polls shards automatically (5 records or 6 MB per batch)
• Use cases: real-time streaming ETL, log aggregation
• Checkpointing via DynamoDB to track shard position

SQS Queues

• Decouples producer and consumer processing
• Supports batch sizes of 1–10 messages
• Use cases: reliable asynchronous processing, retry logic

EventBridge (CloudWatch Events)

• Scheduled rules (cron/rate) for batch orchestration
• Event-driven rules for cross-service integration

Lambda Limits and Best Practices

Timeout Best Practices

Real-World Project Structure

Production Lambda Pipeline

lambda-pipeline/
├── lambda/
│   ├── validate-input/
│   │   ├── lambda_function.py
│   │   ├── requirements.txt
│   │   └── tests/
│   │       └── test_handler.py
│   ├── transform-data/
│   │   ├── lambda_function.py
│   │   └── requirements.txt
│   └── notify-completion/
│       └── lambda_function.py
├── layers/
│   ├── shared-utils/
│   │   ├── python/
│   │   │   └── lib/
│   │   │       └── python3.11/
│   │   │           └── site-packages/
│   │   └── requirements.txt
│   └── database-drivers/
│       ├── python/
│       └── requirements.txt
├── infrastructure/
│   ├── template.yaml
│   └── parameters.json
├── monitoring/
│   ├── alarms.json
│   └── dashboards.json
└── scripts/
    ├── deploy.sh
    └── test-invoke.sh

Production Python Code with Error Handling

import json
import boto3
import logging
from datetime import datetime

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3 = boto3.client('s3')
glue = boto3.client('glue')

def handler(event, context):
    try:
        for record in event['Records']:
            bucket = record['s3']['bucket']['name']
            key = record['s3']['object']['key']
            
            logger.info(json.dumps({
                'event': 'processing_file',
                'bucket': bucket,
                'key': key,
                'request_id': context.aws_request_id
            }))
            
            response = glue.start_job_run(
                JobName='etl-transform',
                Arguments={
                    '--input_path': f's3://{bucket}/{key}',
                    '--output_path': f's3://data-lake/processed/{key}',
                    '--format': 'parquet'
                }
            )
            
            logger.info(f"Started Glue job: {response['JobRunId']}")
        
        return {'statusCode': 200, 'body': 'Jobs triggered'}
        
    except Exception as e:
        logger.error(f"Error processing records: {str(e)}")
        raise

#!/bin/bash
# Deploy Lambda functions
set -euo pipefail

FUNCTION_NAME="data-processor"
RUNTIME="python3.11"
HANDLER="lambda_function.handler"
ROLE_ARN="arn:aws:iam::123456789012:role/LambdaExecutionRole"

echo "Deploying ${FUNCTION_NAME}"

zip -r function.zip lambda_function.py

aws lambda create-function \
    --function-name "${FUNCTION_NAME}" \
    --runtime "${RUNTIME}" \
    --role "${ROLE_ARN}" \
    --handler "${HANDLER}" \
    --zip-file fileb://function.zip \
    --timeout 300 \
    --memory-size 512 \
    --environment Variables="{ENV=production,LOG_LEVEL=INFO}"

echo "Deployment complete"

Mathematical Formulas

Lambda Cost Calculation

Total Cost = Request Cost + Compute Cost

Request Cost = Number of Requests × $0.20 / 1,000,000

Compute Cost = (Memory in GB × Duration in seconds) × $0.0000166667

Example:
  1,000,000 requests × 256MB × 2 seconds average
  = (1M × $0.20/1M) + (1M × 0.256GB × 2s × $0.0000166667)
  = $0.20 + $8.53
  = $8.73 per month

Throughput Calculation

Lambda Concurrency = (Request Rate × Average Duration) / 1 second

Example:
  1,000 requests/second × 2 second average duration
  = 2,000 concurrent executions

Reserved Concurrency = Account Limit - Used by Other Functions

Performance Considerations

Security Considerations

Interview Questions & Answers

Q1: When would you choose Lambda over Glue for a data transformation task?

Answer: Lambda is ideal for event-driven, micro-batch transformations on small data volumes (< 1 GB per invocation) where you need sub-second latency — for example, validating and routing incoming S3 files, transforming individual Kinesis records, or enriching data before writing to a staging area. Glue is better for scheduled, large-scale ETL (1 GB to TB) that requires Spark-based processing, automatic schema discovery via Crawlers, and integration with the Glue Data Catalog.

Q2: How do you handle Lambda cold starts in data pipelines?

Answer: Strategies include: (1) Provisioned Concurrency — pre-warm instances for latency-sensitive triggers, (2) Keep packages small — remove unused dependencies, use Lambda Layers, (3) Choose fast runtimes — Python and Node.js have shorter cold starts, (4) Avoid VPC attachment unless necessary, (5) Use Lambda SnapStart for Java, (6) Design for idempotency to handle retries gracefully.

Q3: Explain the difference between synchronous and asynchronous Lambda invocation.

Answer: Synchronous invocation: the caller waits for Lambda to execute and return a response. Used by API Gateway and Step Functions. Asynchronous invocation: Lambda queues the event and invokes the function independently. Used by S3 event notifications, SNS, and CloudWatch Events. Lambda manages retries (2 by default), and failed events can be sent to a DLQ. In data engineering, asynchronous invocation is more common.

Q4: How do you design a Lambda-based data pipeline that processes 1 million S3 files per day?

Answer: Key design considerations: (1) Batch files by prefix/date, (2) Set reserved concurrency to avoid overwhelming downstream systems, (3) Use SQS buffer between S3 notifications and Lambda processing, (4) Configure DLQ for failed files, (5) Implement idempotency with DynamoDB, (6) Set CloudWatch alarms on error rate and throttle count, (7) Tune timeout per file size.

Q5: What is the maximum execution time for Lambda?

Answer: The maximum execution time is 15 minutes (900 seconds). This impacts data pipeline design: large file processing must split files or use Glue/EMR, streaming workloads may need to increase batch size or add shards, and pipelines requiring more than 15 minutes should use Step Functions to chain multiple Lambda invocations.

Q6: How do you handle schema evolution in a Lambda-based ingestion pipeline?

Answer: Use a combination of techniques: (1) Centralized schema registry, (2) Lambda validation against registered schema, (3) Schema change detection by comparing incoming vs. known schema, (4) Backward compatibility for additive changes, (5) Versioned output with schema version metadata, (6) Route breaking changes to DLQ and alert the team.

Q7: Explain Lambda concurrency and how it affects data pipeline throughput.

Answer: Lambda concurrency refers to simultaneous invocations. Key concepts: (1) Account concurrency limit — default 1,000 per region, (2) Reserved concurrency — function-level limit that reserves account capacity, (3) Provisioned concurrency — pre-initialized instances that eliminate cold starts. For data pipelines, more concurrency = higher throughput, but high concurrency can overwhelm downstream systems.

Q8: What are Lambda Layers, and when should data engineers use them?

Answer: Lambda Layers are a distribution mechanism for shared code and dependencies mounted to /opt. Use layers when: (1) Multiple functions share the same dependencies, (2) Deployment packages exceed limits, (3) You want to update dependencies without redeploying code, (4) You need to share custom utilities. Common contents: database drivers, data processing libraries, custom utility modules.

Common Pitfalls

QuizBox

See Also

• AWS Step Functions
• AWS Glue
• AWS DynamoDB Streams