AWS Step Functions for Data Engineers

Mastering Workflow Orchestration for Data Pipelines

What is Step Functions?

AWS Step Functions is a serverless orchestration service that coordinates multiple AWS services into visual workflows. For data engineers, it eliminates the need to build custom orchestration frameworks by providing a fully managed way to build, run, and debug complex data pipelines.

Core Concepts

Why Step Functions for Data Engineering?

• Visual Workflows: See your entire pipeline at a glance in the AWS Console
• Built-in Error Handling: Retry and catch mechanisms reduce boilerplate code
• Serverless: No infrastructure to manage, patch, or scale
• Service Integration: Direct integration with Glue, Lambda, EMR, Redshift, Athena, and more
• Cost Efficient: Pay only for state transitions, no idle costs
• Debugging: Visual execution history shows exactly where failures occur
• Versioning: Tag and version your workflow definitions for CI/CD

Key Benefits Over Custom Orchestration

Building your own orchestrator (e.g., with Airflow or custom Lambda chains) requires handling retries, state tracking, and failure recovery manually. Step Functions provides all of this out of the box with a managed, fault-tolerant runtime.

State Machine Concept

Standard vs Express Workflows

Step Functions offers two workflow types designed for different use cases:

Standard Workflows

Express Workflows

Decision Framework

Do you need executions longer than 5 minutes?
  Yes --> Standard Workflow
  No -->
    Do you need more than 2,000 executions per second?
      Yes --> Express Workflow
      No -->
        Is cost your primary concern at high volume?
          Yes --> Express Workflow (cheaper per-exec)
          No --> Either works; Standard for full history

Workflow Types Comparison

State Types

Step Functions provides five core state types. Mastering each one is essential for building robust data pipelines.

1. Task State

The workhorse of Step Functions. Invokes AWS services or Lambda functions to perform actual work.

{
  "Type": "Task",
  "Resource": "arn:aws:glue:us-east-1:123456789012:job/my-etl-job",
  "Next": "LoadToRedshift",
  "Retry": [
    {
      "ErrorEquals": ["States.TaskFailed"],
      "IntervalSeconds": 30,
      "MaxAttempts": 3,
      "BackoffRate": 2
    }
  ],
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "Next": "HandleFailure"
    }
  ]
}

Data Engineering Uses:

• Run Glue ETL jobs
• Invoke Lambda transformations
• Start EMR or SageMaker jobs
• Execute Redshift Data API queries
• Call Athena named queries

2. Choice State

Makes decisions based on input. Essential for conditional data routing in pipelines.

{
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.recordCount",
      "NumericGreaterThan": 1000000,
      "Next": "ParallelProcess"
   }, {
      "Variable": "$.recordCount",
      "NumericLessThanOrEqual": 1000000,
      "Next": "SequentialProcess"
    }
  ],
  "Default": "ErrorState"
}

Data Engineering Uses:

• Route based on data volume (small vs large datasets)
• Select processing path by data type (CSV vs JSON vs Parquet)
• Skip downstream steps when source has no new data
• Fan-out to different pipelines based on region or tenant

3. Wait State

Pauses execution for a fixed duration or until a specific timestamp.

{
  "Type": "Wait",
  "Seconds": 300,
  "Next": "CheckDataReady"
}

Data Engineering Uses:

• Wait for upstream data arrival on a schedule
• Rate limiting external API calls
• Retry after a cooldown period
• Wait for a specific business time window

4. Parallel State

Executes multiple branches concurrently. Each branch runs its own sub-workflow.

{
  "Type": "Parallel",
  "Branches": [
    {
      "StartAt": "ProcessRegionUS",
      "States": {
        "ProcessRegionUS": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:us-east-1:function:process-us",
          "End": true
        }
      }
   }, {
      "StartAt": "ProcessRegionEU",
      "States": {
        "ProcessRegionEU": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:eu-west-1:function:process-eu",
          "End": true
        }
      }
    }
  ],
  "Next": "MergeResults"
}

Data Engineering Uses:

• Process multiple data regions in parallel
• Run independent validation checks simultaneously
• Aggregate results from multiple source systems
• Partition-based parallel ETL

5. Map State

Iterates over a collection of items. Essential for dynamic partitioning and batch processing.

{
  "Type": "Map",
  "ItemsPath": "$.fileList",
  "MaxConcurrency": 10,
  "Iterator": {
    "StartAt": "ProcessFile",
    "States": {
      "ProcessFile": {
        "Type": "Task",
        "Resource": "arn:aws:lambda:us-east-1:function:process-file",
        "End": true
      }
    }
  },
  "Next": "AggregateResults"
}

Data Engineering Uses:

• Process a dynamic list of S3 files
• Iterate over database partitions
• Parallel record-by-record transformation
• Dynamic fan-out based on input data

State Types Reference Diagram

Error Handling

Robust error handling is critical for production data pipelines. Step Functions provides Catch and Retry mechanisms built into every Task state.

Retry Configuration

"Retry": [
  {
    "ErrorEquals": ["States.TaskFailed"],
    "IntervalSeconds": 30,
    "MaxAttempts": 3,
    "BackoffRate": 2
  }
]

Catch Configuration

"Catch": [
  {
    "ErrorEquals": ["States.TaskFailed"],
    "ResultPath": "$.error",
    "Next": "FallbackState"
  }
]

Built-in Error Types

Error Handling Flow

Practical Example: Glue Job with Full Error Handling

{
  "Type": "Task",
  "Resource": "arn:aws:states:::glue:startJobRun.sync",
  "Parameters": {
    "JobName": "daily-etl-transform",
    "Arguments": {
      "--source_path.$": "$.sourcePath",
      "--target_path.$": "$.targetPath"
    }
  },
  "ResultPath": "$.glueResult",
  "Retry": [
    {
      "ErrorEquals": ["States.TaskFailed"],
      "IntervalSeconds": 60,
      "MaxAttempts": 3,
      "BackoffRate": 2
   }, {
      "ErrorEquals": ["States.Timeout"],
      "IntervalSeconds": 30,
      "MaxAttempts": 2,
      "BackoffRate": 1.5
    }
  ],
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "ResultPath": "$.error",
      "Next": "NotifyFailure"
    }
  ],
  "Next": "ValidateOutput"
}

Data Engineering Patterns with Step Functions

Pattern 1: ETL Orchestration

The most common pattern: coordinate extraction, transformation, and loading across multiple AWS services.

Pattern 2: Fan-Out / Fan-In

Use Parallel state to process data partitions concurrently, then merge results.

Use Case: Process 100 S3 partitions in parallel, aggregate counts, load summary to Redshift.

{
  "Type": "Parallel",
  "Branches": [
    { "StartAt": "ProcessPartition1", "States": { "ProcessPartition1": { "Type": "Task", "Resource": "arn:aws:lambda:process", "End": true } }}, { "StartAt": "ProcessPartition2", "States": { "ProcessPartition2": { "Type": "Task", "Resource": "arn:aws:lambda:process", "End": true } } }
  ],
  "ResultPath": "$.partitionResults",
  "Next": "AggregateResults"
}

Pattern 3: Dynamic Partitioning with Map

Process a variable number of files without knowing the count at definition time.

{
  "Type": "Map",
  "ItemsPath": "$.files",
  "MaxConcurrency": 20,
  "Iterator": {
    "StartAt": "ProcessFile",
    "States": {
      "ProcessFile": {
        "Type": "Task",
        "Resource": "arn:aws:lambda:process-file",
        "End": true
      }
    }
  }
}

Pattern 4: Human-in-the-Loop

Use Step Functions Activity for manual approval steps.

{
  "Type": "Task",
  "Resource": "arn:aws:states:activity:DataReview",
  "TimeoutSeconds": 86400,
  "Next": "ProcessApproval"
}

Pattern 5: Scheduled Pipeline with Wait

Orchestrate time-dependent data loads with scheduled triggers.

{
  "Type": "Wait",
  "Timestamp": "2026-01-15T06:00:00Z",
  "Next": "LoadDailyData"
}

Amazon States Language (ASL) Deep Dive

Input and Output Processing

Every state can transform data using three JSONPath fields:

Example: Data Transformation Chain

{
  "ExtractData": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:extract",
    "InputPath": "$.config",
    "ResultPath": "$.rawData",
    "OutputPath": "$",
    "Next": "TransformData"
  },
  "TransformData": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:transform",
    "Parameters": {
      "data.$": "$.rawData",
      "timestamp.$": "$$.Execution.StartTime",
      "mode": "full"
    },
    "ResultPath": "$.transformedData",
    "Next": "LoadData"
  }
}

Context Object

Step Functions provides a built-in context object via $$:

{
  "Execution": {
    "Id": "arn:aws:states:us-east-1:123456789012:execution:myMachine:exec-123",
    "Name": "exec-123",
    "StartTime": "2026-01-15T10:00:00Z"
  },
  "State": {
    "Name": "CurrentState",
    "EnteredTime": "2026-01-15T10:00:05Z",
    "RetryCount": 0
  },
  "StateMachine": {
    "Id": "arn:aws:states:us-east-1:123456789012:stateMachine:myMachine"
  }
}

Best Practices for Data Engineers

1. Use .sync for AWS Service Integration

Always use the .sync integration pattern for services like Glue so Step Functions waits for completion:

"Resource": "arn:aws:states:::glue:startJobRun.sync"

2. Implement Exponential Backoff

Configure retry with increasing delays to handle transient failures:

"Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5, "MaxAttempts": 5, "BackoffRate": 2 }]

3. Set Appropriate Timeouts

Prevent hung executions from consuming resources:

"TimeoutSeconds": 3600

4. Use ResultPath Carefully

Avoid overwriting important input data by choosing non-conflicting ResultPath values:

"ResultPath": "$.etlOutput"

5. Tag Your Executions

Use execution names with context for debugging:

daily-sales-etl-2026-01-15-us-east-1

6. Use Map for Dynamic Fan-Out

Replace hardcoded Parallel branches with Map state for variable partition counts.

7. Monitor with CloudWatch

Create alarms for failed executions and track execution metrics:

• ExecutionsFailed - count of failed runs
• ExecutionTime - duration tracking
• StateTransitionCount - cost monitoring

Architecture Flow

Interview Q&A

Q1: What is the difference between Standard and Express workflows in Step Functions?

Answer: Standard workflows support executions up to 1 year with 2,000 executions per second and full execution history. Express workflows support up to 5 minutes but can handle 100,000 executions per second. Standard charges per state transition (1.00/M). Standard is ideal for long-running ETL orchestration; Express is better for high-volume streaming or micro-batch processing.

Q2: How do you handle errors in Step Functions data pipelines?

Answer: Use Retry for transient failures with exponential backoff (e.g., API throttling, temporary network issues). Configure MaxAttempts, IntervalSeconds, and BackoffRate. For non-recoverable errors, use Catch to route to a fallback state that sends alerts via SNS or logs to CloudWatch. Always catch States.ALL as a safety net. Store error details using ResultPath for debugging.

Q3: When would you use Map state vs Parallel state?

Answer: Use Parallel when you know the exact number of branches at definition time (e.g., process exactly 3 AWS regions). Use Map when the number of iterations is dynamic and determined at runtime from input data (e.g., process a variable number of S3 files). Map supports MaxConcurrency to control parallelism and iterates over arrays in the input.

Q4: How does Step Functions integrate with AWS Glue?

Answer: Step Functions integrates with Glue using resource ARNs like arn:aws:states:::glue:startJobRun.sync (synchronous) or .waitForTaskToken (callback). The .sync pattern is most common: Step Functions starts the Glue job and polls for completion. You pass job arguments via Parameters and receive the result in the state output. Use Retry for States.TaskFailed to handle Glue job failures.

Q5: What is Amazon States Language (ASL)?

Answer: ASL is a JSON-based language used to define Step Functions state machines. It declares states (Task, Choice, Wait, Parallel, Map, Pass, Succeed, Fail), transitions between them, and input/output processing rules. Key features include JSONPath for data selection, the context object ($$) for execution metadata, and support for Catch/Retry for error handling. It is a vendor-neutral specification supported by the Serverless Workflow community.

Q6: How do you implement idempotency in Step Functions?

Answer: Use unique execution names derived from the data being processed (e.g., file name + date). Before starting a new execution, check if one with that name already exists using the ListExecutions API. For Express workflows, the service provides built-in duplicate detection. For Standard workflows, implement a check in a Lambda function at the start of your state machine to detect and skip duplicate runs.

Q7: What is the maximum concurrency of Parallel and Map states?

Answer: Parallel state runs all branches simultaneously (no configurable limit on branch count, but practically limited by service quotas). Map state supports MaxConcurrency to limit parallel iterations (default is 0, meaning no limit). For data engineering, set MaxConcurrency to match downstream service limits (e.g., Glue job slots, Lambda concurrent executions) to avoid throttling.

Q8: How do you pass data between states in Step Functions?

Answer: Each state receives JSON input and produces JSON output. Use InputPath to select input fields, Parameters to construct new input from static values and JSONPath references (using .$ suffix), ResultPath to place output into the original input, and OutputPath to filter what gets passed to the next state. The context object ($$) provides execution metadata like start time and execution ID.

Q9: When would you use Step Functions over Apache Airflow?

Answer: Step Functions is serverless with no infrastructure management, built-in AWS service integration, and pay-per-use pricing. It is ideal for AWS-native pipelines. Airflow provides a richer UI, more operators, and supports multi-cloud/hybrid environments but requires infrastructure (workers, scheduler, metadata DB). Choose Step Functions for simplicity and AWS integration; choose Airflow for complex DAG management, cross-platform needs, or when you need a mature open-source ecosystem.

Q10: How do you monitor Step Functions data pipelines in production?

Answer: Enable CloudWatch Logs for execution history. Create CloudWatch Alarms for ExecutionsFailed and ExecutionsTimedOut metrics. Use Step Functions execution history to trace input/output at each state. Integrate with SNS for real-time failure notifications. Use AWS X-Ray for tracing across Lambda and Glue steps. Tag executions with pipeline name and date for filtering. Set up dashboards to track execution duration trends and state transition counts for cost monitoring.

Summary

This topic covered the key concepts of AWS data engineering. Review the architecture diagrams, practice the interview questions, and understand the trade-offs between different service options.

Next Steps

Continue to the next topic to build on your AWS data engineering knowledge.

Knowledge Check

See Also

• Step Functions
• AWS Lambda
• AWS Glue