Troubleshooting Overview

Effective troubleshooting in AWS data engineering requires a systematic approach. Every pipeline failure, performance degradation, or unexpected behavior can be traced through a structured diagnostic process.

Debugging Checklist

Before diving into specific service errors, run through this universal checklist:

Common Glue Errors

AWS Glue is one of the most commonly used ETL services. Understanding its failure patterns is essential for data engineers.

Glue Job Failure � Code Examples

# Issue 1: Schema mismatch after data source change
# Symptom: AnalysisException in CloudWatch logs
from awsglue.transforms import ResolveChoice

resolved_frame = ResolveChoice.apply(
    frame=dynamic_frame,
    choice="match_catalog",
    database="my_database",
    table_name="my_table"
)

# Issue 2: Out of memory on large datasets
# Symptom: Java Heap Space error in logs
# Solution: Increase worker type from Standard to G.2X
glue_context.create_dynamic_frame.from_options(
    connection_type="s3",
    connection_options={"paths": ["s3://bucket/data/"}],
    format="parquet",
    transform_options={"partitionKeys": ["year", "month"}]
)

# Issue 3: Job timeout � default 48 hours
# Solution: Set custom timeout or split into smaller jobs
job.init("my_job_name", args)
job.commit()

Common Redshift Errors

Redshift is the analytical backbone for many organizations. Errors here can halt entire reporting pipelines.

Redshift COPY Command Troubleshooting

-- Most common COPY failure: IAM role not configured
-- ERROR: S3 Query Failed [AccessDenied]
-- FIX: Ensure the cluster's IAM role has:
--   - s3:GetObject on the S3 bucket/prefix
--   - Trust relationship for redshift.amazonaws.com

-- Copy with explicit error logging
COPY sales_data
FROM 's3://data-lake/sales/'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftS3Access'
FORMAT AS PARQUET
REGION 'us-east-1'
-- Log all errors to a table
MAXERROR 0;

-- Check COPY errors after load
SELECT * FROM stl_load_errors ORDER BY starttime DESC LIMIT 10;

-- Common fix: data type mismatch
-- If source has mixed types, use VARCHAR and cast later
COPY staging_table (id, amount, timestamp_col)
FROM 's3://bucket/data/'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftRole'
FORMAT AS CSV
IGNOREHEADER 1;

Common Kinesis Errors

Kinesis streams power real-time data ingestion. Errors here cause data loss or pipeline backpressure.

Kinesis Debugging � Code Examples

import boto3
from datetime import datetime, timedelta

kinesis = boto3.client('kinesis')

# Diagnostic 1: Check stream status and shard count
response = kinesis.describe_stream(StreamName='my-stream')
stream_desc = response['StreamDescription']

print(f"Status: {stream_desc['StreamStatus'
  ]}")
print(f"Shards: {len(stream_desc['Shards'])}")
print(f"Retention: {stream_desc['RetentionPeriodHours'
  ]}h")

# Diagnostic 2: Check iterator age for each shard
for shard in stream_desc['Shards']:
    shard_id = shard['ShardId']
    
    # Get latest sequence number
    shard_iterator = kinesis.get_shard_iterator(
        StreamName='my-stream',
        ShardId=shard_id,
        ShardIteratorType='LATEST'
    )['ShardIterator']
    
    records = kinesis.get_records(
        ShardIterator=shard_iterator,
        Limit=1
    )
    
    print(f"Shard {shard_id}: Age = {records.get('MillisBehindLatest', 0)}ms")

# Diagnostic 3: Check producer errors with PutRecords
records = [
    {'Data': b'test', 'PartitionKey': 'key1'}, {'Data': b'test', 'PartitionKey': 'key2'},
]

response = kinesis.put_records(
    StreamName='my-stream',
    Records=records
)

# Check for failed records
failed = [r for r in response['Records'] if 'ErrorCode' in r]
if failed:
    print(f"Failed records: {len(failed)}")
    for f in failed:
        print(f"  Error: {f['ErrorCode'
  ]} - {f['ErrorMessage'
  ]}")

Architecture Flow

Interview Q&A

Q1: What is your approach when a production data pipeline fails at 2 AM?

Answer: I follow a structured incident response process. First, I check if there's a PagerDuty alert and acknowledge it. Then I assess the blast radius � how many downstream systems are affected and how much data is at risk. I check CloudWatch alarms for the specific pipeline and examine the Glue/EMR job logs. If it's a transient error like a network blip, I retry the job. If it's a code bug, I hotfix and deploy. I always document the incident in a post-mortem within 48 hours and add monitoring to prevent recurrence.

Q2: How do you debug a Glue job that keeps failing with "Out of Memory"?

Answer: First, I check the CloudWatch logs under /aws-glue/jobs/errors. The OOM error typically means the data volume exceeds the worker capacity. I look at the job's execution plan to identify which step consumes the most memory. Common fixes include upgrading from Standard to G.2X or G.4X workers, adding partitioning to reduce data per task, using persist() strategically to avoid recomputation, and tuning spark.sql.shuffle.partitions. If the job reads many small files, I compact them first.

Q3: Your Redshift COPY command is failing. Walk me through debugging it.

Answer: I start by checking stl_load_errors � this table captures every COPY failure with the exact error message and the line/byte offset of the bad data. Common issues include: IAM role not having S3 access (check trust policy for redshift.amazonaws.com), data format mismatch (verify FORMAT parameter matches actual file format), incorrect S3 path or missing trailing slash, and data type mismatches between file and table definition. I also verify the cluster has COPY permissions via the IAM role and that the S3 bucket policy allows the role.

Q4: How do you identify and resolve data skew in Spark/EMR jobs?

Answer: Data skew manifests as some tasks finishing much faster than others. I check the Spark UI's Stages tab to see task duration distribution. If one partition takes 10x longer, that's skew. Solutions include salting the skewed key (adding a random prefix and rebalancing), using broadcast() joins for small-large table joins, repartitioning by a more uniform key, using skewJoin hint in Spark 3.0+, and analyzing key distribution with groupBy().count() before the join.

Q5: A Kinesis stream consumer is falling behind. How do you fix it?

Answer: I first check GetRecords.IteratorAgeMilliseconds in CloudWatch � this shows how far behind the consumer is. If it's growing, the consumer can't keep up. I check if the consumer is throttled via ReadProvisionedThroughputExceeded. Solutions include: enabling enhanced fan-out for dedicated throughput, scaling up the number of consumer instances, optimizing the consumer's processing logic, implementing batch GetRecords with Limit parameter, and adding a dead-letter queue for records that fail processing.

Q6: How do you troubleshoot an S3 503 Slow Down error?

Answer: The 503 Slow Down error means S3 is throttling requests. This happens when you exceed the request rate limit (3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD per prefix per second). I implement exponential backoff with jitter using botocore.config.Config. I also distribute objects across multiple prefixes (a technique called "key name partitioning") to spread the load. For write-heavy workloads, I use multipart uploads to parallelize within a single object.

Q7: How would you debug an EMR cluster that keeps terminating unexpectedly?

Answer: I check the EMR console for the cluster's state change history and termination reasons. Common causes include: spot instance interruption (check CloudTrail for EC2 spot termination notices), master node failures (check /mnt/var/log/ logs on S3), application failures (Spark, Hive logs), and bootstrap action errors. I enable termination protection, use On-Demand instances for master nodes, implement checkpointing in Spark jobs, and configure step-level failure handling with --failure-action CONTINUE.

Q8: Your Lambda function processing Kinesis events times out intermittently. How do you fix this?

Answer: I check CloudWatch logs for the specific invocation that timed out. I look at the function's duration vs. timeout setting. If the function is approaching timeout, I increase memory (which proportionally increases CPU). I also check if the Lambda is blocked on an external call � database connections, API calls � and add connection pooling. For Kinesis specifically, I adjust the batch size to process fewer records per invocation and configure a bisect-on-function-error to split failing batches.

Q9: How do you diagnose and fix a schema evolution mismatch in Glue?

Answer: The error "Cannot up cast" or "Schema mismatch" means the source data has changed since the last catalog update. I first re-run the Glue crawler to update the catalog. Then I use ResolveChoice with choice="match_catalog" to map old columns to the new schema. For columns that are added, I use apply_mapping with stageDataKeys. For deleted columns, I drop them explicitly. I implement schema validation as a pre-step in the ETL job using a try-catch on create_dynamic_frame.

Q10: Describe your debugging process for a slow-running Athena query.

Answer: I first run EXPLAIN on the query to see the execution plan. I check the CloudWatch metrics for the query � bytes scanned vs. bytes returned ratio. If the ratio is high, the query isn't using partitions effectively. I add WHERE clauses on partition columns (year, month, day). I check if there are too many small files (each file = a separate Lambda). I convert frequently queried columns to ORC or Parquet with proper compression. I also set up Athena workgroups with query result caching for repeated patterns.

Q11: A Glue Crawler fails with "Access Denied" errors. What do you check?

Answer: I check the IAM role attached to the crawler. It needs: AWSGlueServiceRole managed policy, s3:GetObject and s3:ListBucket on the source bucket, and glue:CreateTable / glue:UpdateTable on the target database. I also check if there's a VPC endpoint blocking S3 access � crawlers in VPC need proper endpoint configuration. If the crawler is reading from a KMS-encrypted bucket, the role needs kms:Decrypt permission. I use the IAM Policy Simulator to validate each permission.

Q12: How do you handle a situation where Redshift spectrum queries fail after an S3 bucket policy change?

Answer: Redshift Spectrum uses the cluster's IAM role to access S3. After a bucket policy change, I verify: the role still has s3:GetObject permission, the bucket policy hasn't added a deny for the role's ARN, the external schema's IAM role ARN hasn't changed, and there's no SCP (Service Control Policy) blocking access. I also check if the bucket's encryption changed � if it switched to SSE-KMS, the role needs kms:Decrypt on the KMS key.

Q13: How do you debug a Step Functions state machine that silently fails?

Answer: I check the execution history in the Step Functions console. I look at each state's input and output. If a Lambda step fails, I check its CloudWatch logs. If a Choice state isn't matching, I verify the input payload structure. Common issues include: the Catch block is redirecting to a fallback state, the state machine is stuck waiting for a callback token, a Map state is hitting concurrency limits, or IAM permissions for the state machine role are missing for a service integration step.

Q14: Your Spark job writes to an HDFS-compatible path but downstream readers see empty files. Why?

Answer: This is typically an output commit issue. Spark writes use a two-phase commit protocol. If the job crashes after writing but before committing, the files exist but aren't visible to the output committer. I check if spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version is set correctly. I also verify the output path is being cleaned between runs (_SUCCESS file handling). If using EMR, I enable emrfs for consistent view. The fix is usually to use OUTPUT_COMMITTER = PathOutputCommitter and ensure the job completes the commit phase.

Q15: How do you debug inconsistent data between a source database and a Glue-based ETL target?

Answer: I first compare record counts at the source and target. Then I check for duplicate records using SELECT key, COUNT(*) FROM target GROUP BY key HAVING COUNT(*) > 1. I verify the ETL job's extract timestamp filter isn't truncating data. I check if the Glue job uses last_modified timestamps correctly. I also verify that the target table isn't using a MERGE/UPSERT that's leaving stale records. Finally, I check if CDC (Change Data Capture) is configured correctly if using Debezium or DMS.

Q16: How do you debug a KMS key access error when decrypting data in S3 or Redshift?

Answer: I check the key policy in KMS. Common issues include: the IAM role doesn't have kms:Decrypt permission, the key policy restricts access to specific principals, the key is disabled or pending deletion, and the key's region doesn't match the service region. I verify the key's grant using kms:list-grants and ensure the service's service-linked role is authorized. For Redshift Spectrum, I check that the external schema references the correct IAM role that has KMS permissions.

Q17: Describe how you would debug a Glue job that completes but produces zero output records.

Answer: I add logging at each transformation step to count records: dynamic_frame.count(). I check if the filter or apply_mapping steps are removing all records. I verify the source path has data (sometimes S3 paths change). I check if resolveChoice is mapping columns incorrectly, causing null filters. I also check the Glue job's commit log � if the job writes to a temp location and fails to commit, the output may appear empty. Adding explicit print statements in PySpark ETL code is my go-to approach.

Q18: How do you troubleshoot a situation where Redshift COPY loads fewer rows than expected?

Answer: I check stl_load_errors for rejected records. Common causes include: MAXERROR threshold is too low, data format issues (malformed JSON/CSV), encoding mismatches between source and target, null byte corruption in source files, and extra delimiters creating empty rows. I also verify the S3 prefix path � sometimes files are in subdirectories the COPY doesn't traverse. I query svv_load_commits to see exactly which files were loaded.

Q19: How do you debug a Glue Streaming job that stops processing after running for hours?

Answer: I check CloudWatch logs for Continuous failing errors. Common causes include: CloudWatch log group hitting storage limits, Glue streaming job's micro-batch timeout, network issues between Glue and the source Kinesis/Kafka stream, and memory leaks causing eventual OOM. I set up a CloudWatch alarm on Glue.CUSTOM_METRIC.RunTime and check the Glue job's heartbeat. I also verify the DynamoDB lease table used by Glue's streaming consumers hasn't hit write capacity limits.

Q20: How do you debug an EMR cluster where Spark executors are being killed by YARN?

Answer: I check YARN's Resource Manager logs for container kill reasons. Common causes include: container exceeding memory limit (PmemContainers), node manager disk space full, and garbage collection overhead exceeding the threshold. I increase spark.executor.memory and spark.executor.memoryOverhead proportionally. I also check if the cluster has enough headroom � ideally 30-40% free memory. I enable YARN's node health check and add logging for container preemption events.

Summary

Effective AWS data engineering troubleshooting requires:

• Systematic approach: Detect ? Diagnose ? Isolate ? Resolve ? Prevent
• Service-specific knowledge: Understanding Glue, Redshift, Kinesis, EMR, Lambda, and S3 error patterns
• Tooling mastery: CloudWatch Logs/Metrics, CloudTrail, X-Ray, and service-specific diagnostic tables
• Prevention over cure: Proactive monitoring, alerting, and runbook creation
• Data-driven debugging: Always check metrics, logs, and diagnostic queries before applying fixes

These troubleshooting skills separate effective data engineers from those who struggle with production systems. Practice these patterns in your own environments and build debugging muscle memory.

Knowledge Check

See Also

• AWS Data Monitoring
• AWS Architecture
• AWS Cost