Why This Matters

AWS Glue is the central nervous system of modern data lakes on AWS. It provides the metadata layer through its Data Catalog, the compute engine through serverless ETL jobs, and the schema discovery through Crawlers. Every query in Athena, every table in Redshift Spectrum, and every ETL pipeline in a data lake relies on Glue. Understanding Glue internals, DPU optimization, and bookmark management is essential for building scalable, cost-effective data platforms.

Key Insight: Glue ETL jobs run on Apache Spark but abstract away cluster management entirely. This makes Glue ideal for teams that want Spark power without operational overhead, but it also means you must understand Spark tuning to optimize Glue performance.

AWS Glue Architecture

AWS Glue Ecosystem Architecture

Glue ETL Jobs

Glue ETL jobs run on Apache Spark without managing clusters. You provide transformation logic in Python or Scala, and Glue provisions, scales, and terminates compute automatically.

ETL Job Types

Production PySpark ETL Job

import sys
import logging
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame
from pyspark.context import SparkContext
from pyspark.sql import functions as F
from pyspark.sql.window import Window

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DATABASE', 'SOURCE_TABLE', 'TARGET_PATH'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

try:
    logger.info(f"Starting ETL job: {args['JOB_NAME']}")

    # Read from Data Catalog
    dynamic_frame = glueContext.create_dynamic_frame.from_catalog(
        database=args['DATABASE'],
        table_name=args['SOURCE_TABLE'],
        transformation_ctx="datasource0"
    )

    # Convert to DataFrame for complex transformations
    df = dynamic_frame.toDF()

    # Apply transformations
    df_transformed = (
        df
        .filter(F.col("status") != "DELETED")
        .withColumn("processed_at", F.current_timestamp())
        .withColumn("year", F.year("order_date"))
        .withColumn("month", F.month("order_date"))
        .withColumn("day", F.dayofmonth("order_date"))
        .withColumn(
            "amount_bucket",
            F.when(F.col("amount") < 100, "small")
             .when(F.col("amount") < 1000, "medium")
             .otherwise("large")
        )
        .withColumn(
            "row_number",
            F.row_number().over(
                Window.partitionBy("customer_id").orderBy(F.desc("order_date"))
            )
        )
    )

    # Write to S3 in Parquet format, partitioned by date
    df_transformed.write \
        .mode("overwrite") \
        .partitionBy("year", "month", "day") \
        .option("compression", "snappy") \
        .parquet(args['TARGET_PATH'])

    logger.info(f"ETL job completed successfully. Rows processed: {df_transformed.count()}")

except Exception as e:
    logger.error(f"ETL job failed: {str(e)}")
    raise

finally:
    job.commit()

Real-World Project Structure

glue-etl-project/
├── scripts/
│   ├── extract_orders.py
│   ├── transform_customers.py
│   ├── load_to_redshift.py
│   ├── streaming_ingestion.py
│   └── validation.py
├── crawlers/
│   ├── orders_crawler.json
│   ├── customers_crawler.json
│   └── events_crawler.json
├── jobs/
│   ├── etl_orders_full.json
│   ├── etl_orders_incremental.json
│   └── streaming_orders.json
├── connections/
│   ├── rds_postgres_connection.json
│   └── redshift_connection.json
├── classifiers/
│   ├── custom_json_classifier.json
│   └── custom_csv_classifier.json
├── infrastructure/
│   ├── cloudformation/
│   │   ├── glue_catalog.yaml
│   │   ├── glue_jobs.yaml
│   │   └── glue_security.yaml
│   └── terraform/
│       ├── main.tf
│       ├── crawlers.tf
│       └── variables.tf
├── monitoring/
│   ├── cloudwatch_dashboard.json
│   └── alarms.yaml
├── tests/
│   ├── test_transform_orders.py
│   └── test_data_quality.py
└── docs/
    ├── etl_design.md
    └── runbook.md

Glue Crawlers

Crawlers scan data stores, infer schemas, and populate the Data Catalog automatically.

Crawler Configuration

import boto3
import json
import logging

logger = logging.getLogger(__name__)


def create_glue_crawler(
    crawler_name: str,
    database_name: str,
    role_arn: str,
    s3_targets: list,
    schedule: str = "cron(0 2 * * ? *)",
    classifiers: list = None
) -> dict:
    """
    Create a Glue crawler with production-grade settings.

    Args:
        crawler_name: Unique crawler identifier
        database_name: Target database in Data Catalog
        role_arn: IAM role ARN for the crawler
        s3_targets: List of S3 paths to crawl
        schedule: Cron expression for scheduling
        classifiers: List of custom classifier names

    Returns:
        dict with crawler details
    """
    glue_client = boto3.client('glue')

    targets = {
        'S3Targets': [{'Path': path, 'Exclusions': ['_temporary/**', '.spark-staging*/**']} for path in s3_targets]
    }

    crawler_config = {
        'Name': crawler_name,
        'Role': role_arn,
        'DatabaseName': database_name,
        'Targets': targets,
        'SchemaChangePolicy': {
            'UpdateBehavior': 'UPDATE_IN_DATABASE',
            'DeleteBehavior': 'LOG'
        },
        'RecrawlPolicy': {
            'RecrawlBehavior': 'CRAWL_EVERYTHING'
        },
        'Schedule': schedule,
        'Configuration': json.dumps({
            'Version': 1,
            'Grouping': {'TableGroupingPolicy': 'CombineCompatibleSchemas'}
        })
    }

    if classifiers:
        crawler_config['Classifiers'] = classifiers

    try:
        response = glue_client.create_crawler(**crawler_config)
        logger.info(f"Created crawler: {crawler_name}")
        return response
    except glue_client.exceptions.EntityNotFoundException:
        logger.error(f"Role {role_arn} not found. Check IAM permissions.")
        raise
    except Exception as e:
        logger.error(f"Failed to create crawler: {str(e)}")
        raise


def run_crawler(crawler_name: str) -> str:
    """Start a Glue crawler and return the run ID."""
    glue_client = boto3.client('glue')
    response = glue_client.start_crawler(Name=crawler_name)
    logger.info(f"Started crawler: {crawler_name}")
    return response


def get_crawler_status(crawler_name: str) -> dict:
    """Get the current status and last run information for a crawler."""
    glue_client = boto3.client('glue')
    response = glue_client.get_crawler(Name=crawler_name)
    crawler = response['Crawler']

    status = {
        'name': crawler['Name'],
        'state': crawler['State'],
        'last_crawl': crawler.get('LastCrawl', {}),
        'creation_time': str(crawler['CreationTime']),
        'version': crawler['Version']
    }

    logger.info(f"Crawler {crawler_name} state: {status['state']}")
    return status

Glue Job Bookmarks

Job bookmarks track processed data, enabling incremental loads without reprocessing.

# Enable job bookmarks (default behavior)
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Bookmark logic:
# First run: processes File A, B, C -> saves bookmark
# Second run: skips A, B, C -> processes only new File D, E

# Manually manage bookmarks for custom scenarios
from awsglue.utils import getResolvedOptions

# Reset bookmark to reprocess all data
glueContext.purge_staging_path(
    target_path="s3://my-bucket/processed/",
    options={"jobBookmarkOption": "reset"}
)

# Disable bookmarks for full reprocessing
# Set --job-bookmark-option disable in job parameters

Mathematical Formulas

DPU Cost Calculation

Total DPUs = Workers x DPUs_per_Worker

Cost_Per_Hour = Total_DPUs x $0.44/DPU-hour

Daily_Cost = Cost_Per_Hour x Hours_Running

Example:
  10 Workers x 1 DPU/Worker (G.1X) = 10 DPUs
  Cost: 10 DPUs x $0.44/hour = $4.40/hour
  For 8-hour job: $4.40 x 8 = $35.20/day

Throughput Estimation

Read_Rate = Shard_Count x 1 MB/sec (per shard)
Write_Rate = Shard_Count x 1 MB/sec (per shard)

Partitions_Needed = Input_Size_GB / Target_Partition_Size_GB

Recommended: 128MB - 1GB per partition for optimal performance

Performance Considerations

Security Considerations

Interview Questions & Answers

Q1: What is the difference between DynamicFrame and DataFrame in AWS Glue?

Answer: DynamicFrame is Glue's schema-flexible abstraction over Spark DataFrame. Key differences:

1. Schema handling - DynamicFrame can handle schema mismatches and null values gracefully. DataFrame requires a fixed schema.
2. Transforms - DynamicFrame provides Glue-specific transforms like ApplyMapping, DropFields, and ResolveChoice. DataFrame uses standard Spark SQL.
3. Serialization - DynamicFrame adds serialization overhead for schema flexibility. DataFrame is faster for pure Spark operations.
4. Null handling - DynamicFrame has built-in null-safe transforms. DataFrame requires manual null handling.
5. Conversion - Use toDF() to convert DynamicFrame to DataFrame and toDynamicFrame() for the reverse.

Best practice: Use DynamicFrame for ETL with messy data or schema evolution. Use DataFrame for performance-critical operations after initial data cleansing.

Q2: How do Glue Crawlers handle schema evolution?

Answer: Crawlers handle schema evolution through the SchemaChangePolicy configuration:

• UPDATE_IN_DATABASE: Automatically updates existing table definitions when new columns or data types are discovered
• LOG: Logs changes but does not update the catalog, allowing manual review
• DELETE: Removes table definitions if the source no longer exists

The RecrawlBehavior setting controls what the crawler re-scans:

• CRAWL_EVERYTHING: Re-scans all data (slower, most thorough)
• CRAWL_NEW_FOLDERS: Only scans new folders since last crawl (faster)
• CRAWL_EVENT_MODE: Uses S3 events to detect changes

Best practice: Use UPDATE_IN_DATABASE with CRAWL_NEW_FOLDERS for production environments to balance freshness and performance.

Q3: What are Glue Job Bookmarks and how do they work?

Answer: Job bookmarks track which data files have been processed by a job. When enabled:

1. First run processes all data and saves a bookmark with file state
2. Subsequent runs compare current file state with bookmark
3. Only new or modified files are processed
4. Bookmark is updated after successful job completion

Bookmarks work with S3 sources by tracking:

• File path and ETag (content hash)
• Last modified timestamp
• File size

Limitations: Bookmarks do not work with JDBC sources or streaming. For JDBC, implement custom incremental logic using high-water marks.

Q4: How do you optimize Glue job performance for large datasets?

Answer: Performance optimization strategies:

1. Partitioning: Use partitionKeys to organize data, target 128MB-1GB per partition
2. File format: Use Parquet with Snappy compression for columnar storage
3. Worker sizing: Start with G.1X, upgrade to G.2X if memory-bound
4. Caching: Cache DynamicFrames used multiple times in the same job
5. Coalesce: Reduce partitions before writing to avoid small files
6. Broadcast joins: Use for small dimension tables to avoid shuffle
7. Kryo serialization: Enable for faster serialization between stages
8. Adaptive Query Execution: Enable Spark 3.x AQE for automatic optimization

Monitor CloudWatch metrics like driverHeapUsage, executorCPUUtilization, and shuffleWrite to identify bottlenecks.

Q5: What is the Glue Data Catalog and how does it integrate with other services?

Answer: The Data Catalog is a central metadata repository storing table definitions, schema information, and location references. It is compatible with Apache Hive Metastore.

Integrations:

• Athena: Uses catalog tables for SQL queries on S3 data
• Redshift Spectrum: Extends Redshift to query catalog tables
• EMR: Uses catalog for Hive Metastore compatibility
• Glue ETL Jobs: Reads/writes catalog metadata
• Lake Formation: Applies fine-grained permissions on catalog objects
• Apache Hive: Compatible via Hive Metastore protocol

The catalog eliminates hard-coded schemas in ETL scripts and provides a single source of truth for data definitions across the organization.

Q6: Compare AWS Glue with Amazon EMR for data processing.

Answer:

Choose Glue for simplicity, serverless execution, and built-in catalog integration. Choose EMR for maximum control, custom libraries, Hadoop ecosystem tools, or when you need fine-grained Spark configuration.

Q7: How do you debug a failing Glue job?

Answer: Debugging steps:

1. Check CloudWatch Logs - Job logs are automatically sent to CloudWatch Log Groups
2. View Spark UI - Accessible during job execution in the Glue console
3. Enable Spark Event Logging - Configure spark.eventLog.enabled=true for detailed logs
4. Review Error Messages - Look for specific exceptions in driver/executor logs
5. Check IAM Permissions - Ensure Glue role has access to source and target
6. Validate Schema - Check Data Catalog table definitions match source data
7. Test Locally - Use --job-bookmark-option disable and sample data for debugging
8. Enable Metrics - Monitor DPU usage, shuffle read/write, and GC time

Common issues: IAM permission errors, schema mismatches, missing Python libraries, VPC connectivity problems, and data skew causing executor OOM.

Q8: How would you design a serverless data lake using AWS Glue?

Answer: Serverless data lake architecture:

1. Raw Zone: S3 bucket for ingested data (CSV, JSON, logs) with lifecycle policies
2. Crawlers: Scan raw zone, populate Data Catalog with table definitions
3. ETL Jobs: Transform and clean data, write to processed zone
4. Processed Zone: Parquet/ORC in S3, partitioned by date
5. Curated Zone: Aggregated, business-ready datasets for analytics
6. Athena: SQL queries on processed/curated data
7. Lake Formation: Fine-grained access control on catalog
8. EventBridge: Schedule crawlers and trigger jobs on events
9. CloudWatch: Monitor job performance and costs

Benefits: No clusters to manage, pay-per-use pricing, auto-scaling, integrated governance, and automatic schema discovery.

Common Pitfalls

Why This Matters for Your Career

AWS Glue is one of the most widely used services in modern data architectures. Mastering Glue ETL jobs, crawlers, and the Data Catalog is essential for building scalable data lakes. Interview questions frequently test your understanding of DPU optimization, bookmark management, and the trade-offs between DynamicFrame and DataFrame. Glue expertise directly translates to cost savings and operational efficiency in production environments.

Key Takeaways

• Glue is fully managed and serverless, eliminating infrastructure management overhead
• Data Catalog is the central metadata layer used by Athena, Redshift Spectrum, and EMR
• Job bookmarks enable efficient incremental processing without reprocessing all data
• DynamicFrame provides schema flexibility while DataFrame offers better performance
• Proper partitioning and file format selection are critical for job performance

See Also

• AWS EMR for Data Engineers
• AWS Redshift
• Glue Data Catalog Deep Dive