Why This Matters

Databricks on AWS is the industry-standard Lakehouse platform for enterprises running Spark workloads. Understanding the integration between Databricks' managed Spark, Delta Lake's ACID transactions, and AWS-native services (S3, IAM, Glue, Redshift) is critical for building reliable, scalable data pipelines. Interviewers expect you to explain not just the components, but the trade-offs, failure modes, and cost optimization strategies that separate production-ready architectures from proof-of-concept demos.

Databricks on AWS Architecture

Real-World Project Structure

A production Databricks on AWS deployment follows a layered architecture:

databricks-aws-platform/
├── infrastructure/
│   ├── terraform/
│   │   ├── main.tf                  # VPC, subnets, security groups
│   │   ├── databricks.tf            # Workspace, clusters, policies
│   │   ├── s3.tf                    # Data lake buckets
│   │   ├── iam.tf                   # Roles, policies, instance profiles
│   │   └── kms.tf                   # Encryption keys
│   └── cloudformation/
│       └── dr-stack.yaml            # Disaster recovery stack
├── data/
│   ├── raw/                         # Landing zone (append-only)
│   ├── bronze/                      # Raw + schema-on-read
│   ├── silver/                      # Cleansed + validated
│   ├── gold/                        # Business-level aggregates
│   └── delta_log/                   # Transaction logs per table
├── notebooks/
│   ├── ingestion/
│   │   ├── auto_loader.py           # CloudFiles streaming
│   │   └── batch_ingest.py          # Full load from sources
│   ├── transform/
│   │   ├── bronze_to_silver.py      # Data cleansing
│   │   └── silver_to_gold.py        # Business logic
│   └── quality/
│       ├── validations.py           # Great Expectations checks
│       └── anomaly_detection.py     # Statistical outlier detection
├── jobs/
│   ├── daily_etl.json               # Scheduled job definition
│   └── streaming_job.json           # Continuous processing job
└── tests/
    ├── unit/                        # Pytest unit tests
    └── integration/                 # Integration test notebooks

Delta Lake on AWS

Delta Lake provides ACID transactions, schema enforcement, and time travel on top of Parquet files stored in S3.

Delta Lake Operations in PySpark

from pyspark.sql import SparkSession
from delta.tables import DeltaTable
from pyspark.sql import functions as F
import logging

logger = logging.getLogger(__name__)

def create_spark_session(app_name="DeltaLakeOnAWS"):
    """Initialize Spark with Delta Lake extensions and AWS configurations."""
    try:
        spark = SparkSession.builder \
            .appName(app_name) \
            .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
            .config("spark.sql.catalog.spark_catalog",
                    "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
            .config("spark.databricks.delta.properties.defaults.enableChangeDataFeed", "true") \
            .config("spark.databricks.delta.schema.autoMerge.enabled", "true") \
            .config("spark.sql.shuffle.partitions", "200") \
            .config("spark.sql.adaptive.enabled", "true") \
            .config("spark.sql.adaptive.coalescePartitions.enabled", "true") \
            .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
            .getOrCreate()
        logger.info(f"Spark session created: {app_name}")
        return spark
    except Exception as e:
        logger.error(f"Failed to create Spark session: {e}")
        raise


def write_delta_table(spark, df, path, partition_cols=None, mode="overwrite"):
    """Write DataFrame to Delta Lake with partitioning and error handling."""
    try:
        writer = df.write.format("delta").mode(mode)
        if partition_cols:
            writer = writer.partitionBy(*partition_cols)
        writer.save(path)
        logger.info(f"Successfully wrote Delta table to {path}")
    except Exception as e:
        logger.error(f"Failed to write Delta table: {e}")
        raise


def merge_delta_table(spark, source_df, target_path, merge_condition):
    """Perform upsert (merge) operation on Delta Lake table."""
    try:
        delta_table = DeltaTable.forPath(spark, target_path)
        merge_result = delta_table.alias("target").merge(
            source=source_df.alias("source"),
            condition=merge_condition
        ).whenMatchedUpdateAll() \
         .whenNotMatchedInsertAll() \
         .execute()
        logger.info(f"Merge completed on {target_path}")
        return merge_result
    except Exception as e:
        logger.error(f"Merge operation failed: {e}")
        raise


def time_travel_query(spark, path, version=None, timestamp=None):
    """Query historical version of Delta table using time travel."""
    try:
        reader = spark.read.format("delta")
        if version is not None:
            reader = reader.option("versionAsOf", version)
        elif timestamp is not None:
            reader = reader.option("timestampAsOf", timestamp)
        else:
            raise ValueError("Provide either version or timestamp")
        return reader.load(path)
    except Exception as e:
        logger.error(f"Time travel query failed: {e}")
        raise


def optimize_delta_table(spark, path, z_order_cols=None):
    """Optimize Delta table with compaction and optional Z-ordering."""
    try:
        delta_table = DeltaTable.forPath(spark, path)
        delta_table.optimize().executeCompaction()
        if z_order_cols:
            delta_table.optimize().executeZOrderBy(*z_order_cols)
        delta_table.vacuum(168)  # Retain 7 days of history
        logger.info(f"Optimization completed for {path}")
    except Exception as e:
        logger.error(f"Optimization failed: {e}")
        raise


# --- Example Usage ---
if __name__ == "__main__":
    spark = create_spark_session("ProductionETL")

    # Read source data
    source_df = spark.read.parquet("s3a://raw-bucket/events/")

    # Write to Delta Lake with partitioning
    write_delta_table(
        spark, source_df,
        "s3a://delta-bucket/events",
        partition_cols=["event_date", "event_type"]
    )

    # Upsert new data
    new_data = spark.read.parquet("s3a://raw-bucket/events_new/")
    merge_delta_table(
        spark, new_data,
        "s3a://delta-bucket/events",
        "target.event_id = source.event_id AND target.event_date = source.event_date"
    )

    # Optimize for query performance
    optimize_delta_table(spark, "s3a://delta-bucket/events", z_order_cols=["user_id"])

Mathematical Formulas

The key cost and performance formulas for Databricks on AWS:

Performance Considerations

Security Considerations

Interview Questions & Answers

Q1: What is the difference between Databricks and AWS EMR?

Answer: Databricks provides a unified platform with built-in collaboration tools, Delta Lake integration, and Unity Catalog for governance. It is optimized for data science and ML workloads. AWS EMR is a managed Hadoop/Spark service that offers more infrastructure control and is better suited for large-scale batch processing. Databricks uses a control plane/data plane architecture with proprietary optimizations (Photon engine, auto-optimize), while EMR runs open-source Apache Spark with AWS-native integrations. Databricks charges per DBU (compute unit), while EMR charges per EC2 instance hour.

Q2: How does Delta Lake ensure ACID transactions on S3?

Answer: Delta Lake uses a transaction log (delta log) stored alongside the data files in _delta_log/ directory. Each transaction is recorded atomically as a JSON entry. Writes follow a two-phase protocol: 1) Write new Parquet data files, 2) Atomically commit the transaction log entry. S3's strong consistency guarantees (added in 2021) ensure that only one writer can succeed per version. Readers always see a consistent snapshot by reading the latest committed log version. This eliminates partial writes and dirty reads without requiring a traditional database.

Q3: Explain the Databricks Lakehouse architecture.

Answer: The Lakehouse combines data lake flexibility with data warehouse reliability. It uses Delta Lake as the storage layer, providing ACID transactions, schema enforcement, and time travel. Data is stored in open formats (Parquet) on cloud storage (S3). The architecture separates compute from storage, allowing independent scaling. It supports both batch and streaming workloads through a single platform. Unity Catalog provides unified governance across all data and AI assets. The result is a single system for ETL, analytics, ML, and BI without data movement.

Q4: How would you optimize a slow-running Spark job on Databricks?

Answer: Key optimizations include: 1) Enable Adaptive Query Execution (AQE) for dynamic shuffle partition optimization, 2) Use Delta Lake OPTIMIZE for file compaction, 3) Partition data by high-cardinality filter columns, 4) Cache frequently accessed DataFrames with .cache(), 5) Use broadcast joins for tables under 10MB, 6) Tune shuffle partition count based on data size (2x input GB), 7) Apply column pruning and predicate pushdown, 8) Enable Photon engine for 2-8x SQL gains, 9) Monitor Spark UI for data skew and spills, 10) Use Z-ordering on frequently filtered columns.

Q5: What are best practices for securing Databricks on AWS?

Answer: 1) Use IAM instance profiles for AWS service access (never store credentials), 2) Enable SSE-KMS encryption with customer-managed keys, 3) Deploy clusters in VPC private subnets, 4) Configure security groups to restrict traffic, 5) Implement Unity Catalog for fine-grained access control, 6) Enable CloudTrail audit logging, 7) Use Databricks Secret Scopes backed by Secrets Manager, 8) Apply cluster policies for governance, 9) Use VPC endpoints for AWS service access, 10) Conduct regular security assessments with IAM Access Analyzer.

Q6: Explain Delta Lake time travel and its use cases.

Answer: Time travel allows querying historical versions of data. Delta Lake maintains a transaction log that records all changes. Each version is immutable and accessible by version number or timestamp. Use cases: 1) Data auditing and compliance (regulatory requirements), 2) Debugging data pipeline issues (identify when bad data entered), 3) Reproducing ML experiments (recreate exact training dataset), 4) Recovering from accidental deletion (restore to pre-deletion state), 5) Comparing data between time periods (before/after transformation), 6) Implementing slowly changing dimensions (SCD Type 4).

Q7: How does Databricks handle auto-scaling on AWS?

Answer: Databricks uses a control plane to monitor cluster utilization. For job clusters, it scales based on pending tasks from the DAG. For interactive clusters, it uses a configurable grace period (default 120 seconds) before scaling down idle workers. Under the hood, Databricks calls AWS EC2 Auto Scaling Groups APIs to add/remove instances. Spot instances provide 60-90% cost savings with automatic fallback to on-demand if spot capacity is unavailable. The first_on_demand parameter controls how many workers start as on-demand for stability.

Q8: What is Unity Catalog and why is it important?

Answer: Unity Catalog is Databricks' unified governance solution providing: 1) Centralized access control across all workspaces, 2) Fine-grained permissions at table, column, and row level, 3) Automated data lineage tracking, 4) Data discovery and search across catalogs, 5) Comprehensive audit logging, 6) Integration with AWS IAM and SSO. It is important because it enables consistent security policies across the organization, simplifies compliance (GDPR, HIPAA), and provides a single source of truth for data governance. It works across multiple clouds and connects to external data sources.

Common Pitfalls

QuizBox

See Also

• AWS EMR
• EMR Spark
• AWS Glue
• AWS Redshift
• AWS S3