Databricks on AWS � Overview

Databricks provides a unified lakehouse platform that combines the best of data warehouses and data lakes. Running on AWS, Databricks leverages S3 for durable storage, EC2 for compute, and AWS IAM for identity � all wrapped in a fully managed Spark environment with collaborative notebooks, Delta Lake, Unity Catalog, and MLflow.

Why Databricks on AWS?

Key AWS Components Used

• Amazon S3 � Primary data storage for Delta tables, DBFS, and artifacts
• AWS IAM � Authentication, role mapping, and cross-account access
• Amazon VPC � Network isolation with private subnets and security groups
• AWS PrivateLink � Secure connectivity without public internet exposure
• AWS KMS � Customer-managed encryption keys for data at rest
• AWS CloudTrail � Audit logging for compliance and governance

SVG: Databricks + AWS Architecture

Unity Catalog

Unity Catalog is Databricks' centralized governance solution for all data and AI assets. It provides a single pane of glass for access control, auditing, lineage, and data discovery across all Databricks workspaces.

Three-Level Namespace

Unity Catalog uses a catalog ? schema ? table namespace that maps naturally to organizational structures:

• Catalog � Top-level grouping (e.g., production, sandbox, finance)
• Schema � Logical grouping within a catalog (e.g., raw, curated, analytics)
• Table � The actual data asset (Delta, Parquet, CSV, etc.)

-- Three-level namespace in action
SELECT * FROM production.analytics.revenue_by_region;

-- Create catalog and schema
CREATE CATALOG IF NOT EXISTS production;
USE CATALOG production;
CREATE SCHEMA IF NOT EXISTS analytics;

-- Grant access
GRANT SELECT ON TABLE production.analytics.revenue_by_region TO `data-analyst-group`;
GRANT MODIFY ON SCHEMA production.raw TO `data-engineer-group`;

Key Features

SVG: Unity Catalog Architecture

Delta Sharing

Delta Sharing is an open protocol for secure real-time data sharing across organizations. It allows Databricks users to share Delta Lake tables with recipients outside their organization � without requiring the recipient to run Databricks.

How Delta Sharing Works

1. Share � A Databricks admin creates a share containing one or more tables
2. Recipient � An external user or organization is registered as a recipient
3. Permission � The admin grants the recipient access to specific shares
4. Access � The recipient retrieves data using a secure token

# Create a share
spark.sql("CREATE SHARE my_share")
spark.sql("ALTER SHARE my_share ADD TABLE production.analytics.revenue")

# Register a recipient
spark.sql("""
  CREATE RECIPIENT external_partner 
  USING TYPE 'EMAIL' VALUE 'partner@example.com'
""")

# Grant access
spark.sql("GRANT ACCESS ON SHARE my_share TO RECIPIENT external_partner")

Delta Sharing vs. alternatives

SVG: Delta Sharing Flow

Databricks + AWS Services Integration

Databricks integrates natively with a wide range of AWS services to deliver a production-grade lakehouse platform. Understanding these integrations is essential for data engineers designing scalable, secure, and cost-efficient data pipelines.

Core Integrations

Typical Data Pipeline on Databricks + AWS

# 1. Read from Kinesis stream
raw_stream = (spark.readStream
  .format("kinesis")
  .option("streamName", "orders-stream")
  .option("region", "us-east-1")
  .option("initialPosition", "TRIM_HORIZON")
  .load())

# 2. Write to Delta Lake on S3 (Bronze)
(raw_stream.writeStream
  .format("delta")
  .outputMode("append")
  .option("checkpointLocation", "s3://my-bucket/checkpoints/bronze")
  .start("s3://my-bucket/lake/bronze/orders"))

# 3. Transform and write to Silver
spark.sql("""
  CREATE OR REPLACE TEMPORARY VIEW clean_orders
  AS SELECT 
    order_id, customer_id, 
    CAST(amount AS DECIMAL(10,2)) as amount,
    event_time
  FROM delta.`s3://my-bucket/lake/bronze/orders`
  WHERE amount > 0 AND customer_id IS NOT NULL
""")

# 4. Merge into Silver Delta table
from delta.tables import DeltaTable

silver = DeltaTable.forPath(spark, "s3://my-bucket/lake/silver/orders")
silver.alias("target").merge(
    spark.table("clean_orders").alias("source"),
    "target.order_id = source.order_id"
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

DBFS and S3 Mapping

Databricks File System (DBFS) is a distributed file system that mounts to your S3 bucket. All data written to DBFS is actually stored in S3.

# DBFS paths map to S3
dbfs:/mnt/my-storage/data  ?  s3://my-bucket/data

# Direct S3 access
df = spark.read.parquet("s3://my-bucket/lake/gold/revenue/")

# DBFS FUSE (mounted on driver/worker nodes)
%fs ls /mnt/my-storage/

SVG: Databricks + AWS Integration Diagram

Architecture Flow

Interview Q&A

Q1: What is Databricks and how does it relate to AWS?

Answer: Databricks is a unified analytics platform built by the creators of Apache Spark. On AWS, it runs on EC2 instances with data stored in S3. It provides collaborative notebooks, Delta Lake for ACID transactions, Unity Catalog for governance, and MLflow for machine learning � all fully managed with DBU-based pricing.

Q2: What is the difference between Databricks and Amazon EMR?

Answer:

Choose Databricks when you need collaborative notebooks, built-in ML tools, and centralized governance. Choose EMR when you need full cluster control, custom JARs, or cost is the primary concern.

Q3: Explain the Delta Lake ACID transaction model.

Answer: Delta Lake provides ACID transactions on data lakes by using a transaction log (_delta_log) that records every change made to a table. Each commit creates a new JSON file in the log. Readers use optimistic concurrency control � if two writers conflict, one succeeds and the other retries. This ensures atomicity (all-or-nothing writes), consistency (schema enforcement), isolation (concurrent reads/writes), and durability (backed by S3).

# Example: ACID merge operation
delta_table.alias("target").merge(
    source.alias("source"),
    "target.id = source.id"
).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()

Q4: What is Unity Catalog and why is it important?

Answer: Unity Catalog is Databricks' centralized governance solution. It provides:

• Three-level namespace (catalog.schema.table) for organizing data assets
• Fine-grained access control with GRANT/REVOKE at table, column, and row levels
• Automatic data lineage tracking column-level dependencies
• Audit logging via system tables for compliance
• Data discovery with searchable data marketplace
• Cross-workspace governance � unified policies across all Databricks workspaces

Q5: How does Delta Sharing differ from traditional data sharing methods?

Answer: Delta Sharing uses an open protocol that enables secure, real-time data sharing across organizations without requiring the recipient to run Databricks. Unlike S3 pre-signed URLs (AWS-specific, no time travel) or database links (vendor lock-in), Delta Sharing supports:

• Cross-cloud sharing (AWS to Azure to GCP)
• Time travel queries on shared data
• Token-based authentication with fine-grained access control
• Open-source client libraries (Python, R, Java, Scala, Rust)

Q6: What is the multi-hop architecture pattern in Databricks?

Answer: The multi-hop architecture separates data into three layers:

1. Bronze � Raw data ingested as-is. Append-only. Schema on read. Source of truth.
2. Silver � Cleaned, validated, and deduplicated data. Schema enforced. Business rules applied.
3. Gold � Business-level aggregates, dimension/fact tables. Optimized for queries and dashboards.

Each hop adds quality and structure. Delta Live Tables (DLT) is the recommended tool for implementing this pattern declaratively.

# Bronze
@dlt.table
def bronze_orders():
    return spark.readStream.format("cloudFiles").load("/raw/orders")

# Silver
@dlt.table
def silver_orders():
    return dlt.read("bronze_orders").filter("amount > 0")

# Gold
@dlt.table
def gold_revenue():
    return dlt.read("silver_orders").groupBy("region").sum("amount")

Q7: How do you optimize Delta Lake performance?

Answer:

-- Compaction + Z-Order
OPTIMIZE events ZORDER BY (event_date, user_id);

-- Check file statistics
DESCRIBE HISTORY events;
SELECT * FROM table_events('event_date') LIMIT 5;

Q8: How do you implement CI/CD for Databricks?

Answer:

1. Source Control � Use Databricks Repos (Git integration) with branches and pull requests
2. Testing � Unit tests with pytest, integration tests against test workspaces
3. Deployment � Databricks CLI, REST API, or Terraform for infrastructure-as-code
4. Environments � Separate dev/staging/prod workspaces with Unity Catalog access controls
5. Monitoring � Job run history, alerting via email/Slack/webhooks

# Databricks CLI deployment
databricks workspace import_dir ./notebooks /Shared/pipeline --overwrite
databricks jobs create --json-file job_spec.json

Q9: How do you handle data quality in Databricks?

Answer:

• Delta Live Tables (DLT) � Built-in expectations for data quality checks
• Great Expectations � Integration with Databricks for validation
• Custom validation � Define quality rules and alert on failures
• Lakehouse Monitoring � Built-in data quality metrics and dashboards

# DLT with expectations
@dlt.table
def validated_orders():
    return dlt.read("raw_orders")

@dlt.expect("valid_amount", "amount > 0")
@dlt.expect_or_drop("valid_customer", "customer_id IS NOT NULL")
def clean_orders():
    return dlt.read("validated_orders")

Q10: What are the cost optimization strategies for Databricks on AWS?

Answer:

# Cluster configuration for cost optimization
cluster_config = {
    "spark_conf": {
        "spark.databricks.cluster.profile": "serverless",
        "spark.master": "local[*]"
    },
    "autotermination_minutes": 15,
    "enable_elastic_disk": True,
    "disk_spec": {"disk_type": "ebs", "disk_size": 100}
}

Q11: How does Databricks integrate with AWS Glue?

Answer: Databricks can use AWS Glue Data Catalog as an external metastore, allowing Databricks and Glue-based ETL jobs to share the same table metadata. However, Databricks recommends using Unity Catalog for governance when possible. The integration works via Hive metastore compatibility.

Q12: Explain time travel in Delta Lake.

Answer: Delta Lake stores every version of a table via its transaction log. You can query any historical version by:

# By version number
df = spark.read.format("delta").option("versionAsOf", 5).load("/delta/events")

# By timestamp
df = spark.read.format("delta").option("timestampAsOf", "2025-01-15").load("/delta/events")

# View history
deltaTable.history()

# Restore to previous version
deltaTable.restore(5)

Use cases: debugging, auditing, regulatory compliance, and rollback after bad writes.

Q13: What is MLflow and how does it work on Databricks?

Answer: MLflow is an open-source platform for managing the ML lifecycle. On Databricks, it is fully integrated:

• MLflow Tracking � Log parameters, metrics, and artifacts from any notebook
• MLflow Models � Package models in multiple formats (sklearn, pytorch, etc.)
• Model Registry � Version models, stage transitions (Staging ? Production)
• Model Serving � Real-time or batch inference endpoints

import mlflow

with mlflow.start_run():
    mlflow.log_param("learning_rate", 0.01)
    mlflow.log_metric("accuracy", 0.95)
    mlflow.sklearn.log_model(model, "model")

Q14: How do you implement streaming with Delta Lake?

Answer:

# Read from Kafka, write to Delta Lake
stream_df = (spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "broker:9092")
  .option("subscribe", "events")
  .load())

# Parse and write to Delta
parsed = stream_df.selectExpr("CAST(value AS STRING)")

(parsed.writeStream
  .format("delta")
  .outputMode("append")
  .option("checkpointLocation", "s3://bucket/checkpoints/events")
  .start("s3://bucket/lake/events"))

Auto Loader for file-based streaming:

(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "json")
  .load("s3://bucket/incoming/")
  .writeStream
  .format("delta")
  .start("s3://bucket/lake/events"))

Q15: What security features does Databricks provide on AWS?

Answer:

• Network � VPC deployment, PrivateLink, IP access lists, security groups
• Data Encryption � At rest (AWS KMS with CMK), in transit (TLS 1.2+)
• Access Control � Unity Catalog GRANT/REVOKE, column/row-level security
• Identity � SSO (SAML 2.0), SCIM provisioning, MFA
• Audit � Unity Catalog system tables, CloudTrail integration
• Secrets � Databricks Secrets + AWS Secrets Manager

Q16: How do you migrate from on-premises Hadoop to Databricks on AWS?

Answer:

Tools: Databricks Accelerate, code conversion utilities, and migration assessment tools.

Q17: What is the role of DBFS in Databricks?

Answer: DBFS (Databricks File System) is a distributed file system that provides a unified interface to data stored in S3. It maps paths like dbfs:/mnt/data to s3://bucket/data. DBFS supports:

• FUSE mount on driver/worker nodes for local file access
• Caching hot data on SSDs for faster reads
• Integration with Delta Lake for ACID transactions

Q18: How do you handle schema evolution in Delta Lake?

Answer:

# Merge schema (add new columns)
df.write.format("delta").option("mergeSchema", "true").mode("append").save("/delta/events")

# Overwrite schema (replace all columns)
df.write.format("delta").option("overwriteSchema", "true").mode("overwrite").save("/delta/events")

# SQL
ALTER TABLE events ADD COLUMN (new_col STRING);
ALTER TABLE events ALTER COLUMN old_col RENAME TO new_name;

Q19: What is the difference between DBU and EC2 costs?

Answer:

• DBU (Databricks Unit) � The compute billing unit for Databricks services (notebooks, jobs, SQL). Price varies by tier (Standard, Premium, Enterprise).
• EC2 Costs � The underlying AWS infrastructure costs (instances, storage, networking). Billed directly by AWS.

Total cost = DBU cost + EC2 cost. This separation allows you to optimize each independently.

Q20: How do you monitor Databricks workloads?

Answer:

• Spark UI � Visualize job/stage/task execution, identify bottlenecks
• Databricks SQL Dashboards � Monitor query performance and costs
• System Tables � Unity Catalog audit logs, billing, and usage data
• Workspace Alerts � Email/Slack notifications for job failures
• CloudWatch Integration � Forward metrics to AWS CloudWatch
• Lakehouse Monitoring � Built-in data quality and freshness metrics

Summary

Mastering Databricks on AWS requires understanding:

• Lakehouse Architecture � Unified batch + streaming on Delta Lake
• Unity Catalog � Centralized governance with fine-grained access control
• Delta Sharing � Open protocol for cross-organization data sharing
• AWS Integration � Native connectivity with S3, IAM, KMS, VPC, and CloudTrail
• Cost Optimization � Spot instances, auto-termination, DBU monitoring
• ML Integration � MLflow for end-to-end ML lifecycle management

These concepts form the foundation for building scalable, secure, and cost-efficient data platforms on Databricks and AWS.

Knowledge Check

See Also

• AWS Databricks
• AWS EMR
• EMR Spark