Databricks Provider Integration with Airflow

Architecture Diagram

Formal Definitions

Detailed Explanation

Databricks Integration Overview

The Databricks provider enables Airflow to orchestrate Spark jobs, notebooks, and cluster operations on Databricks. It abstracts the complexity of the Databricks REST API.

Key Insight: Use DatabricksRunNowOperator for long-running jobs (>30 min) to avoid the 15-minute timeout on SubmitRunOperator.

Operator Selection Guide

Connection Setup

DatabricksSubmitRunOperator

The primary operator for submitting new Databricks jobs. It creates a new cluster or uses an existing one, submits the job, and monitors until completion.

DatabricksNotebookOperator

Execute Databricks notebooks as part of Airflow workflows.

DatabricksRunNowOperator

Trigger an existing Databricks job by ID.

Sensor for Job Completion

Full ETL Pipeline

Key Concepts Table

Configuration Reference

Best Practices

Cluster Management

1. Use existing clusters when possible — cluster startup can take 5-10 minutes.
2. Tag clusters with team/project for cost tracking and management.
3. Monitor job runs through Databricks job history and Airflow task logs.

Operator Configuration

1. Set timeout_seconds on all operators to prevent indefinite waits.
2. Use mode='reschedule' on sensors to free worker slots during waits.
3. Use libraries for dependency management — wheel files, pip packages, or maven coordinates.

Error Handling

1. Handle TERMINATED_WITH_ERRORS state with retries and failure callbacks.
2. Leverage json_params for dynamic job parameterization via Jinja templates.

Cost Optimization

See Also

• Spark Provider Integration — Apache Spark with Airflow
• BigQuery Provider — BigQuery integration patterns
• Operators and Hooks — Operator lifecycle and hook architecture
• Sensors and Operators — Sensor patterns and poke modes