Apache Spark Provider Integration with Airflow
Architecture Diagram
Formal Definitions
Detailed Explanation
Spark Integration Overview
The Spark provider enables Airflow to submit and monitor Spark jobs across YARN, Kubernetes, and standalone clusters. It wraps the spark-submit CLI command.
Key Insight: Use cluster deploy mode for production to decouple the Spark driver from Airflow workers. This prevents worker failures from killing Spark jobs.
Operator Selection Guide
| Operator | Use Case | Best For |
|---|---|---|
| SparkSubmitOperator | Submit any Spark app | PySpark, Scala, Java |
| SparkJDBCOperator | JDBC read/write | Database transfers |
| SparkSQLOperator | Execute SQL queries | Hive/Spark SQL |
| SparkBatchOperator | Batch processing | Structured Streaming |
Deploy Mode Comparison
| Mode | Driver Location | Worker Dependency | Best For |
|---|---|---|---|
| Client | Airflow worker | High | Development, debugging |
| Cluster | Spark cluster | Low | Production, long jobs |
Connection Setup
Basic SparkSubmitOperator
SparkJDBCOperator
PySpark Job File
Deferrable Spark Job Monitoring
Key Concepts Table
| Operator | Purpose | Connection | Best For |
|---|---|---|---|
| SparkSubmitOperator | Submit any Spark app | spark_default | PySpark, Scala, Java |
| SparkJDBCOperator | JDBC read/write | spark_default | Database transfers |
| SparkSQLOperator | Execute SQL queries | spark_default | Hive/Spark SQL |
| SparkBatchOperator | Batch processing | spark_default | Structured Streaming |
Configuration Reference
| Parameter | Description | YARN | Kubernetes |
|---|---|---|---|
conn_id | Spark connection | yarn | k8s://https://<api>:6443 |
deploy_mode | client or cluster | cluster | cluster |
num_executors | Executor count | Fixed or dynamic | Pod replicas |
executor_memory | RAM per executor | 4-16g | Configured via resources |
executor_cores | CPU per executor | 2-8 | Configured via resources |
driver_memory | Driver RAM | 2-8g | 2-8g |
packages | Maven coordinates | Downloaded on submit | Downloaded on submit |
py_files | Python dependencies | Local or HDFS | Mounted volumes |
Best Practices
Job Configuration
- Use cluster deploy mode for production â client mode ties the driver to Airflow's worker.
- Enable dynamic allocation to optimize resource usage:
spark.dynamicAllocation.enabled=true. - Set
num_partitionsappropriately for JDBC reads to avoid OOM or under-parallelization. - Use
packagesfor dependency management instead of pre-installing on cluster.
Monitoring and Performance
- Monitor Spark UI for job performance â check stages, shuffle, and spill metrics.
- Use deferrable operators for long-running Spark jobs to free worker slots.
Operational Guidelines
- Set
timeouton SparkSubmitOperator to prevent indefinite hangs. - Tag Spark applications with Airflow run IDs for traceability.
Spark Configuration Reference
| Configuration | Description | Recommended Value |
|---|---|---|
spark.sql.shuffle.partitions | Number of shuffle partitions | 200 (default) |
spark.sql.adaptive.enabled | Adaptive query execution | true |
spark.dynamicAllocation.enabled | Dynamic executor allocation | true |
spark.dynamicAllocation.minExecutors | Minimum executors | 2 |
spark.dynamicAllocation.maxExecutors | Maximum executors | 20 |
See Also
- Databricks Provider â Databricks-managed Spark clusters
- BigQuery Provider â Google BigQuery integration
- Snowflake Provider â Snowflake data warehouse integration
- Operators and Hooks â Operator lifecycle and hook architecture