BigQuery Provider Integration with Airflow

Architecture Diagram

Formal Definitions

Detailed Explanation

BigQuery Integration Overview

The BigQuery provider enables Airflow to execute queries, load data, and manage datasets in Google BigQuery. It wraps the BigQuery Python client library.

Key Insight: Partition pruning reduces query scan from O(N) to O(N/P) where P is partition count. Always include partition columns in WHERE clauses.

Operator Selection Guide

Partitioning Strategies

Connection Setup

Query Execution

GCS to BigQuery Loading

Dataset Management

Sensors

Partitioned Query with Parameters

Key Concepts Table

Partitioning Strategies

Best Practices

Query Optimization

1. Always specify location to match your dataset's geographic location.
2. Use partitioned tables for time-series data to reduce query costs and improve performance.
3. Leverage clustering on frequently filtered columns for additional query optimization.

Data Loading

1. Use WRITE_TRUNCATE for full table refreshes, WRITE_APPEND for incremental loads.
2. Handle schema_fields explicitly for CSV/JSON loads to avoid schema detection overhead.
3. Use create_disposition='CREATE_IF_NEEDED' for idempotent table creation.

Cost Management

1. Set priority on jobs: INTERACTIVE for ad-hoc, BATCH for scheduled ETL.
2. Monitor BQ job costs through configuration.dryRun and GCP billing exports.

Cost Optimization Strategies

See Also

• Databricks Provider — Databricks cluster and job management
• Snowflake Provider — Snowflake integration patterns
• Operators and Hooks — Operator lifecycle and hook architecture
• XCom Communications — Task communication and data passing