Spark Fundamentals for Big Data
Apache Spark is the unified analytics engine for large-scale data processing. This module covers the core abstractions: RDDs, DataFrames, Spark SQL, and the Catalyst optimizer.
Spark Architecture
1. RDDs (Resilient Distributed Datasets)
RDDs are Spark's foundational abstraction — immutable, partitioned collections of elements that can be operated on in parallel.
Partition count determines parallelism:
Key Properties
- Resilient: Fault-tolerant via lineage graph
- Distributed: Data split across partitions on multiple nodes
- Dataset: Collection of partitioned data with primitives
Transformations vs Actions
Narrow vs Wide Dependencies
2. DataFrames and Spark SQL
3. Lazy Evaluation and Catalyst Optimizer
4. Partitioning Strategy
Partition Tuning Rules
| Scenario | Recommendation |
|---|---|
| Small files problem | Coalesce before write |
| Large shuffle operations | Increase spark.sql.shuffle.partitions |
| Join on key | Partition by join key |
| Time-series range | Partition by date |
5. Spark MLlib
6. Performance Tuning Checklist
- Cache wisely:
df.cache()for repeated use;df.persist()with storage level - Avoid shuffles: Use broadcast joins for small tables (
broadcast(small_df)) - Tune memory:
spark.executor.memory,spark.driver.memory - AQE: Adaptive Query Execution (
spark.sql.adaptive.enabled=true) - Broadcast threshold:
spark.sql.autoBroadcastJoinThreshold
Key Takeaways
- RDDs provide low-level control; DataFrames leverage Catalyst optimization
- Lazy evaluation enables whole-plan optimization before execution
- Partitioning is the primary lever for performance tuning
- Shuffle is expensive — design pipelines to minimize data movement