Advanced Pandas: Performance and Patterns

Master advanced Pandas — performance optimization, multi-index, and production patterns.

MultiIndex (Hierarchical Indexing)

MultiIndex Selection

MultiIndex Aggregation

Window Functions

Rolling Apply with Arguments

apply() vs Vectorized Operations

Performance Benchmark

Method Chaining

Memory Optimization

Sparse Arrays

Chunked Processing

Production Patterns

Error Handling

Schema Validation

Logging and Profiling

Performance Tips

Key Takeaways

Practice Exercise

1. Create a MultiIndex DataFrame with 3 levels (country, city, year). Compute year-over-year growth rate for each city using groupby() and pct_change().
2. Implement a rolling window function that computes both SMA and EWMA, then generates buy/sell signals when SMA crosses EWMA.
3. Benchmark apply() vs vectorized vs np.select() for a conditional column creation with 5 conditions on a 1M row DataFrame. Report the speedup factor.
4. Build a production-ready ETL pipeline using method chaining: validate schema → clean data → transform → profile → export. Each step should log its results.
5. Optimize a large DataFrame (10M rows) by downcasting dtypes and converting strings to categories. Measure memory savings and verify data integrity.