⚡ PySpark UDF Optimization

DfUser Defined Function (UDF)

A UDF is a custom function defined by the user that extends Spark's built-in functions. Python UDFs are serialized via pickle and executed in a separate Python process per executor, incurring serialization overhead per row.

DfPandas UDF (Vectorized UDF)

A Pandas UDF uses Apache Arrow to transfer data between JVM and Python in batches, enabling vectorized operations via pandas/Series. This avoids row-by-row serialization overhead, achieving 3x–100x speedup over regular Python UDFs.

T_{udf} = N_{rows} \\times (T_{serialize} + T_{process} + T_{deserialize}) + T_{arrow}

Pandas UDF Speedup Factor

Speedup = \\frac{T_{python\\_udf}}{T_{pandas\\_udf}} = \\frac{N_{rows} \\times (T_{ser} + T_{deser})}{T_{arrow} + T_{vectorized}}

Here,

$T_{python\_udf}$ =Execution time with regular Python UDF
$T_{pandas\_udf}$ =Execution time with Pandas UDF
$T_{arrow}$ =Arrow batch transfer time (amortized per row)
$T_{vectorized}$ =Vectorized computation time via pandas

Python UDFs are 10x–100x slower than built-in functions due to per-row serialization between JVM and Python. Pandas UDFs reduce this overhead by transferring data in batches via Apache Arrow.

Use built-in Spark SQL functions whenever possible — they are implemented in JVM and optimized by Catalyst. Only use UDFs when built-in functions cannot express the required logic.

ThArrow Batch Transfer Optimization

Theorem: Pandas UDFs achieve a minimum speedup of N_{rows} / B_{batch} over Python UDFs, where B_{batch} is the Arrow batch size (default 1000 rows). Larger batches amortize Arrow overhead further, with diminishing returns beyond B_{batch} = 10,000.

Python UDFs: row-by-row serialization, 10x–100x slower than built-in functions
Pandas UDFs: batch processing via Arrow, 3x–100x faster than Python UDFs
Always prefer built-in functions; use UDFs only when necessary
Tune arrow.maxRecordsPerBatch for optimal batch size (default 1000)

🏗️ Architecture Diagram

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    UDF EXECUTION ARCHITECTURE                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                  PYTHON UDF FLOW                         │   │
│  │                                                          │   │
│  │  ┌─────────────┐                                        │   │
│  │  │  Driver     │  Send UDF function (pickle)            │   │
│  │  │  (Python)   │ ──────────────────────────────────►   │   │
│  │  └─────────────┘                                        │   │
│  │                                                          │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  Executor (JVM + Python)                         │   │   │
│  │  │                                                  │   │   │
│  │  │  ┌─────────────────────────────────────────┐   │   │   │
│  │  │  │  Spark JVM Process                      │   │   │   │
│  │  │  │  ┌─────────────────────────────────┐   │   │   │   │
│  │  │  │  │  Row-by-Row Processing           │   │   │   │   │
│  │  │  │  │  Row 1 → Python → Result 1      │   │   │   │   │
│  │  │  │  │  Row 2 → Python → Result 2      │   │   │   │   │
│  │  │  │  │  Row 3 → Python → Result 3      │   │   │   │   │
│  │  │  │  │  ...                            │   │   │   │   │
│  │  │  │  └─────────────────────────────────┘   │   │   │   │
│  │  │  │         ▲                               │   │   │   │
│  │  │  │         │ Serialization (Py4J)         │   │   │   │
│  │  │  │         ▼                               │   │   │   │
│  │  │  │  ┌─────────────────────────────────┐   │   │   │   │
│  │  │  │  │  Python Worker Process          │   │   │   │   │
│  │  │  │  │  def my_udf(x):                 │   │   │   │   │
│  │  │  │  │      return x * 2               │   │   │   │   │
│  │  │  │  └─────────────────────────────────┘   │   │   │   │
│  │  │  └─────────────────────────────────────────┘   │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  │                                                          │   │
│  │  OVERHEAD: Serialization per row + Python process switch │   │
│  │  PERFORMANCE: ~100x slower than native Spark operations  │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                  PANDAS UDF FLOW                         │   │
│  │                                                          │   │
│  │  ┌─────────────┐                                        │   │
│  │  │  Driver     │  Send UDF function (pickle)            │   │
│  │  │  (Python)   │ ──────────────────────────────────►   │   │
│  │  └─────────────┘                                        │   │
│  │                                                          │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  Executor (JVM + Python)                         │   │   │   │
│  │  │                                                  │   │   │   │
│  │  │  ┌─────────────────────────────────────────┐   │   │   │   │
│  │  │  │  Spark JVM Process                      │   │   │   │   │
│  │  │  │  ┌─────────────────────────────────┐   │   │   │   │
│  │  │  │  │  Batch Processing                │   │   │   │   │
│  │  │  │  │  Batch 1 (1000 rows) → Pandas   │   │   │   │   │
│  │  │  │  │  Batch 2 (1000 rows) → Pandas   │   │   │   │   │
│  │  │  │  │  Batch 3 (1000 rows) → Pandas   │   │   │   │   │
│  │  │  │  └─────────────────────────────────┘   │   │   │   │
│  │  │  │         ▲                               │   │   │   │
│  │  │  │         │ Arrow Serialization          │   │   │   │
│  │  │  │         ▼                               │   │   │   │
│  │  │  │  ┌─────────────────────────────────┐   │   │   │   │
│  │  │  │  │  Python Worker Process          │   │   │   │   │
│  │  │  │  │  def my_pandas_udf(pdf):        │   │   │   │   │
│  │  │  │  │      return pdf * 2             │   │   │   │   │
│  │  │  │  └─────────────────────────────────┘   │   │   │   │
│  │  │  └─────────────────────────────────────────┘   │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  │                                                          │   │
│  │  ADVANTAGE: Vectorized operations, Arrow serialization   │   │
│  │  PERFORMANCE: ~10x faster than Python UDFs              │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                  ARROW SERIALIZATION                      │   │
│  │                                                          │   │
│  │  JVM (Spark)                    Python (Pandas)          │   │
│  │  ┌──────────────┐              ┌──────────────┐         │   │
│  │  │  Columnar    │    Arrow     │  Columnar    │         │   │
│  │  │  Format      │ ──────────► │  Format      │         │   │
│  │  │  (Internal)  │  Zero-copy   │  (Pandas)    │         │   │
│  │  └──────────────┘              └──────────────┘         │   │
│  │                                                          │   │
│  │  • Zero-copy data transfer                               │   │
│  │  • Columnar format (cache-friendly)                      │   │
│  │  • Vectorized operations                                 │   │
│  │  • No per-row serialization                              │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│              UDF TYPE COMPARISON MATRIX                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                  PERFORMANCE COMPARISON                   │   │
│  │                                                          │   │
│  │  Operation: Process 1M rows                              │   │
│  │                                                          │   │
│  │  ┌─────────────────┬──────────┬──────────┬──────────┐  │   │
│  │  │    Method       │   Time   │  Memory  │   GC     │  │   │
│  │  ├─────────────────┼──────────┼──────────┼──────────┤  │   │
│  │  │ Native Spark    │  100ms   │   50MB   │   Low    │  │   │
│  │  │ Pandas UDF      │  500ms   │  100MB   │  Medium  │  │   │
│  │  │ Python UDF      │ 5000ms   │  200MB   │   High   │  │   │
│  │  │ Python UDF+map  │ 4500ms   │  180MB   │   High   │  │   │
│  │  └─────────────────┴──────────┴──────────┴──────────┘  │   │
│  │                                                          │   │
│  │  Speedup: Native > Pandas (10x) > Python (100x)        │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                  MEMORY USAGE COMPARISON                  │   │
│  │                                                          │   │
│  │  Native Spark:                                          │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  InternalRow[] (compact, columnar)               │   │   │
│  │  │  Memory: 1x data size                           │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  │                                                          │   │
│  │  Pandas UDF:                                            │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  Arrow → Pandas DataFrame → Arrow               │   │   │
│  │  │  Memory: 2x data size (Arrow + Pandas)          │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  │                                                          │   │
│  │  Python UDF:                                            │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  InternalRow → Python object → InternalRow      │   │   │
│  │  │  Memory: 3-5x data size (Python objects)        │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │                  SERIALIZATION OVERHEAD                    │   │
│  │                                                          │   │
│  │  Python UDF (Py4J):                                     │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  Row 1: Serialize → Process → Deserialize       │   │   │
│  │  │  Row 2: Serialize → Process → Deserialize       │   │   │
│  │  │  Row 3: Serialize → Process → Deserialize       │   │   │
│  │  │  ...                                            │   │   │
│  │  │  Overhead: ~100ns per row                       │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  │                                                          │   │
│  │  Pandas UDF (Arrow):                                    │   │
│  │  ┌─────────────────────────────────────────────────┐   │   │
│  │  │  Batch: Serialize once → Process → Deserialize  │   │   │
│  │  │  Overhead: ~1ms per batch (1000 rows)           │   │   │
│  │  └─────────────────────────────────────────────────┘   │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│              UDF OPTIMIZATION DECISION TREE                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │  Can you implement with built-in Spark functions?        │   │
│  │                                                          │   │
│  │  YES ──────────────────────────────────────────────►    │   │
│  │  │       Use Spark functions (fastest)                  │   │
│  │  │       - col(), when(), lit(), etc.                   │   │
│  │  │       - 10-100x faster than UDFs                     │   │
│  │  │                                                      │   │
│  │  NO                                                     │   │
│  │  │                                                      │   │
│  │  ▼                                                      │   │
│  │  Can you use Pandas/NumPy for vectorized ops?          │   │
│  │                                                          │   │
│  │  YES ──────────────────────────────────────────────►    │   │
│  │  │       Use Pandas UDF (10x faster)                    │   │
│  │  │       @pandas_udf(returnType)                        │   │
│  │  │       def my_udf(pdf: pd.DataFrame) -> pd.DataFrame │   │
│  │  │                                                      │   │
│  │  NO                                                     │   │
│  │  │                                                      │   │
│  │  ▼                                                      │   │
│  │  Use Python UDF (slowest, but flexible)                │   │
│  │  @udf(returnType)                                       │   │
│  │  def my_udf(x):                                         │   │
│  │      return ...                                         │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │  OPTIMIZATION CHECKLIST                                  │   │
│  │                                                          │   │
│  │  □ Try built-in functions first                         │   │
│  │  □ Use Pandas UDF instead of Python UDF                 │   │
│  │  □ Minimize data serialization                          │   │
│  │  □ Use @pandas_udf("double") for simple transforms     │   │
│  │  □ Use @pandas_udf(returnType, PandasUDFType.GROUPED_  │   │
│  │    MAP) for group operations                            │   │
│  │  □ Enable Arrow: spark.sql.execution.arrow.enabled=true│   │
│  │  □ Tune batch size: spark.sql.execution.arrow.maxRecords│   │
│  │    PerBatch (default 1000)                              │   │
│  │  □ Avoid global variables in UDFs                       │   │
│  │  □ Use broadcast for large lookup tables                │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

📚 Detailed Explanation

1. What is a UDF?

A User Defined Function (UDF) is a function defined by the user that can be used in Spark SQL and DataFrame operations. UDFs extend Spark's built-in function library to implement custom logic that isn't available out of the box.

Types of UDFs in PySpark:

Python UDF: Standard Python function processed row-by-row
Pandas UDF: Vectorized function using Pandas/NumPy operations
Grouped Map UDF: Processes groups of rows as Pandas DataFrames

2. Python UDF Performance Characteristics

Python UDFs have significant performance overhead due to:

Serialization Overhead:

Each row is serialized from JVM to Python (Py4J)
Python processes the row
Result is serialized back to JVM
This happens for EVERY row

Process Switch Overhead:

Context switching between JVM and Python
Memory copying between processes
Garbage collection in both JVM and Python

Python Execution Overhead:

Python is interpreted (no JIT compilation)
GIL limits true parallelism
Object creation/destruction overhead

Performance Impact:

~10-100x slower than native Spark operations
Significant memory overhead (Python object duplication)
High GC pressure in both JVM and Python

3. Pandas UDF Performance Characteristics

Pandas UDFs (introduced in Spark 2.3) dramatically improve performance by:

Vectorized Processing:

Process batches of rows at once (default: 1000 rows)
Use Pandas/NumPy for vectorized operations
Avoid per-row function call overhead

Arrow Serialization:

Zero-copy data transfer between JVM and Python
Columnar format (cache-friendly)
No per-row serialization/deserialization

Performance Improvement:

~10x faster than Python UDFs
Lower memory usage (Arrow format)
Better GC behavior

4. When to Use UDFs

Use Built-in Functions When Possible:

# GOOD: Built-in function
from pyspark.sql.functions import col, upper, concat
df.withColumn("name_upper", upper(col("name")))

# BAD: UDF for same thing
@udf(StringType())
def my_upper(s):
    return s.upper() if s else None
df.withColumn("name_upper", my_upper(col("name")))

Use Pandas UDF When:

Complex logic not supported by built-in functions
Need to use Pandas/NumPy libraries
Processing batches of rows
Group-by operations with complex logic

Use Python UDF When:

Simple row-level logic
Cannot be vectorized
Need to call external services
Legacy code compatibility

5. Pandas UDF Types

Scalar Pandas UDF:

@pandas_udf(DoubleType())
def multiply_by_two(pdf: pd.Series) -> pd.Series:
    return pdf * 2

Grouped Map Pandas UDF:

@pandas_udf(schema, PandasUDFType.GROUPED_MAP)
def normalize(pdf: pd.DataFrame) -> pd.DataFrame:
    pdf['value'] = (pdf['value'] - pdf['value'].mean()) / pdf['value'].std()
    return pdf

Grouped Aggregate Pandas UDF:

@pandas_udf(DoubleType(), PandasUDFType.GROUPED_AGG)
def mean_udf(v: pd.Series) -> float:
    return v.mean()

6. Arrow Configuration

Apache Arrow enables efficient data transfer between JVM and Python:

Configuration:

spark.conf.set("spark.sql.execution.arrow.enabled", "true")
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "1000")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

Batch Size Tuning:

Default: 1000 rows per batch
Larger batches: Better throughput, more memory
Smaller batches: Lower latency, less memory

7. UDF Testing and Debugging

Testing UDFs:

# Unit test Python UDF
def test_my_udf():
    assert my_udf("hello") == "HELLO"
    assert my_udf(None) is None

# Test with Spark
df = spark.createDataFrame([("hello",)], ["text"])
result = df.withColumn("upper", my_udf(col("text"))).collect()
assert result[0]["upper"] == "HELLO"

Debugging UDFs:

# Add logging
import logging
logging.basicConfig(level=logging.DEBUG)

@udf(StringType())
def debug_udf(x):
    logging.debug(f"Input: {x}")
    result = x.upper() if x else None
    logging.debug(f"Output: {result}")
    return result

8. Common UDF Pitfalls

Pitfall 1: Using global variables

# BAD: Global variable not broadcast
LOOKUP = {"a": 1, "b": 2}  # Not serialized to executors

@udf(IntegerType())
def lookup_value(key):
    return LOOKUP.get(key)  # Error on executors!

# GOOD: Use broadcast variable
broadcast_lookup = spark.sparkContext.broadcast({"a": 1, "b": 2})

@udf(IntegerType())
def lookup_value(key):
    return broadcast_lookup.value.get(key)

Pitfall 2: Not handling nulls

# BAD: No null handling
@udf(DoubleType())
def process_value(x):
    return x * 2  # Error if x is None

# GOOD: Handle nulls
@udf(DoubleType())
def process_value(x):
    return x * 2 if x is not None else None

Pitfall 3: Using Python UDF when Pandas UDF would work

# BAD: Python UDF for simple transformation
@udf(DoubleType())
def double_value(x):
    return x * 2

# GOOD: Pandas UDF for vectorized operation
@pandas_udf(DoubleType())
def double_value(pdf: pd.Series) -> pd.Series:
    return pdf * 2

🔑 Key Concepts Table

UDF Type	Processing	Serialization	Speed	Memory	Use Case
Built-in	JVM native	None	★★★★★	★★★★★	Always prefer
Pandas UDF	Batch (Vectorized)	Arrow	★★★★☆	★★★★☆	Complex logic
Python UDF	Row-by-row	Py4J	★★☆☆☆	★★☆☆☆	Simple logic
Grouped Map	Group batch	Arrow	★★★★☆	★★★★☆	Group operations

💻 Code Examples

Example 1: Python UDF vs Pandas UDF

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf, pandas_udf, col
from pyspark.sql.types import DoubleType
import pandas as pd

spark = SparkSession.builder.appName("UDFOptimization").getOrCreate()

# Create test data
df = spark.range(1000000).withColumn("value", col("id") * 1.0)

# Python UDF (slow)
@udf(DoubleType())
def python_double(x):
    return x * 2 if x is not None else None

# Pandas UDF (fast)
@pandas_udf(DoubleType())
def pandas_double(pdf: pd.Series) -> pd.Series:
    return pdf * 2

# Benchmark
import time

# Python UDF
start = time.time()
result_python = df.withColumn("doubled", python_double(col("value")))
result_python.count()
python_time = time.time() - start

# Pandas UDF
start = time.time()
result_pandas = df.withColumn("doubled", pandas_double(col("value")))
result_pandas.count()
pandas_time = time.time() - start

print(f"Python UDF: {python_time:.2f}s")
print(f"Pandas UDF: {pandas_time:.2f}s")
print(f"Speedup: {python_time / pandas_time:.1f}x")

Example 2: Grouped Map Pandas UDF

from pyspark.sql.functions import pandas_udf, PandasUDFType
from pyspark.sql.types import *
import pandas as pd

# Define schema
schema = StructType([
    StructField("group", IntegerType()),
    StructField("value", DoubleType()),
    StructField("normalized", DoubleType())
])

# Grouped Map UDF
@pandas_udf(schema, PandasUDFType.GROUPED_MAP)
def normalize(pdf: pd.DataFrame) -> pd.DataFrame:
    pdf['normalized'] = (pdf['value'] - pdf['value'].mean()) / pdf['value'].std()
    return pdf

# Create data
df = spark.range(100000).withColumn(
    "group", col("id") % 10
).withColumn(
    "value", col("id") * 1.0
)

# Apply UDF
result = df.groupby("group").apply(normalize)
result.show()

Example 3: Arrow Configuration

# Enable Arrow for better performance
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "1000")
spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")

# Pandas UDF with Arrow
@pandas_udf(DoubleType())
def efficient_udf(pdf: pd.Series) -> pd.Series:
    return pdf * 2

# This will use Arrow for efficient data transfer
result = df.withColumn("doubled", efficient_udf(col("value")))

Example 4: UDF with Broadcast Variable

from pyspark.sql.functions import udf, broadcast
from pyspark.sql.types import IntegerType

# Create lookup table
lookup_data = {i: i * 10 for i in range(100)}
broadcast_lookup = spark.sparkContext.broadcast(lookup_data)

# UDF using broadcast
@udf(IntegerType())
def lookup_value(key):
    return broadcast_lookup.value.get(key)

# Apply UDF
df = spark.range(1000).withColumn("key", col("id") % 100)
result = df.withColumn("lookup_result", lookup_value(col("key")))
result.show()

📊 Performance Metrics

Method	1M Rows	10M Rows	Memory	GC	Complexity
Built-in	100ms	800ms	50MB	Low	N/A
Pandas UDF	500ms	4s	100MB	Medium	Medium
Python UDF	5s	50s	200MB	High	High
Arrow Batch	400ms	3.5s	80MB	Low	Medium
Grouped Map	800ms	7s	150MB	Medium	High

✅ Best Practices

1. Prefer Built-in Functions

# GOOD: Use built-in functions
from pyspark.sql.functions import upper, concat, when
df.withColumn("name_upper", upper(col("name")))

# BAD: UDF for same thing
@udf(StringType())
def my_upper(s):
    return s.upper() if s else None

2. Use Pandas UDF When Possible

# BAD: Python UDF
@udf(DoubleType())
def process(x):
    return x * 2

# GOOD: Pandas UDF
@pandas_udf(DoubleType())
def process(pdf: pd.Series) -> pd.Series:
    return pdf * 2

3. Enable Arrow

spark.conf.set("spark.sql.execution.arrow.enabled", "true")
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "1000")

4. Handle Nulls

@pandas_udf(DoubleType())
def safe_process(pdf: pd.Series) -> pd.Series:
    return pdf * 2  # Pandas handles NaN automatically

5. Use Broadcast for Large Lookups

lookup = spark.sparkContext.broadcast(large_dict)

@udf(IntegerType())
def lookup_value(key):
    return lookup.value.get(key)

6. Tune Batch Size

# Larger batches for better throughput
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "5000")

# Smaller batches for lower latency
spark.conf.set("spark.sql.execution.arrow.maxRecordsPerBatch", "100")

PySpark UDF Optimization: UDFs, Pandas UDFs, and Performance

⚡ PySpark UDF Optimization

DfUser Defined Function (UDF)

DfPandas UDF (Vectorized UDF)

Pandas UDF Speedup Factor

ThArrow Batch Transfer Optimization

🏗️ Architecture Diagram

📚 Detailed Explanation

1. What is a UDF?

2. Python UDF Performance Characteristics

3. Pandas UDF Performance Characteristics

4. When to Use UDFs

5. Pandas UDF Types

6. Arrow Configuration

7. UDF Testing and Debugging

8. Common UDF Pitfalls

🔑 Key Concepts Table

💻 Code Examples

Example 1: Python UDF vs Pandas UDF

Example 2: Grouped Map Pandas UDF

Example 3: Arrow Configuration

Example 4: UDF with Broadcast Variable

📊 Performance Metrics

✅ Best Practices

1. Prefer Built-in Functions

2. Use Pandas UDF When Possible

3. Enable Arrow

4. Handle Nulls

5. Use Broadcast for Large Lookups

6. Tune Batch Size

See Also

Need Expert PySpark Help?