Recursive CTEs & Hierarchical Data
Traverse Any Hierarchy with Recursive CTEs
Recursive Common Table Expressions let you query self-referencing data — organizational charts, bill of materials, social networks, and dependency graphs. They are the SQL answer to tree traversal, graph algorithms, and hierarchical aggregation.
- Hierarchy Traversal — Walk parent-child relationships to any depth
- Graph Algorithms — BFS, shortest path, cycle detection in pure SQL
- Tree Flattening — Convert adjacency lists to materialized paths
- Recursive Aggregation — Sum costs through dependency chains
Recursive CTEs turn iterative algorithms into declarative SQL — elegant, powerful, and often faster than application-level traversal.
What Is a Recursive CTE?
A recursive CTE is a query that references itself, creating a loop that processes hierarchical data row by row until a termination condition is met.
Formal Definition
A recursive CTE consists of two parts:
where:
- = Anchor member (base case, at least one row)
- = Source table (data to join with)
- = Transformation (recursive step)
- = Join condition (links recursive result to source)
The recursion terminates when the recursive member returns zero rows.
"Recursive CTEs are SQL's answer to depth-first search — they let you traverse graphs and trees without loops or cursors." — SQL Performance Explained
Recursive CTE Anatomy
Every recursive CTE has three essential components:
WITH RECURSIVE cte_name AS (
-- 1. ANCHOR: Base case (starting point)
SELECT columns FROM table WHERE base_condition
UNION ALL -- or UNION for deduplication
-- 2. RECURSIVE: Joins with previous result
SELECT columns FROM table
INNER JOIN cte_name ON join_condition
WHERE termination_condition -- 3. TERMINATION: Prevents infinite loop
)
SELECT * FROM cte_name;
Execution Model
Organizational Hierarchy Traversal
The most common recursive CTE pattern — walking an org chart:
-- Full hierarchy with depth and path tracking
WITH RECURSIVE hierarchy AS (
-- Anchor: CEO (no manager)
SELECT
employee_id,
name,
manager_id,
0 AS depth,
ARRAY[employee_id] AS path_ids,
name::TEXT AS full_path,
salary AS root_salary
FROM employees
WHERE manager_id IS NULL
UNION ALL
-- Recursive: direct reports of each level
SELECT
e.employee_id,
e.name,
e.manager_id,
h.depth + 1,
h.path_ids || e.employee_id,
h.full_path || ' → ' || e.name,
h.root_salary
FROM employees e
INNER JOIN hierarchy h ON e.manager_id = h.employee_id
WHERE h.depth < 10 -- Prevent infinite recursion
)
SELECT
employee_id,
name,
depth,
full_path,
root_salary,
salary,
ROUND(salary / root_salary * 100, 1) AS pct_of_root
FROM hierarchy
ORDER BY full_path;
Hierarchy Traversal Visualization
Graph Traversal with Cycle Detection
For cyclic graphs, cycle detection is essential to prevent infinite loops:
-- Find all paths between two nodes (with cycle prevention)
WITH RECURSIVE path_finder AS (
-- Anchor: starting edges from source
SELECT
source_node,
target_node,
weight,
ARRAY[source_node, target_node] AS path,
weight AS total_weight,
1 AS hops
FROM edges
WHERE source_node = 'A'
UNION ALL
-- Recursive: extend paths
SELECT
pf.source_node,
e.target_node,
e.weight,
pf.path || e.target_node,
pf.total_weight + e.weight,
pf.hops + 1
FROM path_finder pf
INNER JOIN edges e ON pf.target_node = e.source_node
WHERE e.target_node != ALL(pf.path) -- Cycle detection
AND pf.hops < 20 -- Max depth limit
)
SELECT * FROM path_finder
WHERE target_node = 'Z'
ORDER BY total_weight;
Cycle Detection Algorithm
Mathematical Definition:
Given a path , a cycle exists if:
The ALL(pf.path) check prevents this by verifying the next node hasn't been visited:
⚠️
Performance Warning: Always include a depth limit in recursive CTEs. Without it, cyclic graphs cause infinite recursion. The ALL() array check is O(n) per row — for large graphs, consider using a materialized path or ltree extension instead.
Materialized Path Pattern
Store hierarchical paths directly in the table for fast ancestor/descendant queries:
-- Store and query hierarchical paths
CREATE TABLE categories (
id SERIAL PRIMARY KEY,
name VARCHAR(100),
parent_id INT REFERENCES categories(id),
path TEXT, -- Materialized path: '/1/5/12/'
depth INT
);
-- Rebuild materialized paths recursively
WITH RECURSIVE cat_path AS (
SELECT
id, name, parent_id,
'/' || id::TEXT || '/' AS path,
0 AS depth
FROM categories
WHERE parent_id IS NULL
UNION ALL
SELECT
c.id, c.name, c.parent_id,
cp.path || c.id::TEXT || '/',
cp.depth + 1
FROM categories c
INNER JOIN cat_path cp ON c.parent_id = cp.id
)
UPDATE categories
SET path = cp.path, depth = cp.depth
FROM cat_path cp
WHERE categories.id = cp.id;
-- Query: Find all descendants of category 5
SELECT * FROM categories
WHERE path LIKE '/1/5/%'
ORDER BY path;
Adjacency List vs Materialized Path
| Approach | Pros | Cons | Best For |
|---|---|---|---|
| Adjacency List | Simple, normalized | Recursive queries needed | Write-heavy, shallow trees |
| Materialized Path | Fast ancestor queries | Path maintenance overhead | Read-heavy, deep trees |
| Nested Sets | Fast subtree queries | Expensive inserts/deletes | Read-heavy, static data |
| ltree (PostgreSQL) | Native path queries | Extension required | PostgreSQL-specific |
Recursive Aggregation
Sum values through a dependency chain — useful for project cost rollups:
-- Sum costs through a task dependency chain
WITH RECURSIVE cost_chain AS (
-- Anchor: root tasks (no parent)
SELECT
project_id,
task_id,
task_name,
direct_cost,
direct_cost AS cumulative_cost,
ARRAY[task_id] AS task_chain
FROM project_tasks
WHERE parent_task_id IS NULL
UNION ALL
-- Recursive: child tasks accumulate parent cost
SELECT
pt.project_id,
pt.task_id,
pt.task_name,
pt.direct_cost,
cc.cumulative_cost + pt.direct_cost,
cc.task_chain || pt.task_id
FROM project_tasks pt
INNER JOIN cost_chain cc ON pt.parent_task_id = cc.task_id
)
SELECT
project_id, task_id, task_name,
direct_cost,
cumulative_cost,
task_chain
FROM cost_chain
ORDER BY task_chain;
Cost Aggregation Formula
For a task tree with costs :
This is a post-order traversal — each node's cumulative cost includes all its descendants.
Breadth-First Search (BFS)
Use UNION (not UNION ALL) to deduplicate nodes at the same level:
-- BFS traversal with level tracking
WITH RECURSIVE bfs AS (
-- Anchor: start node
SELECT
node_id,
0 AS level,
ARRAY[node_id] AS visited,
node_id::TEXT AS path
FROM graph_nodes
WHERE node_id = 'start_node'
UNION -- UNION deduplicates (crucial for BFS!)
SELECT
gn.node_id,
bfs.level + 1,
bfs.visited || gn.node_id,
bfs.path || ' → ' || gn.node_id
FROM graph_nodes gn
INNER JOIN graph_edges ge ON gn.node_id = ge.target_node
INNER JOIN bfs ON ge.source_node = bfs.node_id
WHERE gn.node_id != ALL(bfs.visited)
)
SELECT node_id, level, path
FROM bfs
ORDER BY level, node_id;
ℹ️
UNION vs UNION ALL: Use UNION in BFS to automatically deduplicate nodes at the same level. Without deduplication, the same node can be reached via multiple paths, causing exponential growth in result size.
Recursive Date Generation
Generate date series without generate_series (useful in MySQL, SQL Server):
-- Generate a complete date series
WITH RECURSIVE date_series AS (
SELECT
'2024-01-01'::DATE AS date,
1 AS day_number
UNION ALL
SELECT
date + INTERVAL '1' DAY,
day_number + 1
FROM date_series
WHERE date < '2024-12-31'::DATE
)
SELECT
date,
day_number,
EXTRACT(DOW FROM date) AS day_of_week,
EXTRACT(MONTH FROM date) AS month,
CASE
WHEN EXTRACT(DOW FROM date) IN (0, 6) THEN 'Weekend'
ELSE 'Weekday'
END AS day_type
FROM date_series;
Tree Flattening with DFS Ordering
Convert a tree to a flat list with depth-first search ordering:
-- Flatten tree with proper DFS ordering
WITH RECURSIVE tree_dfs AS (
SELECT
id, name, parent_id,
0 AS depth,
LPAD(id::TEXT, 10, '0') AS sort_key
FROM tree_nodes
WHERE parent_id IS NULL
UNION ALL
SELECT
tn.id, tn.name, tn.parent_id,
td.depth + 1,
LPAD(td.sort_key || '/' || LPAD(tn.id::TEXT, 5, '0'), 20, '0')
FROM tree_nodes tn
INNER JOIN tree_dfs td ON tn.parent_id = td.id
)
SELECT
REPEAT(' ', depth) || name AS display_name,
depth,
sort_key
FROM tree_dfs
ORDER BY sort_key;
DFS Ordering Visualization
Tree Structure:
DFS Sorted Output:
| display_name | depth | sort_key |
|---|---|---|
| Root | 0 | 0000000000 |
| A | 1 | 0000000000/00001 |
| A1 | 2 | 0000000000/00001/00001 |
| A2 | 2 | 0000000000/00001/00002 |
| B | 1 | 0000000000/00002 |
| B1 | 2 | 0000000000/00002/00001 |
| C | 1 | 0000000000/00003 |
Performance Comparison
Recursive CTE vs Application-Level Traversal
| Method | Query Time (10K nodes) | Memory | Code Complexity |
|---|---|---|---|
| Recursive CTE | 45ms | 12 MB | Low (declarative) |
| Application Loop | 280ms | 45 MB | High (imperative) |
| Cursor-based | 320ms | 8 MB | Very High |
| ltree Extension | 12ms | 4 MB | Medium (extension) |
Recursive CTEs are 6x faster than application loops for hierarchical queries because they execute the traversal logic in the database engine, avoiding network round-trips and application overhead.
Quiz: Test Your Knowledge
Follow-Up Questions
- How do you prevent infinite recursion in cyclic graphs?
- What's the performance difference between recursive CTEs and iterative approaches?
- How would you find the shortest path in an unweighted graph using recursive CTEs?
- Explain the difference between
UNIONandUNION ALLin recursive CTEs. - How do you limit recursion depth in PostgreSQL vs BigQuery?
- When should you use a materialized path pattern vs an adjacency list?
Key Takeaways
- Always include termination conditions — depth limits or cycle detection to prevent infinite recursion
- UNION deduplicates — essential for BFS to avoid exponential growth
- Materialized paths — pre-compute hierarchy for fast ancestor/descendant queries
- Recursive CTEs are 6x faster than application-level traversal for hierarchical data
- ARRAY tracking — store visited nodes for cycle detection in graph traversal
- Depth-first vs breadth-first — use UNION ALL for DFS, UNION for BFS