🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

Recursive CTEs — Hierarchical Data, Graph Traversal & Tree Algorithms

Advanced SQLCommon Table Expressions⭐ Premium

Advertisement

Recursive CTEs & Hierarchical Data

Advanced SQL

Traverse Any Hierarchy with Recursive CTEs

Recursive Common Table Expressions let you query self-referencing data — organizational charts, bill of materials, social networks, and dependency graphs. They are the SQL answer to tree traversal, graph algorithms, and hierarchical aggregation.

  • Hierarchy Traversal — Walk parent-child relationships to any depth
  • Graph Algorithms — BFS, shortest path, cycle detection in pure SQL
  • Tree Flattening — Convert adjacency lists to materialized paths
  • Recursive Aggregation — Sum costs through dependency chains

Recursive CTEs turn iterative algorithms into declarative SQL — elegant, powerful, and often faster than application-level traversal.


What Is a Recursive CTE?

A recursive CTE is a query that references itself, creating a loop that processes hierarchical data row by row until a termination condition is met.

Formal Definition

A recursive CTE consists of two parts:

where:

  • = Anchor member (base case, at least one row)
  • = Source table (data to join with)
  • = Transformation (recursive step)
  • = Join condition (links recursive result to source)

The recursion terminates when the recursive member returns zero rows.

"Recursive CTEs are SQL's answer to depth-first search — they let you traverse graphs and trees without loops or cursors." — SQL Performance Explained


Recursive CTE Anatomy

Every recursive CTE has three essential components:

WITH RECURSIVE cte_name AS (
    -- 1. ANCHOR: Base case (starting point)
    SELECT columns FROM table WHERE base_condition

    UNION ALL  -- or UNION for deduplication

    -- 2. RECURSIVE: Joins with previous result
    SELECT columns FROM table
    INNER JOIN cte_name ON join_condition
    WHERE termination_condition  -- 3. TERMINATION: Prevents infinite loop
)
SELECT * FROM cte_name;

Execution Model

Step 1: AnchorExecute base queryResult: R₀ = {{CEO}}Step 2: JoinR₀ ⨝ employeesFind direct reportsStep 3: TransformCompute level, pathR₁ = {{VP1, VP2}}CheckRows > 0?Yes → LoopRepeat until no new rowsFinal Result = R₀ ∪ R₁ ∪ R₂ ∪ ... ∪ Rₙ (all iterations combined)

Organizational Hierarchy Traversal

The most common recursive CTE pattern — walking an org chart:

-- Full hierarchy with depth and path tracking
WITH RECURSIVE hierarchy AS (
  -- Anchor: CEO (no manager)
  SELECT
    employee_id,
    name,
    manager_id,
    0 AS depth,
    ARRAY[employee_id] AS path_ids,
    name::TEXT AS full_path,
    salary AS root_salary
  FROM employees
  WHERE manager_id IS NULL

  UNION ALL

  -- Recursive: direct reports of each level
  SELECT
    e.employee_id,
    e.name,
    e.manager_id,
    h.depth + 1,
    h.path_ids || e.employee_id,
    h.full_path || ' → ' || e.name,
    h.root_salary
  FROM employees e
  INNER JOIN hierarchy h ON e.manager_id = h.employee_id
  WHERE h.depth < 10  -- Prevent infinite recursion
)
SELECT
  employee_id,
  name,
  depth,
  full_path,
  root_salary,
  salary,
  ROUND(salary / root_salary * 100, 1) AS pct_of_root
FROM hierarchy
ORDER BY full_path;

Hierarchy Traversal Visualization

CEO (depth=0)VP Engineering (1)VP Marketing (1)VP Sales (1)Manager (2)Manager (2)Manager (2)Manager (2)Manager (2)Level 3: Individual Contributors (depth=3) — recursion stops when no direct reports found

Graph Traversal with Cycle Detection

For cyclic graphs, cycle detection is essential to prevent infinite loops:

-- Find all paths between two nodes (with cycle prevention)
WITH RECURSIVE path_finder AS (
  -- Anchor: starting edges from source
  SELECT
    source_node,
    target_node,
    weight,
    ARRAY[source_node, target_node] AS path,
    weight AS total_weight,
    1 AS hops
  FROM edges
  WHERE source_node = 'A'

  UNION ALL

  -- Recursive: extend paths
  SELECT
    pf.source_node,
    e.target_node,
    e.weight,
    pf.path || e.target_node,
    pf.total_weight + e.weight,
    pf.hops + 1
  FROM path_finder pf
  INNER JOIN edges e ON pf.target_node = e.source_node
  WHERE e.target_node != ALL(pf.path)  -- Cycle detection
    AND pf.hops < 20  -- Max depth limit
)
SELECT * FROM path_finder
WHERE target_node = 'Z'
ORDER BY total_weight;

Cycle Detection Algorithm

Mathematical Definition:

Given a path , a cycle exists if:

The ALL(pf.path) check prevents this by verifying the next node hasn't been visited:

⚠️

Performance Warning: Always include a depth limit in recursive CTEs. Without it, cyclic graphs cause infinite recursion. The ALL() array check is O(n) per row — for large graphs, consider using a materialized path or ltree extension instead.


Materialized Path Pattern

Store hierarchical paths directly in the table for fast ancestor/descendant queries:

-- Store and query hierarchical paths
CREATE TABLE categories (
  id SERIAL PRIMARY KEY,
  name VARCHAR(100),
  parent_id INT REFERENCES categories(id),
  path TEXT,  -- Materialized path: '/1/5/12/'
  depth INT
);

-- Rebuild materialized paths recursively
WITH RECURSIVE cat_path AS (
  SELECT
    id, name, parent_id,
    '/' || id::TEXT || '/' AS path,
    0 AS depth
  FROM categories
  WHERE parent_id IS NULL

  UNION ALL

  SELECT
    c.id, c.name, c.parent_id,
    cp.path || c.id::TEXT || '/',
    cp.depth + 1
  FROM categories c
  INNER JOIN cat_path cp ON c.parent_id = cp.id
)
UPDATE categories
SET path = cp.path, depth = cp.depth
FROM cat_path cp
WHERE categories.id = cp.id;

-- Query: Find all descendants of category 5
SELECT * FROM categories
WHERE path LIKE '/1/5/%'
ORDER BY path;

Adjacency List vs Materialized Path

ApproachProsConsBest For
Adjacency ListSimple, normalizedRecursive queries neededWrite-heavy, shallow trees
Materialized PathFast ancestor queriesPath maintenance overheadRead-heavy, deep trees
Nested SetsFast subtree queriesExpensive inserts/deletesRead-heavy, static data
ltree (PostgreSQL)Native path queriesExtension requiredPostgreSQL-specific

Recursive Aggregation

Sum values through a dependency chain — useful for project cost rollups:

-- Sum costs through a task dependency chain
WITH RECURSIVE cost_chain AS (
  -- Anchor: root tasks (no parent)
  SELECT
    project_id,
    task_id,
    task_name,
    direct_cost,
    direct_cost AS cumulative_cost,
    ARRAY[task_id] AS task_chain
  FROM project_tasks
  WHERE parent_task_id IS NULL

  UNION ALL

  -- Recursive: child tasks accumulate parent cost
  SELECT
    pt.project_id,
    pt.task_id,
    pt.task_name,
    pt.direct_cost,
    cc.cumulative_cost + pt.direct_cost,
    cc.task_chain || pt.task_id
  FROM project_tasks pt
  INNER JOIN cost_chain cc ON pt.parent_task_id = cc.task_id
)
SELECT
  project_id, task_id, task_name,
  direct_cost,
  cumulative_cost,
  task_chain
FROM cost_chain
ORDER BY task_chain;

Cost Aggregation Formula

For a task tree with costs :

This is a post-order traversal — each node's cumulative cost includes all its descendants.


Breadth-First Search (BFS)

Use UNION (not UNION ALL) to deduplicate nodes at the same level:

-- BFS traversal with level tracking
WITH RECURSIVE bfs AS (
  -- Anchor: start node
  SELECT
    node_id,
    0 AS level,
    ARRAY[node_id] AS visited,
    node_id::TEXT AS path
  FROM graph_nodes
  WHERE node_id = 'start_node'

  UNION  -- UNION deduplicates (crucial for BFS!)

  SELECT
    gn.node_id,
    bfs.level + 1,
    bfs.visited || gn.node_id,
    bfs.path || ' → ' || gn.node_id
  FROM graph_nodes gn
  INNER JOIN graph_edges ge ON gn.node_id = ge.target_node
  INNER JOIN bfs ON ge.source_node = bfs.node_id
  WHERE gn.node_id != ALL(bfs.visited)
)
SELECT node_id, level, path
FROM bfs
ORDER BY level, node_id;

ℹ️

UNION vs UNION ALL: Use UNION in BFS to automatically deduplicate nodes at the same level. Without deduplication, the same node can be reached via multiple paths, causing exponential growth in result size.


Recursive Date Generation

Generate date series without generate_series (useful in MySQL, SQL Server):

-- Generate a complete date series
WITH RECURSIVE date_series AS (
  SELECT
    '2024-01-01'::DATE AS date,
    1 AS day_number

  UNION ALL

  SELECT
    date + INTERVAL '1' DAY,
    day_number + 1
  FROM date_series
  WHERE date < '2024-12-31'::DATE
)
SELECT
  date,
  day_number,
  EXTRACT(DOW FROM date) AS day_of_week,
  EXTRACT(MONTH FROM date) AS month,
  CASE
    WHEN EXTRACT(DOW FROM date) IN (0, 6) THEN 'Weekend'
    ELSE 'Weekday'
  END AS day_type
FROM date_series;

Tree Flattening with DFS Ordering

Convert a tree to a flat list with depth-first search ordering:

-- Flatten tree with proper DFS ordering
WITH RECURSIVE tree_dfs AS (
  SELECT
    id, name, parent_id,
    0 AS depth,
    LPAD(id::TEXT, 10, '0') AS sort_key
  FROM tree_nodes
  WHERE parent_id IS NULL

  UNION ALL

  SELECT
    tn.id, tn.name, tn.parent_id,
    td.depth + 1,
    LPAD(td.sort_key || '/' || LPAD(tn.id::TEXT, 5, '0'), 20, '0')
  FROM tree_nodes tn
  INNER JOIN tree_dfs td ON tn.parent_id = td.id
)
SELECT
  REPEAT('  ', depth) || name AS display_name,
  depth,
  sort_key
FROM tree_dfs
ORDER BY sort_key;

DFS Ordering Visualization

Tree Structure:

Root├── A1└── A2└── B1

DFS Sorted Output:

display_namedepthsort_key
Root00000000000
A10000000000/00001
A120000000000/00001/00001
A220000000000/00001/00002
B10000000000/00002
B120000000000/00002/00001
C10000000000/00003

Performance Comparison

Recursive CTE vs Application-Level Traversal

MethodQuery Time (10K nodes)MemoryCode Complexity
Recursive CTE45ms12 MBLow (declarative)
Application Loop280ms45 MBHigh (imperative)
Cursor-based320ms8 MBVery High
ltree Extension12ms4 MBMedium (extension)

Recursive CTEs are 6x faster than application loops for hierarchical queries because they execute the traversal logic in the database engine, avoiding network round-trips and application overhead.


Quiz: Test Your Knowledge


Follow-Up Questions

  1. How do you prevent infinite recursion in cyclic graphs?
  2. What's the performance difference between recursive CTEs and iterative approaches?
  3. How would you find the shortest path in an unweighted graph using recursive CTEs?
  4. Explain the difference between UNION and UNION ALL in recursive CTEs.
  5. How do you limit recursion depth in PostgreSQL vs BigQuery?
  6. When should you use a materialized path pattern vs an adjacency list?

Key Takeaways

  • Always include termination conditions — depth limits or cycle detection to prevent infinite recursion
  • UNION deduplicates — essential for BFS to avoid exponential growth
  • Materialized paths — pre-compute hierarchy for fast ancestor/descendant queries
  • Recursive CTEs are 6x faster than application-level traversal for hierarchical data
  • ARRAY tracking — store visited nodes for cycle detection in graph traversal
  • Depth-first vs breadth-first — use UNION ALL for DFS, UNION for BFS
🔒

Premium Content

Recursive CTEs — Hierarchical Data, Graph Traversal & Tree Algorithms

You've previewed the first section. Unlock this full lesson and 900+ advanced tutorials with a Premium plan.

🎯End-to-end Projects
💼Interview Prep
📜Certificates
🤝Community Access

Already a member? Log in

Advertisement