Tree-of-Thought Planner for Agents
Planning Architecture â Complete Production Flow
What is Planning and Reasoning?
Planning and reasoning enables agents to decompose complex goals into actionable steps, evaluate multiple approaches, and select optimal strategies. Unlike reactive agents that respond to immediate inputs, planning agents look ahead and structure their actions.
Chain-of-Thought (CoT): Step-by-step reasoning that makes the model's thinking process explicit. Each step builds on previous reasoning, enabling complex multi-step problem solving.
Tree-of-Thought (ToT): Explores multiple reasoning branches simultaneously, evaluating each path's promise before committing. This parallel exploration prevents getting stuck in local optima.
Plan-and-Execute: First creates a complete plan, then executes each step. This two-phase approach separates planning (cheap, can be revised) from execution (expensive, irreversible).
Why This Matters
Without planning, agents react to problems rather than solving them strategically. Planning enables agents to:
- Anticipate resource needs before execution
- Evaluate multiple approaches before committing
- Recover from failures by re-planning
- Optimize for efficiency, not just completion
Real-world analogy: Planning is like using GPS navigation. Without it, you might take wrong turns and waste time. With it, you see the full route, know the estimated arrival time, and can reroute if there's traffic.
Planning Strategies Comparison
| Strategy | Approach | Best For | Cost | Quality |
|---|---|---|---|---|
| Chain-of-Thought | Linear reasoning | Simple problems | Low | Good |
| Tree-of-Thought | Branch exploration | Complex problems | Medium | Best |
| Plan-and-Execute | Plan first, then execute | Multi-step tasks | Medium | Good |
| ReAct | Reasoning + Acting | Tool-use tasks | Variable | Variable |
Project Overview
We will build a planning agent that:
- Generates multiple candidate plans for a given goal
- Evaluates each plan using a scoring function
- Executes the best plan step-by-step
- Re-plans when steps fail
- Maintains a plan tree for analysis
Expected outcome: An agent that can solve complex multi-step tasks by planning ahead.
Difficulty: Advanced (requires understanding of prompt engineering, search algorithms, and error recovery)
Tools & Setup
| Tool | Version | Purpose |
|---|---|---|
| Python | 3.11+ | Core language |
| OpenAI | 1.0+ | LLM backbone |
| pydantic | 2.0+ | Data models |
| networkx | 3.0+ | Plan graph structure |
| rich | 13.0+ | Visualization |
Step 1: Environment Setup
python -m venv venv
source venv/bin/activate
pip install openai pydantic networkx rich
export OPENAI_API_KEY="sk-your-key"
Step 2: Data Models
# models.py
from pydantic import BaseModel
from typing import List, Optional
from enum import Enum
import logging
logger = logging.getLogger(__name__)
class StepStatus(str, Enum):
"""Status of a plan step."""
PENDING = "pending"
IN_PROGRESS = "in_progress"
COMPLETED = "completed"
FAILED = "failed"
class PlanStep(BaseModel):
"""A single step in a plan."""
id: str
description: str
tool_needed: Optional[str] = None
status: StepStatus = StepStatus.PENDING
result: Optional[str] = None
dependencies: List[str] = []
class Plan(BaseModel):
"""A complete plan for achieving a goal."""
id: str
goal: str
steps: List[PlanStep]
score: float = 0.0
reasoning: str = ""
class PlanEvaluation(BaseModel):
"""Evaluation scores for a plan."""
plan_id: str
feasibility_score: float
efficiency_score: float
completeness_score: float
overall_score: float
feedback: str
Step 3: Chain-of-Thought Planner
# planner/cot_planner.py
from openai import OpenAI
from models import Plan, PlanStep
import json
import logging
logger = logging.getLogger(__name__)
COT_SYSTEM = """You are a planning expert. Break down goals into clear,
executable steps. For each step, specify:
1. What needs to be done
2. What tool/information is needed
3. What dependencies exist
Always think step-by-step before generating the plan."""
class CoTPlanner:
"""
Chain-of-Thought planner that generates a single plan.
Uses step-by-step reasoning to create actionable plans.
Fast but may miss better alternatives.
"""
def __init__(self, model: str = "gpt-4-turbo-preview"):
self.client = OpenAI()
self.model = model
def generate_plan(self, goal: str, available_tools: list[str]) -> Plan:
"""
Generate a plan using chain-of-thought reasoning.
Args:
goal: The objective to achieve
available_tools: List of available tool names
Returns:
Plan with ordered steps
"""
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": COT_SYSTEM},
{"role": "user", "content": f"""Goal: {goal}
Available tools: {', '.join(available_tools)}
Think step-by-step, then generate a JSON plan with these fields:
- goal: the original goal
- steps: array of objects with id, description, tool_needed, dependencies
- reasoning: your chain-of-thought reasoning
Return ONLY valid JSON."""},
],
temperature=0.0,
)
content = response.choices[0].message.content
plan_data = self._parse_response(content)
steps = [
PlanStep(
id=s["id"],
description=s["description"],
tool_needed=s.get("tool_needed"),
dependencies=s.get("dependencies", []),
)
for s in plan_data["steps"]
]
plan = Plan(
id="plan_cot_001",
goal=goal,
steps=steps,
reasoning=plan_data.get("reasoning", ""),
)
logger.info(f"Generated CoT plan with {len(steps)} steps")
return plan
def _parse_response(self, content: str) -> dict:
"""Parse LLM response as JSON."""
try:
return json.loads(content)
except json.JSONDecodeError:
import re
json_match = re.search(r'\{[\s\S]*\}', content)
if json_match:
return json.loads(json_match.group())
return {"steps": [], "reasoning": content}
Step 4: Tree-of-Thought Planner
# planner/tot_planner.py
from openai import OpenAI
from models import Plan, PlanStep, PlanEvaluation
from typing import List
import json
import logging
logger = logging.getLogger(__name__)
TOT_SYSTEM = """You are a strategic planner. For each goal, generate
3 different candidate plans. For each plan:
1. Describe the approach
2. List the steps
3. Evaluate feasibility (0-1), efficiency (0-1), completeness (0-1)
Choose the BEST plan and explain why."""
class ToTPlanner:
"""
Tree-of-Thought planner that explores multiple alternatives.
Generates N candidate plans, evaluates each, and selects
the best one. More thorough but costs more tokens.
"""
def __init__(self, model: str = "gpt-4-turbo-preview", num_candidates: int = 3):
self.client = OpenAI()
self.model = model
self.num_candidates = num_candidates
def generate_plans(self, goal: str, available_tools: list[str]) -> List[Plan]:
"""
Generate multiple candidate plans.
Args:
goal: The objective to achieve
available_tools: List of available tool names
Returns:
List of candidate plans
"""
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": TOT_SYSTEM},
{"role": "user", "content": f"""Goal: {goal}
Available tools: {', '.join(available_tools)}
Generate {self.num_candidates} different plans. For each plan provide:
- approach: description of the strategy
- steps: array of step objects (id, description, tool_needed, dependencies)
- scores: object with feasibility, efficiency, completeness (0-1 each)
Then select the best plan and explain your reasoning.
Return JSON: {{"plans": [...], "best_plan_id": "...", "reasoning": "..."}}"""},
],
temperature=0.3,
max_tokens=2000,
)
content = response.choices[0].message.content
data = self._parse_response(content)
plans = []
for plan_data in data.get("plans", []):
steps = [
PlanStep(
id=s["id"],
description=s["description"],
tool_needed=s.get("tool_needed"),
dependencies=s.get("dependencies", []),
)
for s in plan_data.get("steps", [])
]
scores = plan_data.get("scores", {})
plan = Plan(
id=plan_data.get("id", f"plan_{len(plans)}"),
goal=goal,
steps=steps,
score=(
scores.get("feasibility", 0.5) +
scores.get("efficiency", 0.5) +
scores.get("completeness", 0.5)
) / 3,
reasoning=plan_data.get("approach", ""),
)
plans.append(plan)
logger.info(f"Generated {len(plans)} candidate plans")
return plans
def select_best(self, plans: List[Plan]) -> Plan:
"""Select the highest-scoring plan."""
return max(plans, key=lambda p: p.score)
def _parse_response(self, content: str) -> dict:
"""Parse LLM response as JSON."""
try:
return json.loads(content)
except json.JSONDecodeError:
import re
match = re.search(r'\{[\s\S]*\}', content)
if match:
return json.loads(match.group())
return {"plans": [], "reasoning": content}
Step 5: Plan-and-Execute Agent
# planner/plan_and_execute.py
from __future__ import annotations
import json
from openai import OpenAI
from models import Plan, PlanStep, StepStatus
from planner.cot_planner import CoTPlanner
from planner.tot_planner import ToTPlanner
from typing import Callable, Awaitable, Any
import logging
logger = logging.getLogger(__name__)
class PlanAndExecuteAgent:
"""
Agent that plans before executing.
Generates plans, executes steps, and re-plans
on failures for robust completion.
"""
def __init__(self, model: str = "gpt-4-turbo-preview"):
self.client = OpenAI()
self.model = model
self.cot_planner = CoTPlanner(model)
self.tot_planner = ToTPlanner(model)
self.tools: dict[str, Callable] = {}
def register_tool(self, name: str, func: Callable) -> None:
"""Register a tool for step execution."""
self.tools[name] = func
async def execute_goal(
self,
goal: str,
strategy: str = "tot",
) -> dict:
"""
Execute a goal using planning.
Args:
goal: The objective to achieve
strategy: Planning strategy ("tot" or "cot")
Returns:
Execution results
"""
# Generate plan(s)
if strategy == "tot":
plans = self.tot_planner.generate_plans(
goal, list(self.tools.keys())
)
plan = self.tot_planner.select_best(plans)
else:
plan = self.cot_planner.generate_plan(
goal, list(self.tools.keys())
)
# Execute steps
execution_results = []
for step in plan.steps:
step.status = StepStatus.IN_PROGRESS
result = await self._execute_step(step)
execution_results.append(result)
if result["success"]:
step.status = StepStatus.COMPLETED
step.result = result["output"]
else:
step.status = StepStatus.FAILED
step.result = result["error"]
# Re-plan on failure
new_plan = await self._replan(goal, plan, step)
if new_plan:
plan = new_plan
execution_results = []
completed = all(
s.status == StepStatus.COMPLETED for s in plan.steps
)
return {
"goal": goal,
"plan": plan.model_dump(),
"results": execution_results,
"completed": completed,
}
async def _execute_step(self, step: PlanStep) -> dict:
"""Execute a single plan step."""
if not step.tool_needed or step.tool_needed not in self.tools:
return await self._llm_step(step)
tool_func = self.tools[step.tool_needed]
try:
if callable(tool_func):
result = tool_func(step.description)
return {"success": True, "output": str(result)}
return {"success": False, "error": "Invalid tool"}
except Exception as e:
return {"success": False, "error": str(e)}
async def _llm_step(self, step: PlanStep) -> dict:
"""Execute a step using LLM reasoning."""
response = await self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": "Execute the given step precisely."},
{"role": "user", "content": step.description},
],
temperature=0.0,
)
return {
"success": True,
"output": response.choices[0].message.content,
}
async def _replan(
self, goal: str, current_plan: Plan, failed_step: PlanStep
) -> Plan | None:
"""Generate a new plan after a step failure."""
logger.info(f"Re-planning after failure: {failed_step.description}")
response = self.client.chat.completions.create(
model=self.model,
messages=[
{"role": "system", "content": """A step failed. Generate a new
plan that avoids the failed approach. Be concise."""},
{"role": "user", "content": f"""Goal: {goal}
Failed step: {failed_step.description}
Error: {failed_step.result}
Generate a new plan as JSON with steps array."""},
],
temperature=0.2,
)
content = response.choices[0].message.content
try:
data = json.loads(content)
steps = [
PlanStep(
id=s["id"],
description=s["description"],
tool_needed=s.get("tool_needed"),
)
for s in data.get("steps", [])
]
return Plan(id="replan_001", goal=goal, steps=steps)
except (json.JSONDecodeError, KeyError):
return None
Mathematical Foundation
Plan Evaluation Score:
Where each parameter means:
- â feasibility score (can this be executed?)
- â efficiency score (how many steps/resources?)
- â completeness score (does it cover all requirements?)
- , , â weights (typically 0.4, 0.3, 0.3)
Intuition: Balances whether a plan can be done, how efficiently, and how completely it addresses the goal.
ToT Exploration Budget:
Intuition: Total token cost equals number of candidates times planning cost plus evaluation cost. Budget constraints limit how many branches can be explored.
Planning ROI:
Intuition: Planning should save more than it costs. If planning costs 0.50, ROI is 3.3x.
Performance Metrics
| Metric | Value | Notes |
|---|---|---|
| Plan Generation Time | 2-5s | GPT-4 with CoT |
| ToT Candidates | 3-5 | Balanced quality/cost |
| Plan Success Rate | 85%+ | With re-planning |
| Re-plan Frequency | 15% | Steps requiring re-planning |
| Avg Steps per Goal | 4-7 | Depends on complexity |
Real-World Examples
Example 1: Data Pipeline
Planning a data processing pipeline:
agent = PlanAndExecuteAgent()
result = await agent.execute_goal(
"Extract data from CSV, transform to JSON, load into database",
strategy="tot"
)
# Plans: sequential, parallel, or hybrid approaches
# Selects best based on data size and dependencies
Example 2: Research Task
Planning a research workflow:
result = await agent.execute_goal(
"Research competitor pricing, analyze trends, create report",
strategy="cot"
)
# Steps: search â analyze â visualize â write report
Common Pitfalls & Solutions
| Pitfall | Solution |
|---|---|
| Over-planning | Set maximum plan depth |
| Plan rigidity | Implement re-planning on failures |
| Token waste | Cache plan evaluations |
| Circular dependencies | Detect and break dependency cycles |
| Goal drift | Re-validate against original goal periodically |
| High latency | Use parallel plan evaluation |
| Inconsistent scoring | Standardize evaluation criteria |
| Complexity explosion | Limit ToT candidates to 3-5 |
Security Considerations
Critical security measures for planning agents:
- Goal Validation: Ensure the goal doesn't request harmful actions
- Step Validation: Check each step before execution
- Tool Scoping: Plans should only use available tools
- Cost Limits: Set maximum planning budget
- Audit Logging: Track all plans and executions
# Example: Safe planning with validation
class SafePlanAgent(PlanAndExecuteAgent):
def validate_plan(self, plan: Plan) -> bool:
for step in plan.steps:
if step.tool_needed and step.tool_needed not in self.tools:
return False
return True
Summary with Key Takeaways
- CoT provides transparent reasoning; ToT explores alternatives for better plans
- Plan-and-Execute separates cheap planning from expensive execution
- Re-planning on failures is essential for robustness
- Parallel plan evaluation reduces latency significantly
- Always validate plans against the original goal to prevent drift
- Planning cost should be <20% of total execution cost
- Use 3-5 ToT candidates for optimal quality-cost tradeoff
Interview Questions
1. What is the difference between CoT and ToT planning?
Answer: CoT (Chain-of-Thought) generates a single plan by reasoning step-by-step. It's fast but may miss better alternatives. ToT (Tree-of-Thought) generates multiple candidate plans (typically 3-5), evaluates each, and selects the best. ToT is more thorough but costs more tokens. CoT is suitable for straightforward problems; ToT excels at complex problems where the optimal approach isn't obvious. ToT prevents getting stuck in local optima by exploring multiple branches.
2. How does re-planning work when a step fails?
Answer: Re-planning triggers when a step fails: 1) Capture the error and failed step details, 2) Send the original goal + failure context to the LLM, 3) Generate a new plan that avoids the failed approach, 4) Continue execution with the new plan. Key considerations: preserve completed steps, avoid repeating failed approaches, and maintain progress toward the original goal. The re-planner should be given context about what was already accomplished.
3. What is the plan evaluation criteria?
Answer: Plans are evaluated on three dimensions: 1) Feasibility (0-1) â Can this plan be executed with available tools and resources? 2) Efficiency (0-1) â How many steps/resources does it require? Fewer is better. 3) Completeness (0-1) â Does it address all aspects of the goal? The overall score is a weighted combination. ToT generates multiple plans and selects the highest-scoring one. Evaluation can be done by the LLM or with automated metrics.
4. How do you handle circular dependencies in plans?
Answer: Circular dependencies occur when Step A depends on Step B, which depends on Step A. Detection: build a dependency graph and check for cycles using topological sort. Prevention: validate plans before execution. Resolution: 1) Break the cycle by removing one dependency, 2) Merge circular steps into a single step, 3) Use parallel execution for mutually dependent steps. NetworkX can detect cycles in dependency graphs.
5. What is the cost tradeoff between planning and execution?
Answer: Planning is cheap (1-2 LLM calls, ~0.10-1.00). The key insight: spend more on planning to avoid expensive execution failures. ToT with 3 candidates costs ~0.50+ by selecting a better plan. However, over-planning wastes tokens. Optimal strategy: plan with 3-5 candidates for complex tasks, use CoT for simple tasks. Budget constraint: planning cost should be <20% of total cost.
6. How do you evaluate plan quality?
Answer: Evaluation methods: 1) Automated scoring â LLM evaluates feasibility/efficiency/completeness, 2) Simulation â Dry-run the plan to check for issues, 3) Historical analysis â Compare to past successful plans, 4) Human review â Expert evaluation for critical tasks. Metrics: plan completion rate, average steps to complete, re-plan frequency, total cost. Track plan quality over time and adjust weights based on outcomes.
7. When should you use plan-and-execute vs ReAct?
Answer: Use plan-and-execute when: 1) Task requires multiple steps with dependencies, 2) Execution is expensive (API calls, database writes), 3) You need to preview the approach before committing. Use ReAct when: 1) Task requires exploration and adaptation, 2) The path forward depends on intermediate results, 3) You need real-time information from tools. Plan-and-execute is more efficient for known patterns; ReAct is more flexible for novel problems.
8. How do you prevent goal drift during execution?
Answer: Goal drift occurs when the agent deviates from the original objective. Prevention: 1) Re-validate against original goal after each step, 2) Track progress toward goal completion, 3) Set termination conditions â stop if drift detected, 4) Include goal reminders in prompts during execution, 5) Use checkpoints â compare intermediate results to goal. Implement a drift detector that scores each step's relevance to the original goal. If score drops below threshold, trigger re-planning.
KnowledgeCheck
-
What is the key advantage of Tree-of-Thought over Chain-of-Thought?
- a) ToT is faster
- b) ToT explores multiple plan alternatives
- c) ToT uses fewer tokens
- d) ToT doesn't need an LLM
-
What triggers re-planning in a plan-and-execute agent?
- a) After every step
- b) When a step fails
- c) When tokens run out
- d) When the user asks
-
What are the three plan evaluation criteria?
- a) Speed, cost, accuracy
- b) Feasibility, efficiency, completeness
- c) Memory, latency, throughput
- d) Size, weight, color
-
Why is planning cheaper than execution?
- a) Planning uses smaller models
- b) Planning requires fewer LLM calls
- c) Planning doesn't use tools
- d) Both b and c
-
How do you prevent circular dependencies?
- a) Use more steps
- b) Detect cycles with topological sort
- c) Ignore dependencies
- d) Use ReAct instead
-
What is goal drift?
- a) The plan becomes more expensive
- b) The agent deviates from the original objective
- c) The LLM hallucinates
- d) Tools fail to execute
Answers: 1-b, 2-b, 3-b, 4-d, 5-b, 6-b