🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
đŸ’ŧ Servicesâ„šī¸ Aboutâœ‰ī¸ ContactView Pricing Plansfrom $10

Tree-of-Thought Planner for Agents

AI AgentsPlanning and ReasoningđŸŸĸ Free Lesson

Advertisement

Tree-of-Thought Planner for Agents

Planning Architecture — Complete Production Flow

Tree-of-Thought Planning — Production ArchitectureGoalInputPlan A: SequentialScore: 0.75Steps: 4 | Cost: LowPlan B: Parallel (Best)Score: 0.92Steps: 3 | Cost: MediumPlan C: IterativeScore: 0.68Steps: 6 | Cost: HighSELECT BEST PLANEvaluate feasibility, efficiency, completenessEXECUTE: Run selected plan step-by-step with re-planning on failuresKey Insight: Planning is cheap compared to execution — explore multiple plans before committing

What is Planning and Reasoning?

Planning and reasoning enables agents to decompose complex goals into actionable steps, evaluate multiple approaches, and select optimal strategies. Unlike reactive agents that respond to immediate inputs, planning agents look ahead and structure their actions.

Chain-of-Thought (CoT): Step-by-step reasoning that makes the model's thinking process explicit. Each step builds on previous reasoning, enabling complex multi-step problem solving.

Tree-of-Thought (ToT): Explores multiple reasoning branches simultaneously, evaluating each path's promise before committing. This parallel exploration prevents getting stuck in local optima.

Plan-and-Execute: First creates a complete plan, then executes each step. This two-phase approach separates planning (cheap, can be revised) from execution (expensive, irreversible).

Why This Matters

Without planning, agents react to problems rather than solving them strategically. Planning enables agents to:

  • Anticipate resource needs before execution
  • Evaluate multiple approaches before committing
  • Recover from failures by re-planning
  • Optimize for efficiency, not just completion

Real-world analogy: Planning is like using GPS navigation. Without it, you might take wrong turns and waste time. With it, you see the full route, know the estimated arrival time, and can reroute if there's traffic.

Planning Strategies Comparison

StrategyApproachBest ForCostQuality
Chain-of-ThoughtLinear reasoningSimple problemsLowGood
Tree-of-ThoughtBranch explorationComplex problemsMediumBest
Plan-and-ExecutePlan first, then executeMulti-step tasksMediumGood
ReActReasoning + ActingTool-use tasksVariableVariable

Project Overview

We will build a planning agent that:

  • Generates multiple candidate plans for a given goal
  • Evaluates each plan using a scoring function
  • Executes the best plan step-by-step
  • Re-plans when steps fail
  • Maintains a plan tree for analysis

Expected outcome: An agent that can solve complex multi-step tasks by planning ahead.

Difficulty: Advanced (requires understanding of prompt engineering, search algorithms, and error recovery)

Tools & Setup

ToolVersionPurpose
Python3.11+Core language
OpenAI1.0+LLM backbone
pydantic2.0+Data models
networkx3.0+Plan graph structure
rich13.0+Visualization

Step 1: Environment Setup

python -m venv venv
source venv/bin/activate
pip install openai pydantic networkx rich
export OPENAI_API_KEY="sk-your-key"

Step 2: Data Models

# models.py
from pydantic import BaseModel
from typing import List, Optional
from enum import Enum
import logging

logger = logging.getLogger(__name__)


class StepStatus(str, Enum):
    """Status of a plan step."""
    PENDING = "pending"
    IN_PROGRESS = "in_progress"
    COMPLETED = "completed"
    FAILED = "failed"


class PlanStep(BaseModel):
    """A single step in a plan."""
    id: str
    description: str
    tool_needed: Optional[str] = None
    status: StepStatus = StepStatus.PENDING
    result: Optional[str] = None
    dependencies: List[str] = []


class Plan(BaseModel):
    """A complete plan for achieving a goal."""
    id: str
    goal: str
    steps: List[PlanStep]
    score: float = 0.0
    reasoning: str = ""


class PlanEvaluation(BaseModel):
    """Evaluation scores for a plan."""
    plan_id: str
    feasibility_score: float
    efficiency_score: float
    completeness_score: float
    overall_score: float
    feedback: str

Step 3: Chain-of-Thought Planner

# planner/cot_planner.py
from openai import OpenAI
from models import Plan, PlanStep
import json
import logging

logger = logging.getLogger(__name__)

COT_SYSTEM = """You are a planning expert. Break down goals into clear, 
executable steps. For each step, specify:
1. What needs to be done
2. What tool/information is needed
3. What dependencies exist

Always think step-by-step before generating the plan."""


class CoTPlanner:
    """
    Chain-of-Thought planner that generates a single plan.
    
    Uses step-by-step reasoning to create actionable plans.
    Fast but may miss better alternatives.
    """
    
    def __init__(self, model: str = "gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def generate_plan(self, goal: str, available_tools: list[str]) -> Plan:
        """
        Generate a plan using chain-of-thought reasoning.
        
        Args:
            goal: The objective to achieve
            available_tools: List of available tool names
            
        Returns:
            Plan with ordered steps
        """
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": COT_SYSTEM},
                {"role": "user", "content": f"""Goal: {goal}
Available tools: {', '.join(available_tools)}

Think step-by-step, then generate a JSON plan with these fields:
- goal: the original goal
- steps: array of objects with id, description, tool_needed, dependencies
- reasoning: your chain-of-thought reasoning

Return ONLY valid JSON."""},
            ],
            temperature=0.0,
        )

        content = response.choices[0].message.content
        plan_data = self._parse_response(content)
        
        steps = [
            PlanStep(
                id=s["id"],
                description=s["description"],
                tool_needed=s.get("tool_needed"),
                dependencies=s.get("dependencies", []),
            )
            for s in plan_data["steps"]
        ]
        
        plan = Plan(
            id="plan_cot_001",
            goal=goal,
            steps=steps,
            reasoning=plan_data.get("reasoning", ""),
        )
        
        logger.info(f"Generated CoT plan with {len(steps)} steps")
        return plan

    def _parse_response(self, content: str) -> dict:
        """Parse LLM response as JSON."""
        try:
            return json.loads(content)
        except json.JSONDecodeError:
            import re
            json_match = re.search(r'\{[\s\S]*\}', content)
            if json_match:
                return json.loads(json_match.group())
            return {"steps": [], "reasoning": content}

Step 4: Tree-of-Thought Planner

# planner/tot_planner.py
from openai import OpenAI
from models import Plan, PlanStep, PlanEvaluation
from typing import List
import json
import logging

logger = logging.getLogger(__name__)

TOT_SYSTEM = """You are a strategic planner. For each goal, generate 
3 different candidate plans. For each plan:
1. Describe the approach
2. List the steps
3. Evaluate feasibility (0-1), efficiency (0-1), completeness (0-1)

Choose the BEST plan and explain why."""


class ToTPlanner:
    """
    Tree-of-Thought planner that explores multiple alternatives.
    
    Generates N candidate plans, evaluates each, and selects
    the best one. More thorough but costs more tokens.
    """
    
    def __init__(self, model: str = "gpt-4-turbo-preview", num_candidates: int = 3):
        self.client = OpenAI()
        self.model = model
        self.num_candidates = num_candidates

    def generate_plans(self, goal: str, available_tools: list[str]) -> List[Plan]:
        """
        Generate multiple candidate plans.
        
        Args:
            goal: The objective to achieve
            available_tools: List of available tool names
            
        Returns:
            List of candidate plans
        """
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": TOT_SYSTEM},
                {"role": "user", "content": f"""Goal: {goal}
Available tools: {', '.join(available_tools)}

Generate {self.num_candidates} different plans. For each plan provide:
- approach: description of the strategy
- steps: array of step objects (id, description, tool_needed, dependencies)
- scores: object with feasibility, efficiency, completeness (0-1 each)

Then select the best plan and explain your reasoning.

Return JSON: {{"plans": [...], "best_plan_id": "...", "reasoning": "..."}}"""},
            ],
            temperature=0.3,
            max_tokens=2000,
        )

        content = response.choices[0].message.content
        data = self._parse_response(content)
        
        plans = []
        for plan_data in data.get("plans", []):
            steps = [
                PlanStep(
                    id=s["id"],
                    description=s["description"],
                    tool_needed=s.get("tool_needed"),
                    dependencies=s.get("dependencies", []),
                )
                for s in plan_data.get("steps", [])
            ]
            scores = plan_data.get("scores", {})
            plan = Plan(
                id=plan_data.get("id", f"plan_{len(plans)}"),
                goal=goal,
                steps=steps,
                score=(
                    scores.get("feasibility", 0.5) +
                    scores.get("efficiency", 0.5) +
                    scores.get("completeness", 0.5)
                ) / 3,
                reasoning=plan_data.get("approach", ""),
            )
            plans.append(plan)

        logger.info(f"Generated {len(plans)} candidate plans")
        return plans

    def select_best(self, plans: List[Plan]) -> Plan:
        """Select the highest-scoring plan."""
        return max(plans, key=lambda p: p.score)

    def _parse_response(self, content: str) -> dict:
        """Parse LLM response as JSON."""
        try:
            return json.loads(content)
        except json.JSONDecodeError:
            import re
            match = re.search(r'\{[\s\S]*\}', content)
            if match:
                return json.loads(match.group())
            return {"plans": [], "reasoning": content}

Step 5: Plan-and-Execute Agent

# planner/plan_and_execute.py
from __future__ import annotations
import json
from openai import OpenAI
from models import Plan, PlanStep, StepStatus
from planner.cot_planner import CoTPlanner
from planner.tot_planner import ToTPlanner
from typing import Callable, Awaitable, Any
import logging

logger = logging.getLogger(__name__)


class PlanAndExecuteAgent:
    """
    Agent that plans before executing.
    
    Generates plans, executes steps, and re-plans
    on failures for robust completion.
    """
    
    def __init__(self, model: str = "gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model
        self.cot_planner = CoTPlanner(model)
        self.tot_planner = ToTPlanner(model)
        self.tools: dict[str, Callable] = {}

    def register_tool(self, name: str, func: Callable) -> None:
        """Register a tool for step execution."""
        self.tools[name] = func

    async def execute_goal(
        self,
        goal: str,
        strategy: str = "tot",
    ) -> dict:
        """
        Execute a goal using planning.
        
        Args:
            goal: The objective to achieve
            strategy: Planning strategy ("tot" or "cot")
            
        Returns:
            Execution results
        """
        # Generate plan(s)
        if strategy == "tot":
            plans = self.tot_planner.generate_plans(
                goal, list(self.tools.keys())
            )
            plan = self.tot_planner.select_best(plans)
        else:
            plan = self.cot_planner.generate_plan(
                goal, list(self.tools.keys())
            )

        # Execute steps
        execution_results = []
        for step in plan.steps:
            step.status = StepStatus.IN_PROGRESS
            result = await self._execute_step(step)
            execution_results.append(result)
            
            if result["success"]:
                step.status = StepStatus.COMPLETED
                step.result = result["output"]
            else:
                step.status = StepStatus.FAILED
                step.result = result["error"]
                
                # Re-plan on failure
                new_plan = await self._replan(goal, plan, step)
                if new_plan:
                    plan = new_plan
                    execution_results = []

        completed = all(
            s.status == StepStatus.COMPLETED for s in plan.steps
        )
        
        return {
            "goal": goal,
            "plan": plan.model_dump(),
            "results": execution_results,
            "completed": completed,
        }

    async def _execute_step(self, step: PlanStep) -> dict:
        """Execute a single plan step."""
        if not step.tool_needed or step.tool_needed not in self.tools:
            return await self._llm_step(step)

        tool_func = self.tools[step.tool_needed]
        try:
            if callable(tool_func):
                result = tool_func(step.description)
                return {"success": True, "output": str(result)}
            return {"success": False, "error": "Invalid tool"}
        except Exception as e:
            return {"success": False, "error": str(e)}

    async def _llm_step(self, step: PlanStep) -> dict:
        """Execute a step using LLM reasoning."""
        response = await self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": "Execute the given step precisely."},
                {"role": "user", "content": step.description},
            ],
            temperature=0.0,
        )
        return {
            "success": True,
            "output": response.choices[0].message.content,
        }

    async def _replan(
        self, goal: str, current_plan: Plan, failed_step: PlanStep
    ) -> Plan | None:
        """Generate a new plan after a step failure."""
        logger.info(f"Re-planning after failure: {failed_step.description}")
        
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": """A step failed. Generate a new 
                plan that avoids the failed approach. Be concise."""},
                {"role": "user", "content": f"""Goal: {goal}
Failed step: {failed_step.description}
Error: {failed_step.result}

Generate a new plan as JSON with steps array."""},
            ],
            temperature=0.2,
        )
        
        content = response.choices[0].message.content
        try:
            data = json.loads(content)
            steps = [
                PlanStep(
                    id=s["id"],
                    description=s["description"],
                    tool_needed=s.get("tool_needed"),
                )
                for s in data.get("steps", [])
            ]
            return Plan(id="replan_001", goal=goal, steps=steps)
        except (json.JSONDecodeError, KeyError):
            return None

Mathematical Foundation

Plan Evaluation Score:

Where each parameter means:

  • — feasibility score (can this be executed?)
  • — efficiency score (how many steps/resources?)
  • — completeness score (does it cover all requirements?)
  • , , — weights (typically 0.4, 0.3, 0.3)

Intuition: Balances whether a plan can be done, how efficiently, and how completely it addresses the goal.

ToT Exploration Budget:

Intuition: Total token cost equals number of candidates times planning cost plus evaluation cost. Budget constraints limit how many branches can be explored.

Planning ROI:

Intuition: Planning should save more than it costs. If planning costs 0.50, ROI is 3.3x.

Performance Metrics

MetricValueNotes
Plan Generation Time2-5sGPT-4 with CoT
ToT Candidates3-5Balanced quality/cost
Plan Success Rate85%+With re-planning
Re-plan Frequency15%Steps requiring re-planning
Avg Steps per Goal4-7Depends on complexity

Real-World Examples

Example 1: Data Pipeline

Planning a data processing pipeline:

agent = PlanAndExecuteAgent()
result = await agent.execute_goal(
    "Extract data from CSV, transform to JSON, load into database",
    strategy="tot"
)
# Plans: sequential, parallel, or hybrid approaches
# Selects best based on data size and dependencies

Example 2: Research Task

Planning a research workflow:

result = await agent.execute_goal(
    "Research competitor pricing, analyze trends, create report",
    strategy="cot"
)
# Steps: search → analyze → visualize → write report

Common Pitfalls & Solutions

PitfallSolution
Over-planningSet maximum plan depth
Plan rigidityImplement re-planning on failures
Token wasteCache plan evaluations
Circular dependenciesDetect and break dependency cycles
Goal driftRe-validate against original goal periodically
High latencyUse parallel plan evaluation
Inconsistent scoringStandardize evaluation criteria
Complexity explosionLimit ToT candidates to 3-5

Security Considerations

Critical security measures for planning agents:

  1. Goal Validation: Ensure the goal doesn't request harmful actions
  2. Step Validation: Check each step before execution
  3. Tool Scoping: Plans should only use available tools
  4. Cost Limits: Set maximum planning budget
  5. Audit Logging: Track all plans and executions
# Example: Safe planning with validation
class SafePlanAgent(PlanAndExecuteAgent):
    def validate_plan(self, plan: Plan) -> bool:
        for step in plan.steps:
            if step.tool_needed and step.tool_needed not in self.tools:
                return False
        return True

Summary with Key Takeaways

  • CoT provides transparent reasoning; ToT explores alternatives for better plans
  • Plan-and-Execute separates cheap planning from expensive execution
  • Re-planning on failures is essential for robustness
  • Parallel plan evaluation reduces latency significantly
  • Always validate plans against the original goal to prevent drift
  • Planning cost should be <20% of total execution cost
  • Use 3-5 ToT candidates for optimal quality-cost tradeoff

Interview Questions

1. What is the difference between CoT and ToT planning?

Answer: CoT (Chain-of-Thought) generates a single plan by reasoning step-by-step. It's fast but may miss better alternatives. ToT (Tree-of-Thought) generates multiple candidate plans (typically 3-5), evaluates each, and selects the best. ToT is more thorough but costs more tokens. CoT is suitable for straightforward problems; ToT excels at complex problems where the optimal approach isn't obvious. ToT prevents getting stuck in local optima by exploring multiple branches.

2. How does re-planning work when a step fails?

Answer: Re-planning triggers when a step fails: 1) Capture the error and failed step details, 2) Send the original goal + failure context to the LLM, 3) Generate a new plan that avoids the failed approach, 4) Continue execution with the new plan. Key considerations: preserve completed steps, avoid repeating failed approaches, and maintain progress toward the original goal. The re-planner should be given context about what was already accomplished.

3. What is the plan evaluation criteria?

Answer: Plans are evaluated on three dimensions: 1) Feasibility (0-1) — Can this plan be executed with available tools and resources? 2) Efficiency (0-1) — How many steps/resources does it require? Fewer is better. 3) Completeness (0-1) — Does it address all aspects of the goal? The overall score is a weighted combination. ToT generates multiple plans and selects the highest-scoring one. Evaluation can be done by the LLM or with automated metrics.

4. How do you handle circular dependencies in plans?

Answer: Circular dependencies occur when Step A depends on Step B, which depends on Step A. Detection: build a dependency graph and check for cycles using topological sort. Prevention: validate plans before execution. Resolution: 1) Break the cycle by removing one dependency, 2) Merge circular steps into a single step, 3) Use parallel execution for mutually dependent steps. NetworkX can detect cycles in dependency graphs.

5. What is the cost tradeoff between planning and execution?

Answer: Planning is cheap (1-2 LLM calls, ~0.10-1.00). The key insight: spend more on planning to avoid expensive execution failures. ToT with 3 candidates costs ~0.50+ by selecting a better plan. However, over-planning wastes tokens. Optimal strategy: plan with 3-5 candidates for complex tasks, use CoT for simple tasks. Budget constraint: planning cost should be <20% of total cost.

6. How do you evaluate plan quality?

Answer: Evaluation methods: 1) Automated scoring — LLM evaluates feasibility/efficiency/completeness, 2) Simulation — Dry-run the plan to check for issues, 3) Historical analysis — Compare to past successful plans, 4) Human review — Expert evaluation for critical tasks. Metrics: plan completion rate, average steps to complete, re-plan frequency, total cost. Track plan quality over time and adjust weights based on outcomes.

7. When should you use plan-and-execute vs ReAct?

Answer: Use plan-and-execute when: 1) Task requires multiple steps with dependencies, 2) Execution is expensive (API calls, database writes), 3) You need to preview the approach before committing. Use ReAct when: 1) Task requires exploration and adaptation, 2) The path forward depends on intermediate results, 3) You need real-time information from tools. Plan-and-execute is more efficient for known patterns; ReAct is more flexible for novel problems.

8. How do you prevent goal drift during execution?

Answer: Goal drift occurs when the agent deviates from the original objective. Prevention: 1) Re-validate against original goal after each step, 2) Track progress toward goal completion, 3) Set termination conditions — stop if drift detected, 4) Include goal reminders in prompts during execution, 5) Use checkpoints — compare intermediate results to goal. Implement a drift detector that scores each step's relevance to the original goal. If score drops below threshold, trigger re-planning.


KnowledgeCheck

  1. What is the key advantage of Tree-of-Thought over Chain-of-Thought?

    • a) ToT is faster
    • b) ToT explores multiple plan alternatives
    • c) ToT uses fewer tokens
    • d) ToT doesn't need an LLM
  2. What triggers re-planning in a plan-and-execute agent?

    • a) After every step
    • b) When a step fails
    • c) When tokens run out
    • d) When the user asks
  3. What are the three plan evaluation criteria?

    • a) Speed, cost, accuracy
    • b) Feasibility, efficiency, completeness
    • c) Memory, latency, throughput
    • d) Size, weight, color
  4. Why is planning cheaper than execution?

    • a) Planning uses smaller models
    • b) Planning requires fewer LLM calls
    • c) Planning doesn't use tools
    • d) Both b and c
  5. How do you prevent circular dependencies?

    • a) Use more steps
    • b) Detect cycles with topological sort
    • c) Ignore dependencies
    • d) Use ReAct instead
  6. What is goal drift?

    • a) The plan becomes more expensive
    • b) The agent deviates from the original objective
    • c) The LLM hallucinates
    • d) Tools fail to execute

Answers: 1-b, 2-b, 3-b, 4-d, 5-b, 6-b

Need Expert AI Agents Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement