LLM Agent Architectures

1. The Agent Paradigm

An LLM agent is an autonomous system that uses a language model as its core reasoning engine, augmented with tools, memory, and planning capabilities to accomplish complex tasks. Formally, an agent can be defined as a tuple:

where:

• : the language model (policy)
• : available tools
• : memory system
• : planning strategy

The agent operates in an environment through an observe-think-act loop:

where is the current state (observation + history), is the chosen action, and is the environment transition function.

2. ReAct Framework

2.1 Overview

ReAct (Yao et al., 2023) interleaves reasoning (thinking) and acting within a single prompt structure. The agent generates a thought, takes an action, observes the result, and repeats.

2.2 The ReAct Loop

2.3 ReAct Prompt Template

Answer the following questions as best you can. You have access to the following tools:

{tools}

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

2.4 Formal ReAct Algorithm

function ReAct(query q, tools T, LLM M, max_steps N):
    history h ← []
    for t = 1 to N:
        thought_t ← M("Thought based on: " + h)
        action_t, input_t ← M("Action to take: " + h + thought_t)
        if action_t == "Finish":
            return input_t
        observation_t ← Execute(action_t, input_t, T)
        h ← h + (thought_t, action_t, input_t, observation_t)
    return "Max steps reached"

3. Chain-of-Thought (CoT) Reasoning

3.1 Prompting-Based CoT

Chain-of-thought prompting (Wei et al., 2022) elicits step-by-step reasoning by including examples with intermediate reasoning steps:

where each represents a reasoning step rather than a final answer token.

3.2 Self-Consistency

Self-consistency (Wang et al., 2023) samples multiple reasoning paths and takes the majority vote:

where is the answer from the -th sampled reasoning path.

The probability of the correct answer under self-consistency is:

where can be uniform or weighted by reasoning path confidence.

3.3 Tree-of-Thought (ToT)

Tree-of-Thought (Yao et al., 2023) generalizes CoT by maintaining a tree of reasoning states:

where are states (partial solutions) and are transitions (reasoning steps).

At each step, the agent:

1. Generates candidate next states from the current state
2. Evaluates each candidate using the LLM as a value function
3. Selects the most promising state(s) to expand

The value of a state is estimated by:

4. Tool Use

4.1 Tool Selection

Given a set of available tools , the agent selects a tool based on:

In practice, this is done via function calling or structured output parsing.

4.2 Tool Use Patterns

Parallel tool use: Execute multiple independent tools simultaneously:

Sequential tool use: Chain tool outputs as inputs to subsequent tools:

Conditional tool use: Select next tool based on previous observations:

4.3 Tool Representation

Tools are typically described with:

tools = [
    {
        "name": "search",
        "description": "Search the web for information",
        "parameters": {
            "query": {"type": "string", "description": "Search query"}
        }
    },
    {
        "name": "calculator",
        "description": "Evaluate mathematical expressions",
        "parameters": {
            "expression": {"type": "string", "description": "Math expression"}
        }
    },
    {
        "name": "code_executor",
        "description": "Execute Python code",
        "parameters": {
            "code": {"type": "string", "description": "Python code to execute"}
        }
    }
]

5. Agent Memory Systems

5.1 Short-Term Memory (Context Window)

The context window serves as the agent's working memory:

Limited by the model's context length . Strategies to manage short-term memory:

• Sliding window: Keep the most recent interactions
• Summarization: Periodically summarize older interactions
• Priority queue: Keep interactions ranked by relevance

5.2 Long-Term Memory

External memory systems that persist across sessions:

where is the embedding, is the memory content, and is a timestamp or access frequency.

Retrieval from long-term memory:

5.3 Episodic Memory

Stores specific past experiences as trajectories:

When faced with a new task, the agent retrieves similar past episodes and uses them as few-shot examples or to guide planning.

5.4 Working Memory vs. Reference Memory

• Working memory: Active information being processed (context window)
• Reference memory: Archived experiences for retrieval (vector database)

6. Planning Strategies

6.1 ReAct (Reasoning + Acting)

Single-path planning: think → act → observe → think...

6.2 Reflexion

Reflexion (Shinn et al., 2023) adds self-reflection after each attempt:

The reflection is stored in memory and used to improve future attempts.

6.3 Plan-and-Solve

Generate a complete plan before execution:

Then execute each step sequentially, potentially re-planning if a step fails.

6.4 LATS (Language Agent Tree Search)

Combines Monte Carlo Tree Search (MCTS) with LLM agents:

where is the estimated value, is the visit count, and is the exploration constant.

7. Action Selection and Trajectory Optimization

7.1 Action Scoring

The agent scores each possible action:

7.2 Trajectory Optimization

Given a task with reward , optimize the agent's policy over entire trajectories :

In practice, this is approximated through:

• RLHF on trajectories: Train a reward model on agent trajectories
• Rejection sampling: Generate multiple trajectories, select the best
• Fine-tuning on successful trajectories: SFT on demonstrations

7.3 Reward Shaping for Agents

where:

• : progress toward task completion
• : penalize unnecessary steps
• : penalize dangerous or harmful actions

7.4 Credit Assignment in Multi-Step Tasks

For a trajectory with steps, assigning credit to each action is challenging:

where is the advantage estimate.

8. Multi-Agent Systems

8.1 Debate

Multiple agents argue and refine answers:

8.2 Collaboration

Specialized agents handle different aspects:

8.3 Competitive

Agents compete to provide the best solution, evaluated by a judge:

9. Implementation

from typing import List, Dict, Callable
import openai

class Agent:
    def __init__(self, model: str, tools: List[Dict], max_steps: int = 10):
        self.model = model
        self.tools = {t["name"]: t for t in tools}
        self.max_steps = max_steps
        self.history = []

    def run(self, task: str) -> str:
        system_prompt = self._build_system_prompt()
        messages = [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": task}
        ]

        for step in range(self.max_steps):
            response = openai.ChatCompletion.create(
                model=self.model,
                messages=messages,
                tools=self._format_tools(),
                tool_choice="auto"
            )

            msg = response["choices"][0]["message"]

            if msg.get("tool_calls"):
                for tool_call in msg["tool_calls"]:
                    result = self._execute_tool(
                        tool_call["function"]["name"],
                        tool_call["function"]["arguments"]
                    )
                    messages.append({
                        "role": "tool",
                        "tool_call_id": tool_call["id"],
                        "content": str(result)
                    })
            else:
                return msg["content"]

        return "Max steps reached"

    def _execute_tool(self, name: str, args: str) -> str:
        tool = self.tools[name]
        return tool["function"](**eval(args))

    def _build_system_prompt(self) -> str:
        tool_descriptions = "\n".join([
            f"- {t['name']}: {t['description']}"
            for t in self.tools.values()
        ])
        return f"""You are a helpful agent with access to these tools:
{tool_descriptions}

Use tools when needed. Think step by step."""

    def _format_tools(self) -> List[Dict]:
        return [{
            "type": "function",
            "function": {
                "name": t["name"],
                "description": t["description"],
                "parameters": t["parameters"]
            }
        } for t in self.tools.values()]

10. Evaluation Metrics

10.1 Task Success Rate

10.2 Efficiency

10.3 Tool Accuracy

10.4 Safety Rate

11. Open Challenges

1. Robustness: Handling tool failures and unexpected observations
2. Long-horizon planning: Maintaining coherent plans over many steps
3. Generalization: Transferring skills across tasks
4. Evaluation: Standardized benchmarks for agent capabilities
5. Safety: Preventing harmful actions in real-world environments

References

1. Yao et al. (2023). "ReAct: Synergizing Reasoning and Acting in Language Models." ICLR.
2. Wei et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS.
3. Wang et al. (2023). "Self-Consistency Improves Chain of Thought Reasoning in Language Models." ICLR.
4. Yao et al. (2023). "Tree of Thoughts: Deliberate Problem Solving with Large Language Models." arXiv.
5. Shinn et al. (2023). "Reflexion: Language Agents with Verbal Reinforcement Learning." NeurIPS.
6. Zhou et al. (2023). "Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models." ICML.