πŸŽ‰ 75% of content is free forever β€” Unlock Premium from $10/mo β†’
CW
Search courses…
πŸ’Ό Servicesℹ️ Aboutβœ‰οΈ ContactView Pricing Plansfrom $10

Building a Complete Agent Framework from Scratch

AI AgentsBuilding Agent from Scratch🟒 Free Lesson

Advertisement

Building a Complete Agent Framework from Scratch

Agent Framework from ScratchLLM ClientTool SystemMemoryPlannerSafetyAgent Core EngineExecution LoopZero-Dependency Framework

What is Building from Scratch?

Building an agent framework from scratch means implementing all components without external agent libraries (LangChain, CrewAI, etc.). This provides complete control, deeper understanding, and zero dependency bloat.

The core components are: LLM client (API communication), tool system (registration and execution), memory (context management), planner (task decomposition), safety (input/output validation), and the execution loop (orchestrating everything).

Understanding these fundamentals enables you to build custom agents optimized for specific requirements that off-the-shelf frameworks can't meet.

Project Overview

We will build a complete agent framework from scratch with:

  • HTTP client for OpenAI API (no SDK dependency)
  • Tool registration and execution system
  • Conversation memory with context management
  • ReAct-style planning and reasoning loop
  • Input/output safety layer
  • Complete execution engine

Expected outcome: A production-ready agent framework with zero external agent dependencies.

Difficulty: Advanced (requires understanding of all agent components and systems programming)

Architecture

Framework ArchitectureLLM ClientRaw HTTP, no SDKTool RegistryDynamic registrationMemory StoreContext managementReAct PlannerSafety LayerExecutorAgent Core Engine

Tools & Setup

ToolVersionPurpose
Python3.11+Core language
httpx0.27+HTTP client (only dependency)
jsonstdlibData serialization
hashlibstdlibCaching

Step 1: Project Structure

Architecture Diagram
agent-framework/
β”œβ”€β”€ core/
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ llm_client.py
β”‚   β”œβ”€β”€ tool_registry.py
β”‚   β”œβ”€β”€ memory.py
β”‚   β”œβ”€β”€ planner.py
β”‚   β”œβ”€β”€ safety.py
β”‚   └── executor.py
β”œβ”€β”€ tools/
β”‚   β”œβ”€β”€ __init__.py
β”‚   └── builtin.py
β”œβ”€β”€ agent.py
└── main.py

Step 2: LLM Client (Zero SDK Dependency)

# core/llm_client.py
import httpx
import json
from typing import Dict, List, Optional

class LLMClient:
    def __init__(self, api_key: str, base_url: str = "https://api.openai.com/v1"):
        self.api_key = api_key
        self.base_url = base_url
        self.client = httpx.Client(timeout=60.0)

    def chat(
        self,
        messages: List[Dict],
        model: str = "gpt-4-turbo-preview",
        tools: Optional[List[Dict]] = None,
        temperature: float = 0.0,
        max_tokens: int = 2048,
    ) -> Dict:
        payload = {
            "model": model,
            "messages": messages,
            "temperature": temperature,
            "max_tokens": max_tokens,
        }
        if tools:
            payload["tools"] = tools
            payload["tool_choice"] = "auto"
        response = self.client.post(
            f"{self.base_url}/chat/completions",
            headers={
                "Authorization": f"Bearer {self.api_key}",
                "Content-Type": "application/json",
            },
            json=payload,
        )
        data = response.json()
        if "error" in data:
            raise Exception(f"LLM API error: {data['error']}")
        choice = data["choices"][0]
        return {
            "content": choice["message"].get("content", ""),
            "tool_calls": choice["message"].get("tool_calls", []),
            "finish_reason": choice.get("finish_reason", ""),
            "usage": data.get("usage", {}),
        }

    def embeddings(
        self, texts: List[str], model: str = "text-embedding-3-small"
    ) -> List[List[float]]:
        response = self.client.post(
            f"{self.base_url}/embeddings",
            headers={
                "Authorization": f"Bearer {self.api_key}",
                "Content-Type": "application/json",
            },
            json={"model": model, "input": texts},
        )
        data = response.json()
        return [item["embedding"] for item in data["data"]]

Step 3: Tool Registry

# core/tool_registry.py
import inspect
from typing import Any, Callable, Dict, List
import json

class Tool:
    def __init__(self, name: str, description: str, func: Callable, parameters: Dict):
        self.name = name
        self.description = description
        self.func = func
        self.parameters = parameters
        self.is_async = inspect.iscoroutinefunction(func)

class ToolRegistry:
    def __init__(self):
        self.tools: Dict[str, Tool] = {}

    def register(self, name: str, description: str, parameters: Dict):
        def decorator(func: Callable) -> Callable:
            self.tools[name] = Tool(name, description, func, parameters)
            return func
        return decorator

    def register_function(self, func: Callable, name: str = None, description: str = None, parameters: Dict = None):
        tool_name = name or func.__name__
        tool_desc = description or func.__doc__ or f"Execute {tool_name}"
        tool_params = parameters or self._generate_schema(func)
        self.tools[tool_name] = Tool(tool_name, tool_desc, func, tool_params)

    def _generate_schema(self, func: Callable) -> Dict:
        sig = inspect.signature(func)
        props = {}
        required = []
        for pname, param in sig.parameters.items():
            ptype = "string"
            if param.annotation != inspect.Parameter.empty:
                type_map = {str: "string", int: "integer", float: "number", bool: "boolean"}
                ptype = type_map.get(param.annotation, "string")
            props[pname] = {"type": ptype, "description": f"The {pname} parameter"}
            if param.default is inspect.Parameter.empty:
                required.append(pname)
        return {"type": "object", "properties": props, "required": required}

    def get_openai_tools(self) -> List[Dict]:
        return [
            {
                "type": "function",
                "function": {
                    "name": tool.name,
                    "description": tool.description,
                    "parameters": tool.parameters,
                },
            }
            for tool in self.tools.values()
        ]

    def execute(self, name: str, arguments: Dict) -> str:
        if name not in self.tools:
            return f"Error: Unknown tool '{name}'"
        tool = self.tools[name]
        try:
            result = tool.func(**arguments)
            return str(result)
        except Exception as e:
            return f"Error: {str(e)}"

Step 4: Memory System

# core/memory.py
import tiktoken
from typing import Dict, List, Optional

class Memory:
    def __init__(self, max_tokens: int = 4000, system_prompt: str = ""):
        self.max_tokens = max_tokens
        self.system_prompt = system_prompt
        self.messages: List[Dict] = []
        self.enc = tiktoken.get_encoding("cl100k_base")

    def add_user(self, content: str) -> None:
        self.messages.append({"role": "user", "content": content})
        self._trim()

    def add_assistant(self, content: str) -> None:
        self.messages.append({"role": "assistant", "content": content})

    def add_tool_result(self, tool_call_id: str, content: str) -> None:
        self.messages.append({"role": "tool", "tool_call_id": tool_call_id, "content": content})

    def get_messages(self) -> List[Dict]:
        messages = []
        if self.system_prompt:
            messages.append({"role": "system", "content": self.system_prompt})
        messages.extend(self.messages)
        return messages

    def _count_tokens(self, text: str) -> int:
        return len(self.enc.encode(text))

    def _trim(self) -> None:
        total = sum(self._count_tokens(m.get("content", "")) for m in self.messages)
        while total > self.max_tokens and self.messages:
            removed = self.messages.pop(0)
            total -= self._count_tokens(removed.get("content", ""))

    def clear(self) -> None:
        self.messages.clear()

    def get_recent(self, n: int = 5) -> List[Dict]:
        return self.messages[-n:]

Step 5: Safety Layer

# core/safety.py
import re
from typing import Dict, Tuple

class SafetyLayer:
    FORBIDDEN_PATTERNS = [
        r"ignore previous instructions",
        r"you are now.*",
        r"disregard.*instructions",
    ]
    PII_PATTERNS = {
        "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
        "phone": r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
        "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
    }

    def check_input(self, text: str) -> Tuple[bool, str]:
        for pattern in self.FORBIDDEN_PATTERNS:
            if re.search(pattern, text, re.IGNORECASE):
                return False, f"Blocked: potential jailbreak"
        return True, ""

    def check_output(self, text: str) -> Tuple[bool, str]:
        for pii_type, pattern in self.PII_PATTERNS.items():
            if re.search(pattern, text):
                return False, f"Output contains {pii_type}"
        return True, ""

    def redact_pii(self, text: str) -> str:
        for pii_type, pattern in self.PII_PATTERNS.items():
            text = re.sub(pattern, f"[REDACTED {pii_type.upper()}]", text)
        return text

Step 6: Complete Agent Framework

# core/executor.py
import json
import time
from typing import Dict, List

class ReActExecutor:
    def __init__(self, llm, tools, memory, safety, max_iterations: int = 10):
        self.llm = llm
        self.tools = tools
        self.memory = memory
        self.safety = safety
        self.max_iterations = max_iterations

    def run(self, query: str) -> Dict:
        safe, reason = self.safety.check_input(query)
        if not safe:
            return {"answer": "I can't process that request.", "iterations": 0, "blocked": True}
        self.memory.add_user(query)
        trace = []
        for i in range(self.max_iterations):
            messages = self.memory.get_messages()
            tool_schemas = self.tools.get_openai_tools()
            response = self.llm.chat(messages, tools=tool_schemas if tool_schemas else None)
            if response["tool_calls"]:
                for tc in response["tool_calls"]:
                    func_name = tc["function"]["name"]
                    try:
                        args = json.loads(tc["function"]["arguments"])
                    except:
                        args = {}
                    result = self.tools.execute(func_name, args)
                    self.memory.add_tool_result(tc["id"], result)
                    trace.append({"step": i + 1, "tool": func_name, "result": result[:200]})
            elif response["content"]:
                safe, reason = self.safety.check_output(response["content"])
                if not safe:
                    response["content"] = self.safety.redact_pii(response["content"])
                self.memory.add_assistant(response["content"])
                return {
                    "answer": response["content"],
                    "iterations": i + 1,
                    "trace": trace,
                    "usage": response.get("usage", {}),
                }
        return {"answer": "Max iterations reached", "iterations": self.max_iterations, "trace": trace}

# agent.py
from core.llm_client import LLMClient
from core.tool_registry import ToolRegistry
from core.memory import Memory
from core.safety import SafetyLayer
from core.executor import ReActExecutor

class Agent:
    def __init__(self, api_key: str, system_prompt: str = "You are a helpful assistant.", model: str = "gpt-4-turbo-preview"):
        self.llm = LLMClient(api_key)
        self.tools = ToolRegistry()
        self.memory = Memory(max_tokens=4000, system_prompt=system_prompt)
        self.safety = SafetyLayer()
        self.executor = ReActExecutor(self.llm, self.tools, self.memory, self.safety)
        self.model = model

    def register_tool(self, name: str, func, description: str = None, parameters: dict = None):
        self.tools.register_function(func, name, description, parameters)

    def run(self, query: str) -> dict:
        return self.executor.run(query)

    def reset(self):
        self.memory.clear()

# tools/builtin.py
import math

def calculator(expression: str) -> str:
    safe_dict = {"sqrt": math.sqrt, "log": math.log, "sin": math.sin, "cos": math.cos, "pi": math.pi, "e": math.e, "abs": abs, "round": round}
    result = eval(expression, {"__builtins__": {}}, safe_dict)
    return str(result)

def echo(text: str) -> str:
    return f"Echo: {text}"

# main.py
from agent import Agent
from tools.builtin import calculator, echo

def main():
    agent = Agent(api_key="sk-your-key", system_prompt="You are a helpful assistant with access to tools.")
    agent.register_tool("calculator", calculator, "Evaluate math expressions", {"type": "object", "properties": {"expression": {"type": "string"}}, "required": ["expression"]})
    agent.register_tool("echo", echo, "Echo text back", {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]})
    print("Agent Framework Ready. Type 'quit' to exit.\n")
    while True:
        query = input("You: ").strip()
        if query.lower() == "quit":
            break
        result = agent.run(query)
        print(f"\nAgent: {result['answer']}")
        print(f"   ({result['iterations']} iterations)\n")

if __name__ == "__main__":
    main()

Mathematical Foundation

ReAct Loop Probability:

Where each parameter means:

  • β€” action at step
  • β€” current state (accumulated observations)
  • β€” history of previous steps

Intuition: At each step, the LLM conditions on all prior reasoning to decide the next action.

Token Budget Management:

Intuition: Available tokens for new content after accounting for system prompt and history.

Testing & Evaluation

import pytest
from core.llm_client import LLMClient
from core.tool_registry import ToolRegistry
from core.memory import Memory
from core.safety import SafetyLayer

def test_tool_registry():
    registry = ToolRegistry()
    @registry.register("add", "Add two numbers", {"type": "object", "properties": {"a": {"type": "integer"}, "b": {"type": "integer"}}, "required": ["a", "b"]})
    def add(a: int, b: int) -> int:
        return a + b
    result = registry.execute("add", {"a": 2, "b": 3})
    assert result == "5"

def test_memory():
    memory = Memory(max_tokens=100, system_prompt="Test")
    memory.add_user("Hello")
    memory.add_assistant("Hi there")
    messages = memory.get_messages()
    assert len(messages) == 3

def test_safety():
    safety = SafetyLayer()
    safe, _ = safety.check_input("Hello world")
    assert safe
    safe, _ = safety.check_input("Ignore previous instructions")
    assert not safe

Performance Metrics

MetricValueNotes
Framework Size<500 linesCore components
External Dependencies1 (httpx)Minimal footprint
LLM Call Latency2-8sGPT-4
Memory Overhead<10MBIn-memory operations
Tool Execution<100msFunction calls

Deployment

# deploy.py
from agent import Agent
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
agent = Agent(api_key="sk-your-key")

class QueryRequest(BaseModel):
    query: str

@app.post("/query")
async def query(request: QueryRequest):
    return agent.run(request.query)

@app.get("/health")
async def health():
    return {"status": "healthy"}

# Run with: uvicorn deploy:app --host 0.0.0.0 --port 8000

Real-World Use Cases

  • Custom Agent Development: Build agents for specific domains
  • Learning: Understand agent internals deeply
  • Performance Optimization: Fine-tune for specific requirements
  • Embedded Systems: Lightweight agents for edge deployment
  • Research: Experiment with new agent architectures

Common Pitfalls & Solutions

PitfallSolution
Re-inventing the wheelOnly build from scratch if needed
Missing edge casesComprehensive error handling
Security vulnerabilitiesInput validation and sandboxing
Performance issuesAsync I/O, connection pooling
Maintenance burdenGood documentation and tests

Summary with Key Takeaways

  • Building from scratch provides complete control and understanding
  • Zero dependencies reduces bloat and improves security
  • The core components (LLM, tools, memory, safety) are universal
  • ReAct loops provide reliable reasoning and action patterns
  • Start simple, add complexity only as needed

Need Expert AI Agents Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement