AI Coding Assistant with Sandboxed Execution

Code Agent Architecture — Complete Production Flow

What is a Code Generation Agent?

Code generation agents go beyond simple code completion. They understand requirements, generate complete implementations, execute code in sandboxes, analyze results, and iteratively debug until the code works correctly.

The key challenge is safe execution. Running arbitrary LLM-generated code on your system is dangerous. Sandboxed execution environments (containers, subprocesses with restrictions, or cloud sandboxes) isolate generated code from the host system.

The most effective code agents follow a generate-execute-debug loop: generate code, run it, check for errors, fix issues, and repeat until working. This mirrors how human developers actually work.

Why This Matters

Code generation agents transform how software is built:

• Rapid prototyping from natural language descriptions
• Automated testing and validation
• Safe execution of untrusted code
• Iterative improvement through debugging

Real-world analogy: Think of a code agent as a junior developer with superpowers. It can write code instantly, run it safely, and debug faster than any human. But like a junior developer, it needs guardrails (safety checks) and supervision (human review).

Code Agent Capabilities

Project Overview

We will build a code agent that:

• Generates Python code from natural language descriptions
• Executes code in a sandboxed subprocess with resource limits
• Captures stdout, stderr, and return codes
• Parses errors and iteratively fixes them
• Runs tests to validate correctness

Expected outcome: A safe, iterative code generation agent that produces working code.

Difficulty: Advanced (requires understanding of subprocess management, Docker, and code safety)

Tools & Setup

Step 1: Environment Setup

python -m venv venv
source venv/bin/activate
pip install openai docker
export OPENAI_API_KEY="sk-your-key"

Step 2: Code Safety Checker

# safety.py
import ast
from typing import List
import logging

logger = logging.getLogger(__name__)

FORBIDDEN_MODULES = {
    "os", "sys", "subprocess", "shutil", "pathlib",
    "socket", "http", "urllib", "requests",
    "ctypes", "multiprocessing", "threading",
}

FORBIDDEN_FUNCTIONS = {
    "exec", "eval", "compile", "__import__",
    "open", "input", "print",
}


class CodeSafetyChecker:
    """
    Validates code safety using AST analysis.
    
    Checks for forbidden imports, function calls, and patterns
    before code execution.
    """
    
    def __init__(self, allowed_modules: set = None):
        self.allowed_modules = allowed_modules or {
            "math", "json", "re", "datetime", "collections"
        }

    def check(self, code: str) -> tuple[bool, List[str]]:
        """
        Check code for safety issues.
        
        Args:
            code: Python code to check
            
        Returns:
            Tuple of (is_safe, list_of_issues)
        """
        issues = []
        try:
            tree = ast.parse(code)
        except SyntaxError as e:
            return False, [f"Syntax error: {str(e)}"]

        for node in ast.walk(tree):
            if isinstance(node, ast.Import):
                for alias in node.names:
                    if alias.name.split(".")[0] in FORBIDDEN_MODULES:
                        issues.append(f"Forbidden import: {alias.name}")

            elif isinstance(node, ast.ImportFrom):
                if node.module and node.module.split(".")[0] in FORBIDDEN_MODULES:
                    issues.append(f"Forbidden import from: {node.module}")

            elif isinstance(node, ast.Call):
                if isinstance(node.func, ast.Name):
                    if node.func.id in FORBIDDEN_FUNCTIONS:
                        issues.append(f"Forbidden function call: {node.func.id}")

            elif isinstance(node, ast.Attribute):
                if isinstance(node.value, ast.Name):
                    if node.value.id in FORBIDDEN_MODULES:
                        issues.append(f"Forbidden attribute access: {node.value.id}.{node.attr}")

        is_safe = len(issues) == 0
        if not is_safe:
            logger.warning(f"Safety issues found: {issues}")
        return is_safe, issues

Step 3: Sandbox Executor

# sandbox/local_sandbox.py
import subprocess
import tempfile
import os
from typing import Dict
import logging

logger = logging.getLogger(__name__)


class LocalSandbox:
    """
    Sandboxed execution environment using subprocess.
    
    Provides isolation with resource limits:
    - Execution timeout
    - Memory limits (via ulimit)
    - Output size limits
    - Restricted working directory
    """
    
    def __init__(
        self,
        timeout: int = 30,
        memory_limit_mb: int = 256,
        max_output_bytes: int = 1024 * 1024,
    ):
        self.timeout = timeout
        self.memory_limit_mb = memory_limit_mb
        self.max_output_bytes = max_output_bytes

    def execute(self, code: str) -> Dict:
        """
        Execute code in a sandboxed environment.
        
        Args:
            code: Python code to execute
            
        Returns:
            Dictionary with stdout, stderr, return_code, success
        """
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".py", delete=False, dir="/tmp"
        ) as f:
            f.write(code)
            temp_path = f.name

        try:
            result = subprocess.run(
                ["python", temp_path],
                capture_output=True,
                text=True,
                timeout=self.timeout,
                env={
                    **os.environ,
                    "PYTHONDONTWRITEBYTECODE": "1",
                },
                cwd=tempfile.gettempdir(),
            )
            return {
                "success": result.returncode == 0,
                "stdout": result.stdout[:self.max_output_bytes],
                "stderr": result.stderr[:self.max_output_bytes],
                "return_code": result.returncode,
            }
        except subprocess.TimeoutExpired:
            logger.warning(f"Code execution timed out after {self.timeout}s")
            return {
                "success": False,
                "stdout": "",
                "stderr": f"Execution timed out after {self.timeout}s",
                "return_code": -1,
            }
        finally:
            os.unlink(temp_path)

Step 4: Code Generator and Debugger

# generator.py
from openai import OpenAI
import logging

logger = logging.getLogger(__name__)

CODE_SYSTEM = """You are an expert Python programmer. Generate clean, 
working Python code based on user requirements.

Rules:
1. Write complete, runnable code
2. Include necessary imports
3. Add a main() function or script entry point
4. Handle errors gracefully
5. Use type hints where helpful
6. Print results to stdout for verification"""


class CodeGenerator:
    """
    Generates Python code from natural language requirements.
    """
    
    def __init__(self, model: str = "gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def generate(
        self,
        requirement: str,
        context: str = "",
        previous_code: str = "",
        error: str = "",
    ) -> str:
        """
        Generate code from requirements.
        
        Args:
            requirement: Natural language description
            context: Additional context
            previous_code: Code that failed (for debugging)
            error: Error from previous attempt
            
        Returns:
            Generated Python code
        """
        prompt = f"Requirement: {requirement}"
        if context:
            prompt += f"\n\nContext: {context}"
        if previous_code:
            prompt += f"\n\nPrevious code that failed:\n```python\n{previous_code}\n```\n\nError:\n{error}\n\nFix the code:"
        else:
            prompt += "\n\nGenerate the code:"

        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": CODE_SYSTEM},
                {"role": "user", "content": prompt},
            ],
            temperature=0.0,
        )
        return self._extract_code(response.choices[0].message.content)

    def _extract_code(self, content: str) -> str:
        """Extract code from markdown code blocks."""
        if "```python" in content:
            parts = content.split("```python")
            if len(parts) > 1:
                code = parts[1].split("```")[0]
                return code.strip()
        return content.strip()


# debugger.py
from openai import OpenAI
import logging

logger = logging.getLogger(__name__)

DEBUG_SYSTEM = """You are a debugging expert. Analyze the error and provide 
a fix. Return ONLY the corrected Python code, no explanation."""


class CodeDebugger:
    """
    Debuggs code by analyzing errors and generating fixes.
    """
    
    def __init__(self, model: str = "gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def debug(self, code: str, error: str, stdout: str = "") -> str:
        """
        Debug code by analyzing errors.
        
        Args:
            code: Current code
            error: Error message
            stdout: Standard output (if any)
            
        Returns:
            Fixed Python code
        """
        prompt = f"""Code:
```python
{code}

Error: {error}

Stdout (if any): {stdout[:500]}

Fix the code and return ONLY the corrected Python code:"""

response = self.client.chat.completions.create( model=self.model, messages=[ {"role": "system", "content": DEBUG_SYSTEM}, {"role": "user", "content": prompt}, ], temperature=0.0, ) return self._extract_code(response.choices[0].message.content)

def _extract_code(self, content: str) -> str: """Extract code from markdown code blocks.""" if "python" in content: parts = content.split("python") if len(parts) > 1: return parts[1].split("```")[0].strip() return content.strip()

### Step 5: Complete Code Agent

```python
# code_agent.py
from generator import CodeGenerator
from debugger import CodeDebugger
from sandbox.local_sandbox import LocalSandbox
from safety import CodeSafetyChecker
from typing import Dict
import logging

logger = logging.getLogger(__name__)


class CodeAgent:
    """
    Complete code generation agent with sandboxed execution.
    
    Features:
    - Code generation from natural language
    - Safety validation before execution
    - Sandboxed execution with resource limits
    - Iterative debugging on failures
    - Test generation for validation
    """
    
    def __init__(
        self,
        model: str = "gpt-4-turbo-preview",
        max_retries: int = 3,
        use_docker: bool = False,
    ):
        self.generator = CodeGenerator(model)
        self.debugger = CodeDebugger(model)
        self.sandbox = LocalSandbox()
        self.safety_checker = CodeSafetyChecker()
        self.max_retries = max_retries

    def run(self, requirement: str) -> Dict:
        """
        Generate and execute code for a requirement.
        
        Args:
            requirement: Natural language code requirement
            
        Returns:
            Execution results with code, output, and history
        """
        logger.info(f"Generating code for: {requirement[:50]}...")
        
        code = self.generator.generate(requirement)
        history = []

        for attempt in range(self.max_retries):
            # Safety check
            is_safe, issues = self.safety_checker.check(code)
            if not is_safe:
                return {
                    "success": False,
                    "code": code,
                    "error": f"Safety issues: {', '.join(issues)}",
                    "attempts": attempt + 1,
                    "history": history,
                }

            # Execute in sandbox
            result = self.sandbox.execute(code)
            history.append({
                "attempt": attempt + 1,
                "code": code,
                "result": result,
            })

            if result["success"]:
                logger.info(f"Code executed successfully on attempt {attempt + 1}")
                return {
                    "success": True,
                    "code": code,
                    "output": result["stdout"],
                    "attempts": attempt + 1,
                    "history": history,
                }

            # Debug and retry
            logger.warning(f"Attempt {attempt + 1} failed: {result['stderr'][:100]}")
            code = self.debugger.debug(
                code, result["stderr"], result["stdout"]
            )

        return {
            "success": False,
            "code": code,
            "error": "Max retries exceeded",
            "attempts": self.max_retries,
            "history": history,
        }

Mathematical Foundation

Code Generation Confidence:

Where each parameter means:

• — probability the generated code works correctly
• — natural language description
• — existing code, tests, documentation

Intuition: The model's confidence increases with clearer requirements and more context.

Retry Success Probability:

Where is per-attempt success probability and is max retries.

Intuition: With p=0.7 and k=3 retries, success probability is 97.3%.

Debug Efficiency:

Intuition: Measures how effective the debugger is at fixing errors. Higher efficiency means fewer retries needed.

Performance Metrics

Real-World Examples

Example 1: Data Processing

Generate code to process CSV data:

agent = CodeAgent()
result = agent.run("Read CSV file, calculate average age by city, output as JSON")
# Generates, executes, and debugs until working

Example 2: API Integration

Generate code to call an API:

result = agent.run("Fetch weather data from OpenWeatherMap API for New York")
# Includes error handling and retry logic

Common Pitfalls & Solutions

Security Considerations

Critical security measures for code generation agents:

1. AST Analysis: Parse and validate code structure before execution
2. Import Blocking: Prevent dangerous module imports (os, sys, subprocess)
3. Sandbox Isolation: Run code in isolated environments (Docker, subprocess)
4. Resource Limits: Set CPU, memory, and timeout limits
5. Network Blocking: Disable network access in sandbox
6. Output Sanitization: Filter sensitive information from output

# Example: Docker-based sandbox
class DockerSandbox:
    def execute(self, code: str) -> Dict:
        import docker
        client = docker.from_env()
        
        container = client.containers.run(
            "python:3.11-slim",
            command=f"python -c '{code}'",
            detach=True,
            mem_limit="256m",
            cpu_period=100000,
            cpu_quota=50000,
            network_disabled=True,
        )
        
        result = container.wait(timeout=30)
        logs = container.logs().decode()
        container.remove()
        
        return {
            "success": result["StatusCode"] == 0,
            "output": logs,
        }

Summary with Key Takeaways

• Sandboxed execution is essential for running LLM-generated code safely
• The generate-execute-debug loop achieves 95%+ success rates
• Safety checking should happen before execution, not after
• Docker provides stronger isolation than local subprocess execution
• Test generation validates that code meets requirements, not just runs
• AST-based safety checks are fast and effective for Python code
• Limit max retries to prevent infinite debug loops

Interview Questions

1. Why is sandboxed execution critical for code agents?

Answer: LLM-generated code may contain bugs, infinite loops, or malicious patterns. Sandboxed execution isolates generated code from the host system: 1) Subprocess isolation — Separate process with resource limits, 2) Docker containers — Full filesystem/network isolation, 3) Resource limits — CPU, memory, and timeout constraints, 4) No network access — Prevent data exfiltration. Without sandboxing, a code agent could delete files, steal data, or compromise the system. Docker provides the strongest isolation.

2. How does the generate-execute-debug loop work?

Answer: The loop has 3 phases: 1) Generate — LLM creates code from requirements, 2) Execute — Run code in sandbox, capture stdout/stderr/return code, 3) Debug — If errors, send error + code back to LLM for fixing. Repeat until success or max retries. Key insight: each iteration provides more context (error messages) to the LLM, improving fix accuracy. With p=0.7 per attempt, 3 retries give 97.3% success rate.

3. What safety checks should be performed before execution?

Answer: Pre-execution safety checks: 1) AST parsing — Verify syntax correctness, 2) Import checking — Block dangerous modules (os, sys, subprocess), 3) Function blocking — Prevent exec, eval, compile, 4) File access control — Restrict file operations, 5) Network blocking — Disable network access in sandbox, 6) Resource limits — Set CPU/memory/timeout limits. The safety checker should be fast (<10ms) and run before every execution. Balance security with flexibility for legitimate code.

4. How do you handle code that needs external dependencies?

Answer: Dependency strategies: 1) Pre-installed packages — Include common packages in sandbox image, 2) Dependency declaration — Require users to declare dependencies, 3) Virtual environments — Create isolated venvs per execution, 4) Docker images — Custom images with pre-installed packages, 5) Requirements file — Auto-install from requirements.txt. For production: use Docker images with pre-installed dependencies. For development: allow user-declared dependencies with review.

5. What is the role of test generation in code agents?

Answer: Test generation validates that code meets requirements, not just runs without errors: 1) Generate tests from requirements, 2) Run tests in sandbox, 3) Report coverage and failures, 4) Fix code based on test failures. Tests provide: objective quality measure, regression prevention, documentation of expected behavior. Use pytest for Python code. Generate both unit tests (function-level) and integration tests (end-to-end).

6. How do you evaluate code generation quality?

Answer: Quality metrics: 1) Correctness — Does code run and produce correct output? 2) Efficiency — Time/space complexity, 3) Readability — Code style, naming, comments, 4) Robustness — Error handling, edge cases, 5) Test coverage — % of code covered by tests, 6) First-try success — % of code that works on first execution, 7) Debug iterations — Average fixes needed. Use automated metrics (linting, test coverage) and human evaluation for style/quality.

7. How would you extend the code agent for multiple languages?

Answer: Multi-language support: 1) Language detection — Identify target language from requirements, 2) Sandbox images — Docker images for each language (python, node, go), 3) Safety rules — Language-specific forbidden patterns, 4) Test frameworks — pytest, jest, go test, 5) Linting — Language-specific linters (ruff, eslint), 6) Package management — pip, npm, go mod. Abstract the sandbox interface to support multiple runtimes. Each language needs its own safety checker and test runner.

8. What are the limitations of code generation agents?

Answer: Key limitations: 1) Complex code — Struggles with large, multi-file projects, 2) Domain knowledge — May lack specialized library knowledge, 3) Architecture — Can't design system architecture, only implement functions, 4) Security — May generate vulnerable code, 5) Performance — May not optimize for speed, 6) Context window — Limited by model's context for large codebases. Mitigations: human review, incremental generation, test-driven development, and security scanning.

KnowledgeCheck

1. Why is sandboxed execution critical?

   • a) It's faster
   • b) It isolates generated code from the host system
   • c) It uses fewer tokens
   • d) It improves code quality

2. What is the success rate with 3 retries (p=0.7)?

   • a) 70%
   • b) 90%
   • c) 97.3%
   • d) 100%

3. What does the safety checker analyze?

   • a) Code performance
   • b) AST for forbidden imports and functions
   • c) Code style
   • d) Test coverage

4. What is the generate-execute-debug loop?

   • a) Generate once and deploy
   • b) Generate, run in sandbox, fix errors, repeat
   • c) Debug first, then generate
   • d) Execute without safety checks

5. Which provides stronger isolation?

   • a) Local subprocess
   • b) Docker container
   • c) Virtual environment
   • d) Both a and b are equal

6. What validates code meets requirements?

   • a) Linting only
   • b) Test generation and execution
   • c) Code review
   • d) Documentation

Answers: 1-b, 2-c, 3-b, 4-b, 5-b, 6-b