🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
đŸ’ŧ Servicesâ„šī¸ Aboutâœ‰ī¸ ContactView Pricing Plansfrom $10

Building Multi-Agent Systems with CrewAI

AI AgentsMulti-Agent SystemsđŸŸĸ Free Lesson

Advertisement

Building Multi-Agent Systems with CrewAI

Multi-Agent Architecture — Production Flow

Multi-Agent System — Production CoordinationUSER TASK: Complex objective requiring multiple specialized capabilitiesOrchestrator AgentTask decomposition | Agent selection | Result synthesisResearch AgentWeb searchData collectionSource verificationFact gatheringWriter AgentContent creationStructure planningDraft generationEditor AgentGrammar checkStyle consistencyQuality assuranceSEO AgentKeyword researchMeta optimizationStructure reviewShared MemoryConversation history | Task outputs | Agent stateTask QueueDependency graph | Priority schedulingFinal OutputPublication-ready articleAgents communicate through shared memory and message passing, coordinated by the orchestrator

What is Multi-Agent Systems?

Multi-agent systems coordinate multiple specialized agents to solve complex tasks that exceed single-agent capabilities. Each agent has distinct roles, tools, and expertise, collaborating through shared memory and message passing.

The key advantage is decomposition: complex problems are broken into manageable subproblems assigned to domain experts. A researcher gathers information, a writer synthesizes it, and a reviewer ensures quality. This mirrors human team dynamics and produces superior results for multifaceted tasks.

Why This Matters

Single agents hit limitations when tasks require multiple specializations. A research agent that also writes and edits will produce mediocre results across all three. Multi-agent systems enable each agent to focus on what it does best, producing higher quality outputs.

Real-world analogy: Think of a multi-agent system as a newsroom. A reporter researches, a writer drafts, an editor polishes, and a fact-checker verifies. Each specialist produces better work than a generalist trying to do everything.

Multi-Agent vs Single-Agent

AspectSingle-AgentMulti-Agent
ComplexitySimple tasksComplex, multi-faceted tasks
SpecializationGeneralistDomain experts
ScalabilityLimited by contextScales with agents
CostLower per queryHigher, but better quality
DebuggingStraightforwardRequires tracing across agents
QualityGood for simple tasksSuperior for complex tasks

Coordination Patterns

PatternDescriptionBest ForTrade-offs
SequentialPipeline: Agent A → B → CLinear workflowsSimple but no parallelism
HierarchicalManager delegates to workersComplex projectsMore overhead, better control
ParallelIndependent agents work simultaneouslyIndependent subtasksFaster but needs synchronization
DebateAgents argue different perspectivesDecision makingSlower but more thorough

Project Overview

We will build a content research team with 4 specialized agents:

  • Researcher: Gathers information from multiple sources
  • Writer: Synthesizes findings into coherent articles
  • Editor: Reviews and improves quality
  • SEO Specialist: Optimizes for search engines

Expected outcome: A multi-agent system that produces publication-ready articles from a topic prompt.

Difficulty: Advanced (requires understanding of agent design, task decomposition, and inter-agent communication)

Tools & Setup

ToolVersionPurpose
Python3.11+Core language
CrewAI0.30+Multi-agent framework
OpenAI1.0+LLM backbone
duckduckgo-search4.0+Web search tool
pydantic2.0+Data validation

Step 1: Environment Setup

python -m venv venv
source venv/bin/activate
pip install crewai[tools] duckduckgo-search openai
export OPENAI_API_KEY="sk-your-key"

Step 2: Custom Tools

# tools/search_tools.py
from crewai import Tool
from duckduckgo_search import DDGS
from typing import List, Dict
import logging

logger = logging.getLogger(__name__)


def search_web(query: str, num_results: int = 5) -> List[Dict]:
    """Search the web using DuckDuckGo."""
    try:
        with DDGS() as ddgs:
            results = list(ddgs.text(query, max_results=num_results))
            return [
                {"title": r["title"], "snippet": r["body"], "url": r["href"]}
                for r in results
            ]
    except Exception as e:
        logger.error(f"Web search failed: {e}")
        return []


def get_research_tool() -> Tool:
    """Create a web search tool for research agents."""
    def _search(query: str) -> str:
        results = search_web(query, num_results=5)
        if not results:
            return "No results found for the query."
        
        formatted = []
        for i, r in enumerate(results):
            formatted.append(
                f"[{i+1}] {r['title']}\n{r['snippet']}\nURL: {r['url']}"
            )
        return "\n\n".join(formatted)

    return Tool(
        name="Web Search",
        description="Search the web for current information on any topic",
        func=_search,
    )


def get_article_analysis_tool() -> Tool:
    """Create a URL content extraction tool."""
    def _analyze(url: str) -> str:
        import httpx
        try:
            response = httpx.get(url, follow_redirects=True, timeout=10)
            response.raise_for_status()
            from bs4 import BeautifulSoup
            soup = BeautifulSoup(response.text, "html.parser")
            
            # Remove script and style elements
            for element in soup(["script", "style", "nav", "footer"]):
                element.decompose()
            
            text = soup.get_text(separator="\n", strip=True)
            
            # Limit to meaningful content
            lines = [line.strip() for line in text.splitlines() if line.strip()]
            return "\n".join(lines)[:3000]
        except Exception as e:
            return f"Error analyzing article: {str(e)}"

    return Tool(
        name="Article Analyzer",
        description="Extract and analyze content from a URL",
        func=_analyze,
    )

Step 3: Agent Definitions

# agents/researcher.py
from crewai import Agent, Tool
from tools.search_tools import get_research_tool, get_article_analysis_tool


def create_researcher() -> Agent:
    """Create a research agent with web search capabilities."""
    tools = [get_research_tool(), get_article_analysis_tool()]
    return Agent(
        role="Senior Research Analyst",
        goal="Gather comprehensive, accurate information on the given topic",
        backstory="""You are an expert researcher with a talent for finding 
        relevant information quickly. You verify multiple sources and identify 
        key insights, trends, and data points. You always cite your sources 
        and prioritize authoritative references.""",
        tools=tools,
        verbose=True,
        allow_delegation=False,
        max_iter=15,
        memory=True,
    )


# agents/writer.py
from crewai import Agent


def create_writer() -> Agent:
    """Create a content writer agent."""
    return Agent(
        role="Content Writer",
        goal="Transform research into engaging, well-structured articles",
        backstory="""You are a skilled writer who creates compelling content.
        You structure information logically, use clear language, and maintain
        reader engagement throughout. You can adapt tone for different audiences
        and always maintain factual accuracy.""",
        tools=[],
        verbose=True,
        allow_delegation=False,
        max_iter=10,
        memory=True,
    )


# agents/editor.py
from crewai import Agent


def create_editor() -> Agent:
    """Create an editor agent for quality assurance."""
    return Agent(
        role="Senior Editor",
        goal="Ensure content quality, accuracy, and readability",
        backstory="""You are a meticulous editor with exceptional attention to 
        detail. You check for grammar, clarity, flow, and factual accuracy.
        You improve content without changing the author's voice.
        You provide specific, actionable feedback.""",
        tools=[],
        verbose=True,
        allow_delegation=False,
        max_iter=10,
        memory=True,
    )


# agents/seo_specialist.py
from crewai import Agent


def create_seo_specialist() -> Agent:
    """Create an SEO optimization agent."""
    return Agent(
        role="SEO Specialist",
        goal="Optimize content for search engine visibility",
        backstory="""You are an SEO expert who understands search algorithms
        and user intent. You optimize titles, headings, meta descriptions,
        and content structure for maximum search visibility while maintaining
        readability for human audiences.""",
        tools=[],
        verbose=True,
        allow_delegation=False,
        max_iter=8,
        memory=True,
    )

Step 4: Task Definitions and Crew

# tasks/content_tasks.py
from crewai import Task


def create_research_task(agent, topic: str) -> Task:
    """Create a research task for the given topic."""
    return Task(
        description=f"""Research the topic: "{topic}"

1. Search for 5-7 authoritative sources
2. Extract key facts, statistics, and insights
3. Identify different perspectives and debates
4. Note any recent developments or trends
5. Compile a structured research summary with citations

Output format:
- Key Findings (5-7 bullet points)
- Supporting Evidence (with source URLs)
- Different Perspectives
- Recent Developments
- Conclusion""",
        expected_output="Structured research summary with citations",
        agent=agent,
    )


def create_writing_task(agent, topic: str) -> Task:
    """Create a writing task based on research."""
    return Task(
        description=f"""Write a comprehensive article on: "{topic}"

Using the research provided:
1. Create an engaging introduction with a hook
2. Structure content with clear H2/H3 headings
3. Include relevant examples and data points
4. Write 1500-2000 words
5. End with a strong conclusion and call to action

Style guidelines:
- Professional but accessible tone
- Short paragraphs (2-3 sentences)
- Use bullet points for lists
- Include transition sentences between sections""",
        expected_output="Complete 1500-2000 word article",
        agent=agent,
    )


def create_editing_task(agent) -> Task:
    """Create an editing task for quality review."""
    return Task(
        description="""Review and edit the article for:

1. Grammar and spelling errors
2. Clarity and readability
3. Logical flow and structure
4. Factual accuracy (flag any unsupported claims)
5. Tone consistency
6. Engagement level

Provide specific improvements and a cleaned-up version.""",
        expected_output="Edited article with improvements noted",
        agent=agent,
    )


def create_seo_task(agent) -> Task:
    """Create an SEO optimization task."""
    return Task(
        description="""Optimize the article for SEO:

1. Suggest primary keyword and 3-5 secondary keywords
2. Optimize title tag (under 60 characters)
3. Write meta description (under 160 characters)
4. Ensure proper heading hierarchy (H1, H2, H3)
5. Add internal linking suggestions
6. Optimize image alt text suggestions
7. Check keyword density (1-2%)

Provide the optimized version with SEO changes highlighted.""",
        expected_output="SEO-optimized article with metadata",
        agent=agent,
    )

Step 5: Crew Orchestration

# crew.py
from crewai import Crew, Process
from agents.researcher import create_researcher
from agents.writer import create_writer
from agents.editor import create_editor
from agents.seo_specialist import create_seo_specialist
from tasks.content_tasks import (
    create_research_task,
    create_writing_task,
    create_editing_task,
    create_seo_task,
)
import logging

logger = logging.getLogger(__name__)


class ContentCrew:
    """
    Multi-agent crew for content creation.
    
    Coordinates researcher, writer, editor, and SEO specialist
    to produce publication-ready articles.
    """
    
    def __init__(self, topic: str):
        self.topic = topic
        self.researcher = create_researcher()
        self.writer = create_writer()
        self.editor = create_editor()
        self.seo_specialist = create_seo_specialist()

    def run(self) -> dict:
        """Execute the content creation pipeline."""
        logger.info(f"Starting content creation for: {self.topic}")
        
        research_task = create_research_task(self.researcher, self.topic)
        writing_task = create_writing_task(self.writer, self.topic)
        editing_task = create_editing_task(self.editor)
        seo_task = create_seo_task(self.seo_specialist)

        crew = Crew(
            agents=[
                self.researcher,
                self.writer,
                self.editor,
                self.seo_specialist,
            ],
            tasks=[
                research_task,
                writing_task,
                editing_task,
                seo_task,
            ],
            process=Process.sequential,
            verbose=True,
            memory=True,
        )

        result = crew.kickoff()
        
        return {
            "topic": self.topic,
            "output": result,
            "usage_metrics": crew.usage_metrics,
        }

Mathematical Foundation

Task Dependency Scoring:

Where each parameter means:

  • , , — weight coefficients for scoring factors
  • — estimated difficulty (1-10)
  • — business importance (1-10)
  • — number of dependent tasks (higher = higher priority)

Intuition: Tasks are scored to determine execution order. High-priority, low-dependency tasks execute first, ensuring optimal parallelism.

Agent Utilization:

Intuition: Measures what percentage of time each agent is actively working vs idle. Higher utilization indicates better task assignment.

Multi-Agent Speedup:

Intuition: Parallel execution reduces total time to the slowest agent plus coordination overhead.

Performance Metrics

MetricValueNotes
Articles/Hour4-6With GPT-4 backbone
Avg Quality Score8.5/10Human evaluation
Task Completion95%+Successful crew runs
Agent Utilization70%+Well-balanced tasks
Cost per Article$0.50-2.00GPT-4 pricing

Real-World Examples

Example 1: Marketing Content Team

A marketing team uses multi-agent systems to produce blog posts:

crew = ContentCrew(topic="AI in Healthcare 2024")
result = crew.run()
# Produces: Research-backed, well-written, edited, SEO-optimized article

Example 2: Research Report Generation

A consultancy generates research reports:

# Customize agents for specific domains
researcher = create_researcher()
researcher.goal = "Focus on market data and competitor analysis"

writer = create_writer()
writer.backstory = "Write in formal consulting style with executive summaries"

Common Pitfalls & Solutions

PitfallSolution
Agent confusionDefine clear roles and backstories
Circular delegationSet allow_delegation=False for leaf agents
Context lossUse shared memory and task outputs
High API costsUse GPT-3.5 for simpler agents
Quality inconsistencyAdd validation steps between agents
Infinite loopsSet max_iter on each agent
Slow executionUse parallel process for independent tasks
Output hallucinationInclude verification agents in pipeline

Security Considerations

Critical security measures for multi-agent systems:

  1. Tool Permission Scoping: Each agent should only access tools it needs
  2. Output Validation: Validate agent outputs before passing to downstream agents
  3. Cost Controls: Set token limits per agent to prevent runaway costs
  4. Audit Logging: Track all agent interactions for debugging and compliance
  5. Rate Limiting: Prevent agents from overwhelming external APIs
# Example: Cost-controlled multi-agent system
crew = Crew(
    agents=[...],
    tasks=[...],
    max_rpm=10,  # Limit API calls per minute
    token_budget=10000,  # Limit total tokens
)

Summary with Key Takeaways

  • Multi-agent systems decompose complex tasks across specialized agents
  • CrewAI provides built-in delegation, memory, and process orchestration
  • Sequential processes work well for content pipelines; hierarchical for complex projects
  • Agent backstories significantly impact output quality and style
  • Always include review/validation agents to catch errors before final output
  • Set max_iter and allow_delegation appropriately to prevent infinite loops
  • Shared memory reduces redundancy and enables knowledge accumulation

Interview Questions

1. What is the key advantage of multi-agent systems over single-agent systems?

Answer: Multi-agent systems decompose complex problems across specialized agents, each with domain-specific tools and prompts. This mirrors human team dynamics where specialists collaborate. Key advantages: 1) Better quality — Each agent focuses on one task, 2) Scalability — Add agents for new capabilities, 3) Maintainability — Individual agents can be updated independently, 4) Cost optimization — Use cheaper models for simpler agents. The tradeoff is increased complexity in coordination and debugging.

2. How does CrewAI handle task delegation between agents?

Answer: CrewAI supports three process types: 1) Sequential — Tasks execute in order, each receiving previous outputs, 2) Hierarchical — A manager agent delegates tasks to workers dynamically, 3) Parallel — Independent tasks run simultaneously. The allow_delegation flag determines if agents can request help from others. Task dependencies are declared in the Task object, and CrewAI manages the execution graph automatically. Shared memory enables agents to access each other's outputs.

3. What are agent backstories and why do they matter?

Answer: Backstories are persona descriptions that shape agent behavior and output style. They provide context about the agent's expertise, communication style, and approach to problems. For example, a "Senior Research Analyst" backstory produces more thorough, citation-heavy output than a generic agent. Backstories impact: 1) Output quality — More specific personas produce better results, 2) Consistency — Maintains consistent voice across interactions, 3) Tool selection — Influences which tools the agent prefers. Well-crafted backstories are the primary lever for improving multi-agent output quality.

4. How do you prevent circular delegation between agents?

Answer: Circular delegation occurs when Agent A delegates to Agent B, which delegates back to A, creating infinite loops. Prevention strategies: 1) Set allow_delegation=False for leaf agents that shouldn't delegate, 2) Implement delegation depth limits — max 2-3 levels, 3) Use the hierarchical process where only the manager can delegate, 4) Track delegation history and refuse repeated delegations, 5) Set max_iter on each agent to prevent infinite loops. CrewAI's max_iter parameter is the primary safeguard.

5. How do you evaluate multi-agent system performance?

Answer: Multi-level evaluation: 1) Task completion rate — % of tasks completed successfully, 2) Quality metrics — Human evaluation of output quality, 3) Agent utilization — % of time each agent is active, 4) Token efficiency — Total tokens used per output, 5) Latency — End-to-end execution time, 6) Cost — Dollar cost per execution, 7) Error rate — % of runs that fail. Use benchmarks with known correct outputs and A/B test different agent configurations.

6. What is the role of shared memory in multi-agent systems?

Answer: Shared memory enables agents to access each other's outputs and maintain context across the workflow. In CrewAI, memory stores: 1) Task outputs — Previous agent results available to subsequent agents, 2) Conversation history — Full trace of agent interactions, 3) Learned knowledge — Facts and insights accumulated during execution. Memory is configured per-crew with memory=True. It reduces redundancy (agents don't repeat work) and enables quality improvement (later agents can build on earlier findings).

7. How do you handle agent failures in a multi-agent system?

Answer: Failure handling strategies: 1) Retry with different parameters — Adjust temperature or prompt, 2) Fallback agents — Have backup agents for critical roles, 3) Graceful degradation — Continue with partial results if non-critical agent fails, 4) Error propagation — Pass error information to downstream agents so they can adapt, 5) Circuit breaker — Stop execution if repeated failures occur. Implement try/except around each agent execution and log failures for debugging. CrewAI's max_iter prevents infinite retry loops.

8. When should you use multi-agent vs single-agent systems?

Answer: Use multi-agent when: 1) Task requires multiple specializations (research + writing + editing), 2) Output quality is critical (publication-ready content), 3) Task complexity exceeds single context window, 4) You need parallel processing of independent subtasks. Use single-agent when: 1) Task is straightforward (simple Q&A), 2) Latency is critical (real-time responses), 3) Cost is a primary concern, 4) Debugging simplicity is needed. Start with single-agent and add complexity only when quality or capability requirements demand it.


KnowledgeCheck

  1. What is the primary advantage of multi-agent systems?

    • a) Lower cost
    • b) Decomposition of complex tasks across specialists
    • c) Faster execution
    • d) Simpler debugging
  2. Which CrewAI process type runs tasks in a pipeline?

    • a) Parallel
    • b) Hierarchical
    • c) Sequential
    • d) Debate
  3. How do you prevent circular delegation?

    • a) Use more agents
    • b) Set allow_delegation=False for leaf agents
    • c) Increase max_iter
    • d) Use GPT-4 instead of GPT-3.5
  4. What is the purpose of agent backstories?

    • a) To store conversation history
    • b) To shape agent behavior and output style
    • c) To manage task dependencies
    • d) To reduce token usage
  5. When should you use multi-agent over single-agent?

    • a) For simple Q&A tasks
    • b) When task requires multiple specializations
    • c) When latency is critical
    • d) When cost is the primary concern
  6. What enables agents to access each other's outputs?

    • a) Tool registry
    • b) Shared memory
    • c) Task queue
    • d) Process orchestration

Answers: 1-b, 2-c, 3-b, 4-b, 5-b, 6-b

Need Expert AI Agents Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement