Meeting Notes Agent with Whisper

What is an Audio Transcription Agent?

Audio transcription agents convert meeting recordings into structured notes with speaker identification, action items, key decisions, and summaries. They transform hours of audio into actionable meeting minutes automatically.

The key pipeline is: audio preprocessing → speech-to-text (Whisper) → speaker diarization → content analysis → structured output generation. Each step adds structure to the raw audio data.

These agents save hours of manual note-taking while ensuring nothing falls through the cracks.

Project Overview

We will build a meeting notes agent that:

Transcribes audio using OpenAI Whisper
Identifies different speakers (diarization)
Extracts action items and assignees
Identifies key decisions and discussion points
Generates concise meeting summaries
Outputs structured meeting notes

Expected outcome: An agent that produces meeting minutes from audio recordings.

Difficulty: Advanced (requires understanding of audio processing, speech recognition, and NLP)

Architecture

Tools & Setup

Tool	Version	Purpose
Python	3.11+	Core language
openai-whisper	20231117	Speech-to-text
pyannote.audio	3.1+	Speaker diarization
openai	1.0+	LLM backbone
pydub	0.25+	Audio manipulation

Step 1: Environment Setup

python -m venv venv
source venv/bin/activate
pip install openai-whisper pyannote.audio openai pydub
export OPENAI_API_KEY="sk-your-key"
export HUGGINGFACE_TOKEN="hf-your-token"

Step 2: Project Structure

Architecture Diagram

meeting-agent/
├── transcription/
│   ├── __init__.py
│   ├── whisper_client.py
│   └── diarizer.py
├── analysis/
│   ├── __init__.py
│   ├── content_analyzer.py
│   └── action_extractor.py
├── reporting/
│   ├── __init__.py
│   └── meeting_notes.py
├── agent.py
└── main.py

Step 3: Whisper Transcription

# transcription/whisper_client.py
import whisper
from typing import Dict
import tempfile
import os

class WhisperTranscriber:
    def __init__(self, model_size: str = "base"):
        self.model = whisper.load_model(model_size)

    def transcribe(self, audio_path: str, language: str = None) -> Dict:
        options = {}
        if language:
            options["language"] = language
        result = self.model.transcribe(audio_path, **options)
        return {
            "text": result["text"],
            "segments": [
                {
                    "start": seg["start"],
                    "end": seg["end"],
                    "text": seg["text"],
                }
                for seg in result["segments"]
            ],
            "language": result.get("language", "unknown"),
        }

    def transcribe_with_timestamps(self, audio_path: str) -> str:
        result = self.model.transcribe(audio_path, word_timestamps=True)
        output = []
        for seg in result["segments"]:
            start = self._format_time(seg["start"])
            end = self._format_time(seg["end"])
            output.append(f"[{start} -> {end}] {seg['text'].strip()}")
        return "\n".join(output)

    def _format_time(self, seconds: float) -> str:
        mins = int(seconds // 60)
        secs = int(seconds % 60)
        return f"{mins:02d}:{secs:02d}"

Step 4: Speaker Diarization

# transcription/diarizer.py
from pyannote.audio import Pipeline
from typing import Dict, List
import torch

class SpeakerDiarizer:
    def __init__(self, token: str):
        self.pipeline = Pipeline.from_pretrained(
            "pyannote/speaker-diarization-3.1",
            use_auth_token=token,
        )

    def diarize(self, audio_path: str, num_speakers: int = None) -> List[Dict]:
        kwargs = {}
        if num_speakers:
            kwargs["num_speakers"] = num_speakers
        diarization = self.pipeline(audio_path, **kwargs)
        segments = []
        for turn, _, speaker in diarization.itertracks(yield_label=True):
            segments.append({
                "start": turn.start,
                "end": turn.end,
                "speaker": speaker,
            })
        return segments

    def merge_with_transcript(
        self, diarization: List[Dict], transcript: List[Dict]
    ) -> List[Dict]:
        merged = []
        for seg in transcript:
            speaker = self._find_speaker(diarization, seg["start"])
            merged.append({
                **seg,
                "speaker": speaker,
            })
        return merged

    def _find_speaker(
        self, diarization: List[Dict], timestamp: float
    ) -> str:
        for d in diarization:
            if d["start"] <= timestamp <= d["end"]:
                return d["speaker"]
        return "Unknown"

Step 5: Content Analysis and Reporting

# analysis/content_analyzer.py
from openai import OpenAI
from typing import Dict, List
import json

class MeetingContentAnalyzer:
    def __init__(self, model: str = "gpt-4-turbo-preview"):
        self.client = OpenAI()
        self.model = model

    def analyze(self, transcript: str, num_participants: int) -> Dict:
        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": """Analyze this meeting transcript.
                Return JSON:
                {
                    "summary": "2-3 paragraph summary",
                    "key_decisions": [{"decision": "...", "context": "..."}],
                    "action_items": [{"action": "...", "assignee": "...", "deadline": "..."}],
                    "discussion_topics": ["topic1", "topic2"],
                    "follow_ups": ["items needing follow-up"],
                    "sentiment": "positive|neutral|negative",
                    "engagement_level": "high|medium|low"
                }"""},
                {"role": "user", "content": f"Meeting transcript ({num_participants} participants):\n\n{transcript[:8000]}"},
            ],
            temperature=0.2,
        )
        try:
            return json.loads(response.choices[0].message.content)
        except:
            return {"summary": response.choices[0].message.content, "key_decisions": [], "action_items": []}

# reporting/meeting_notes.py
from typing import Dict, List
from datetime import datetime

class MeetingNotesGenerator:
    def generate_notes(
        self,
        analysis: Dict,
        transcript_segments: List[Dict],
        metadata: Dict = None,
    ) -> str:
        metadata = metadata or {}
        notes = f"# Meeting Notes\n\n"
        notes += f"**Date:** {metadata.get('date', datetime.now().strftime('%Y-%m-%d'))}\n"
        notes += f"**Duration:** {metadata.get('duration', 'N/A')}\n"
        notes += f"**Participants:** {metadata.get('participants', 'N/A')}\n\n"
        notes += "## Summary\n\n"
        notes += f"{analysis.get('summary', 'No summary available')}\n\n"
        notes += "## Key Decisions\n\n"
        for i, decision in enumerate(analysis.get("key_decisions", []), 1):
            notes += f"{i}. **{decision.get('decision', 'N/A')}**\n"
            notes += f"   - Context: {decision.get('context', 'N/A')}\n"
        notes += "\n## Action Items\n\n"
        notes += "| Action | Assignee | Deadline |\n"
        notes += "|--------|----------|----------|\n"
        for item in analysis.get("action_items", []):
            notes += f"| {item.get('action', 'N/A')} | {item.get('assignee', 'TBD')} | {item.get('deadline', 'TBD')} |\n"
        notes += "\n## Discussion Topics\n\n"
        for topic in analysis.get("discussion_topics", []):
            notes += f"- {topic}\n"
        notes += "\n## Follow-ups\n\n"
        for follow in analysis.get("follow_ups", []):
            notes += f"- {follow}\n"
        return notes

Step 6: Complete Agent

# agent.py
from transcription.whisper_client import WhisperTranscriber
from transcription.diarizer import SpeakerDiarizer
from analysis.content_analyzer import MeetingContentAnalyzer
from reporting.meeting_notes import MeetingNotesGenerator
from typing import Dict

class MeetingNotesAgent:
    def __init__(
        self,
        whisper_model: str = "base",
        hf_token: str = None,
        llm_model: str = "gpt-4-turbo-preview",
    ):
        self.transcriber = WhisperTranscriber(whisper_model)
        self.diarizer = SpeakerDiarizer(hf_token) if hf_token else None
        self.analyzer = MeetingContentAnalyzer(llm_model)
        self.notes_gen = MeetingNotesGenerator()

    def process_meeting(
        self,
        audio_path: str,
        num_speakers: int = None,
        metadata: Dict = None,
    ) -> Dict:
        transcript = self.transcriber.transcribe(audio_path)
        segments = transcript["segments"]
        if self.diarizer:
            diarization = self.diarizer.diarize(audio_path, num_speakers)
            segments = self.diarizer.merge_with_transcript(diarization, segments)
        full_text = " ".join(s["text"] for s in segments)
        analysis = self.analyzer.analyze(full_text, num_speakers or 2)
        notes = self.notes_gen.generate_notes(analysis, segments, metadata)
        return {
            "transcript": transcript,
            "segments": segments,
            "analysis": analysis,
            "notes": notes,
            "duration_seconds": segments[-1]["end"] if segments else 0,
        }

Mathematical Foundation

Word Error Rate (WER):

Where each parameter means:

— substitutions
— deletions
— insertions
— total words in reference

Intuition: Lower WER indicates better transcription accuracy. Professional transcription targets <5% WER.

Speaker Diarization Error:

Intuition: Measures how accurately speakers are identified and segmented.

Testing & Evaluation

import pytest
from transcription.whisper_client import WhisperTranscriber

def test_transcription():
    transcriber = WhisperTranscriber("tiny")
    result = transcriber.transcribe("test_audio.mp3")
    assert "text" in result
    assert len(result["text"]) > 0

def test_format_time():
    transcriber = WhisperTranscriber("tiny")
    assert transcriber._format_time(65) == "01:05"
    assert transcriber._format_time(125) == "02:05"

Performance Metrics

Metric	Value	Notes
Transcription Speed	1x-5x realtime	Depends on model size
WER (base model)	5-10%	English, clear audio
Diarization Accuracy	85%+	2-5 speakers
Analysis Time	5-15s	GPT-4 per transcript
Notes Generation	3-8s	Structured output

Deployment

# main.py
from agent import MeetingNotesAgent
import os

def main():
    agent = MeetingNotesAgent(
        whisper_model="base",
        hf_token=os.getenv("HUGGINGFACE_TOKEN"),
    )
    audio_path = input("Audio file path: ").strip()
    print("Processing meeting...\n")
    result = agent.process_meeting(audio_path)
    print(result["notes"])
    with open("meeting_notes.md", "w") as f:
        f.write(result["notes"])
    print("\nNotes saved to meeting_notes.md")

if __name__ == "__main__":
    main()

Real-World Use Cases

Meeting Minutes: Automate meeting note-taking
Interview Analysis: Extract insights from interviews
Podcast Production: Generate show notes and transcripts
Legal Depositions: Transcribe and analyze testimony
Customer Calls: Extract action items from sales calls

Common Pitfalls & Solutions

Pitfall	Solution
Poor audio quality	Pre-process with noise reduction
Multiple accents	Use larger Whisper models
Overlapping speech	Diarization handles this natively
Long recordings	Split into chunks for processing
Speaker confusion	Provide speaker count hint

Summary with Key Takeaways

Whisper provides accurate, multilingual speech-to-text
Speaker diarization attributes text to specific speakers
LLM analysis extracts structured insights from transcripts
Meeting notes generation produces actionable minutes
Always validate transcriptions for critical meetings

Meeting Notes Agent with Whisper

Meeting Notes Agent with Whisper

What is an Audio Transcription Agent?

Project Overview

Architecture

Tools & Setup

Step 1: Environment Setup

Step 2: Project Structure

Step 3: Whisper Transcription

Step 4: Speaker Diarization

Step 5: Content Analysis and Reporting

Step 6: Complete Agent

Mathematical Foundation

Testing & Evaluation

Performance Metrics

Deployment

Real-World Use Cases

Common Pitfalls & Solutions

Summary with Key Takeaways

Need Expert AI Agents Help?