🎉 75% of content is free forever — Unlock Premium from $10/mo →
CW
💼 Servicesℹ️ About✉️ ContactView Pricing Plansfrom $10

Meeting Notes Agent with Whisper

AI AgentsAudio Transcription Agent🟢 Free Lesson

Advertisement

Meeting Notes Agent with Whisper

Meeting Notes PipelineAudio InputMP3/WAV/OGGWhisperSpeech-to-TextDiarizationSpeaker IDSummarizerLLM AnalysisReporterMD/PDFAction ExtractorDecision TrackerSentimentMeeting Notes Orchestrator

Why This Matters

Professionals spend 15+ hours per week in meetings, yet most lack actionable notes. Manual note-taking distracts from participation and often misses critical details. An AI meeting notes agent automatically transcribes audio, identifies speakers, extracts action items, and generates structured meeting minutes—saving hours while ensuring nothing falls through the cracks.

Real-World Analogy

Think of an Audio Transcription Agent as a perfect secretary with photographic memory. Just as a skilled secretary records every word, identifies who said what, tracks action items, and produces polished meeting minutes, this agent processes audio recordings with perfect accuracy—never missing a detail, never misattributing a quote, and always producing clear, actionable summaries.

What is an Audio Transcription Agent?

Audio transcription agents convert meeting recordings into structured notes with speaker identification, action items, key decisions, and summaries. The key pipeline is: audio preprocessing → speech-to-text (Whisper) → speaker diarization → content analysis → structured output generation. Each step adds structure to the raw audio data.

Project Overview

We will build a meeting notes agent that:

  • Transcribes audio using OpenAI Whisper
  • Identifies different speakers (diarization)
  • Extracts action items and assignees
  • Identifies key decisions and discussion points
  • Generates concise meeting summaries
  • Outputs structured meeting notes

Expected outcome: An agent that produces meeting minutes from audio recordings.

Architecture

Meeting Notes ArchitectureAudio Preprocessor16kHz mono, normalizeWhisper TranscriberSpeech-to-text + timestampsSpeaker Diarizerpyannote embeddingsContent AnalyzerLLM extractionAction ExtractorTasks & ownersNotes GeneratorStructured outputMeeting Notes Orchestrator

Tools & Setup

ToolVersionPurpose
Python3.11+Core language
openai-whisper20231117Speech-to-text
pyannote.audio3.1+Speaker diarization
openai1.0+LLM backbone
pydub0.25+Audio manipulation

Step 1: Environment Setup

python -m venv venv
source venv/bin/activate
pip install openai-whisper pyannote.audio openai pydub
export OPENAI_API_KEY="sk-your-key"
export HUGGINGFACE_TOKEN="hf-your-token"

Step 2: Whisper Transcriber

import logging
from typing import Any, Dict, List, Optional

logger = logging.getLogger(__name__)


class WhisperTranscriber:
    """Transcribe audio files using OpenAI Whisper with timestamp support."""

    def __init__(self, model_size: str = "base"):
        import whisper
        self.model = whisper.load_model(model_size)

    def transcribe(self, audio_path: str, language: Optional[str] = None) -> Dict[str, Any]:
        options = {}
        if language:
            options["language"] = language
        result = self.model.transcribe(audio_path, **options)
        return {
            "text": result["text"],
            "segments": [
                {
                    "start": seg["start"],
                    "end": seg["end"],
                    "text": seg["text"],
                }
                for seg in result["segments"]
            ],
            "language": result.get("language", "unknown"),
        }

    def transcribe_with_timestamps(self, audio_path: str) -> str:
        result = self.model.transcribe(audio_path, word_timestamps=True)
        output = []
        for seg in result["segments"]:
            start = self._format_time(seg["start"])
            end = self._format_time(seg["end"])
            output.append(f"[{start} -> {end}] {seg['text'].strip()}")
        return "\n".join(output)

    def _format_time(self, seconds: float) -> str:
        mins = int(seconds // 60)
        secs = int(seconds % 60)
        return f"{mins:02d}:{secs:02d}"

Step 3: Speaker Diarization

from typing import Any, Dict, List, Optional


class SpeakerDiarizer:
    """Identify speakers in audio using pyannote diarization."""

    def __init__(self, token: str):
        from pyannote.audio import Pipeline
        self.pipeline = Pipeline.from_pretrained(
            "pyannote/speaker-diarization-3.1",
            use_auth_token=token,
        )

    def diarize(self, audio_path: str, num_speakers: Optional[int] = None) -> List[Dict[str, Any]]:
        kwargs = {}
        if num_speakers:
            kwargs["num_speakers"] = num_speakers
        diarization = self.pipeline(audio_path, **kwargs)
        segments = []
        for turn, _, speaker in diarization.itertracks(yield_label=True):
            segments.append({
                "start": turn.start,
                "end": turn.end,
                "speaker": speaker,
            })
        return segments

    def merge_with_transcript(
        self,
        diarization: List[Dict[str, Any]],
        transcript: List[Dict[str, Any]],
    ) -> List[Dict[str, Any]]:
        merged = []
        for seg in transcript:
            speaker = self._find_speaker(diarization, seg["start"])
            merged.append({**seg, "speaker": speaker})
        return merged

    def _find_speaker(self, diarization: List[Dict[str, Any]], timestamp: float) -> str:
        for d in diarization:
            if d["start"] <= timestamp <= d["end"]:
                return d["speaker"]
        return "Unknown"

Step 4: Content Analysis and Reporting

from openai import AsyncOpenAI
import json
from datetime import datetime
from typing import Any, Dict, List, Optional


class MeetingContentAnalyzer:
    """Analyze meeting transcripts for actions, decisions, and summaries."""

    def __init__(self, model: str = "gpt-4o"):
        self.client = AsyncOpenAI()
        self.model = model

    async def analyze(self, transcript: str, num_participants: int) -> Dict[str, Any]:
        response = await self.client.chat.completions.create(
            model=self.model,
            messages=[
                {
                    "role": "system",
                    "content": """Analyze this meeting transcript.
                    Return JSON:
                    {
                        "summary": "2-3 paragraph summary",
                        "key_decisions": [{"decision": "...", "context": "..."}],
                        "action_items": [{"action": "...", "assignee": "...", "deadline": "..."}],
                        "discussion_topics": ["topic1", "topic2"],
                        "follow_ups": ["items needing follow-up"],
                        "sentiment": "positive|neutral|negative",
                        "engagement_level": "high|medium|low"
                    }""",
                },
                {"role": "user", "content": f"Meeting transcript ({num_participants} participants):\n\n{transcript[:8000]}"},
            ],
            temperature=0.2,
        )
        try:
            return json.loads(response.choices[0].message.content)
        except json.JSONDecodeError:
            return {"summary": response.choices[0].message.content, "key_decisions": [], "action_items": []}


class MeetingNotesGenerator:
    """Generate formatted meeting notes from analysis results."""

    def generate_notes(
        self,
        analysis: Dict[str, Any],
        transcript_segments: List[Dict[str, Any]],
        metadata: Optional[Dict[str, Any]] = None,
    ) -> str:
        metadata = metadata or {}
        notes = f"# Meeting Notes\n\n"
        notes += f"**Date:** {metadata.get('date', datetime.now().strftime('%Y-%m-%d'))}\n"
        notes += f"**Duration:** {metadata.get('duration', 'N/A')}\n"
        notes += f"**Participants:** {metadata.get('participants', 'N/A')}\n\n"
        notes += "## Summary\n\n"
        notes += f"{analysis.get('summary', 'No summary available')}\n\n"
        notes += "## Key Decisions\n\n"
        for i, decision in enumerate(analysis.get("key_decisions", []), 1):
            notes += f"{i}. **{decision.get('decision', 'N/A')}**\n"
            notes += f"   - Context: {decision.get('context', 'N/A')}\n"
        notes += "\n## Action Items\n\n"
        notes += "| Action | Assignee | Deadline |\n"
        notes += "|--------|----------|----------|\n"
        for item in analysis.get("action_items", []):
            notes += f"| {item.get('action', 'N/A')} | {item.get('assignee', 'TBD')} | {item.get('deadline', 'TBD')} |\n"
        notes += "\n## Discussion Topics\n\n"
        for topic in analysis.get("discussion_topics", []):
            notes += f"- {topic}\n"
        notes += "\n## Follow-ups\n\n"
        for follow in analysis.get("follow_ups", []):
            notes += f"- {follow}\n"
        return notes

Step 5: Complete Agent

class MeetingNotesAgent:
    """Orchestrate the complete meeting notes pipeline."""

    def __init__(
        self,
        whisper_model: str = "base",
        hf_token: Optional[str] = None,
        llm_model: str = "gpt-4o",
    ):
        self.transcriber = WhisperTranscriber(whisper_model)
        self.diarizer = SpeakerDiarizer(hf_token) if hf_token else None
        self.analyzer = MeetingContentAnalyzer(llm_model)
        self.notes_gen = MeetingNotesGenerator()

    async def process_meeting(
        self,
        audio_path: str,
        num_speakers: Optional[int] = None,
        metadata: Optional[Dict[str, Any]] = None,
    ) -> Dict[str, Any]:
        transcript = self.transcriber.transcribe(audio_path)
        segments = transcript["segments"]
        if self.diarizer:
            diarization = self.diarizer.diarize(audio_path, num_speakers)
            segments = self.diarizer.merge_with_transcript(diarization, segments)
        full_text = " ".join(s["text"] for s in segments)
        analysis = await self.analyzer.analyze(full_text, num_speakers or 2)
        notes = self.notes_gen.generate_notes(analysis, segments, metadata)
        return {
            "transcript": transcript,
            "segments": segments,
            "analysis": analysis,
            "notes": notes,
            "duration_seconds": segments[-1]["end"] if segments else 0,
        }

Mathematical Foundation

Word Error Rate (WER):

Where = substitutions, = deletions, = insertions, = total reference words. Professional transcription targets <5% WER.

Speaker Diarization Error (DER):

State-of-the-art DER is ~10% for 2-5 speaker meetings.

Performance Considerations

MetricLatencyCostAccuracy
Transcription (base model)1-5x realtime~$0.001/min (local)5-10% WER
Diarization0.5-2x realtimeFree (local)85%+ accuracy
Content analysis5-15s$0.05-0.15High
Notes generation3-8s$0.02-0.05High
Full meeting (1 hour)5-15 min$0.07-0.20High

Security Considerations

  • Encrypt audio files at rest and in transit
  • Store HuggingFace tokens securely in environment variables
  • Process sensitive meetings locally when possible
  • Implement access controls for meeting notes
  • Log all processing for audit trails
  • Implement data retention policies for audio files
  • Consider GDPR implications for recording employees

Interview Q&A

Q1: What is the difference between Whisper model sizes (tiny vs. large)?

Tiny models (~39M params) process audio 10x faster but with higher WER (~10-15%). Large models (~1.5B params) achieve WER <5% but require GPU and are slower. Base model offers the best speed/accuracy tradeoff for most meetings.

Q2: How does speaker diarization work with overlapping speech?

pyannote uses neural embeddings (ECAPA-TDNN) to distinguish speakers, then applies clustering. Overlapping speech is handled by assigning frames to multiple speakers simultaneously. Accuracy drops from 90% to ~70% with heavy overlap.

Q3: How would you handle meetings with 10+ speakers?

Pre-specify num_speakers parameter to improve accuracy. Use larger Whisper models for better speaker separation. Consider chunking audio into segments and processing in parallel. Post-process by merging consecutive segments from the same speaker.

Q4: What preprocessing improves transcription quality?

Convert to 16kHz mono WAV format, apply noise reduction (RNNoise), normalize audio levels, remove silence segments. These steps can improve WER by 2-5 percentage points, especially in noisy environments.

Q5: How do you handle multilingual meetings?

Use Whisper's auto language detection, or specify language if known. For code-switching meetings, use the base model which handles mixed languages better. For pure language meetings, specify the language parameter for 10-20% WER improvement.

Q6: What is the cost structure for processing a 1-hour meeting?

Whisper base model: ~0.05-0.15 depending on transcript length. Total: ~$0.05-0.15 per hour of audio.

Q7: How do you handle background noise and poor audio quality?

Apply spectral subtraction for noise reduction, use Whisper's condition_on_previous_text=True for context, increase beam_size to 5 for better accuracy, and use the large model which is more robust to noise.

Q8: How would you build a real-time meeting transcription system?

Use streaming Whisper for real-time transcription, implement VAD (Voice Activity Detection) for chunking, buffer 30-second windows for diarization, and use WebSocket for live updates. Target latency <500ms for near-real-time experience.

Common Pitfalls & Solutions

PitfallSolution
Poor audio qualityApply noise reduction and normalize levels before transcription
Multiple accentsUse larger Whisper models (large-v3) for better accent handling
Overlapping speechDiarization handles this natively; pre-specify speaker count
Long recordingsSplit into 30-minute chunks for processing
Speaker confusionProvide speaker count hint and use word-level timestamps
Memory overflowProcess audio in chunks with streaming
Language detection failuresManually specify language for known meetings

Knowledge Check

Q1: What does WER measure in speech recognition? A) Audio quality B) Word Error Rate comparing to reference C) Speaker accuracy D) Processing speed

AnswerB) Word Error Rate comparing transcription to reference—combines substitutions, deletions, and insertions.

Q2: Which Whisper model size offers the best speed/accuracy tradeoff? A) Tiny B) Base C) Large D) Turbo

AnswerB) Base—provides ~90% of large model accuracy at 5x the speed.

Q3: What is the primary purpose of speaker diarization? A) Transcribing audio B) Identifying who spoke when C) Reducing noise D) Compressing files

AnswerB) Identifying who spoke when in the recording—segments audio by speaker identity.

Q4: How does specifying num_speakers improve diarization? A) Speeds processing B) Constrains clustering for better accuracy C) Reduces memory D) Enables parallelism

AnswerB) Constrains the clustering algorithm for better accuracy—prevents over/under-segmentation.

Q5: What audio format is recommended for Whisper? A) MP3 at 320kbps B) 16kHz mono WAV C) FLAC at 44.1kHz D) OGG at 128kbps

AnswerB) 16kHz mono WAV—Whisper's training data uses this format for optimal performance.

Q6: What is the typical WER for Whisper base model on clean English audio? A) 1-2% B) 5-10% C) 15-20% D) 25-30%

AnswerB) 5-10%—expected error rate for clear, well-recorded English speech.

Summary with Key Takeaways

  • Whisper provides accurate, multilingual speech-to-text with timestamp support
  • Speaker diarization attributes text to specific speakers using neural embeddings
  • LLM analysis extracts structured insights (actions, decisions, topics) from transcripts
  • Meeting notes generation produces actionable minutes in markdown format
  • Always validate transcriptions for critical meetings using WER benchmarks
  • Audio preprocessing (16kHz, mono, normalized) directly improves transcription quality

Need Expert AI Agents Help?

Get personalized tutoring, project support, or professional consulting.

Advertisement