Meeting Notes Agent with Whisper
Why This Matters
Professionals spend 15+ hours per week in meetings, yet most lack actionable notes. Manual note-taking distracts from participation and often misses critical details. An AI meeting notes agent automatically transcribes audio, identifies speakers, extracts action items, and generates structured meeting minutes—saving hours while ensuring nothing falls through the cracks.
Real-World Analogy
Think of an Audio Transcription Agent as a perfect secretary with photographic memory. Just as a skilled secretary records every word, identifies who said what, tracks action items, and produces polished meeting minutes, this agent processes audio recordings with perfect accuracy—never missing a detail, never misattributing a quote, and always producing clear, actionable summaries.
What is an Audio Transcription Agent?
Audio transcription agents convert meeting recordings into structured notes with speaker identification, action items, key decisions, and summaries. The key pipeline is: audio preprocessing → speech-to-text (Whisper) → speaker diarization → content analysis → structured output generation. Each step adds structure to the raw audio data.
Project Overview
We will build a meeting notes agent that:
- Transcribes audio using OpenAI Whisper
- Identifies different speakers (diarization)
- Extracts action items and assignees
- Identifies key decisions and discussion points
- Generates concise meeting summaries
- Outputs structured meeting notes
Expected outcome: An agent that produces meeting minutes from audio recordings.
Architecture
Tools & Setup
| Tool | Version | Purpose |
|---|---|---|
| Python | 3.11+ | Core language |
| openai-whisper | 20231117 | Speech-to-text |
| pyannote.audio | 3.1+ | Speaker diarization |
| openai | 1.0+ | LLM backbone |
| pydub | 0.25+ | Audio manipulation |
Step 1: Environment Setup
python -m venv venv
source venv/bin/activate
pip install openai-whisper pyannote.audio openai pydub
export OPENAI_API_KEY="sk-your-key"
export HUGGINGFACE_TOKEN="hf-your-token"
Step 2: Whisper Transcriber
import logging
from typing import Any, Dict, List, Optional
logger = logging.getLogger(__name__)
class WhisperTranscriber:
"""Transcribe audio files using OpenAI Whisper with timestamp support."""
def __init__(self, model_size: str = "base"):
import whisper
self.model = whisper.load_model(model_size)
def transcribe(self, audio_path: str, language: Optional[str] = None) -> Dict[str, Any]:
options = {}
if language:
options["language"] = language
result = self.model.transcribe(audio_path, **options)
return {
"text": result["text"],
"segments": [
{
"start": seg["start"],
"end": seg["end"],
"text": seg["text"],
}
for seg in result["segments"]
],
"language": result.get("language", "unknown"),
}
def transcribe_with_timestamps(self, audio_path: str) -> str:
result = self.model.transcribe(audio_path, word_timestamps=True)
output = []
for seg in result["segments"]:
start = self._format_time(seg["start"])
end = self._format_time(seg["end"])
output.append(f"[{start} -> {end}] {seg['text'].strip()}")
return "\n".join(output)
def _format_time(self, seconds: float) -> str:
mins = int(seconds // 60)
secs = int(seconds % 60)
return f"{mins:02d}:{secs:02d}"
Step 3: Speaker Diarization
from typing import Any, Dict, List, Optional
class SpeakerDiarizer:
"""Identify speakers in audio using pyannote diarization."""
def __init__(self, token: str):
from pyannote.audio import Pipeline
self.pipeline = Pipeline.from_pretrained(
"pyannote/speaker-diarization-3.1",
use_auth_token=token,
)
def diarize(self, audio_path: str, num_speakers: Optional[int] = None) -> List[Dict[str, Any]]:
kwargs = {}
if num_speakers:
kwargs["num_speakers"] = num_speakers
diarization = self.pipeline(audio_path, **kwargs)
segments = []
for turn, _, speaker in diarization.itertracks(yield_label=True):
segments.append({
"start": turn.start,
"end": turn.end,
"speaker": speaker,
})
return segments
def merge_with_transcript(
self,
diarization: List[Dict[str, Any]],
transcript: List[Dict[str, Any]],
) -> List[Dict[str, Any]]:
merged = []
for seg in transcript:
speaker = self._find_speaker(diarization, seg["start"])
merged.append({**seg, "speaker": speaker})
return merged
def _find_speaker(self, diarization: List[Dict[str, Any]], timestamp: float) -> str:
for d in diarization:
if d["start"] <= timestamp <= d["end"]:
return d["speaker"]
return "Unknown"
Step 4: Content Analysis and Reporting
from openai import AsyncOpenAI
import json
from datetime import datetime
from typing import Any, Dict, List, Optional
class MeetingContentAnalyzer:
"""Analyze meeting transcripts for actions, decisions, and summaries."""
def __init__(self, model: str = "gpt-4o"):
self.client = AsyncOpenAI()
self.model = model
async def analyze(self, transcript: str, num_participants: int) -> Dict[str, Any]:
response = await self.client.chat.completions.create(
model=self.model,
messages=[
{
"role": "system",
"content": """Analyze this meeting transcript.
Return JSON:
{
"summary": "2-3 paragraph summary",
"key_decisions": [{"decision": "...", "context": "..."}],
"action_items": [{"action": "...", "assignee": "...", "deadline": "..."}],
"discussion_topics": ["topic1", "topic2"],
"follow_ups": ["items needing follow-up"],
"sentiment": "positive|neutral|negative",
"engagement_level": "high|medium|low"
}""",
},
{"role": "user", "content": f"Meeting transcript ({num_participants} participants):\n\n{transcript[:8000]}"},
],
temperature=0.2,
)
try:
return json.loads(response.choices[0].message.content)
except json.JSONDecodeError:
return {"summary": response.choices[0].message.content, "key_decisions": [], "action_items": []}
class MeetingNotesGenerator:
"""Generate formatted meeting notes from analysis results."""
def generate_notes(
self,
analysis: Dict[str, Any],
transcript_segments: List[Dict[str, Any]],
metadata: Optional[Dict[str, Any]] = None,
) -> str:
metadata = metadata or {}
notes = f"# Meeting Notes\n\n"
notes += f"**Date:** {metadata.get('date', datetime.now().strftime('%Y-%m-%d'))}\n"
notes += f"**Duration:** {metadata.get('duration', 'N/A')}\n"
notes += f"**Participants:** {metadata.get('participants', 'N/A')}\n\n"
notes += "## Summary\n\n"
notes += f"{analysis.get('summary', 'No summary available')}\n\n"
notes += "## Key Decisions\n\n"
for i, decision in enumerate(analysis.get("key_decisions", []), 1):
notes += f"{i}. **{decision.get('decision', 'N/A')}**\n"
notes += f" - Context: {decision.get('context', 'N/A')}\n"
notes += "\n## Action Items\n\n"
notes += "| Action | Assignee | Deadline |\n"
notes += "|--------|----------|----------|\n"
for item in analysis.get("action_items", []):
notes += f"| {item.get('action', 'N/A')} | {item.get('assignee', 'TBD')} | {item.get('deadline', 'TBD')} |\n"
notes += "\n## Discussion Topics\n\n"
for topic in analysis.get("discussion_topics", []):
notes += f"- {topic}\n"
notes += "\n## Follow-ups\n\n"
for follow in analysis.get("follow_ups", []):
notes += f"- {follow}\n"
return notes
Step 5: Complete Agent
class MeetingNotesAgent:
"""Orchestrate the complete meeting notes pipeline."""
def __init__(
self,
whisper_model: str = "base",
hf_token: Optional[str] = None,
llm_model: str = "gpt-4o",
):
self.transcriber = WhisperTranscriber(whisper_model)
self.diarizer = SpeakerDiarizer(hf_token) if hf_token else None
self.analyzer = MeetingContentAnalyzer(llm_model)
self.notes_gen = MeetingNotesGenerator()
async def process_meeting(
self,
audio_path: str,
num_speakers: Optional[int] = None,
metadata: Optional[Dict[str, Any]] = None,
) -> Dict[str, Any]:
transcript = self.transcriber.transcribe(audio_path)
segments = transcript["segments"]
if self.diarizer:
diarization = self.diarizer.diarize(audio_path, num_speakers)
segments = self.diarizer.merge_with_transcript(diarization, segments)
full_text = " ".join(s["text"] for s in segments)
analysis = await self.analyzer.analyze(full_text, num_speakers or 2)
notes = self.notes_gen.generate_notes(analysis, segments, metadata)
return {
"transcript": transcript,
"segments": segments,
"analysis": analysis,
"notes": notes,
"duration_seconds": segments[-1]["end"] if segments else 0,
}
Mathematical Foundation
Word Error Rate (WER):
Where = substitutions, = deletions, = insertions, = total reference words. Professional transcription targets <5% WER.
Speaker Diarization Error (DER):
State-of-the-art DER is ~10% for 2-5 speaker meetings.
Performance Considerations
| Metric | Latency | Cost | Accuracy |
|---|---|---|---|
| Transcription (base model) | 1-5x realtime | ~$0.001/min (local) | 5-10% WER |
| Diarization | 0.5-2x realtime | Free (local) | 85%+ accuracy |
| Content analysis | 5-15s | $0.05-0.15 | High |
| Notes generation | 3-8s | $0.02-0.05 | High |
| Full meeting (1 hour) | 5-15 min | $0.07-0.20 | High |
Security Considerations
- Encrypt audio files at rest and in transit
- Store HuggingFace tokens securely in environment variables
- Process sensitive meetings locally when possible
- Implement access controls for meeting notes
- Log all processing for audit trails
- Implement data retention policies for audio files
- Consider GDPR implications for recording employees
Interview Q&A
Q1: What is the difference between Whisper model sizes (tiny vs. large)?
Tiny models (~39M params) process audio 10x faster but with higher WER (~10-15%). Large models (~1.5B params) achieve WER <5% but require GPU and are slower. Base model offers the best speed/accuracy tradeoff for most meetings.
Q2: How does speaker diarization work with overlapping speech?
pyannote uses neural embeddings (ECAPA-TDNN) to distinguish speakers, then applies clustering. Overlapping speech is handled by assigning frames to multiple speakers simultaneously. Accuracy drops from 90% to ~70% with heavy overlap.
Q3: How would you handle meetings with 10+ speakers?
Pre-specify num_speakers parameter to improve accuracy. Use larger Whisper models for better speaker separation. Consider chunking audio into segments and processing in parallel. Post-process by merging consecutive segments from the same speaker.
Q4: What preprocessing improves transcription quality?
Convert to 16kHz mono WAV format, apply noise reduction (RNNoise), normalize audio levels, remove silence segments. These steps can improve WER by 2-5 percentage points, especially in noisy environments.
Q5: How do you handle multilingual meetings?
Use Whisper's auto language detection, or specify language if known. For code-switching meetings, use the base model which handles mixed languages better. For pure language meetings, specify the language parameter for 10-20% WER improvement.
Q6: What is the cost structure for processing a 1-hour meeting?
Whisper base model: ~0.05-0.15 depending on transcript length. Total: ~$0.05-0.15 per hour of audio.
Q7: How do you handle background noise and poor audio quality?
Apply spectral subtraction for noise reduction, use Whisper's condition_on_previous_text=True for context, increase beam_size to 5 for better accuracy, and use the large model which is more robust to noise.
Q8: How would you build a real-time meeting transcription system?
Use streaming Whisper for real-time transcription, implement VAD (Voice Activity Detection) for chunking, buffer 30-second windows for diarization, and use WebSocket for live updates. Target latency <500ms for near-real-time experience.
Common Pitfalls & Solutions
| Pitfall | Solution |
|---|---|
| Poor audio quality | Apply noise reduction and normalize levels before transcription |
| Multiple accents | Use larger Whisper models (large-v3) for better accent handling |
| Overlapping speech | Diarization handles this natively; pre-specify speaker count |
| Long recordings | Split into 30-minute chunks for processing |
| Speaker confusion | Provide speaker count hint and use word-level timestamps |
| Memory overflow | Process audio in chunks with streaming |
| Language detection failures | Manually specify language for known meetings |
Knowledge Check
Q1: What does WER measure in speech recognition? A) Audio quality B) Word Error Rate comparing to reference C) Speaker accuracy D) Processing speed
Answer
B) Word Error Rate comparing transcription to reference—combines substitutions, deletions, and insertions.Q2: Which Whisper model size offers the best speed/accuracy tradeoff? A) Tiny B) Base C) Large D) Turbo
Answer
B) Base—provides ~90% of large model accuracy at 5x the speed.Q3: What is the primary purpose of speaker diarization? A) Transcribing audio B) Identifying who spoke when C) Reducing noise D) Compressing files
Answer
B) Identifying who spoke when in the recording—segments audio by speaker identity.Q4: How does specifying num_speakers improve diarization?
A) Speeds processing B) Constrains clustering for better accuracy C) Reduces memory D) Enables parallelism
Answer
B) Constrains the clustering algorithm for better accuracy—prevents over/under-segmentation.Q5: What audio format is recommended for Whisper? A) MP3 at 320kbps B) 16kHz mono WAV C) FLAC at 44.1kHz D) OGG at 128kbps
Answer
B) 16kHz mono WAV—Whisper's training data uses this format for optimal performance.Q6: What is the typical WER for Whisper base model on clean English audio? A) 1-2% B) 5-10% C) 15-20% D) 25-30%
Answer
B) 5-10%—expected error rate for clear, well-recorded English speech.Summary with Key Takeaways
- Whisper provides accurate, multilingual speech-to-text with timestamp support
- Speaker diarization attributes text to specific speakers using neural embeddings
- LLM analysis extracts structured insights (actions, decisions, topics) from transcripts
- Meeting notes generation produces actionable minutes in markdown format
- Always validate transcriptions for critical meetings using WER benchmarks
- Audio preprocessing (16kHz, mono, normalized) directly improves transcription quality