Why test against cloud APIs
People often assume running transcription on a laptop means trading away accuracy for privacy. We tested LokalBot's default local models—Qwen3-ASR 1.7B (8-bit) and Parakeet TDT 0.6B v3 on an Apple M4 Max—head-to-head against direct API calls to OpenAI (gpt-transcribe, gpt-4o-transcribe, gpt-4o-mini-transcribe), Google (gemini-3.5-transcribe, gemini-3.8-flash), and Alibaba Model Studio (qwen-audio-3.1-asr-flash, qwen3-asr-flash).
Public AMI meetings (human reference)
Sixteen 90–120 second windows from four AMI Mix-Headset test meetings (27.4 minutes, 4,160 reference words), containing heavy overlapped speech on a single mixed channel:
| System | WER | 95% CI | Low-overlap WER | List price / hr | Speed |
|---|---|---|---|---|---|
| Gemini 3.8 Flash (prompted) | 16.7% | 14.0–19.0% | 6.2% | $0.21 | 18× |
| Qwen3-ASR 1.7B (local) | 17.6% | 14.6–20.5% | 7.1% | $0 | 44× |
| Qwen3-ASR Flash (API) | 18.1% | 14.6–20.8% | 6.3% | $0.13 | 25× |
| Qwen-Audio 3.1 ASR Flash | 18.5% | 14.6–21.8% | 5.6% | $0.02 | 10× |
| Parakeet TDT 0.6B v3 (local) | 19.2% | 15.8–22.3% | 8.1% | $0 | 129× |
| GPT-4o mini Transcribe | 21.8% | 17.5–25.7% | 7.8% | $0.18 | 34× |
| GPT Transcribe | 22.2% | 17.6–27.5% | 6.9% | $0.27 | 30× |
| GPT-4o Transcribe | 23.0% | 18.4–27.0% | 6.2% | $0.36 | 25× |
Two-track real meetings: where cloud APIs broke
On 99 two-track chunks (69.9 minutes) from real meetings scored by cross-family ROVER consensus disagreement (measuring agreement, not human ground truth), local Qwen3-ASR 1.7B placed first at 4.2% disagreement (0.4 s median latency, 0 dropped spans), tied within confidence intervals with Gemini 3.5 Transcribe (4.5%, 6.3 s latency) and Qwen-Audio 3.1 ASR Flash (4.6%, 2.9 s latency).
Two failure modes jumped out immediately:
- Gemini 3.8 Flash invented speech on quiet clips. Despite leading AMI WER by 0.9 points, on real meeting chunks it hallucinated 115 words across 15 spans—including turning a 1.2-second clip that every other system heard as one word or silence into a fluent 30-word sentence.
- GPT-4o Transcribe silently dropped microphone speech. It logged 278 dropped words in runs of four or more and 42% disagreement on one meeting's microphone track (vs. 7–14% for the other systems). Only ~20% of those dropped words appeared on the system track, ruling out echo suppression.
- Span length matters inside our own app. The test also caught a bug in LokalBot's older production path: cutting audio at speaker turns before transcription hurt AMI WER (26.5% vs. 17.7%). Assigning speakers after transcribing longer spans closed that gap.
Data and reproduction
Full tables, ROVER methodology, and caveats are in Benchmarks/transcribe/CloudSTT/results/2026-09-30/REPORT.md and the follow-up span-length report. Try the same local models offline in LokalBot.