Benchmark

Local speech-to-text vs. cloud APIs on real meetings

No closed cloud speech-to-text API measurably beat LokalBot's local Qwen3-ASR 1.7B on real two-track meetings—and several cloud APIs failed in ways that are hard to catch in clean benchmarks.

Why test against cloud APIs

People often assume running transcription on a laptop means trading away accuracy for privacy. We tested LokalBot's default local models—Qwen3-ASR 1.7B (8-bit) and Parakeet TDT 0.6B v3 on an Apple M4 Max—head-to-head against direct API calls to OpenAI (gpt-transcribe, gpt-4o-transcribe, gpt-4o-mini-transcribe), Google (gemini-3.5-transcribe, gemini-3.8-flash), and Alibaba Model Studio (qwen-audio-3.1-asr-flash, qwen3-asr-flash).

Public AMI meetings (human reference)

Sixteen 90–120 second windows from four AMI Mix-Headset test meetings (27.4 minutes, 4,160 reference words), containing heavy overlapped speech on a single mixed channel:

AMI Mix-Headset WER comparison across cloud APIs and local models
SystemWER95% CILow-overlap WERList price / hrSpeed
Gemini 3.8 Flash (prompted)16.7%14.0–19.0%6.2%$0.2118×
Qwen3-ASR 1.7B (local)17.6%14.6–20.5%7.1%$044×
Qwen3-ASR Flash (API)18.1%14.6–20.8%6.3%$0.1325×
Qwen-Audio 3.1 ASR Flash18.5%14.6–21.8%5.6%$0.0210×
Parakeet TDT 0.6B v3 (local)19.2%15.8–22.3%8.1%$0129×
GPT-4o mini Transcribe21.8%17.5–25.7%7.8%$0.1834×
GPT Transcribe22.2%17.6–27.5%6.9%$0.2730×
GPT-4o Transcribe23.0%18.4–27.0%6.2%$0.3625×

Two-track real meetings: where cloud APIs broke

On 99 two-track chunks (69.9 minutes) from real meetings scored by cross-family ROVER consensus disagreement (measuring agreement, not human ground truth), local Qwen3-ASR 1.7B placed first at 4.2% disagreement (0.4 s median latency, 0 dropped spans), tied within confidence intervals with Gemini 3.5 Transcribe (4.5%, 6.3 s latency) and Qwen-Audio 3.1 ASR Flash (4.6%, 2.9 s latency).

Two failure modes jumped out immediately:

Data and reproduction

Full tables, ROVER methodology, and caveats are in Benchmarks/transcribe/CloudSTT/results/2026-09-30/REPORT.md and the follow-up span-length report. Try the same local models offline in LokalBot.

Try it locally

Meeting notes that stay on your computer by default.

Free, open source, and no account. For Mac, with Windows and Linux in alpha.

Download for macOS

Apple Silicon · macOS 15+ · or with Homebrew: brew install --cask stevyhacker/tap/lokalbot

macOS · Windows · Linux (alpha) · requirements