Key results
- Speech recognition: Granite Speech 4.1 (Q4) scored a 4.42% normalized word error rate on 44 public clips and processed 33× realtime once warm.
- Speaker diarization: Nemotron 3 cut the diarization error rate from 43.4% to 14.6% on 8 public AMI recordings, found the right number of speakers in all 8, and ran 500× realtime. It is now the default speaker model on new installs.
- Semantic search: EmbeddingGemma 2 ranked the correct passage first for 43 of 48 queries (89.6%), against 40 of 48 for Harrier 0.6B, with half the download and faster queries. EmbeddingGemma 2 is now LokalBot's search model.
- Summaries: Qwen3.5 4B produced a recap and action list for a 26-minute meeting in 33 seconds with a warm runtime, and for a 91-minute meeting in 71–74 seconds.
- Autocomplete: Gemma 4 E2B Base, the default, made no unsafe suggestions in 140 checks. Half of the suggestions it showed were the exact next word, and it took 56 ms median and 102 ms p95 per keystroke.
- Screenshot text: Apple Vision averaged 243 ms per real screenshot with 0.971 token F1 on the ground-truth set. Every open-source OCR engine tested was several times slower.
Every number comes from a report checked into the LokalBot repository. These are engine-level measurements on one machine, an Apple M4 Max MacBook Pro with 48 GB of unified memory. They are not end-to-end UI timings.
Speech recognition accuracy and speed
The fixed set is 44 public clips totalling 271.3 seconds: 24 conversational clips from the AMI meeting corpus and 20 clean LibriSpeech clips. Every model received identical 16 kHz mono audio. Word error rate uses the standard Whisper English normalizer, so contractions such as “I've” and “I have” score the same. Run date: September 7, 2026.
| Measure | Granite 4.1 Q4 | Granite 4.1 Q8 | Granite 5 TurboCTC |
|---|---|---|---|
| Normalized word error rate | 4.42% | 4.93% | 5.18% |
| AMI meeting audio | 8.83% | 9.09% | 10.13% |
| LibriSpeech read speech | 0.25% | 0.99% | 0.49% |
| Audio seconds per second (warm) | 33.2× | 29.3× | 280.2× |
| Median clip latency | 161 ms | 187 ms | 17.5 ms |
| Sampled peak memory (RSS) | 2.87 GiB | 3.62 GiB | 1.05 GiB |
Granite Speech 5 was about 8× faster but English-only and unpunctuated, and it is not integrated into LokalBot. With a set this small, the accuracy differences do not establish a reliable ranking. Qwen3-ASR 1.7B, the default since LokalBot 0.8.1, was not part of this run. Parakeet and Whisper large-v3 turbo have not been scored against this set either. Source: model alternatives report.
Speaker diarization
Diarization decides who spoke when. The test set is four English AMI meetings, each scored as a headset mix and as a distant-microphone recording: 8 conditions and 184.55 minutes of audio, with four speakers in every meeting. Diarization error rate (DER) adds missed speech, false alarms, and speaker confusion, divided by reference speaker time. Overlapping speech is scored, the main comparison uses no boundary collar, and no model was told how many speakers to expect. Run date: September 23, 2026.
| Measure | Pyannote Community-1 (previous default) | Nemotron 3 |
|---|---|---|
| DER, no collar | 43.44% | 14.56% |
| DER, 250 ms collar | 30.45% | 8.34% |
| Correct speaker count | 5/8 | 8/8 |
| Speed, 17.5-minute recording (warm) | 243× realtime | 500× realtime |
| Peak process memory (RSS) | 0.68 GiB | 0.67 GiB |
Nemotron 3 ran as full-precision Core ML through FluidAudio. It reduced speaker confusion from 8.87 to 1.77 points, missed speech from 15.34 to 8.40, and false alarms from 19.23 to 4.38, and it beat the baseline on all eight recordings. The baseline is the pinned FluidAudio implementation of Pyannote Community-1; keeping overlapping output or switching to community defaults did not close the gap. That makes this a comparison with one implementation, not with every Pyannote setup. Pyannote Community-1 remains selectable in the app. Source: diarization report.
Semantic search quality
Search was tested on 48 authored queries: 24 in English and 24 in Serbian/Montenegrin, across Latin and Cyrillic scripts, against 60 synthetic passages. The relevant passage for each query was fixed before inference. Both models used Q8 weights on the same llama.cpp build; Harrier returns 1,024-dimension vectors and EmbeddingGemma 2 returns 768.
| Measure | Harrier 0.6B | EmbeddingGemma 2 |
|---|---|---|
| Correct passage ranked first | 40/48 (83.3%) | 43/48 (89.6%) |
| Correct passage in top five | 47/48 (97.9%) | 48/48 (100%) |
| English, ranked first | 22/24 | 23/24 |
| Serbian/Montenegrin, ranked first | 18/24 | 20/24 |
| Mean reciprocal rank | 0.903 | 0.944 |
| Median query latency | 13.1 ms | 9.0 ms |
| Model download | 0.64 GB | 0.31 GB |
On 48 correlated queries the 6-point gain is within noise (95% interval −4 to +17 points), so EmbeddingGemma 2 was adopted because it matched or beat Harrier at half the download, half the idle memory, and faster queries, not for a proven quality jump. Harrier had replaced Qwen3-Embedding 0.6B (35/48) in September. Sources: EmbeddingGemma 2 report, model alternatives report.
Meeting summary and action-item speed
This replay runs LokalBot's production summary and action extractor against two meeting transcripts with Qwen3.5 4B (Q4_K_M) on llama.cpp b10173. “Cold” includes starting the model runtime; “warm” reuses a loaded runtime. Transcription time is excluded. Run date: September 9, 2026.
| Meeting | Transcript | Cold | Warm | Model calls |
|---|---|---|---|---|
| 26-minute meeting | 4,477 words | 59.8 s | 33.4 s | 5 |
| 91-minute meeting | 11,229 words | 71.5 s | 74.2 s | 8 |
All four runs completed without truncated output. LokalBot's validator dropped notes and actions it could not tie to transcript evidence or to the right owner. That check covers evidence and ownership, not meaning, so factual coverage still needs human review. In the same September 7 pilot, the smaller MiniCPM5 2B model was faster on long context but misattributed action items and looped on one summary. Qwen3.5 4B remains the default. Sources: summary replay results and model alternatives report.
Autocomplete
The autocomplete benchmark runs LokalBot's production autocomplete engine on one model, with fresh-install settings and nothing else from the app. It has three parts:
- Safety checks: 28 scenarios, run 5 times. A result is unsafe if it would damage the text: a leaked prompt, a split or corrupted word, or text where none belongs.
- Next-word accuracy: 788 word positions in 96 original phrases, in English, Serbian, and four other languages.
- Typing latency: per-keystroke time while typing 8 of those phrases one character at a time.
Run date: October 15, 2026, on a Release build.
| Measure | Gemma 4 E2B Base (Q6_K), default | LFM2.5 1.2B Instruct (Q4_K_M), Lightweight |
|---|---|---|
| Unsafe results in 140 checks | 0 | 0 |
| Next-word accuracy | 23.1% | 20.7% |
| Accuracy in English / other languages | 25.3% / 14.6% | 24.2% / 7.3% |
| Suggestion shown | 46% | 48% |
| Right when shown | 50% | 43% |
| Word completions | 12/13 | 12/13 |
| Per keystroke, median / p95 | 56 / 102 ms | 26 / 76 ms |
| Model load | 0.65 s | 0.16 s |
Accuracy counts a missing suggestion as a miss. When unsure of the first word, LokalBot shows nothing rather than guessing, so “right when shown” is closer to what you see while typing.
Gemma 4 E2B Base's suggestions are right more often, and it suggests in Serbian, where LFM2.5 almost never does. In English the two are within noise: the difference is +1.1 points, with a 95% interval of −1.4 to +3.5. LFM2.5 is two to three times faster, and both stay near 100 ms or less per keystroke on this Mac.
Both models showed a suggestion in 24 of the 28 checks:
- In 3 open-ended continuations, the confidence gate held a guess back.
- In 1, inside a word, staying quiet is the expected answer.
These are engine timings. They leave out the pause before a suggestion is requested, reading the text field, and drawing the suggestion. The phrases are synthetic and were written for earlier LokalBot studies. Source: autocomplete engine benchmark.
Earlier versions of this check reported model loading as latency. They also counted a held-back suggestion as a safety failure. Their July and October 14 results are kept in the repository.
Screenshot text recognition
When you opt in to saving screen text, LokalBot reads it with Apple Vision. On 5 synthetic screenshots with known text, Apple Vision scored 0.971 token F1 at 120 ms per image. The closest open-source engine, PP-OCRv6 medium, scored 0.974 but took 6.78 seconds per image and about 15 seconds to load. On 15 real screenshots, Apple Vision averaged 243 ms per image; every alternative tested took 1.3 to 22 seconds. Run date: June 24, 2026. Sources: synthetic OCR results and real-screenshot OCR results.
What has not been measured yet
- Any Mac other than the M4 Max. M1, M2, and M3 results would change the timing picture.
- The Windows and Linux app (alpha), whose engines differ: Qwen3-ASR 0.6B through sherpa-onnx, pyannote and TitaNet for speakers, ocrs for screen text, and llama.cpp on a Vulkan GPU or the CPU.
- Word error rate for Qwen3-ASR, Parakeet, and Whisper large-v3 turbo.
- Speaker diarization beyond four-speaker English meetings.
- Human-rated summary quality on a fixed corpus: faithfulness and action-item completeness.
- Noisy far-field microphones, most supported languages, and power or thermal behavior.
Model choices change between releases. Settings → Models in the app shows your active selections, download sizes, and presets. For choosing a speech engine, see the local transcription model guide.