Key results
- Speech recognition: Granite Speech 4.1 (Q4) scored a 4.42% normalized word error rate on 44 public clips and processed 33× realtime once warm.
- Semantic search: Harrier 0.6B ranked the correct passage first for 40 of 48 queries (83.3%), against 35 of 48 for Qwen3-Embedding 0.6B. Harrier is now LokalBot's search model.
- Summaries: Qwen3.5 4B produced a recap and action list for a 26-minute meeting in 33 seconds with a warm runtime, and for a 91-minute meeting in 71–74 seconds.
- Autocomplete: LFM2.5 1.2B passed all 28 safety scenarios with a 143 ms average and about 490 ms p95 latency.
- Screenshot text: Apple Vision averaged 243 ms per real screenshot with 0.971 token F1 on the ground-truth set. Every open-source OCR engine tested was several times slower.
Every number comes from a report checked into the LokalBot repository. These are engine-level measurements on one machine, an Apple M4 Max MacBook Pro with 48 GB of unified memory. They are not end-to-end UI timings. The same data is summarized on the LokalBot benchmarks Space on Hugging Face.
Speech recognition accuracy and speed
The fixed set is 44 public clips totalling 271.3 seconds: 24 conversational clips from the AMI meeting corpus and 20 clean LibriSpeech clips. Every model received identical 16 kHz mono audio. Word error rate uses the standard Whisper English normalizer, so contractions such as “I've” and “I have” score the same. Run date: September 7, 2026.
| Measure | Granite 4.1 Q4 | Granite 4.1 Q8 | Granite 5 TurboCTC |
|---|---|---|---|
| Normalized word error rate | 4.42% | 4.93% | 5.18% |
| AMI meeting audio | 8.83% | 9.09% | 10.13% |
| LibriSpeech read speech | 0.25% | 0.99% | 0.49% |
| Audio seconds per second (warm) | 33.2× | 29.3× | 280.2× |
| Median clip latency | 161 ms | 187 ms | 17.5 ms |
| Sampled peak memory (RSS) | 2.87 GiB | 3.62 GiB | 1.05 GiB |
Granite Speech 5 was about 8× faster but English-only and unpunctuated, and it is not integrated into LokalBot. With a set this small, the accuracy differences do not establish a reliable ranking. Qwen3-ASR 1.7B, the default since LokalBot 0.8.1, was not part of this run. Parakeet and Whisper large-v3 turbo have not been scored against this set either. Source: model alternatives report.
Semantic search quality
Search was tested on 48 authored queries: 24 in English and 24 in Serbian/Montenegrin, across Latin and Cyrillic scripts. Queries ran against 110 passages, 86 of them distractors. The relevant passage for each query was fixed before inference. Both models used Q8 weights and 1,024-dimension vectors.
| Measure | Qwen3-Embedding 0.6B | Harrier 0.6B |
|---|---|---|
| Correct passage ranked first | 35/48 (72.9%) | 40/48 (83.3%) |
| Correct passage in top five | 45/48 (93.8%) | 47/48 (97.9%) |
| English, ranked first | 22/24 | 22/24 |
| Serbian/Montenegrin, ranked first | 13/24 | 18/24 |
| Mean reciprocal rank | 0.814 | 0.903 |
| Median query latency | 11.85 ms | 11.97 ms |
The gain came almost entirely from non-English queries. Forty-eight correlated queries are a pilot, not a general retrieval benchmark. Source: model alternatives report.
Meeting summary and action-item speed
This replay runs LokalBot's production summary and action extractor against two meeting transcripts with Qwen3.5 4B (Q4_K_M) on llama.cpp b10173. “Cold” includes starting the model runtime; “warm” reuses a loaded runtime. Transcription time is excluded. Run date: September 9, 2026.
| Meeting | Transcript | Cold | Warm | Model calls |
|---|---|---|---|---|
| 26-minute meeting | 4,477 words | 59.8 s | 33.4 s | 5 |
| 91-minute meeting | 11,229 words | 71.5 s | 74.2 s | 8 |
All four runs completed without truncated output. LokalBot's validator dropped notes and actions it could not tie to transcript evidence or to the right owner. That check covers evidence and ownership, not meaning, so factual coverage still needs human review. In the same September 7 pilot, the smaller MiniCPM5 2B model was faster on long context but misattributed action items and looped on one summary. Qwen3.5 4B remains the default. Sources: summary replay results and model alternatives report.
Autocomplete latency
The production --cotyping-bench harness runs 28 scenarios through the in-process llama.cpp engine. A model passes if it clears all 28 safety checks, completes at least 12 of 13 words, and keeps p95 latency at or below 2 seconds. Run date: July 21, 2026.
| Model | Safety | Word completions | Average | p95 | Result |
|---|---|---|---|---|---|
| LFM2.5 1.2B Instruct (Q4_K_M) | 28/28 | 12/13 | 143–151 ms | 484–494 ms | Pass ×3 |
| Gemma 4 E4B Instruct | 28/28 | 12/13 | 399 ms | 1,830 ms | Pass |
| Qwen3.5 2B | 27/28 | 11/13 | 437 ms | 1,629 ms | Fail (quality) |
| Qwen3.5 4B | 27/28 | 11/13 | 434 ms | 1,652 ms | Fail (quality) |
Cold Metal kernel compilation on the first scenario took about 19 seconds for LFM2.5 and about 127 seconds for Gemma 4 E4B. Source: autocomplete benchmark.
Screenshot text recognition
When you opt in to saving screen text, LokalBot reads it with Apple Vision. On 5 synthetic screenshots with known text, Apple Vision scored 0.971 token F1 at 120 ms per image. The closest open-source engine, PP-OCRv6 medium, scored 0.974 but took 6.78 seconds per image and about 15 seconds to load. On 15 real screenshots, Apple Vision averaged 243 ms per image; every alternative tested took 1.3 to 22 seconds. Run date: June 24, 2026. Sources: synthetic OCR results and real-screenshot OCR results.
What has not been measured yet
- Any Mac other than the M4 Max. M1, M2, and M3 results would change the timing picture.
- Word error rate for Qwen3-ASR, Parakeet, and Whisper large-v3 turbo.
- A published speaker-diarization error rate.
- Human-rated summary quality on a fixed corpus: faithfulness and action-item completeness.
- Noisy far-field microphones, most supported languages, and power or thermal behavior.
Model choices change between releases. Settings → Models in the app shows your active selections, download sizes, and presets. For choosing a speech engine, see the local transcription model guide.