Measuring autocomplete without fooling ourselves
Our original 28-scenario autocomplete check had five flaws: it treated an intentional confidence-gate silence as a safety failure, accepted only one exact word tail for partial-word completions, mixed 10–40 s cold startup into steady-state latency, stole keyboard focus by launching the full app, and never measured next-word accuracy.
We replaced it with a headless engine runner (--cotyping-bench) that exercises the exact production LlamaCotypingRuntime with hybrid decoding, certified prefix search, and prefix healing across 28 safety/behaviour checks (5 deterministic passes) plus 96 phrases replayed at all 788 word boundaries and 8 phrases typed one character at a time.
Gemma 4 E2B Base vs. LFM2.5 1.2B Instruct (Apple M4 Max)
| Metric | Gemma 4 E2B Base (Balanced) | LFM2.5 1.2B Instruct (Lightweight) |
|---|---|---|
| Unsafe results (28 checks × 5 passes) | 0 | 0 |
| Word completions | 12 / 13 | 12 / 13 |
| Next-word accuracy (788 positions) | 23.1% | 20.7% |
| English accuracy (624 positions) | 25.3% | 24.2% |
| Non-English accuracy (164 positions) | 14.6% | 7.3% |
| Right when shown (precision) | 50.1% | 43.0% |
| Per keystroke while typing (median / p95) | 56 / 102 ms | 26 / 76 ms |
| Model load time | 0.65 s | 0.16 s |
What the numbers actually say
- In English, the two models are effectively tied (+1.1 points, 95% bootstrap interval −1.4 to +3.5), while LFM2.5 is 2–3× faster per keystroke (26 ms median) and weighs only 0.73 GB.
- Gemma 4 E2B earns the Balanced default on precision and non-English text. When it chooses to show a suggestion, it is right 50.1% of the time vs. 43.0% for LFM2.5, and its non-English lead (+7.3 points) comes largely from languages like Serbian where LFM2.5 stayed silent.
- Silence is a feature. Both models withheld suggestions on the exact same 3 open-ended prompts (
notes-list,comma-clause,sentence-start) and stayed quiet mid-word (rec|eive), producing zero unsafe suggestions across all 140 passes.
Choosing between them in LokalBot
Both presets ship built into LokalBot: Balanced local uses Gemma 4 E2B Base, and Lightweight local switches to LFM2.5 1.2B Instruct for faster typing and a smaller disk footprint. Full category breakdowns and commands to rerun the harness are in Benchmarks/autocomplete/Cotyping/results/2026-10-15-engine-benchmark/REPORT.md.