Benchmark

56 ms local system-wide autocomplete: Gemma 4 E2B vs. LFM2.5 1.2B

System-wide inline autocomplete only feels invisible when it stays under ~100 ms per keystroke and knows when to stay silent. Here are the measured numbers behind LokalBot's Balanced and Lightweight presets.

Measuring autocomplete without fooling ourselves

Our original 28-scenario autocomplete check had five flaws: it treated an intentional confidence-gate silence as a safety failure, accepted only one exact word tail for partial-word completions, mixed 10–40 s cold startup into steady-state latency, stole keyboard focus by launching the full app, and never measured next-word accuracy.

We replaced it with a headless engine runner (--cotyping-bench) that exercises the exact production LlamaCotypingRuntime with hybrid decoding, certified prefix search, and prefix healing across 28 safety/behaviour checks (5 deterministic passes) plus 96 phrases replayed at all 788 word boundaries and 8 phrases typed one character at a time.

Gemma 4 E2B Base vs. LFM2.5 1.2B Instruct (Apple M4 Max)

Autocomplete engine benchmark comparing Gemma 4 E2B Base Q6_K and LFM2.5 1.2B Instruct Q4_K_M
MetricGemma 4 E2B Base (Balanced)LFM2.5 1.2B Instruct (Lightweight)
Unsafe results (28 checks × 5 passes)00
Word completions12 / 1312 / 13
Next-word accuracy (788 positions)23.1%20.7%
English accuracy (624 positions)25.3%24.2%
Non-English accuracy (164 positions)14.6%7.3%
Right when shown (precision)50.1%43.0%
Per keystroke while typing (median / p95)56 / 102 ms26 / 76 ms
Model load time0.65 s0.16 s

What the numbers actually say

Choosing between them in LokalBot

Both presets ship built into LokalBot: Balanced local uses Gemma 4 E2B Base, and Lightweight local switches to LFM2.5 1.2B Instruct for faster typing and a smaller disk footprint. Full category breakdowns and commands to rerun the harness are in Benchmarks/autocomplete/Cotyping/results/2026-10-15-engine-benchmark/REPORT.md.

Try it locally

Meeting notes that stay on your computer by default.

Free, open source, and no account. For Mac, with Windows and Linux in alpha.

Download for macOS

Apple Silicon · macOS 15+ · or with Homebrew: brew install --cask stevyhacker/tap/lokalbot

macOS · Windows · Linux (alpha) · requirements