Benchmark

39 ms and 50 MB vs. 255 ms and 1 GB: benchmarking local meeting retrieval on QMSum

Does a heavier local memory server beat plain SQLite FTS5 plus a 310 MB embedding model? We tested LokalBot's retrieval against GBrain on 232 public meetings and 1,576 questions.

Why we ran this

Every few weeks a new local memory layer promises better recall for coding agents and personal assistants. LokalBot's own retrieval is intentionally plain: SQLite FTS5 BM25 plus EmbeddingGemma 2 (310 MB, 768 dimensions) joined with reciprocal rank fusion. We tested whether replacing it with GBrain (v0.60.734.0 on PGLite over stdio MCP) improved meeting recall enough to justify the extra runtime.

The setup

We used the public QMSum corpus (232 meetings, 129,177 turns, 2.1 million words across AMI product design, ICSI research, and parliamentary committees) and all 1,576 specific questions with human-marked answer spans. No personal data was used. Both systems ingested the exact same meetings and shared one local llama-server:embeddinggemma-2 instance on an Apple M4 Max (48 GB). Confidence intervals come from 2,000 paired bootstrap resamples over questions.

Results

Retrieval comparison on 1,576 QMSum questions (Apple M4 Max)
MetricLokalBot keywordGBrain keywordLokalBot hybridGBrain hybrid
Meeting R@10.1930.2530.2800.239
Meeting R@40.3350.4120.4750.367
Meeting R@100.4560.4680.5950.370
MRR0.2740.3250.3790.295
Gold turn in evidence0.2600.4020.3670.340
Median evidence handed to LLM718 chars97,444 chars1,248 chars96,761 chars
Query latency (p50 / p95)3 / 8 ms159 / 212 ms39 / 52 ms255 / 342 ms

What stood out

Data and reproduction

The corpus builder, harness, scorer, and full report are in Benchmarks/memory/GBrain/2026-11-17/REPORT.md. A keyword-only repeat on a 4-core Intel Xeon produced identical accuracy numbers. See the benchmarks page for the rest of LokalBot's local stack.

Try it locally

Meeting notes that stay on your computer by default.

Free, open source, and no account. For Mac, with Windows and Linux in alpha.

Download for macOS

Apple Silicon · macOS 15+ · or with Homebrew: brew install --cask stevyhacker/tap/lokalbot

macOS · Windows · Linux (alpha) · requirements