Why we ran this
Every few weeks a new local memory layer promises better recall for coding agents and personal assistants. LokalBot's own retrieval is intentionally plain: SQLite FTS5 BM25 plus EmbeddingGemma 2 (310 MB, 768 dimensions) joined with reciprocal rank fusion. We tested whether replacing it with GBrain (v0.60.734.0 on PGLite over stdio MCP) improved meeting recall enough to justify the extra runtime.
The setup
We used the public QMSum corpus (232 meetings, 129,177 turns, 2.1 million words across AMI product design, ICSI research, and parliamentary committees) and all 1,576 specific questions with human-marked answer spans. No personal data was used. Both systems ingested the exact same meetings and shared one local llama-server:embeddinggemma-2 instance on an Apple M4 Max (48 GB). Confidence intervals come from 2,000 paired bootstrap resamples over questions.
Results
| Metric | LokalBot keyword | GBrain keyword | LokalBot hybrid | GBrain hybrid |
|---|---|---|---|---|
| Meeting R@1 | 0.193 | 0.253 | 0.280 | 0.239 |
| Meeting R@4 | 0.335 | 0.412 | 0.475 | 0.367 |
| Meeting R@10 | 0.456 | 0.468 | 0.595 | 0.370 |
| MRR | 0.274 | 0.325 | 0.379 | 0.295 |
| Gold turn in evidence | 0.260 | 0.402 | 0.367 | 0.340 |
| Median evidence handed to LLM | 718 chars | 97,444 chars | 1,248 chars | 96,761 chars |
| Query latency (p50 / p95) | 3 / 8 ms | 159 / 212 ms | 39 / 52 ms | 255 / 342 ms |
What stood out
- Hybrid retrieval favored LokalBot cleanly. With the same embedder, LokalBot led GBrain by +4.1 points on R@1 (95% CI +1.9 to +6.2) and +10.9 points on R@4 (+8.2 to +13.3), at 6.5× lower median latency (39 ms vs. 255 ms) and roughly 50 MB vs. 1 GB resident memory (excluding the shared 0.5 GB embedding server).
- Context budget matters on local models. GBrain's default
queryhands back a median ~97,000 characters (full transcripts) per question. LokalBot hands back 1,248 characters (top 4 meetings, up to 6 moments of 400 characters each), which fits comfortably inside a local 4B model or a coding agent's MCP tool budget. - GBrain's keyword-only search beat LokalBot's keyword-only search. On lexical queries alone, GBrain hit R@1 0.253 vs. 0.193 and got the gold turn into evidence on 40.2% vs. 26.0% of questions. That points directly at an improvement we can take into LokalBot's FTS5 query formulation without adopting a heavier database.
- Restated contradictions are still unsolved in both. Explicit edits and deletions worked cleanly in both systems (40/40 and 39/40), but when a later meeting silently reversed an earlier decision, neither ranked the newer meeting above the older one reliably (7/20 in LokalBot, 8/20 in GBrain).
Data and reproduction
The corpus builder, harness, scorer, and full report are in Benchmarks/memory/GBrain/2026-11-17/REPORT.md. A keyword-only repeat on a 4-core Intel Xeon produced identical accuracy numbers. See the benchmarks page for the rest of LokalBot's local stack.