Recall vs. restraint: which local models know when not to save a memory

Why restraint matters

On Mac, LokalBot's Personal Context learns facts, preferences, goals, and principles from what you type and say, so that agents you allow can tailor their work to you. A missed memory costs little: the next conversation usually brings it up again. A wrong one is worse. It quietly ends up in the context every agent reads, and nobody notices until an answer is off.

So the test asks for both recall and restraint. Three kinds of cases make the difference visible:

• Who said it. “I live in Valencia” should be saved. “My colleague said, ‘I live in Valencia’” should not.
• Already known. A new preference should be saved. One already in the known-memories list should be skipped.
• Policy. A job title mentioned alongside a salary should be kept, and the salary left out.

The test

The benchmark has 120 synthetic cases arranged as 60 contrast pairs. The main comparison uses the 96 English cases: 52 contain one useful memory to extract, and in 44 the correct output is an empty list. A separate 24-case Serbian slice is reported on its own.

Every model received the same application prompt with its native chat template, an enforced JSON schema, temperature 0, seed 0, thinking disabled, and a single pass, with no model-specific prompt tuning. The GGUF models ran one after another through llama.cpp on the same Ryzen 7 9700X server. This measures accuracy, not speed. A recalled memory only counts when it also cites the right source line.

Results

What stood out

• Perfect recall can hide the real problem. Hy-Memory captured every eligible memory and failed all 44 cases where the right answer was nothing, with 76 unwanted proposals. Scored on recall alone, it would look like the best model here.
• The Qwen models held back. Both missed more valid memories, but they handled over 90% of the empty cases and made only three and five unwanted proposals.
• LFM2.5 copied from the instructions. In several cases it proposed “I live in Belgrade”, an example from the prompt with no support in the conversation. It also often returned the source sentence instead of the required source-line ID. Accepting exact quotations recovered 9 of its 18 invalid English citations, raising exact source selection from 51 to 56 of 96 cases, but that only repairs formatting. The table keeps the strict scoring.
• MemT-1.0-Nano was the best balance. A reader on r/LocalLLaMA suggested it, and it was added on October 25: 51 of 52 useful memories, 6 unwanted proposals, and 39 of 44 empty cases. On the Serbian slice it recalled 13 of 13, made 3 unwanted proposals, and passed 8 of 11 empty cases, but three of its outputs switched language or script.

Caveats

• MemT-1.0-Nano ran with experimental NF4 expert offload through Transformers on a Radeon RX 6600 XT; the other models ran as GGUF files in llama.cpp on a CPU. There is no matched run of its Qwen3.6-35B-A3B backbone, so the comparison isn't controlled for hardware or precision.
• Each case starts from a fresh state, so MemT's persistent memory feature isn't exercised.
• An AI assistant wrote the cases and graded the semantic decisions with the model names visible. There is no independent human validation and no claim of statistical significance.

What it means for LokalBot

These results don't change LokalBot's defaults. What they confirm is that a memory writer has to be judged on what it leaves out as much as on what it finds. On Mac, the suggestions Personal Context learns appear under To review in Settings → Personal Context, where you keep or dismiss them.

Data

Everything needed to check or rerun the comparison is in the repository: the benchmark folder with the cases, prompts, and scorer, the MemT report with every failed case, and the raw responses. Other model results are on the benchmarks page.