Changelog

• Dataset: LoCoMo long-term conversation memory benchmark.
• Models tested: Gemma 2 9B (backend), Gemma 3 4B, Gemma 3 12B — all open-source, running locally at 4-bit quantization.
• Honest headline: ~55–60% accuracy on models 17× smaller than frontier. GPT-4o with no memory layer scores ~60% on the same benchmark.
• Run 4 caveat: the 75.1% figure is inflated — the smaller 4b judge passes hallucinated answers. Manual review confirms true accuracy sits at 55–60%. The higher idk_over_refusal count in Run 5 (420 instances, 27.3%) reflects rigorous honest refusal by the larger judge model rather than regression.

Full results with failure analysis → Google Sheets

* Run 4 score is inflated by judge leniency. Manual review of outputs confirms true accuracy ~55–60%. Run 5's lower score reflects stricter honest grading by the larger judge.

• Root cause: phrases like "last Saturday" or "next month" were being baked into stored summaries with no calendar anchor, becoming permanently unresolvable the moment that week passed. Queries asking "when did X happen?" would retrieve the right event but get no usable date back.
• Session date anchoring: the session date is now forwarded through the entire pipeline from ingest to summarization. The summarizer resolves relative references to absolute dates before writing them to storage, so "last Saturday" becomes "6 May 2023" in the actual memory record.
• Multi-hop temporal queries: this was the primary reason multi-hop questions scored near zero on the LoCoMo benchmark despite the data being present. The information existed in storage, it just had no resolvable date attached.

• Determined encryption on backend DBs will be coming to become compliant.

• Did some testing and scored around 10%.
• Determined response model and backend bug are the main issues, fixed in subsequent runs.

• Updated summary logic to stop condensing key data.

• Temporal fact tagging: preferences that flip (e.g. "loves X" → "can't stand X anymore") now mark the old value deprecated instead of both living in the profile forever. Old values stay on disk for audit, but never surface to retrieval as current.
• Race condition fix: concurrent requests for the same user could clobber each other's writes to the fact store under load — this was the root cause of facts silently disappearing. Reads and writes are now serialized per user, across threads and processes.
• Idempotency: near-duplicate facts no longer create separate entries on repeated mentions of the same thing.

• The API key endpoint could be called with a bare email string and no proof of identity, meaning an unverified signup could reach the dashboard and pull a working key without ever clicking a verification link.
• A verified Firebase ID token is now required on every key request, and the backend checks email_verified before issuing or creating a key.
• Added client-side guards so an unverified session gets redirected before it ever reaches the dashboard or plan picker, not just on the login page.