Code graph and memory for your coding agents, in one Rust program. For Claude Code, Codex, Cursor and any MCP client.
Rust agent-memoryai-agentsclaude-codecode-analysiscode-graphcode-intelligencecode-searchcodexcoding-agentscontext-engineeringcursordeveloper-toolsgraphragknowledge-graphlocal-firstmcpmcp-servermodel-context-protocolrusttree-sitter
docs: the 8 October memory runs on GPT models
VikramLexa1da05ed176d
diff --git a/README.md b/README.mdindex 4d03fe4..54357fe 100644--- a/README.md+++ b/README.md@@ -65,6 +65,8 @@ In our October comparison with seven memory systems, Graphicle matched the accurOn the two other development sets, LME-84 and BEAM-100K, Graphicle trailed none of the systems it was compared with. [Every score, with its method and caveats](docs/benchmarks.md#agent-memory-comparison-with-seven-memory-systems).+Earlier, on 8 October, a pre-release build also ran three published protocols on GPT models. Under Mem0's own LoCoMo protocol (its prompts, GPT-5 as reader and judge), it scored 93.5%, against 91.6% in Mem0's published result files (paired difference +1.95 points [+0.55, +3.55]). On LongMemEval-S under the same protocol it scored 92.6%, against Mem0's 93.4%. Against Hindsight's published contexts, it tied with a GPT-5.4 reader (LoCoMo 90.4 vs 90.6, LongMemEval-S 94.6 vs 93.4) and with a GPT-4.1-mini reader on LoCoMo (86.4 vs 87.5), and trailed with GPT-4.1-mini on LongMemEval-S (85.8 vs 89.2). [Every GPT run, with its labels](docs/benchmarks.md#agent-memory-with-gpt-models-the-8-october-runs).+## Quick start```shdiff --git a/docs/benchmarks.md b/docs/benchmarks.mdindex 1551b48..458f9a1 100644--- a/docs/benchmarks.md+++ b/docs/benchmarks.md@@ -1130,6 +1130,59 @@ decision rule.them ended at their output limit. It made no embedding calls to the modelservers, because it embeds in its own process.+## Agent memory with GPT models: the 8 October runs++Before the comparison above, on 8 October 2026, a pre-release build of Graphicle answered LoCoMo (1,540 questions) and LongMemEval-S (500 questions) through three published protocols, on Azure OpenAI models, as memory vendors publish their results:++- **mem0:** Mem0's published prompts, with GPT-5 (2025-08-07) as reader and judge.+- **gpt54-common:** the Omni prompts, with GPT-5.4 (2026-03-05, medium reasoning) as reader and GPT-4o (2024-11-20) as judge. This is not an exact replication of Zep's setup.+- **omni-adapted:** the published Omni prompts, with GPT-4.1-mini (2025-04-14) as reader and GPT-4o (2024-11-20) as judge, in place of the unavailable GPT-4o-mini.++Labels for every number in this section:++- **The build.** A pre-release build of Graphicle, whose memory notes were written by GPT-4.1-mini. `graphicle` is its retrieval alone, `graphicle_observer` adds the notes, and `plain` is a baseline without Graphicle.+- **The other rows are not fresh runs.** `hindsight_replay` and `amb_hybrid_replay` are fresh answers from the published, unchanged retrieval contexts of Hindsight's and the Agent Memory Benchmark's runs. The Mem0 figures are Mem0's own published result files: merged outputs, not a fresh run of the managed service.+- **Missing notes.** When the answers were scored, notes were missing for 142 of the 500 LongMemEval histories and for 1 of the 10 LoCoMo conversations. Every question stayed in the denominator, and a failure scored 0.+- **Intervals and gold labels.** LoCoMo intervals resample its ten conversations. About 6.4% of LoCoMo's gold answers are wrong (an independent audit), which caps a fully correct system near 93.6%.++| Dataset | Protocol | Memory or context | Correct | Accuracy [95% interval] | Reader or judge errors | Context tokens, mean |+| --- | --- | --- | ---: | --- | ---: | ---: |+| LoCoMo | gpt54-common | graphicle_observer | 1,392 / 1,540 | 90.39% [88.9, 91.8] | 0 | 13,232 |+| LoCoMo | gpt54-common | hindsight_replay | 1,395 / 1,540 | 90.58% [88.3, 92.6] | 2 | 35,604 |+| LoCoMo | mem0 | graphicle | 1,413 / 1,540 | 91.75% [89.8, 93.8] | 9 | 6,208 |+| LoCoMo | mem0 | graphicle_observer | 1,440 / 1,540 | 93.51% [92.0, 94.8] | 11 | 12,661 |+| LoCoMo | mem0 | plain | 1,382 / 1,540 | 89.74% [87.9, 91.7] | 7 | 6,233 |+| LoCoMo | omni-adapted | amb_hybrid_replay | 1,201 / 1,540 | 77.99% [75.2, 80.6] | 0 | 21,727 |+| LoCoMo | omni-adapted | graphicle | 1,330 / 1,540 | 86.36% [85.0, 87.8] | 0 | 6,408 |+| LoCoMo | omni-adapted | graphicle_observer | 1,330 / 1,540 | 86.36% [83.4, 89.2] | 0 | 13,232 |+| LoCoMo | omni-adapted | hindsight_replay | 1,347 / 1,540 | 87.47% [85.8, 89.0] | 0 | 35,604 |+| LoCoMo | omni-adapted | plain | 1,284 / 1,540 | 83.38% [82.1, 84.6] | 0 | 6,429 |+| LongMemEval-S | gpt54-common | graphicle_observer | 473 / 500 | 94.60% [92.3, 96.3] | 0 | 25,242 |+| LongMemEval-S | gpt54-common | hindsight_replay | 467 / 500 | 93.40% [90.9, 95.3] | 1 | 43,125 |+| LongMemEval-S | mem0 | graphicle | 411 / 500 | 82.20% [78.6, 85.3] | 1 | 5,771 |+| LongMemEval-S | mem0 | graphicle_observer | 463 / 500 | 92.60% [90.0, 94.6] | 1 | 24,176 |+| LongMemEval-S | mem0 | plain | 385 / 500 | 77.00% [73.1, 80.5] | 0 | 5,806 |+| LongMemEval-S | omni-adapted | amb_hybrid_replay | 385 / 500 | 77.00% [73.1, 80.5] | 0 | 22,907 |+| LongMemEval-S | omni-adapted | graphicle | 386 / 500 | 77.20% [73.3, 80.7] | 0 | 5,997 |+| LongMemEval-S | omni-adapted | graphicle_observer | 429 / 500 | 85.80% [82.5, 88.6] | 0 | 25,242 |+| LongMemEval-S | omni-adapted | hindsight_replay | 446 / 500 | 89.20% [86.2, 91.6] | 0 | 43,125 |+| LongMemEval-S | omni-adapted | plain | 380 / 500 | 76.00% [72.1, 79.5] | 0 | 6,035 |++Paired differences, candidate minus control:++| Dataset | Protocol | Comparison | Difference | Paired 95% interval |+| --- | --- | --- | ---: | --- |+| LongMemEval-S | omni-adapted | graphicle_observer vs hindsight_replay | -3.40 points | [-6.60, -0.20] |+| LongMemEval-S | gpt54-common | graphicle_observer vs hindsight_replay | +1.20 points | [-0.80, +3.20] |+| LoCoMo | omni-adapted | graphicle_observer vs hindsight_replay | -1.10 points | [-3.80, +1.79] |+| LoCoMo | gpt54-common | graphicle_observer vs hindsight_replay | -0.19 points | [-1.28, +0.83] |+| LoCoMo | mem0 | graphicle_observer vs Mem0's published files | +1.95 points | [+0.55, +3.55] |+| LoCoMo | mem0 | graphicle vs Mem0's published files | +0.19 points | [-1.27, +1.88] |++Mem0's published files score 1,410 of 1,540 (91.56%) on LoCoMo and 467 of 500 (93.4%) on LongMemEval-S under the mem0 protocol. These differ from the headlines in Mem0's README. No paired interval was computed for LongMemEval-S.++A citation audit checked the notes' quoted byte spans against the source files: 3,894 spans on LoCoMo and 507,874 on LongMemEval, with 0 integrity errors. It checks that each quote is exact, not that each note is complete or correct.+## Agent memory: how the retrieval defaults were chosenModel-free retrieval runs chose the defaults. They ask whether the evidence