Skip to content

#1 Persistent memory for AI coding agents based on real-world benchmarks

feat(eval): pluggable benchmark harness with in-house coding-agent corpus (#562)

Rohit Ghumare7fb72f401085
diff --git a/.gitignore b/.gitignore
index 585d0f49..ba6af995 100644
--- a/.gitignore
+++ b/.gitignore
@@ -13,6 +13,8 @@ dist/
plugin/scripts/*.map
plugin/scripts/*.d.mts
data/
+!eval/data/
+!eval/data/**
data-*/
agentmemory-debug/
.gstack/
@@ -22,3 +24,8 @@ package-lock.json
pnpm-lock.yaml
yarn.lock
integrations/hermes/__pycache__/
+
+# Eval reports (transient; published scorecards live in docs/benchmarks/)
+eval/reports/
+# LongMemEval download is 278MB; fetched on demand
+eval/data/longmemeval/
diff --git a/README.md b/README.md
index cfa87bc4..96e804d4 100644
--- a/README.md
+++ b/README.md
@@ -208,6 +208,15 @@ npx @agentmemory/agentmemory
### Retrieval Accuracy
+**coding-agent-life-v1** (in-house corpus, sandbox-reproducible)
+
+| Adapter | P@5 | R@5 | Top-5 hit rate | p50 latency |
+|---|---|---|---|---|
+| **agentmemory hybrid** | **0.578** | **0.967** | **15 / 15** | 14 ms |
+| grep baseline | 0.267 | 0.967 | 15 / 15 | 0 ms |
+
+100% top-5 hit rate. **2.2×** better precision than the grep baseline on identical input. Full per-type breakdown: [`docs/benchmarks/2026-05-20-coding-agent-life-v1.md`](docs/benchmarks/2026-05-20-coding-agent-life-v1.md).
+
**LongMemEval-S** (ICLR 2025, 500 questions)
| System | R@5 | R@10 | MRR |
@@ -233,6 +242,8 @@ npx @agentmemory/agentmemory
> Embedding model: `all-MiniLM-L6-v2` (local, free, no API key). Full reports: [`benchmark/LONGMEMEVAL.md`](benchmark/LONGMEMEVAL.md), [`benchmark/QUALITY.md`](benchmark/QUALITY.md), [`benchmark/SCALE.md`](benchmark/SCALE.md). Competitor comparison: [`benchmark/COMPARISON.md`](benchmark/COMPARISON.md) — agentmemory vs mem0, Letta, Khoj, claude-mem, Hippo.
+**Reproduce locally:** [`eval/README.md`](eval/README.md) — adapter-pluggable harness for LongMemEval `_s` (public 500-Q) + `coding-agent-life-v1` (in-house 15-session corpus). Grep / vector / agentmemory adapters score side-by-side, NDJSON output, published scorecards land in [`docs/benchmarks/`](docs/benchmarks/).
+
---
<h2 id="vs-competitors"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/tags/light/section-competitors.svg"><img src="assets/tags/section-competitors.svg" alt="vs Competitors" height="32" /></picture></h2>
diff --git a/docs/benchmarks/2026-05-20-coding-agent-life-v1.md b/docs/benchmarks/2026-05-20-coding-agent-life-v1.md
new file mode 100644
index 00000000..f280b27d
--- /dev/null
+++ b/docs/benchmarks/2026-05-20-coding-agent-life-v1.md
@@ -0,0 +1,76 @@
+# 2026-05-20 — coding-agent-life-v1 (v0.9.21)
+
+**Commit:** `e9dc710`
+**Bench:** coding-agent-life-v1 (15 sessions, 15 queries)
+**N:** 15
+**K:** 5
+**Hardware:** macOS 15 (Apple Silicon)
+**agentmemory:** v0.9.21
+**iii-engine:** v0.11.2
+**Embedding provider:** local default
+**Sandbox:** isolated data dir at `/tmp/agentmemory-eval-sandbox/`, ports 3411/3412
+
+## Headline
+
+`agentmemory-hybrid` hits **100% top-5 hit rate**, R@5 = **0.967**, P@5 = **0.578**.
+
+Same corpus, grep baseline: R@5 = 0.967, P@5 = 0.267 — same recall, but **2.2× worse precision**. Hybrid's top-5 is mostly gold; grep's top-5 is half noise.
+
+## Per-adapter
+
+| Adapter | P@5 | R@5 | Hit rate | p50 latency |
+|---|---|---|---|---|
+| grep (tokenized substring) | 0.267 | 0.967 | 15 / 15 | 0 ms |
+| `agentmemory-hybrid` | **0.578** | **0.967** | **15 / 15** | 14 ms |
+
+`agentmemory-hybrid` runs through the production smart-search endpoint (`POST /agentmemory/smart-search`) so it exercises the full BM25 + embedding + reranker stack.
+
+## Per-question-type
+
+P@5, grep vs `agentmemory-hybrid`:
+
+| Type | grep | hybrid | hybrid lift |
+|---|---|---|---|
+| single-session-bug | 0.20 | 0.33 | 1.7× |
+| single-session-infra (n=2) | 0.20 | 0.50 | 2.5× |
+| single-session-refactor | 0.20 | 0.50 | 2.5× |
+| single-session-feature | 0.50 | 0.50 | tie |
+| single-session-test | 0.20 | 0.33 | 1.7× |
+| single-session-perf | 0.20 | 0.50 | 2.5× |
+| single-session-api | 0.20 | 0.50 | 2.5× |
+| single-session-db | 0.20 | 0.50 | 2.5× |
+| single-session-release | 0.20 | 0.33 | 1.7× |
+| multi-session-causal | 0.40 | 0.40 | tie |
+| preference (n=2) | 0.20 | 0.42 | 2.1× |
+| multi-session-review | 0.40 | 0.67 | 1.7× |
+| temporal (R@5 = 0.50 grep / 1.00 hybrid) | 0.50 | 0.67 | 1.3× |
+
+Temporal queries (`What was shipped on April 8th 2026?`) need both gold sessions to score full recall. grep finds 1/2; hybrid finds 2/2.
+
+## Methodology
+
+- 15 fictional Claude Code sessions across a 10-day stretch of a Rust CLI project (`shipctl`) — bug fixes, refactors, infra, perf, schema migrations, preferences, post-mortem
+- 15 hand-graded queries with `goldSessionIds[]` covering single-session, multi-session causal, multi-session review, preference, temporal
+- Each session ingested via `POST /agentmemory/remember` with `type=eval-session` and `concepts=[session_id]`
+- Each query hits `POST /agentmemory/smart-search` with `limit=50`; dedupe by session ID; truncate to K=5
+- No LLM in the retrieval loop
+- Sandbox: clean `~/.agentmemory` via `HOME` override + alt ports (3411/3412) so no cross-contamination from a user's real store
+
+## Reproduce
+
+```sh
+git checkout e9dc710
+npm install --legacy-peer-deps
+npm run build
+
+source eval/scripts/sandbox.sh
+npm run eval:coding-life -- --adapters grep,agentmemory
+```
+
+Outputs land in `eval/reports/coding-life/`: `scores.ndjson` (per-query rows) and `summary.json` (per-adapter and per-type aggregates).
+
+## Notes
+
+- The single-session-feature tie (`Which PR introduced helm chart support?`) is interesting: query says `PR introduced helm chart` and gold session has `helm chart` literally — grep wins on lexical exactness, hybrid matches but doesn't outperform.
+- The corpus is intentionally small for fast iteration. Hardening targets: paraphrased queries, synonym substitution, in-corpus distractors with shared keywords, longer multi-session chains.
+- Vector adapter not measured here — requires `OPENAI_API_KEY`; will be added in a follow-up scorecard alongside LongMemEval `_s`.
diff --git a/docs/benchmarks/TEMPLATE.md b/docs/benchmarks/TEMPLATE.md
new file mode 100644
index 00000000..b830e24e
--- /dev/null
+++ b/docs/benchmarks/TEMPLATE.md
@@ -0,0 +1,54 @@
+# <YYYY-MM-DD> — <benchmark-name>
+
+**Commit:** `<sha>`
+**Bench:** LongMemEval `_s` / coding-agent-life-v1 / ...
+**N:** 500 / 15 / ...
+**K:** 5
+**Hardware:** macos-15 / ubuntu-22.04 / ...
+**OpenAI model:** text-embedding-3-small
+**Anthropic model:** N/A (no LLM in retrieval loop)
+
+## Headline
+
+agentmemory-hybrid: **R@5 = XX.XX%**, P@5 = XX.XX%, p50 latency = XXms
+
+Beats grep baseline by +X.Xpt R@5, vector by +X.Xpt R@5.
+
+## Per-adapter
+
+| Adapter | P@5 | R@5 | Hit rate | p50 latency |
+|---|---|---|---|---|
+| grep | | | | |
+| vector | | | | |
+| agentmemory-hybrid | | | | |
+
+## Per-question-type
+
+| Type | grep R@5 | vector R@5 | agentmemory R@5 |
+|---|---|---|---|
+| single-session-bug | | | |
+| single-session-refactor | | | |
+| preference | | | |
+| multi-session-causal | | | |
+| temporal | | | |
+
+## Methodology
+
+- Sessions ingested via `POST /agentmemory/remember` with `type=eval-session`
+- Queries hit `POST /agentmemory/smart-search` with `limit=k*4`
+- No LLM in retrieval loop. Direct rank from hybrid scoring.
+- Ranks dedup by sessionId before truncating to K
+- Latency measured as init+query for LongMemEval (per-question fresh state), query-only for coding-life (shared state)
+
+## Reproduce
+
+```sh
+git checkout <sha>
+npm install --legacy-peer-deps
+OPENAI_API_KEY=sk-... AGENTMEMORY_BASE_URL=http://localhost:3111 \
+ npm run eval:longmemeval -- --stratify 10
+```
+
+## Notes
+
+<what surprised, what regressed, what's load-bearing>
diff --git a/eval/README.md b/eval/README.md
new file mode 100644
index 00000000..7f295367
--- /dev/null
+++ b/eval/README.md
@@ -0,0 +1,111 @@
+# agentmemory-evals
+
+Public benchmarks for agentmemory's hybrid memory stack (BM25 + embeddings + consolidation + graph).
+
+Two families, both reproducible:
+
+- **LongMemEval** — public 500-question retrieval benchmark over multi-session chat
+- **coding-agent-life-v1** — in-house corpus of 15 fictional Claude Code sessions for a Rust CLI project (`shipctl`), with 15 hand-graded queries covering bug fixes, refactors, preferences, and multi-session causal reasoning
+
+## Adapters
+
+| Adapter | Backend | API key needed |
+|---|---|---|
+| `grep` | Tokenized substring match | none |
+| `vector` | OpenAI `text-embedding-3-small` + cosine | `OPENAI_API_KEY` |
+| `agentmemory` | Running agentmemory server, smart-search endpoint | none (auth optional via `AGENTMEMORY_SECRET`) |
+
+## Sandbox first
+
+Running the `agentmemory` adapter against your real `~/.agentmemory` directory pollutes the eval with pre-existing memories AND pollutes your real store with eval test data. Always sandbox.
+
+`eval/scripts/sandbox.sh` spins up a clean agentmemory + iii-engine on ports 3411/3412 with state in `/tmp/agentmemory-eval-sandbox/`, exports `AGENTMEMORY_BASE_URL`, and tears down on exit.
+
+```sh
+source eval/scripts/sandbox.sh
+npm run eval:coding-life -- --adapters grep,agentmemory
+```
+
+Requires iii v0.11.2 on PATH (agentmemory pin). If you already have a different version installed, install the pinned build into `~/.local/bin` and make sure that directory comes first on `PATH`:
+
+```sh
+mkdir -p ~/.local/bin
+curl -fsSL https://github.com/iii-hq/iii/releases/download/iii/v0.11.2/iii-aarch64-apple-darwin.tar.gz | tar -xz -C ~/.local/bin
+export PATH="$HOME/.local/bin:$PATH" # add to ~/.zshrc or ~/.bashrc for persistence
+```
+
+## Quickstart
+
+### coding-agent-life-v1 (in-house, no download)
+
+```sh
+# grep baseline, no sandbox needed
+npm run eval:coding-life -- --adapters grep
+
+# add agentmemory + vector (sandbox + OpenAI key)
+source eval/scripts/sandbox.sh
+OPENAI_API_KEY=sk-... npm run eval:coding-life -- --adapters grep,vector,agentmemory
+```
+
+### LongMemEval `_s` (public, 278MB download)
+
+```sh
+mkdir -p ~/datasets/longmemeval
+curl -Lo ~/datasets/longmemeval/longmemeval_s.json \
+ https://huggingface.co/datasets/xiaowu0162/longmemeval/resolve/main/longmemeval_s
+
+source eval/scripts/sandbox.sh
+
+# Stratified sample of 10 per type (fast iteration, ~$0.20 OpenAI cost)
+OPENAI_API_KEY=sk-... LONGMEMEVAL_PATH=~/datasets/longmemeval/longmemeval_s.json \
+ npm run eval:longmemeval -- --stratify 10
+
+# Full 500 questions × 3 adapters (~$2 OpenAI cost)
+OPENAI_API_KEY=sk-... LONGMEMEVAL_PATH=~/datasets/longmemeval/longmemeval_s.json \
+ npm run eval:longmemeval
+```
+
+## Repo layout
+
+```text
+eval/
+├── README.md
+├── runner/
+│ ├── types.ts Adapter, Question, RankedDoc, ScoreRow
+│ ├── score.ts P@K, R@K, aggregation
+│ ├── load.ts LongMemEval JSON → Question[]
+│ ├── adapters/
+│ │ ├── grep.ts tokenized substring baseline
+│ │ ├── vector.ts OpenAI embeddings + cosine
+│ │ └── agentmemory.ts POST /agentmemory/{remember,smart-search}
+│ ├── longmemeval.ts public benchmark runner
+│ └── coding-life.ts in-house benchmark runner
+└── data/
+ └── coding-agent-life-v1/
+ ├── sessions.json 15 fictional sessions (~6KB)
+ └── queries.json 15 queries with gold session IDs
+```
+
+Reports land in `eval/reports/<bench>/` (gitignored): `scores.ndjson` + `summary.json`.
+
+Published scorecards land in `docs/benchmarks/YYYY-MM-DD-<bench>.md`.
+
+## Writing a new adapter
+
+1. Implement `Adapter<State>` from `eval/runner/types.ts`:
+ ```ts
+ import type { Adapter } from "../types.js";
+ export const myAdapter: Adapter<MyState> = {
+ name: "my-adapter",
+ async init(sessions, config) { /* index */ return state; },
+ async query(q, state, k) { /* search */ return ranked; },
+ };
+ ```
+2. Register in `eval/runner/{longmemeval,coding-life}.ts` `ADAPTERS` map.
+3. Run against `coding-agent-life-v1` to sanity-check before committing OpenAI spend on LongMemEval.
+
+## Why a benchmark for agentmemory
+
+agentmemory ships BM25 + embeddings + consolidation + graph retrieval. Numbers from those layers should be measured against grep/vector baselines so the value of each layer is provable.
+
+The in-house corpus is small on purpose (15 sessions) — covers single-session, multi-session, preference, and temporal question types without taking 15 minutes to run. LongMemEval gives the public-comparison axis.
diff --git a/eval/data/coding-agent-life-v1/queries.json b/eval/data/coding-agent-life-v1/queries.json
new file mode 100644
index 00000000..5603e8a0
--- /dev/null
+++ b/eval/data/coding-agent-life-v1/queries.json
@@ -0,0 +1,107 @@
+[
+ {
+ "id": "q-001",
+ "type": "single-session-bug",
+ "question": "Where did we land the auth env var precedence fix?",
+ "answer": "PR #11 with SHIPCTL_TOKEN > SHIP_TOKEN > SC_TOKEN precedence",
+ "goldSessionIds": ["sess-001"]
+ },
+ {
+ "id": "q-002",
+ "type": "single-session-infra",
+ "question": "What was the multi-arch Docker fix?",
+ "answer": "Added --platform=$BUILDPLATFORM and BUILDX_PLATFORMS for amd64+arm64",
+ "goldSessionIds": ["sess-002"]
+ },
+ {
+ "id": "q-003",
+ "type": "single-session-refactor",
+ "question": "Where did we consolidate the retry logic?",
+ "answer": "src/retry.rs with exponential backoff base=200ms cap=30s full jitter",
+ "goldSessionIds": ["sess-003"]
+ },
+ {
+ "id": "q-004",
+ "type": "single-session-feature",
+ "question": "Which PR introduced helm chart support?",
+ "answer": "PR #14",
+ "goldSessionIds": ["sess-004"]
+ },
+ {
+ "id": "q-005",
+ "type": "single-session-test",
+ "question": "Which test was flaky on macos and how was it fixed?",
+ "answer": "fs-watcher emits_changekind_file_delete; bumped wait to 1500ms + retry: 2",
+ "goldSessionIds": ["sess-005"]
+ },
+ {
+ "id": "q-006",
+ "type": "single-session-perf",
+ "question": "How did we fix the memory leak?",
+ "answer": "Replaced unbounded HashMap with LruCache cap=10k in src/cache.rs (PR #16)",
+ "goldSessionIds": ["sess-006"]
+ },
+ {
+ "id": "q-007",
+ "type": "single-session-api",
+ "question": "How did we handle the github API rate limit?",
+ "answer": "Conditional requests with If-None-Match etag and 304 caching via http-cache",
+ "goldSessionIds": ["sess-007"]
+ },
+ {
+ "id": "q-008",
+ "type": "single-session-db",
+ "question": "What was the schema migration approach for run_history?",
+ "answer": "Three-phase: nullable column + dual-write, backfill + flip reads, drop old column",
+ "goldSessionIds": ["sess-008"]
+ },
+ {
+ "id": "q-009",
+ "type": "single-session-infra",
+ "question": "How is the docs site deployed?",
+ "answer": "GitHub Actions docs.yml workflow + mdbook build + Cloudflare Pages on shipctl.dev",
+ "goldSessionIds": ["sess-009"]
+ },
+ {
+ "id": "q-010",
+ "type": "single-session-release",
+ "question": "Which PR set up the cross-platform release pipeline?",
+ "answer": "PR #19 with cross-rs for linux and native macos/windows builds",
+ "goldSessionIds": ["sess-010"]
+ },
+ {
+ "id": "q-011",
+ "type": "multi-session-causal",
+ "question": "What was the root cause of the staging incident, and where was it fixed?",
+ "answer": "SHIPCTL_TOKEN unset caused fallback to bad SC_TOKEN; fixed in PR #11 (sess-001) with precedence test; documented in post-mortem (sess-014)",
+ "goldSessionIds": ["sess-001", "sess-014"]
+ },
+ {
+ "id": "q-012",
+ "type": "preference",
+ "question": "Which async runtime does the team prefer for new code?",
+ "answer": "async-std (decided in arch review, not tokio)",
+ "goldSessionIds": ["sess-015"]
+ },
+ {
+ "id": "q-013",
+ "type": "preference",
+ "question": "What are the user's formatting preferences?",
+ "answer": "2-space indent in yaml, tab in rust, rust-analyzer over rls, cargo clippy before commit",
+ "goldSessionIds": ["sess-012"]
+ },
+ {
+ "id": "q-014",
+ "type": "multi-session-review",
+ "question": "What was the race condition Aria fixed in the schema migration?",
+ "answer": "Dual-write step had race when two writers hit same row; Aria added per-row advisory lock",
+ "goldSessionIds": ["sess-008", "sess-013"]
+ },
+ {
+ "id": "q-015",
+ "type": "temporal",
+ "question": "What was shipped on April 8th 2026?",
+ "answer": "Auth env-var precedence fix in PR #11, then started docker arm64 work",
+ "goldSessionIds": ["sess-001", "sess-011"]
+ }
+]
diff --git a/eval/data/coding-agent-life-v1/sessions.json b/eval/data/coding-agent-life-v1/sessions.json
new file mode 100644
index 00000000..71134134
--- /dev/null
+++ b/eval/data/coding-agent-life-v1/sessions.json
@@ -0,0 +1,77 @@
+[
+ {
+ "id": "sess-001",
+ "timestamp": "2026-04-08T09:14:00Z",
+ "content": "[user] auth check broken on staging when SHIPCTL_TOKEN unset, falling back to wrong env var\n[assistant] looked at src/auth.rs:42. precedence is SHIPCTL_TOKEN > SHIP_TOKEN > legacy SC_TOKEN. legacy path didn't validate. fixing precedence + adding length check.\n[user] yeah ship it\n[assistant] PR #11 opened with fix + regression test. covers all three env names."
+ },
+ {
+ "id": "sess-002",
+ "timestamp": "2026-04-08T14:30:00Z",
+ "content": "[user] docker build failing for arm64 on M2 runner\n[assistant] missing target in Dockerfile FROM line. need --platform=$BUILDPLATFORM on builder stage. updated Dockerfile + Makefile to set BUILDX_PLATFORMS=linux/amd64,linux/arm64. tested locally on apple silicon, builds clean."
+ },
+ {
+ "id": "sess-003",
+ "timestamp": "2026-04-09T10:02:00Z",
+ "content": "[user] retry logic is duplicated across handlers. consolidate.\n[assistant] extracted retry_with_backoff into src/retry.rs. exponential backoff base=200ms cap=30s jitter=full. all 4 callsites now use it. cuts ~80 lines."
+ },
+ {
+ "id": "sess-004",
+ "timestamp": "2026-04-10T11:45:00Z",
+ "content": "[user] add helm chart support\n[assistant] new subcommand `shipctl helm install`. wraps helm CLI, threads through --namespace and --values. PR #14 has the changes + 12 integration tests. Aria reviewed and approved."
+ },
+ {
+ "id": "sess-005",
+ "timestamp": "2026-04-11T16:20:00Z",
+ "content": "[user] test flake on macos-22 runner. emits_changekind_file_delete keeps failing\n[assistant] fsevents propagation slower on apple silicon, the 800ms wait isn't enough. bumped to 1500ms + describe-level retry: 2. all 15 fs-watcher tests pass now."
+ },
+ {
+ "id": "sess-006",
+ "timestamp": "2026-04-12T09:50:00Z",
+ "content": "[user] memory leak in long-running daemon. RSS grows 5MB/min.\n[assistant] traced to unbounded HashMap in src/cache.rs. swapped for LruCache with 10k cap. RSS stable at 180MB after 6h soak test. PR #16."
+ },
+ {
+ "id": "sess-007",
+ "timestamp": "2026-04-13T13:15:00Z",
+ "content": "[user] github API rate-limiting us when listing PRs. 5000/hr isn't enough during ci\n[assistant] added conditional requests with If-None-Match etag. cache 304 responses. drops actual API calls by ~70%. used http-cache crate."
+ },
+ {
+ "id": "sess-008",
+ "timestamp": "2026-04-15T10:30:00Z",
+ "content": "[user] schema migration for run_history table needs zero-downtime\n[assistant] three-phase migration. phase 1: add nullable new column + dual-write. phase 2: backfill + flip reads. phase 3: drop old column. avoids ALTER TABLE locks. ran on staging, took 4s for 2M rows."
+ },
+ {
+ "id": "sess-009",
+ "timestamp": "2026-04-16T15:40:00Z",
+ "content": "[user] docs site is stale. set up auto-deploy from main\n[assistant] added .github/workflows/docs.yml. mdbook build + cloudflare pages deploy. domain shipctl.dev wired up. preview deploys on PRs."
+ },
+ {
+ "id": "sess-010",
+ "timestamp": "2026-04-17T11:00:00Z",
+ "content": "[user] release pipeline. cut v0.3.0 with binaries for linux/macos/windows\n[assistant] new .github/workflows/release.yml. cross-rs for static linux, native for macos + windows. uploads to gh release + crates.io publish. tag-triggered. PR #19 has the workflow."
+ },
+ {
+ "id": "sess-011",
+ "timestamp": "2026-04-08T17:00:00Z",
+ "content": "[user] standup notes: shipped #11 auth fix this morning. starting on docker arm64 next."
+ },
+ {
+ "id": "sess-012",
+ "timestamp": "2026-04-10T18:30:00Z",
+ "content": "[user] preferences: I like rust analyzer over rls. always run cargo clippy before commit. prefer 2-space indent in yaml but tab in rust."
+ },
+ {
+ "id": "sess-013",
+ "timestamp": "2026-04-15T19:00:00Z",
+ "content": "[assistant] reviewed Aria's PR #18 (schema migration). flagged race condition in dual-write step when two writers hit same row. Aria added per-row advisory lock. lgtm now."
+ },
+ {
+ "id": "sess-014",
+ "timestamp": "2026-04-16T20:10:00Z",
+ "content": "[user] post-mortem from prod incident last week: SHIPCTL_TOKEN was unset in staging, fell back to bad SC_TOKEN which had wrong perms. delivery delayed 40min. action items: (1) precedence test (done in #11), (2) startup validation, (3) alert on auth fallback."
+ },
+ {
+ "id": "sess-015",
+ "timestamp": "2026-04-17T16:45:00Z",
+ "content": "[user] preferences: stick to async-std not tokio for new code. team agreed in arch review."
+ }
+]
diff --git a/eval/runner/adapters/agentmemory.ts b/eval/runner/adapters/agentmemory.ts
new file mode 100644
index 00000000..38028a7d
--- /dev/null
+++ b/eval/runner/adapters/agentmemory.ts
@@ -0,0 +1,93 @@
+import type { Adapter, RankedDoc, Session } from "../types.js";
+
+interface AgentMemoryState {
+ baseUrl: string;
+ secret?: string;
+ sessions: Session[];
+ observationToSession: Map<string, string>;
+}
+
+interface RememberResponse {
+ memory?: { id?: string };
+ observationId?: string;
+ id?: string;
+ observation?: { id?: string };
+}
+
+interface SmartSearchResponse {
+ results?: Array<{
+ obsId?: string;
+ id?: string;
+ observationId?: string;
+ sessionId?: string;
+ score?: number;
+ content?: string;
+ }>;
+ observations?: Array<{
+ obsId?: string;
+ id?: string;
+ sessionId?: string;
+ score?: number;
+ content?: string;
+ }>;
+}
+
+function authHeaders(secret?: string): Record<string, string> {
+ const h: Record<string, string> = { "Content-Type": "application/json" };
+ if (secret) h.Authorization = `Bearer ${secret}`;
+ return h;
+}
+
+export const agentmemoryAdapter: Adapter<AgentMemoryState> = {
+ name: "agentmemory-hybrid",
+ async init(sessions, config) {
+ const baseUrl = (config?.baseUrl as string) ?? process.env.AGENTMEMORY_BASE_URL ?? "http://localhost:3111";
+ const secret = (config?.secret as string) ?? process.env.AGENTMEMORY_SECRET;
+ const observationToSession = new Map<string, string>();
+ for (const s of sessions) {
+ const res = await fetch(`${baseUrl}/agentmemory/remember`, {
+ method: "POST",
+ headers: authHeaders(secret),
+ body: JSON.stringify({
+ content: s.content,
+ type: "eval-session",
+ concepts: [s.id],
+ }),
+ });
+ if (!res.ok) {
+ throw new Error(`remember failed for ${s.id}: ${res.status} ${await res.text()}`);
+ }
+ const body = (await res.json()) as RememberResponse;
+ const obsId =
+ body.memory?.id ?? body.observationId ?? body.id ?? body.observation?.id;
+ if (obsId) observationToSession.set(obsId, s.id);
+ }
+ return { baseUrl, secret, sessions, observationToSession };
+ },
+ async query(q, state, k) {
+ const res = await fetch(`${state.baseUrl}/agentmemory/smart-search`, {
+ method: "POST",
+ headers: authHeaders(state.secret),
+ body: JSON.stringify({ query: q, limit: Math.max(k * 10, 50) }),
+ });
+ if (!res.ok) {
+ throw new Error(`smart-search failed: ${res.status} ${await res.text()}`);
+ }
+ const body = (await res.json()) as SmartSearchResponse;
+ const rows = body.results ?? body.observations ?? [];
+ const ranked: RankedDoc[] = [];
+ const seen = new Set<string>();
+ for (const row of rows) {
+ let sessionId = row.sessionId;
+ if (!sessionId) {
+ const memId = row.obsId ?? row.id ?? row.observationId;
+ sessionId = memId ? state.observationToSession.get(memId) : undefined;
+ }
+ if (!sessionId || seen.has(sessionId)) continue;
+ seen.add(sessionId);
+ ranked.push({ sessionId, score: row.score ?? 0 });
+ if (ranked.length >= k) break;
+ }
+ return ranked;
+ },
+};
diff --git a/eval/runner/adapters/grep.ts b/eval/runner/adapters/grep.ts
new file mode 100644
index 00000000..28b18ea6
--- /dev/null
+++ b/eval/runner/adapters/grep.ts
@@ -0,0 +1,36 @@
+import type { Adapter, RankedDoc, Session } from "../types.js";
+
+interface GrepState {
+ sessions: Session[];
+}
+
+function tokenize(s: string): string[] {
+ return s
+ .toLowerCase()
+ .replace(/[^a-z0-9_]+/g, " ")
+ .split(/\s+/)
+ .filter((t) => t.length > 2);
+}
+
+export const grepAdapter: Adapter<GrepState> = {
+ name: "grep",
+ async init(sessions) {
+ return { sessions };
+ },
+ async query(q, state, k) {
+ const terms = tokenize(q);
+ const scored: RankedDoc[] = [];
+ for (const s of state.sessions) {
+ const body = s.content.toLowerCase();
+ let hits = 0;
+ for (const t of terms) {
+ if (body.includes(t)) hits += 1;
+ }
+ if (hits > 0) {
+ scored.push({ sessionId: s.id, score: hits });
+ }
+ }
+ scored.sort((a, b) => b.score - a.score);
+ return scored.slice(0, k);
+ },
+};
diff --git a/eval/runner/adapters/vector.ts b/eval/runner/adapters/vector.ts
new file mode 100644
index 00000000..c40e414d
--- /dev/null
+++ b/eval/runner/adapters/vector.ts
@@ -0,0 +1,108 @@
+import type { Adapter, RankedDoc, Session } from "../types.js";
+
+interface VectorState {
+ sessions: Session[];
+ embeddings: Float32Array[];
+}
+
+const OPENAI_URL = "https://api.openai.com/v1/embeddings";
+const MODEL = "text-embedding-3-small";
+const DIM = 1536;
+
+async function embed(text: string, apiKey: string): Promise<Float32Array> {
+ const res = await fetch(OPENAI_URL, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ Authorization: `Bearer ${apiKey}`,
+ },
+ body: JSON.stringify({ input: text, model: MODEL }),
+ });
+ if (!res.ok) {
+ throw new Error(`OpenAI embed failed: ${res.status} ${await res.text()}`);
+ }
+ const data = (await res.json()) as { data: Array<{ embedding: number[] }> };
+ return Float32Array.from(data.data[0].embedding);
+}
+
+async function embedBatch(texts: string[], apiKey: string): Promise<Float32Array[]> {
+ const res = await fetch(OPENAI_URL, {
+ method: "POST",
+ headers: {
+ "Content-Type": "application/json",
+ Authorization: `Bearer ${apiKey}`,
+ },
+ body: JSON.stringify({ input: texts, model: MODEL }),
+ });
+ if (!res.ok) {
+ throw new Error(`OpenAI batch embed failed: ${res.status} ${await res.text()}`);
+ }
+ const data = (await res.json()) as { data: Array<{ embedding: number[]; index: number }> };
+ if (!Array.isArray(data.data) || data.data.length !== texts.length) {
+ throw new Error(
+ `OpenAI batch embed: expected ${texts.length} embeddings, got ${data.data?.length ?? 0}`,
+ );
+ }
+ const out = new Array<Float32Array>(texts.length);
+ for (const row of data.data) {
+ if (
+ !Number.isInteger(row.index) ||
+ row.index < 0 ||
+ row.index >= texts.length ||
+ out[row.index] !== undefined
+ ) {
+ throw new Error(`OpenAI batch embed: invalid or duplicate index ${row.index}`);
+ }
+ if (!Array.isArray(row.embedding) || row.embedding.length === 0) {
+ throw new Error(`OpenAI batch embed: empty embedding at index ${row.index}`);
+ }
+ out[row.index] = Float32Array.from(row.embedding);
+ }
+ return out;
+}
+
+function cosine(a: Float32Array, b: Float32Array): number {
+ let dot = 0;
+ let na = 0;
+ let nb = 0;
+ for (let i = 0; i < a.length; i++) {
+ dot += a[i] * b[i];
+ na += a[i] * a[i];
+ nb += b[i] * b[i];
+ }
+ const denom = Math.sqrt(na) * Math.sqrt(nb);
+ return denom === 0 ? 0 : dot / denom;
+}
+
+export const vectorAdapter: Adapter<VectorState> = {
+ name: "vector",
+ async init(sessions) {
+ const apiKey = process.env.OPENAI_API_KEY;
+ if (!apiKey) throw new Error("OPENAI_API_KEY required for vector adapter");
+ const embeddings: Float32Array[] = new Array(sessions.length);
+ const BATCH = 50;
+ for (let i = 0; i < sessions.length; i += BATCH) {
+ const batch = sessions.slice(i, i + BATCH);
+ const vecs = await embedBatch(
+ batch.map((s) => s.content.slice(0, 8000)),
+ apiKey,
+ );
+ for (let j = 0; j < vecs.length; j++) embeddings[i + j] = vecs[j];
+ }
+ if (embeddings.length > 0 && embeddings[0].length !== DIM) {
+ throw new Error(`unexpected embedding dim: ${embeddings[0].length}`);
+ }
+ return { sessions, embeddings };
+ },
+ async query(q, state, k) {
+ const apiKey = process.env.OPENAI_API_KEY;
+ if (!apiKey) throw new Error("OPENAI_API_KEY required for vector adapter");
+ const qvec = await embed(q, apiKey);
+ const scored: RankedDoc[] = state.sessions.map((s, i) => ({
+ sessionId: s.id,
+ score: cosine(qvec, state.embeddings[i]),
+ }));
+ scored.sort((a, b) => b.score - a.score);
+ return scored.slice(0, k);
+ },
+};
diff --git a/eval/runner/coding-life.ts b/eval/runner/coding-life.ts
new file mode 100644
index 00000000..753ca87f
--- /dev/null
+++ b/eval/runner/coding-life.ts
@@ -0,0 +1,101 @@
+import { readFileSync, existsSync, mkdirSync, writeFileSync, appendFileSync } from "node:fs";
+import { resolve } from "node:path";
+import { parseArgs } from "node:util";
+import { agentmemoryAdapter } from "./adapters/agentmemory.js";
+import { grepAdapter } from "./adapters/grep.js";
+import { vectorAdapter } from "./adapters/vector.js";
+import { aggregate, scoreQuestion } from "./score.js";
+import type { Adapter, Question, ScoreRow, Session } from "./types.js";
+
+const ADAPTERS: Record<string, Adapter> = {
+ grep: grepAdapter as unknown as Adapter,
+ vector: vectorAdapter as unknown as Adapter,
+ agentmemory: agentmemoryAdapter as unknown as Adapter,
+};
+
+interface CliOptions {
+ data: string;
+ adapters: string;
+ k: string;
+ out: string;
+}
+
+function parse(): CliOptions {
+ const { values } = parseArgs({
+ options: {
+ data: { type: "string", default: "eval/data/coding-agent-life-v1" },
+ adapters: { type: "string", default: "grep,vector,agentmemory" },
+ k: { type: "string", default: "5" },
+ out: { type: "string", default: "eval/reports/coding-life" },
+ },
+ });
+ return values as unknown as CliOptions;
+}
+
+async function main(): Promise<void> {
+ const opts = parse();
+ const k = Number(opts.k);
+ if (!Number.isInteger(k) || k <= 0) {
+ console.error(`--k must be a positive integer, got: ${opts.k}`);
+ process.exit(2);
+ }
+ const sessions = JSON.parse(
+ readFileSync(resolve(opts.data, "sessions.json"), "utf8"),
+ ) as Session[];
+ const queriesRaw = JSON.parse(
+ readFileSync(resolve(opts.data, "queries.json"), "utf8"),
+ ) as Array<Omit<Question, "haystack">>;
+ const questions: Question[] = queriesRaw.map((q) => ({ ...q, haystack: sessions }));
+ const adapterNames = opts.adapters.split(",").map((s) => s.trim()).filter(Boolean);
+ for (const a of adapterNames) {
+ if (!ADAPTERS[a]) {
+ console.error(`unknown adapter: ${a}. options: ${Object.keys(ADAPTERS).join(",")}`);
+ process.exit(2);
+ }
+ }
+ console.log(
+ `loaded ${sessions.length} sessions, ${questions.length} queries, adapters: ${adapterNames.join(",")}, k=${k}`,
+ );
+
+ const outDir = resolve(opts.out);
+ mkdirSync(outDir, { recursive: true });
+ const ndjsonPath = `${outDir}/scores.ndjson`;
+ if (existsSync(ndjsonPath)) writeFileSync(ndjsonPath, "");
+
+ const rows: ScoreRow[] = [];
+ for (const adapterName of adapterNames) {
+ const adapter = ADAPTERS[adapterName];
+ console.log(`\n== ${adapter.name} ==`);
+ const state = await adapter.init(sessions);
+ try {
+ for (const q of questions) {
+ const t0 = performance.now();
+ const ranked = await adapter.query(q.question, state, k);
+ const latencyMs = performance.now() - t0;
+ const row = scoreQuestion(q, ranked, k, adapter.name, latencyMs);
+ rows.push(row);
+ appendFileSync(ndjsonPath, JSON.stringify(row) + "\n");
+ const mark = row.hit ? "+" : "-";
+ console.log(
+ ` ${mark} ${q.id} [${q.type}] R@${k}=${row.recallAtK.toFixed(2)} (${Math.round(latencyMs)}ms)`,
+ );
+ }
+ } finally {
+ if (adapter.teardown) await adapter.teardown(state);
+ }
+ }
+
+ const agg = aggregate(rows);
+ writeFileSync(`${outDir}/summary.json`, JSON.stringify(agg, null, 2));
+ console.log("\n=== Summary ===");
+ for (const [adapter, stats] of Object.entries(agg.byAdapter)) {
+ console.log(
+ ` ${adapter.padEnd(22)} P@${k}=${stats.p.toFixed(3)} R@${k}=${stats.r.toFixed(3)} hit=${stats.hit}/${stats.n} p50=${Math.round(stats.latencyP50)}ms`,
+ );
+ }
+}
+
+main().catch((err) => {
+ console.error(err);
+ process.exit(1);
+});
diff --git a/eval/runner/load.ts b/eval/runner/load.ts
new file mode 100644
index 00000000..aece2452
--- /dev/null
+++ b/eval/runner/load.ts
@@ -0,0 +1,54 @@
+import { readFileSync } from "node:fs";
+import type { Question, Session } from "./types.js";
+
+interface LongMemEvalRaw {
+ question_id: string;
+ question_type: string;
+ question: string;
+ answer?: string;
+ answer_session_ids: string[];
+ haystack_session_ids: string[];
+ haystack_sessions: Array<Array<{ role: string; content: string }>>;
+}
+
+function flattenSession(turns: Array<{ role: string; content: string }>): string {
+ return turns.map((t) => `[${t.role}] ${t.content}`).join("\n\n");
+}
+
+export function loadLongMemEval(path: string, limit?: number): Question[] {
+ const raw = JSON.parse(readFileSync(path, "utf8")) as LongMemEvalRaw[];
+ const slice = typeof limit === "number" ? raw.slice(0, limit) : raw;
+ const questions: Question[] = [];
+ for (const r of slice) {
+ if (r.haystack_session_ids.length !== r.haystack_sessions.length) {
+ throw new Error(
+ `LongMemEval row ${r.question_id}: haystack_session_ids (${r.haystack_session_ids.length}) and haystack_sessions (${r.haystack_sessions.length}) length mismatch`,
+ );
+ }
+ const haystack: Session[] = r.haystack_session_ids.map((id, i) => ({
+ id,
+ content: flattenSession(r.haystack_sessions[i]),
+ }));
+ questions.push({
+ id: r.question_id,
+ type: r.question_type,
+ question: r.question,
+ answer: r.answer,
+ goldSessionIds: r.answer_session_ids,
+ haystack,
+ });
+ }
+ return questions;
+}
+
+export function stratifySample(questions: Question[], perType: number): Question[] {
+ const buckets: Record<string, Question[]> = {};
+ for (const q of questions) {
+ (buckets[q.type] ??= []).push(q);
+ }
+ const out: Question[] = [];
+ for (const type of Object.keys(buckets).sort()) {
+ out.push(...buckets[type].slice(0, perType));
+ }
+ return out;
+}
diff --git a/eval/runner/longmemeval.ts b/eval/runner/longmemeval.ts
new file mode 100644
index 00000000..a906fa21
--- /dev/null
+++ b/eval/runner/longmemeval.ts
@@ -0,0 +1,126 @@
+import { existsSync, mkdirSync, writeFileSync, appendFileSync } from "node:fs";
+import { dirname, resolve } from "node:path";
+import { parseArgs } from "node:util";
+import { agentmemoryAdapter } from "./adapters/agentmemory.js";
+import { grepAdapter } from "./adapters/grep.js";
+import { vectorAdapter } from "./adapters/vector.js";
+import { loadLongMemEval, stratifySample } from "./load.js";
+import { aggregate, scoreQuestion } from "./score.js";
+import type { Adapter, ScoreRow } from "./types.js";
+
+const ADAPTERS: Record<string, Adapter> = {
+ grep: grepAdapter as unknown as Adapter,
+ vector: vectorAdapter as unknown as Adapter,
+ agentmemory: agentmemoryAdapter as unknown as Adapter,
+};
+
+interface CliOptions {
+ data: string;
+ adapters: string;
+ k: string;
+ limit?: string;
+ stratify?: string;
+ out: string;
+}
+
+function parse(): CliOptions {
+ const { values } = parseArgs({
+ options: {
+ data: { type: "string", default: process.env.LONGMEMEVAL_PATH ?? "" },
+ adapters: { type: "string", default: "grep,vector,agentmemory" },
+ k: { type: "string", default: "5" },
+ limit: { type: "string" },
+ stratify: { type: "string" },
+ out: { type: "string", default: "eval/reports/longmemeval" },
+ },
+ });
+ return values as unknown as CliOptions;
+}
+
+async function main(): Promise<void> {
+ const opts = parse();
+ if (!opts.data) {
+ console.error("--data <path/to/longmemeval_s.json> required (or LONGMEMEVAL_PATH env)");
+ process.exit(2);
+ }
+ const k = Number(opts.k);
+ if (!Number.isInteger(k) || k <= 0) {
+ console.error(`--k must be a positive integer, got: ${opts.k}`);
+ process.exit(2);
+ }
+ let limit: number | undefined;
+ if (opts.limit !== undefined) {
+ limit = Number(opts.limit);
+ if (!Number.isInteger(limit) || limit <= 0) {
+ console.error(`--limit must be a positive integer, got: ${opts.limit}`);
+ process.exit(2);
+ }
+ }
+ let perType: number | undefined;
+ if (opts.stratify !== undefined) {
+ perType = Number(opts.stratify);
+ if (!Number.isInteger(perType) || perType <= 0) {
+ console.error(`--stratify must be a positive integer, got: ${opts.stratify}`);
+ process.exit(2);
+ }
+ }
+ const adapterNames = opts.adapters.split(",").map((s) => s.trim()).filter(Boolean);
+ for (const a of adapterNames) {
+ if (!ADAPTERS[a]) {
+ console.error(`unknown adapter: ${a}. options: ${Object.keys(ADAPTERS).join(",")}`);
+ process.exit(2);
+ }
+ }
+ let questions = loadLongMemEval(resolve(opts.data), limit);
+ if (perType) questions = stratifySample(questions, perType);
+ console.log(
+ `loaded ${questions.length} questions, adapters: ${adapterNames.join(",")}, k=${k}`,
+ );
+
+ const outDir = resolve(opts.out);
+ mkdirSync(outDir, { recursive: true });
+ const ndjsonPath = `${outDir}/scores.ndjson`;
+ if (existsSync(ndjsonPath)) writeFileSync(ndjsonPath, "");
+ mkdirSync(dirname(ndjsonPath), { recursive: true });
+
+ const rows: ScoreRow[] = [];
+ for (const adapterName of adapterNames) {
+ const adapter = ADAPTERS[adapterName];
+ console.log(`\n== ${adapter.name} ==`);
+ for (const q of questions) {
+ const t0 = performance.now();
+ const state = await adapter.init(q.haystack);
+ try {
+ const ranked = await adapter.query(q.question, state, k);
+ const latencyMs = performance.now() - t0;
+ const row = scoreQuestion(q, ranked, k, adapter.name, latencyMs);
+ rows.push(row);
+ appendFileSync(ndjsonPath, JSON.stringify(row) + "\n");
+ const mark = row.hit ? "+" : "-";
+ console.log(
+ ` ${mark} ${q.id} [${q.type}] R@${k}=${row.recallAtK.toFixed(2)} (${Math.round(latencyMs)}ms)`,
+ );
+ } finally {
+ if (adapter.teardown) await adapter.teardown(state);
+ }
+ }
+ }
+
+ const agg = aggregate(rows);
+ const summaryPath = `${outDir}/summary.json`;
+ writeFileSync(summaryPath, JSON.stringify(agg, null, 2));
+
+ console.log("\n=== Summary ===");
+ for (const [adapter, stats] of Object.entries(agg.byAdapter)) {
+ console.log(
+ ` ${adapter.padEnd(22)} P@${k}=${stats.p.toFixed(3)} R@${k}=${stats.r.toFixed(3)} hit=${stats.hit}/${stats.n} p50=${Math.round(stats.latencyP50)}ms`,
+ );
+ }
+ console.log(`\nwrote ${ndjsonPath}`);
+ console.log(`wrote ${summaryPath}`);
+}
+
+main().catch((err) => {
+ console.error(err);
+ process.exit(1);
+});
diff --git a/eval/runner/score.ts b/eval/runner/score.ts
new file mode 100644
index 00000000..b21d30ca
--- /dev/null
+++ b/eval/runner/score.ts
@@ -0,0 +1,78 @@
+import type { Question, RankedDoc, ScoreRow } from "./types.js";
+
+export function scoreQuestion(
+ q: Question,
+ ranked: RankedDoc[],
+ k: number,
+ adapter: string,
+ latencyMs: number,
+): ScoreRow {
+ const topK = ranked.slice(0, k).map((r) => r.sessionId);
+ const gold = new Set(q.goldSessionIds);
+ const hits = topK.filter((id) => gold.has(id)).length;
+ const precisionAtK = k > 0 ? hits / k : 0;
+ const recallAtK = gold.size === 0 ? 0 : hits / gold.size;
+ const hit = hits > 0;
+ let topGoldRank: number | null = null;
+ for (let i = 0; i < ranked.length; i++) {
+ if (gold.has(ranked[i].sessionId)) {
+ topGoldRank = i + 1;
+ break;
+ }
+ }
+ return {
+ questionId: q.id,
+ questionType: q.type,
+ adapter,
+ k,
+ precisionAtK,
+ recallAtK,
+ hit,
+ topGoldRank,
+ latencyMs,
+ };
+}
+
+export function aggregate(rows: ScoreRow[]): {
+ byAdapter: Record<string, { p: number; r: number; hit: number; n: number; latencyP50: number }>;
+ byType: Record<string, Record<string, { p: number; r: number; hit: number; n: number }>>;
+} {
+ const byAdapter: Record<
+ string,
+ { p: number; r: number; hit: number; n: number; latencyP50: number }
+ > = {};
+ const latencies: Record<string, number[]> = {};
+ for (const r of rows) {
+ const a = (byAdapter[r.adapter] ??= { p: 0, r: 0, hit: 0, n: 0, latencyP50: 0 });
+ a.p += r.precisionAtK;
+ a.r += r.recallAtK;
+ a.hit += r.hit ? 1 : 0;
+ a.n += 1;
+ (latencies[r.adapter] ??= []).push(r.latencyMs);
+ }
+ for (const adapter of Object.keys(byAdapter)) {
+ const a = byAdapter[adapter];
+ a.p = a.p / a.n;
+ a.r = a.r / a.n;
+ const sorted = latencies[adapter].slice().sort((x, y) => x - y);
+ a.latencyP50 = sorted[Math.floor(sorted.length / 2)] ?? 0;
+ }
+ const byType: Record<string, Record<string, { p: number; r: number; hit: number; n: number }>> =
+ {};
+ for (const r of rows) {
+ const t = (byType[r.questionType] ??= {});
+ const a = (t[r.adapter] ??= { p: 0, r: 0, hit: 0, n: 0 });
+ a.p += r.precisionAtK;
+ a.r += r.recallAtK;
+ a.hit += r.hit ? 1 : 0;
+ a.n += 1;
+ }
+ for (const t of Object.keys(byType)) {
+ for (const adapter of Object.keys(byType[t])) {
+ const a = byType[t][adapter];
+ a.p = a.p / a.n;
+ a.r = a.r / a.n;
+ }
+ }
+ return { byAdapter, byType };
+}
diff --git a/eval/runner/types.ts b/eval/runner/types.ts
new file mode 100644
index 00000000..e72a6408
--- /dev/null
+++ b/eval/runner/types.ts
@@ -0,0 +1,38 @@
+export interface Session {
+ id: string;
+ timestamp?: string;
+ content: string;
+}
+
+export interface Question {
+ id: string;
+ type: string;
+ question: string;
+ answer?: string;
+ goldSessionIds: string[];
+ haystack: Session[];
+}
+
+export interface RankedDoc {
+ sessionId: string;
+ score: number;
+}
+
+export interface Adapter<State = unknown> {
+ name: string;
+ init(sessions: Session[], config?: Record<string, unknown>): Promise<State>;
+ query(q: string, state: State, k: number): Promise<RankedDoc[]>;
+ teardown?(state: State): Promise<void>;
+}
+
+export interface ScoreRow {
+ questionId: string;
+ questionType: string;
+ adapter: string;
+ k: number;
+ precisionAtK: number;
+ recallAtK: number;
+ hit: boolean;
+ topGoldRank: number | null;
+ latencyMs: number;
+}
diff --git a/eval/scripts/sandbox.sh b/eval/scripts/sandbox.sh
new file mode 100755
index 00000000..5d402330
--- /dev/null
+++ b/eval/scripts/sandbox.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Boot a sandboxed agentmemory + iii-engine on alt ports with a clean data dir,
+# so eval runs aren't polluted by (and don't pollute) your real ~/.agentmemory.
+# Source it: `source eval/scripts/sandbox.sh` then run eval scripts;
+# the sandbox is torn down on EXIT.
+
+set -euo pipefail
+
+SANDBOX_ROOT="${SANDBOX_ROOT:-/tmp/agentmemory-eval-sandbox}"
+SANDBOX_PORT="${SANDBOX_PORT:-3411}"
+SANDBOX_STREAM_PORT="${SANDBOX_STREAM_PORT:-3412}"
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+
+if ! command -v iii >/dev/null 2>&1; then
+ echo "iii binary not on PATH. Install pinned version:"
+ echo " curl -fsSL https://github.com/iii-hq/iii/releases/download/iii/v0.11.2/iii-aarch64-apple-darwin.tar.gz | tar -xz -C ~/.local/bin"
+ exit 1
+fi
+
+iii_ver=$(iii --version 2>&1 | head -1)
+if [[ "$iii_ver" != "0.11.2" ]]; then
+ echo "warning: iii version on PATH is $iii_ver; agentmemory pins 0.11.2"
+fi
+
+if [[ ! -f "$REPO_ROOT/dist/index.mjs" ]]; then
+ echo "dist/ missing. Run: npm run build" >&2
+ exit 1
+fi
+
+if [[ -z "${SANDBOX_ROOT:-}" || "$SANDBOX_ROOT" == "/" || "$SANDBOX_ROOT" != /tmp/* ]]; then
+ echo "refusing to wipe SANDBOX_ROOT='$SANDBOX_ROOT' — must be non-empty and under /tmp/" >&2
+ exit 1
+fi
+rm -rf "$SANDBOX_ROOT"
+mkdir -p "$SANDBOX_ROOT/data" "$SANDBOX_ROOT/.agentmemory"
+
+cat > "$SANDBOX_ROOT/iii-config.yaml" <<EOF
+workers:
+ - name: iii-http
+ config:
+ port: $SANDBOX_PORT
+ host: 127.0.0.1
+ default_timeout: 180000
+ cors:
+ allowed_origins: ["http://localhost:$SANDBOX_PORT", "http://127.0.0.1:$SANDBOX_PORT"]
+ allowed_methods: [GET, POST, PUT, DELETE, OPTIONS]
+ - name: iii-state
+ config:
+ adapter:
+ name: kv
+ config:
+ store_method: file_based
+ file_path: $SANDBOX_ROOT/data/state_store.db
+ - name: iii-queue
+ config:
+ adapter:
+ name: builtin
+ - name: iii-pubsub
+ config:
+ adapter:
+ name: local
+ - name: iii-cron
+ config:
+ adapter:
+ name: kv
+ - name: iii-stream
+ config:
+ port: $SANDBOX_STREAM_PORT
+ host: 127.0.0.1
+ adapter:
+ name: kv
+ config:
+ store_method: file_based
+ file_path: $SANDBOX_ROOT/data/stream_store
+ - name: iii-observability
+ config:
+ enabled: true
+ service_name: agentmemory-eval
+ exporter: memory
+ sampling_ratio: 1.0
+ metrics_enabled: true
+ logs_enabled: false
+ logs_console_output: false
+ - name: iii-exec
+ config:
+ exec:
+ - node $REPO_ROOT/dist/index.mjs
+EOF
+
+cd "$SANDBOX_ROOT"
+HOME="$SANDBOX_ROOT" iii --config "$SANDBOX_ROOT/iii-config.yaml" > "$SANDBOX_ROOT/iii.log" 2>&1 &
+SANDBOX_PID=$!
+
+cleanup() {
+ echo "tearing down sandbox (pid $SANDBOX_PID)"
+ kill "$SANDBOX_PID" 2>/dev/null || true
+ sleep 1
+ kill -9 "$SANDBOX_PID" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+# wait for livez
+for i in $(seq 1 30); do
+ if curl -sS --max-time 1 "http://localhost:$SANDBOX_PORT/agentmemory/livez" 2>/dev/null | grep -q '"status":"ok"'; then
+ export AGENTMEMORY_BASE_URL="http://localhost:$SANDBOX_PORT"
+ echo "sandbox ready: $AGENTMEMORY_BASE_URL"
+ echo " state: $SANDBOX_ROOT/data/"
+ echo " logs: $SANDBOX_ROOT/iii.log"
+ return 0 2>/dev/null || exit 0
+ fi
+ sleep 1
+done
+
+echo "sandbox failed to come up within 30s. last log lines:" >&2
+tail -10 "$SANDBOX_ROOT/iii.log" >&2
+exit 1
diff --git a/package.json b/package.json
index e775c305..ae7b92ed 100644
--- a/package.json
+++ b/package.json
@@ -25,7 +25,9 @@
"test:watch": "vitest --exclude test/integration.test.ts",
"test:integration": "vitest run test/integration.test.ts",
"test:all": "vitest run",
- "bench:load": "node --import tsx benchmark/load-100k.ts"
+ "bench:load": "node --import tsx benchmark/load-100k.ts",
+ "eval:longmemeval": "tsx eval/runner/longmemeval.ts",
+ "eval:coding-life": "tsx eval/runner/coding-life.ts"
},
"keywords": [
"ai",
diff --git a/test/eval-adapters.test.ts b/test/eval-adapters.test.ts
new file mode 100644
index 00000000..90f914f5
--- /dev/null
+++ b/test/eval-adapters.test.ts
@@ -0,0 +1,92 @@
+import { describe, it, expect } from "vitest";
+import { readFileSync } from "node:fs";
+import { resolve } from "node:path";
+import { grepAdapter } from "../eval/runner/adapters/grep.js";
+import { aggregate, scoreQuestion } from "../eval/runner/score.js";
+import type { Question, Session } from "../eval/runner/types.js";
+
+const DATA_DIR = resolve(__dirname, "..", "eval", "data", "coding-agent-life-v1");
+const sessions = JSON.parse(readFileSync(`${DATA_DIR}/sessions.json`, "utf8")) as Session[];
+const queries = JSON.parse(readFileSync(`${DATA_DIR}/queries.json`, "utf8")) as Array<
+ Omit<Question, "haystack">
+>;
+
+describe("eval scaffold", () => {
+ it("coding-agent-life-v1 corpus is well-formed", () => {
+ expect(sessions.length).toBeGreaterThan(0);
+ expect(queries.length).toBeGreaterThan(0);
+ const sessionIds = new Set(sessions.map((s) => s.id));
+ for (const q of queries) {
+ expect(q.goldSessionIds.length).toBeGreaterThan(0);
+ for (const id of q.goldSessionIds) {
+ expect(sessionIds.has(id)).toBe(true);
+ }
+ }
+ });
+
+ it("grep adapter ranks gold session in top-5 for most queries", async () => {
+ const state = await grepAdapter.init(sessions);
+ let hits = 0;
+ for (const q of queries) {
+ const ranked = await grepAdapter.query(q.question, state, 5);
+ const topIds = new Set(ranked.map((r) => r.sessionId));
+ if (q.goldSessionIds.some((id) => topIds.has(id))) hits += 1;
+ }
+ expect(hits / queries.length).toBeGreaterThan(0.5);
+ });
+
+ it("scoreQuestion computes P@K, R@K, hit, topGoldRank", () => {
+ const q: Question = {
+ id: "test",
+ type: "single-session",
+ question: "?",
+ goldSessionIds: ["a", "b"],
+ haystack: [],
+ };
+ const ranked = [
+ { sessionId: "x", score: 0.9 },
+ { sessionId: "a", score: 0.7 },
+ { sessionId: "y", score: 0.5 },
+ { sessionId: "b", score: 0.3 },
+ ];
+ const row = scoreQuestion(q, ranked, 5, "test", 12);
+ expect(row.hit).toBe(true);
+ expect(row.recallAtK).toBe(1);
+ expect(row.precisionAtK).toBeCloseTo(2 / 5);
+ expect(row.topGoldRank).toBe(2);
+ });
+
+ it("scoreQuestion handles miss", () => {
+ const q: Question = {
+ id: "test",
+ type: "x",
+ question: "?",
+ goldSessionIds: ["a"],
+ haystack: [],
+ };
+ const ranked = [
+ { sessionId: "x", score: 1 },
+ { sessionId: "y", score: 0.5 },
+ ];
+ const row = scoreQuestion(q, ranked, 5, "test", 5);
+ expect(row.hit).toBe(false);
+ expect(row.recallAtK).toBe(0);
+ expect(row.topGoldRank).toBeNull();
+ });
+
+ it("aggregate computes per-adapter and per-type means", () => {
+ const q: Question = {
+ id: "1",
+ type: "t1",
+ question: "?",
+ goldSessionIds: ["a"],
+ haystack: [],
+ };
+ const row1 = scoreQuestion(q, [{ sessionId: "a", score: 1 }], 5, "grep", 10);
+ const row2 = scoreQuestion(q, [{ sessionId: "x", score: 1 }], 5, "grep", 20);
+ const agg = aggregate([row1, row2]);
+ expect(agg.byAdapter.grep.hit).toBe(1);
+ expect(agg.byAdapter.grep.n).toBe(2);
+ expect(agg.byType.t1.grep.n).toBe(2);
+ });
+});