Solution for long term memory for agent coding CLIs and to facilitate handoff between different agent vendors
Rust
Merge branch 'main' into pr-557; port the eval ProviderConfig field to ab.rs
AkitaOnRails448483dc2e2b
diff --cc CHANGELOG.mdindex f9ba47e9,66e4e6e8..321811e7--- a/CHANGELOG.md+++ b/CHANGELOG.md@@@ -7,19 -7,249 +7,262 @@@ and this project adheres to [Semantic V## [Unreleased]+### Added++- `status` tells more of the truth (2.0 status audit): the wiki-format++ line (OKF migration state and the pre-migration backup archive while++ it exists), stored embedding triples by provider/model/dim, typed-edge++ counts by relation, and a write-queue depth gauge that surfaces a++ backpressured (wedged) writer — shown only when non-zero.++- An opt-in cross-session "experience" pass++ (`[auto_improve.scheduler] experience_every_sessions`): every N newly++ completed sessions, the last few session summaries are reviewed side++ by side and knowledge visible only ACROSS trajectories — repeated++ workflows, re-stated preferences, architecture facts every session++ re-discovers, contradictions with stored decisions — is proposed++ through the identical schema-constrained, validated, eval-gated,++ pending-writes staging path as per-session auto-improve. Evidence++ must span at least two sessions; off by default and LLM-hosted, so++ the zero-LLM path is untouched. See `docs/experience.md`.++- `AI_MEMORY_EMBEDDING_PROVIDER=local` — in-process sentence embeddings++ (pure-Rust candle BERT, `all-MiniLM-L6-v2`, 384-dim) with no API key++ and no external server. The ~87 MB model is fetched once into++ `<data_dir>/models/` with source-pinned sha256s (offline installs drop++ the files in manually); vectors coexist with any provider's via the++ stored `(provider, model, dim)` triple, and switching is a config++ change. Feature-gated (`local-embeddings`, default on). The eval++ harness gained `--embeddings local` and publishes the hybrid++ LongMemEval row next to the zero-LLM baseline. See++ `docs/local-embeddings.md`.++- The entity index carries ingestion-time validity windows++ (bi-temporal-lite): entity links open at their page version's creation++ and close when the version is superseded, backfilled for existing++ stores by an additive migration. `memory_query` gains an `as_of`++ ISO-8601 argument that turns the call into an entity-timeline lookup —++ "what did we know about X then" — returning the page versions valid at++ that instant, including ones superseded since. Ingestion time only,++ documented honestly as such. See `docs/temporal.md`.++- Pages can declare typed relation edges in `relations:` frontmatter —++ a closed `causes` / `fixes` / `contradicts` vocabulary riding the++ existing `links.link_type` column (no schema migration). Declared++ `contradicts` edges surface as zero-LLM `contradiction` lint findings++ (including stale declarations whose target no longer resolves), and++ the session-end consolidation prompt may emit relations under the++ same schema-constrained vocabulary. See `docs/typed-edges.md`.++- `ai-memory export-okf --project <p> -o <bundle.tar.gz>` and++ `POST /admin/export-okf` — stream one project's wiki as a validated++ OKF v0.2 bundle with a freshly generated index; a non-conformant page++ fails the export. Importing needs no command: unpack a bundle's++ concept files into a project's wiki directory and the watcher (or++ `reindex`) ingests them.++- Added a LongMemEval retrieval benchmark to the evaluation harness++ (`cargo run -p ai-memory-eval -- retrieval`). It drives a real++ `ai-memory serve` subprocess end to end: haystack chat histories replay++ through `POST /hook/batch` at the production hook cadence, questions run++ through MCP `memory_query`, and results are scored session-level++ (`hit@k` / `recall@k` per question category, abstention questions++ reported separately). The 278 MB dataset is fetched on demand with a++ pinned sha256; baselines are published under `docs/benchmarks/`. The++ existing LLM A/B harness moved to the `ab` subcommand.+- `llm_reasoning_effort` / `AI_MEMORY_LLM_REASONING_EFFORT` is a typed+ enum (`none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`,+ `ultra`, `persistent`); unknown values fail at config load. Unset keeps+ the model default. Each chat provider maps the value to its native+ field: OpenAI `reasoning_effort`, OpenRouter `reasoning.effort`,+ xAI Grok `reasoning_effort`, Anthropic `output_config.effort` (plus+ adaptive/disabled thinking on models that accept it), and Codex+ Responses `reasoning.effort`. Gemini and Copilot ignore the key.+ Host-unsupported values are clamped (`ultra`/`persistent` → `max` on+ OpenAI/OpenRouter/Codex; Anthropic `xhigh`/`max` per model; `none`+ does not send `thinking: disabled` on always-on Claude models).+++||||||| a613bf6d+ ### Fixed+ - Merging a project no longer reports a spurious content conflict when+ the only difference is `generated.at`: the merge comparison now uses+ the same modulo-timestamp projection as the store's idempotency rule.+ The old raw comparison made conflict detection timing-dependent — the+ identical page re-written across a second boundary 409'd as+ "different content" (the `copy_purge_rerun_is_idempotent` flake,+ root-caused via its diagnosable failure body on the Windows matrix+ run and now pinned by a deterministic cross-boundary test).+ - `install-hooks` now writes the capture-mode opt-in atomically. It was+ written in place, and every reader maps an unrecognised value onto+ `denylist` — the less private mode — so a crash, a full disk or a killed+ `ai-memory upgrade` mid-write left a truncated file that silently reverted+ an `--capture-mode allowlist` opt-in to capture-by-default, which is exactly+ what `persist_capture_mode` documents it must never do. The fallback itself+ is correct and stays; the torn write that triggered it is gone. Now routed+ through the same `write_atomic` (tmp + fsync + rename) the rest of the CLI+ already uses, per the project's atomic-writes invariant.+ ||||||| parent of f14775e5 (fix: post-audit findings across the 2.0 range)+ - The 2.0 pre-release audit findings (post-merge adversarial review of+ the whole range): `export-okf` no longer fails on real deployments —+ scope manifests are OKF-typed at their writer (ending a startup+ tug-of-war that reverted the migration's typing on the same boot and+ self-healing manifests reverted by pre-fix binaries), approved+ auto-improve pages land conformant like every other write (also+ preventing phantom supersedes on the first reindex after a binary+ upgrade), and `_pending/` proposal sidecars are excluded from the+ export walk. Pages retired WITHOUT a successor (decay tombstones,+ purge-regenerate, workspace merges) now close their entity-link+ validity windows so `as_of` cannot resurrect retired knowledge, with+ a V58 backfill for previously retired rows. The experience pass no+ longer burns its cadence window or stages an empty run when a scope+ has too few session pages. Backup destination guards canonicalize+ paths; `as_of` parse errors return invalid-params; the FTS5 probe+ connection is cached per thread.+ - `status` no longer overstates missing embeddings: empty-body pages —+ which no embedder can ever cover and the backfill skips by rule — are+ excluded from the "latest pages missing" figure and reported on their+ own line (observed live as a permanently stuck 427).+ - Natural-language searches no longer surface stopword trash: bare+ queries drop English stopwords before the FTS5 OR-join, so a page+ containing five "the"s cannot outrank the page whose content matches,+ and pages matching only stopwords no longer appear at all. Quoted+ phrases and explicit-operator queries are untouched, and an+ all-stopword query still searches its literal terms. Guarded by a+ CI-runnable search-quality suite asserting ranking usefulness, not+ just machinery.+ - `memory_query` no longer fails with `fts5: syntax error` when the query is+ natural language containing parentheses, apostrophe-quoted phrases, or a+ stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and+ the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now+ validated against the engine's own parser and degrade to an always-valid+ quoted bag of words when the preserved operator form does not parse;+ deliberate well-formed operator queries are preserved as before. Found by+ the new LongMemEval retrieval harness on its first full run.++ - `ai-memory serve` now takes an exclusive lock on `<data-dir>/.serve.lock`+ before opening the store, so a second server pointed at the same data+ directory refuses at startup and names the holding process instead of+ silently contending the single-writer actor, the wiki git handle, and the+ active-project pointer. The OS releases the lock when the process exits,+ so a crashed server never leaves the operator locked out; `--force` starts+ unguarded for an operator who knows the previous server is gone, and a+ filesystem that cannot lock at all downgrades the guard to a warning+ instead of refusing to start. (#563)++ - `install-hooks` now writes the capture-mode opt-in atomically. It was+ written in place, and every reader maps an unrecognised value onto+ `denylist` — the less private mode — so a crash, a full disk or a killed+ `ai-memory upgrade` mid-write left a truncated file that silently reverted+ an `--capture-mode allowlist` opt-in to capture-by-default, which is exactly+ what `persist_capture_mode` documents it must never do. The fallback itself+ is correct and stays; the torn write that triggered it is gone. Now routed+ through the same `write_atomic` (tmp + fsync + rename) the rest of the CLI+ already uses, per the project's atomic-writes invariant.+ - `status` no longer overstates missing embeddings: empty-body pages —+ which no embedder can ever cover and the backfill skips by rule — are+ excluded from the "latest pages missing" figure and reported on their+ own line (observed live as a permanently stuck 427).+ - Natural-language searches no longer surface stopword trash: bare+ queries drop English stopwords before the FTS5 OR-join, so a page+ containing five "the"s cannot outrank the page whose content matches,+ and pages matching only stopwords no longer appear at all. Quoted+ phrases and explicit-operator queries are untouched, and an+ all-stopword query still searches its literal terms. Guarded by a+ CI-runnable search-quality suite asserting ranking usefulness, not+ just machinery.+ - `memory_query` no longer fails with `fts5: syntax error` when the query is+ natural language containing parentheses, apostrophe-quoted phrases, or a+ stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and+ the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now+ validated against the engine's own parser and degrade to an always-valid+ quoted bag of words when the preserved operator form does not parse;+ deliberate well-formed operator queries are preserved as before. Found by+ the new LongMemEval retrieval harness on its first full run.++ - `ai-memory serve` now takes an exclusive lock on `<data-dir>/.serve.lock`+ before opening the store, so a second server pointed at the same data+ directory refuses at startup and names the holding process instead of+ silently contending the single-writer actor, the wiki git handle, and the+ active-project pointer. The OS releases the lock when the process exits,+ so a crashed server never leaves the operator locked out; `--force` starts+ unguarded for an operator who knows the previous server is gone, and a+ filesystem that cannot lock at all downgrades the guard to a warning+ instead of refusing to start. (#563)++ - `install-hooks` now writes the capture-mode opt-in atomically. It was+ written in place, and every reader maps an unrecognised value onto+ `denylist` — the less private mode — so a crash, a full disk or a killed+ `ai-memory upgrade` mid-write left a truncated file that silently reverted+ an `--capture-mode allowlist` opt-in to capture-by-default, which is exactly+ what `persist_capture_mode` documents it must never do. The fallback itself+ is correct and stays; the torn write that triggered it is gone. Now routed+ through the same `write_atomic` (tmp + fsync + rename) the rest of the CLI+ already uses, per the project's atomic-writes invariant.+ - `status` no longer overstates missing embeddings: empty-body pages —+ which no embedder can ever cover and the backfill skips by rule — are+ excluded from the "latest pages missing" figure and reported on their+ own line (observed live as a permanently stuck 427).+ - Natural-language searches no longer surface stopword trash: bare+ queries drop English stopwords before the FTS5 OR-join, so a page+ containing five "the"s cannot outrank the page whose content matches,+ and pages matching only stopwords no longer appear at all. Quoted+ phrases and explicit-operator queries are untouched, and an+ all-stopword query still searches its literal terms. Guarded by a+ CI-runnable search-quality suite asserting ranking usefulness, not+ just machinery.+ - `memory_query` no longer fails with `fts5: syntax error` when the query is+ natural language containing parentheses, apostrophe-quoted phrases, or a+ stray `OR`/`AND` (e.g. *"my visit to the Museum of Modern Art (MoMA) and+ the 'Ancient Civilizations' exhibit"*). Prepared FTS5 queries are now+ validated against the engine's own parser and degrade to an always-valid+ quoted bag of words when the preserved operator form does not parse;+ deliberate well-formed operator queries are preserved as before. Found by+ the new LongMemEval retrieval harness on its first full run.++ - `ai-memory serve` now takes an exclusive lock on `<data-dir>/.serve.lock`+ before opening the store, so a second server pointed at the same data+ directory refuses at startup and names the holding process instead of+ silently contending the single-writer actor, the wiki git handle, and the+ active-project pointer. The OS releases the lock when the process exits,+ so a crashed server never leaves the operator locked out; `--force` starts+ unguarded for an operator who knows the previous server is gone, and a+ filesystem that cannot lock at all downgrades the guard to a warning+ instead of refusing to start. (#563)++ - Preserved third-party string-form lifecycle hooks whose commands merely+ contain `ai-memory` or `ai_memory` when `install-hooks --apply` refreshes+ ai-memory's own entries. Legacy ai-memory script and native hook commands+ remain recognized by their specific hook signatures.+ ||||||| a613bf6d-### Added-- `status` tells more of the truth (2.0 status audit): the wiki-format- line (OKF migration state and the pre-migration backup archive while- it exists), stored embedding triples by provider/model/dim, typed-edge- counts by relation, and a write-queue depth gauge that surfaces a- backpressured (wedged) writer — shown only when non-zero.-- An opt-in cross-session "experience" pass- (`[auto_improve.scheduler] experience_every_sessions`): every N newly- completed sessions, the last few session summaries are reviewed side- by side and knowledge visible only ACROSS trajectories — repeated- workflows, re-stated preferences, architecture facts every session- re-discovers, contradictions with stored decisions — is proposed- through the identical schema-constrained, validated, eval-gated,- pending-writes staging path as per-session auto-improve. Evidence- must span at least two sessions; off by default and LLM-hosted, so- the zero-LLM path is untouched. See `docs/experience.md`.-- `AI_MEMORY_EMBEDDING_PROVIDER=local` — in-process sentence embeddings- (pure-Rust candle BERT, `all-MiniLM-L6-v2`, 384-dim) with no API key- and no external server. The ~87 MB model is fetched once into- `<data_dir>/models/` with source-pinned sha256s (offline installs drop- the files in manually); vectors coexist with any provider's via the- stored `(provider, model, dim)` triple, and switching is a config- change. Feature-gated (`local-embeddings`, default on). The eval- harness gained `--embeddings local` and publishes the hybrid- LongMemEval row next to the zero-LLM baseline. See- `docs/local-embeddings.md`.-- The entity index carries ingestion-time validity windows- (bi-temporal-lite): entity links open at their page version's creation- and close when the version is superseded, backfilled for existing- stores by an additive migration. `memory_query` gains an `as_of`- ISO-8601 argument that turns the call into an entity-timeline lookup —- "what did we know about X then" — returning the page versions valid at- that instant, including ones superseded since. Ingestion time only,- documented honestly as such. See `docs/temporal.md`.-- Pages can declare typed relation edges in `relations:` frontmatter —- a closed `causes` / `fixes` / `contradicts` vocabulary riding the- existing `links.link_type` column (no schema migration). Declared- `contradicts` edges surface as zero-LLM `contradiction` lint findings- (including stale declarations whose target no longer resolves), and- the session-end consolidation prompt may emit relations under the- same schema-constrained vocabulary. See `docs/typed-edges.md`.-- `ai-memory export-okf --project <p> -o <bundle.tar.gz>` and- `POST /admin/export-okf` — stream one project's wiki as a validated- OKF v0.2 bundle with a freshly generated index; a non-conformant page- fails the export. Importing needs no command: unpack a bundle's- concept files into a project's wiki directory and the watcher (or- `reindex`) ingests them.-- Added a LongMemEval retrieval benchmark to the evaluation harness- (`cargo run -p ai-memory-eval -- retrieval`). It drives a real- `ai-memory serve` subprocess end to end: haystack chat histories replay- through `POST /hook/batch` at the production hook cadence, questions run- through MCP `memory_query`, and results are scored session-level- (`hit@k` / `recall@k` per question category, abstention questions- reported separately). The 278 MB dataset is fetched on demand with a- pinned sha256; baselines are published under `docs/benchmarks/`. The- existing LLM A/B harness moved to the `ab` subcommand.+ ### Changed+ - Hybrid retrieval is on by default: an install with no+ `embedding_provider` configured now gets in-process local embeddings+ best-effort — the model downloads in the background on first start+ (hybrid search enables on the next restart), existing pages are+ backfilled by a one-shot startup pass, and hosts that cannot fetch+ the model keep the previous FTS-only behaviour with a warning. Opt+ out with `embedding_provider = "none"`; configured providers are+ never overridden. Measured on LongMemEval-S: overall hit@5 rises+ from 0.617 (FTS-only) to 0.779 with local embeddings.+ - An unscoped `memory_write_page` / `memory_handoff_begin` from a caller whose+ active-project pointer does not resolve is now refused instead of written to+ the server's default project. The page such a write produced was real,+ attributed and searchable, and in a project nobody goes looking in. The error+ names both remedies — pass explicit `workspace`/`project`, or install the+ lifecycle hooks so the pointer is populated. Reads are unchanged: a read+ answering from the default is a wrong answer the caller can see, a write is a+ misfile they cannot. Callers with no identity coordinate at all+ (anonymous/legacy) keep resolving through the shared slot, and an install with+ no pointer information anywhere still resolves to the configured default+ ([#564]).+ - The wiki's on-disk format is now natively the Open Knowledge Format+ (OKF) v0.2: every page write fills the spec's `type`, `generated`,+ `sources` and `stale_after` keys (ai-memory's own fields ride along as+ spec-safe extensions), and each project directory is a conformant OKF+ bundle with an `okf_version` index. Existing stores are migrated+ automatically on the first 2.0 start — **backup-gated**: the entire+ data dir is archived to the user's home (or `AI_MEMORY_BACKUP_DIR`)+ and verified first, or the migration refuses to run; pages are+ rewritten in place (same ids, same version rows, timestamps+ untouched), and the wiki homepage shows where the archive is until it+ is deleted. A data dir migrated by a newer binary is refused by older+ 2.0+ binaries instead of silently mixing formats. See+ `docs/okf.md` and `docs/MIGRATION-2.0.md`.++ ||||||| 7907fd4d## [1.39.0] - 2026-09-01### Addeddiff --cc Cargo.lockindex e671137e,b6def3b7..a1a3246a--- a/Cargo.lock+++ b/Cargo.lock@@@ -2188,18 -2718,9 +2732,18 @@@ source = "registry+https://github.com/rchecksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"dependencies = ["proc-macro2",- "syn",+ "syn 2.0.117",]+[[package]]+name = "proc-macro-crate"+version = "3.5.0"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f"+dependencies = [+ "toml_edit 0.25.13+spec-1.1.0",+]+[[package]]name = "proc-macro2"version = "1.0.106"@@@ -2646,38 -3220,9 +3249,38 @@@ dependencies ="proc-macro2","quote","serde_json",- "syn",+ "syn 2.0.117",]+[[package]]+name = "rstest"+version = "0.26.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "f5a3193c063baaa2a95a33f03035c8a72b83d97a54916055ba22d35ed3839d49"+dependencies = [+ "futures-timer",+ "futures-util",+ "rstest_macros",+]++[[package]]+name = "rstest_macros"+version = "0.26.1"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "9c845311f0ff7951c5506121a9ad75aec44d083c31583b2ea5a30bcb0b0abba0"+dependencies = [+ "cfg-if",+ "glob",+ "proc-macro-crate",+ "proc-macro2",+ "quote",+ "regex",+ "relative-path",+ "rustc_version",- "syn",++ "syn 2.0.117",+ "unicode-ident",+]+[[package]]name = "rusqlite"version = "0.32.1"diff --cc evals/src/ab.rsindex 00000000,a7960404..94c991e8mode 000000,100644..100644--- a/evals/src/ab.rs+++ b/evals/src/ab.rs@@@ -1,0 -1,605 +1,609 @@@+ //! Live A/B harness for the ai-memory consolidation prompt.+ //!+ //! ## What it does+ //!+ //! For each fixture under `evals/fixtures/*.json` (each one a small,+ //! synthetic session log), the runner:+ //!+ //! 1. Calls [`ai_memory_consolidate::build_batch_request`] to build+ //! the EXACT ChatRequest the production consolidator would send.+ //! 2. Sends that request to two providers concurrently — a+ //! *baseline* and a *candidate* — via+ //! [`ai_memory_llm::complete_structured`], which is the same+ //! parse-extract-and-validate path the live system uses.+ //! 3. Saves the deserialised `ConsolidatedBatch`, the raw JSON, a+ //! flattened markdown rendering, and a `meta.json` (timing,+ //! update count, parse status) per fixture per provider under+ //! `evals/runs/<timestamp>/{baseline,candidate}/`.+ //! 4. Prints a side-by-side summary table to stdout.+ //!+ //! Quality judgement is left to the human reading the outputs — the+ //! harness only reports the OBJECTIVE deltas (latency, update+ //! counts, schema-parse success). Anything subtler (faithfulness,+ //! hallucination, structure) is for you to eyeball.+ //!+ //! ## Running it+ //!+ //! See `evals/README.md` for canonical invocations. The short+ //! version:+ //!+ //! ```bash+ //! cargo run -p ai-memory-eval -- ab \+ //! --baseline-provider openai-compat \+ //! --baseline-base-url https://openrouter.ai/api/v1 \+ //! --baseline-model moonshotai/kimi-k2.6 \+ //! --baseline-api-key-env OPENROUTER_API_KEY \+ //! --candidate-provider openai-compat \+ //! --candidate-base-url http://192.168.0.90:11434/v1 \+ //! --candidate-model qwen3:32b \+ //! --candidate-api-key ollama-local+ //! ```++ use std::io::Write as _;+ use std::path::{Path, PathBuf};+ use std::sync::Arc;+ use std::time::{Duration, Instant};++ use anyhow::{Context, Result, bail};+ use jiff::Timestamp;+ use secrecy::SecretString;+ use serde::{Deserialize, Serialize};++ use ai_memory_consolidate::{ConsolidatedBatch, build_batch_request};+ use ai_memory_core::{Observation, ObservationKind, ProjectId, SessionId, WorkspaceId};+ use ai_memory_llm::{AuthRequirement, LlmProvider, ProviderAuth, ProviderChoice, ProviderConfig};++ #[derive(clap::Args, Debug)]+ pub struct AbArgs {+ /// Fixture directory (each `*.json` is one synthetic session log).+ #[arg(long, default_value = "evals/fixtures")]+ fixtures: PathBuf,++ /// Output root. A timestamped subdir gets created here per run.+ #[arg(long, default_value = "evals/runs")]+ out: PathBuf,++ /// Baseline provider (`anthropic` / `openai` / `openai-compat` / `openai-oauth` / `copilot`).+ #[arg(long)]+ baseline_provider: String,++ /// Baseline base URL (omit for native Anthropic/OpenAI).+ #[arg(long)]+ baseline_base_url: Option<String>,++ /// Baseline model id.+ #[arg(long)]+ baseline_model: String,++ /// Baseline API key (raw value). Prefer `--baseline-api-key-env`.+ #[arg(long)]+ baseline_api_key: Option<String>,++ /// Env var to read the baseline API key from.+ #[arg(long)]+ baseline_api_key_env: Option<String>,++ /// Baseline OpenAI OAuth token file for `--baseline-provider openai-oauth`.+ #[arg(long)]+ baseline_token_file: Option<PathBuf>,++ /// Candidate provider.+ #[arg(long)]+ candidate_provider: String,++ /// Candidate base URL.+ #[arg(long)]+ candidate_base_url: Option<String>,++ /// Candidate model id.+ #[arg(long)]+ candidate_model: String,++ /// Candidate API key (raw value).+ #[arg(long)]+ candidate_api_key: Option<String>,++ /// Env var to read the candidate API key from.+ #[arg(long)]+ candidate_api_key_env: Option<String>,++ /// Candidate OpenAI OAuth token file for `--candidate-provider openai-oauth`.+ #[arg(long)]+ candidate_token_file: Option<PathBuf>,+ }++ /// JSON shape of an eval fixture file. We only care about+ /// observations the consolidation prompt actually consumes; the+ /// other Observation fields get populated with dummies at runtime.+ #[derive(Debug, Deserialize)]+ struct Fixture {+ name: String,+ /// Free-text describing the scenario; not consumed by the runner+ /// but useful when reading fixture files by hand.+ #[allow(dead_code)]+ #[serde(default)]+ description: String,+ observations: Vec<FixtureObservation>,+ }++ #[derive(Debug, Deserialize)]+ struct FixtureObservation {+ kind: String,+ title: String,+ #[serde(default)]+ body: String,+ }++ /// One result of one provider on one fixture.+ #[derive(Debug, Serialize)]+ struct CaseRun {+ provider: String,+ fixture: String,+ elapsed_ms: u128,+ parsed_ok: bool,+ error: Option<String>,+ update_count: Option<usize>,+ }++ pub async fn run(args: AbArgs) -> Result<()> {+ let baseline = resolve_provider("baseline", &args)?;+ let candidate = resolve_provider("candidate", &args)?;++ let baseline_provider =+ make_provider(baseline.clone()).context("building baseline provider")?;+ let candidate_provider =+ make_provider(candidate.clone()).context("building candidate provider")?;++ let fixtures = load_fixtures(&args.fixtures)+ .with_context(|| format!("loading fixtures from {}", args.fixtures.display()))?;+ if fixtures.is_empty() {+ bail!("no *.json fixtures found in {}", args.fixtures.display());+ }++ // Timestamped run dir; filesystem-safe (colons + dots → dashes).+ let stamp: String = Timestamp::now()+ .to_string()+ .chars()+ .map(|c| if c == ':' || c == '.' { '-' } else { c })+ .collect();+ let run_dir = args.out.join(&stamp);+ std::fs::create_dir_all(run_dir.join("baseline"))?;+ std::fs::create_dir_all(run_dir.join("candidate"))?;++ println!();+ println!("ai-memory eval — {stamp}");+ println!(+ " fixtures : {} ({})",+ fixtures.len(),+ args.fixtures.display()+ );+ println!(+ " baseline : {} {} @ {}",+ baseline.provider_name(),+ baseline.model,+ baseline.base_url.as_deref().unwrap_or("(default)")+ );+ println!(+ " candidate: {} {} @ {}",+ candidate.provider_name(),+ candidate.model,+ candidate.base_url.as_deref().unwrap_or("(default)")+ );+ println!(" out : {}", run_dir.display());+ println!();++ let mut results: Vec<(CaseRun, CaseRun)> = Vec::with_capacity(fixtures.len());+ for (path, fx) in fixtures {+ println!("→ {} ({} observations)", fx.name, fx.observations.len());+ let observations = synthesise_observations(&fx);+ let session_id = SessionId::new();++ // Run both providers concurrently per fixture so a slow one+ // doesn't double the wall-clock.+ let req_a = build_batch_request(session_id, &observations);+ let req_b = build_batch_request(session_id, &observations);+ let (a_res, b_res) = tokio::join!(+ run_one(baseline_provider.clone(), "baseline", req_a),+ run_one(candidate_provider.clone(), "candidate", req_b),+ );++ let a = persist_case("baseline", &fx.name, &path, &run_dir, a_res)?;+ let b = persist_case("candidate", &fx.name, &path, &run_dir, b_res)?;++ println!(+ " baseline : {:>5} ms parsed={} updates={}",+ a.elapsed_ms,+ a.parsed_ok,+ a.update_count.map_or("-".into(), |n| n.to_string())+ );+ println!(+ " candidate: {:>5} ms parsed={} updates={}",+ b.elapsed_ms,+ b.parsed_ok,+ b.update_count.map_or("-".into(), |n| n.to_string())+ );+ // Force-flush per fixture so progress is visible when stdout+ // is piped (e.g. via `tee` or backgrounded runs). Without+ // this, full-buffering swallows every println! until exit.+ std::io::stdout().flush().ok();+ results.push((a, b));+ }++ // Tail summary so the user can grok the run at a glance.+ println!();+ println!("=== summary ===");+ let n = results.len();+ let baseline_total: u128 = results.iter().map(|(a, _)| a.elapsed_ms).sum();+ let candidate_total: u128 = results.iter().map(|(_, b)| b.elapsed_ms).sum();+ let baseline_ok = results.iter().filter(|(a, _)| a.parsed_ok).count();+ let candidate_ok = results.iter().filter(|(_, b)| b.parsed_ok).count();+ println!(+ "baseline : {baseline_ok}/{n} parsed ok total {baseline_total} ms avg {} ms",+ baseline_total / n.max(1) as u128+ );+ println!(+ "candidate: {candidate_ok}/{n} parsed ok total {candidate_total} ms avg {} ms",+ candidate_total / n.max(1) as u128+ );+ println!();+ println!("outputs under: {}", run_dir.display());+ println!(+ "compare with: diff -ru {}/baseline {}/candidate",+ run_dir.display(),+ run_dir.display()+ );+ Ok(())+ }++ /// Resolve `ProviderConfig` for one side from CLI args + env.+ fn resolve_provider(side: &str, args: &AbArgs) -> Result<ResolvedConfig> {+ let (provider_str, base_url, model, api_key_inline, api_key_env, token_file) = match side {+ "baseline" => (+ args.baseline_provider.as_str(),+ args.baseline_base_url.clone(),+ args.baseline_model.clone(),+ args.baseline_api_key.clone(),+ args.baseline_api_key_env.clone(),+ args.baseline_token_file.clone(),+ ),+ "candidate" => (+ args.candidate_provider.as_str(),+ args.candidate_base_url.clone(),+ args.candidate_model.clone(),+ args.candidate_api_key.clone(),+ args.candidate_api_key_env.clone(),+ args.candidate_token_file.clone(),+ ),+ _ => bail!("unknown side {side}"),+ };++ let provider = match provider_str {+ "anthropic" => ProviderChoice::Anthropic,+ "openai" => ProviderChoice::OpenAi,+ "openai-compat" | "openai_compat" => ProviderChoice::OpenAiCompat,+ "openai-oauth" | "openai_oauth" => ProviderChoice::OpenAiOAuth,+ "copilot" | "github-copilot" | "github_copilot" => ProviderChoice::Copilot,+ other => {+ bail!(+ "{side}: provider {other} not one of anthropic|openai|openai-compat|openai-oauth|copilot"+ )+ }+ };++ let api_key = match (api_key_inline, api_key_env) {+ (Some(s), _) if !s.is_empty() => Some(SecretString::from(s)),+ (_, Some(env)) => match std::env::var(&env) {+ Ok(v) if !v.is_empty() => Some(SecretString::from(v)),+ _ => bail!("{side}: env var {env} is not set or empty"),+ },+ _ => None,+ };++ let auth = match provider.auth_requirement() {+ AuthRequirement::RequiredApiKey { env_var } => {+ ProviderAuth::required_api_key_from_env(env_var, api_key)+ }+ AuthRequirement::OptionalApiKey { env_var } => {+ ProviderAuth::optional_api_key_from_env(env_var, api_key)+ }+ AuthRequirement::OpenAiOAuthToken => {+ ProviderAuth::openai_oauth_token_file(token_file.ok_or_else(|| {+ anyhow::anyhow!("{side}: --{side}-token-file is required for openai-oauth")+ })?)+ }+ AuthRequirement::CopilotToken => ProviderAuth::copilot(+ token_file.ok_or_else(|| {+ anyhow::anyhow!("{side}: --{side}-token-file is required for copilot")+ })?,+ None,+ None,+ base_url.clone(),+ ),+ AuthRequirement::AnthropicOAuthToken => ProviderAuth::anthropic_oauth_token(api_key),+ };++ Ok(ResolvedConfig {+ provider,+ provider_str: provider_str.to_string(),+ model,+ base_url,+ auth,+ })+ }++ #[derive(Clone)]+ struct ResolvedConfig {+ provider: ProviderChoice,+ provider_str: String,+ model: String,+ base_url: Option<String>,+ auth: ProviderAuth,+ }++ impl ResolvedConfig {+ fn provider_name(&self) -> &str {+ self.provider_str.as_str()+ }+ }++ impl From<ResolvedConfig> for ProviderConfig {+ fn from(r: ResolvedConfig) -> Self {+ Self {+ provider: r.provider,+ model: r.model,+ auth: r.auth,+ base_url: r.base_url,+ // Match the product default so provider comparisons exercise the+ // same schema-constrained path operators receive.+ compat_strict: true,+ request_timeout_secs: ai_memory_llm::DEFAULT_REQUEST_TIMEOUT_SECS,++ // The A/B harness compares consolidation quality; reasoning++ // effort stays at each model's own default unless a future++ // flag threads it through.++ reasoning_effort: None,+ }+ }+ }++ fn make_provider(r: ResolvedConfig) -> Result<Arc<dyn LlmProvider>> {+ ai_memory_llm::build_provider(ProviderConfig::from(r))+ .map_err(anyhow::Error::from)+ .context("building provider")+ }++ fn load_fixtures(dir: &Path) -> Result<Vec<(PathBuf, Fixture)>> {+ let mut paths: Vec<_> = std::fs::read_dir(dir)?+ .filter_map(|e| e.ok().map(|e| e.path()))+ .filter(|p| p.extension().is_some_and(|e| e == "json"))+ .collect();+ paths.sort();+ let mut out = Vec::with_capacity(paths.len());+ for p in paths {+ let bytes = std::fs::read(&p).with_context(|| format!("reading {}", p.display()))?;+ let fx: Fixture =+ serde_json::from_slice(&bytes).with_context(|| format!("parsing {}", p.display()))?;+ out.push((p, fx));+ }+ Ok(out)+ }++ fn synthesise_observations(fx: &Fixture) -> Vec<Observation> {+ let workspace_id = WorkspaceId::new();+ let project_id = ProjectId::new();+ let session_id = SessionId::new();+ fx.observations+ .iter()+ .map(|f| Observation {+ id: ai_memory_core::ObservationId::new(),+ session_id,+ workspace_id,+ project_id,+ kind: f+ .kind+ .parse::<ObservationKind>()+ .unwrap_or(ObservationKind::Other),+ extension: None,+ source_event: None,+ title: f.title.clone(),+ body: f.body.clone(),+ importance: 5,+ created_at: Timestamp::now(),+ })+ .collect()+ }++ struct ProviderResult {+ elapsed: Duration,+ /// Raw text the model emitted — always saved, even when parsing+ /// fails. This is the load-bearing artifact for debugging+ /// schema mismatches: you can see exactly what the model said.+ raw_text: String,+ /// Parsed-from-text JSON, if the response was valid JSON at all.+ raw_json: Option<serde_json::Value>,+ /// Schema-validated batch, if the JSON also satisfied the schema.+ batch: Option<ConsolidatedBatch>,+ error: Option<String>,+ }++ async fn run_one(+ provider: Arc<dyn LlmProvider>,+ side: &str,+ request: ai_memory_llm::ChatRequest,+ ) -> ProviderResult {+ let start = Instant::now();+ // Call `complete` directly — bypassing `complete_structured` —+ // so we can capture the raw text the model emitted, even if it+ // doesn't deserialise. The cost is that we re-implement the+ // "extract first balanced {…}" fallback locally, but knowing+ // exactly what the model said is what makes the harness useful.+ let chat_res = provider.complete(request).await;+ let elapsed = start.elapsed();+ let raw_text = match &chat_res {+ Ok(r) => r.text.clone(),+ Err(e) => format!("(provider error before any response: {e})"),+ };+ let (raw_json, batch, error) = match chat_res {+ Ok(_) => parse_response(&raw_text),+ Err(e) => (None, None, Some(e.to_string())),+ };+ if let Some(err) = &error {+ tracing::warn!(side, error = %err, "model output did not satisfy schema");+ } else {+ tracing::info!(+ side,+ ms = elapsed.as_millis() as u64,+ updates = batch.as_ref().map_or(0, |b| b.updates.len()),+ "parsed ok"+ );+ }+ ProviderResult {+ elapsed,+ raw_text,+ raw_json,+ batch,+ error,+ }+ }++ /// Parse a model response into (json, batch, error).+ ///+ /// 1. Try the whole response as JSON.+ /// 2. If that fails, search for the first balanced `{…}` block.+ /// 3. If we have JSON, try to deserialise as `ConsolidatedBatch`.+ fn parse_response(+ text: &str,+ ) -> (+ Option<serde_json::Value>,+ Option<ConsolidatedBatch>,+ Option<String>,+ ) {+ let json = serde_json::from_str::<serde_json::Value>(text.trim())+ .ok()+ .or_else(|| {+ extract_first_balanced_object(text).and_then(|s| serde_json::from_str(&s).ok())+ });+ let Some(json) = json else {+ return (None, None, Some("response is not valid JSON".into()));+ };+ match serde_json::from_value::<ConsolidatedBatch>(json.clone()) {+ Ok(batch) => (Some(json), Some(batch), None),+ Err(e) => (Some(json), None, Some(format!("schema mismatch: {e}"))),+ }+ }++ /// Best-effort `{…}` extractor — ignores braces inside strings.+ fn extract_first_balanced_object(text: &str) -> Option<String> {+ let bytes = text.as_bytes();+ let start = bytes.iter().position(|&b| b == b'{')?;+ let mut depth = 0i32;+ let mut in_string = false;+ let mut escape = false;+ for (i, &b) in bytes.iter().enumerate().skip(start) {+ if in_string {+ if escape {+ escape = false;+ } else if b == b'\\' {+ escape = true;+ } else if b == b'"' {+ in_string = false;+ }+ continue;+ }+ if b == b'"' {+ in_string = true;+ continue;+ }+ if b == b'{' {+ depth += 1;+ } else if b == b'}' {+ depth -= 1;+ if depth == 0 {+ return Some(text[start..=i].to_string());+ }+ }+ }+ None+ }++ fn persist_case(+ side: &str,+ fixture_name: &str,+ fixture_path: &Path,+ run_dir: &Path,+ res: ProviderResult,+ ) -> Result<CaseRun> {+ let safe_name = fixture_path+ .file_stem()+ .and_then(|s| s.to_str())+ .unwrap_or(fixture_name);+ let dir = run_dir.join(side);+ std::fs::create_dir_all(&dir)?;++ // 1. Raw model text — always saved, even on parse failure.+ // This is the load-bearing debug artifact: a side-by-side+ // diff of these files tells you exactly what each model said.+ let raw_path = dir.join(format!("{safe_name}.raw.txt"));+ std::fs::write(&raw_path, &res.raw_text)?;++ // 2. Parsed JSON (if response was valid JSON at all).+ if let Some(json) = &res.raw_json {+ let json_path = dir.join(format!("{safe_name}.json"));+ std::fs::write(&json_path, serde_json::to_string_pretty(json)?)?;+ }++ // 3. Flattened markdown rendering for easy eyeballing.+ let md_path = dir.join(format!("{safe_name}.md"));+ let md = render_markdown(side, fixture_name, &res);+ std::fs::write(&md_path, md)?;++ // 4. Per-case meta.+ let case = CaseRun {+ provider: side.to_string(),+ fixture: fixture_name.to_string(),+ elapsed_ms: res.elapsed.as_millis(),+ parsed_ok: res.batch.is_some(),+ error: res.error.clone(),+ update_count: res.batch.as_ref().map(|b| b.updates.len()),+ };+ let meta_path = dir.join(format!("{safe_name}.meta.json"));+ std::fs::write(&meta_path, serde_json::to_string_pretty(&case)?)?;+ Ok(case)+ }++ fn render_markdown(side: &str, fixture_name: &str, res: &ProviderResult) -> String {+ let mut out = String::new();+ out.push_str(&format!("# {side} — {fixture_name}\n\n"));+ out.push_str(&format!("- elapsed: {} ms\n", res.elapsed.as_millis()));+ if let Some(err) = &res.error {+ out.push_str(&format!("- error: `{err}`\n"));+ }+ // Always include the raw response so the human reviewer can+ // judge structure/style even when schema validation failed.+ if !res.raw_text.is_empty() {+ out.push_str("\n## Raw response\n\n```\n");+ out.push_str(&res.raw_text);+ out.push_str("\n```\n");+ }+ let Some(batch) = &res.batch else {+ return out;+ };+ out.push_str(&format!("\n- updates: {}\n", batch.updates.len()));+ if !batch.rationale.trim().is_empty() {+ out.push_str("\n## Rationale\n\n");+ out.push_str(batch.rationale.trim());+ out.push('\n');+ }+ for (i, u) in batch.updates.iter().enumerate() {+ out.push_str(&format!("\n## Update {} — `{}`\n\n", i + 1, u.path));+ out.push_str(&format!("**Title:** {} \n", u.title));+ out.push_str(&format!("**Kind:** `{}` \n", u.kind.as_str()));+ out.push_str(&format!("**Tier:** `{}` \n", u.tier.as_str()));+ if !u.tags.is_empty() {+ out.push_str(&format!("**Tags:** {} \n", u.tags.join(", ")));+ }+ out.push('\n');+ out.push_str(u.body_markdown.trim());+ out.push('\n');+ }+ out+ }