Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
29.2 KB
Raw Blame
/**
 * LongMemEval benchmark suite adapter (recall-quality wave, t_607d429d).
 *
 * LongMemEval (Wu et al., 2024) is the long-term-memory benchmark the
 * field reports recall_all@5 on. This module adds it beside LoCoMo as an
 * ADDITIVE named suite: a fail-fast parser over the official dataset
 * shape (a JSON array of question entries, each carrying its haystack
 * sessions), and a converter into the existing {@link BenchFixture} the
 * unchanged `runMemoryBench` consumes.
 *
 * Network-free by design: the loader is pure over its input file. The
 * committed sample fixtures certify CI: `longmemeval-official-sample.json`
 * pins the OFFICIAL dataset shape, `longmemeval-sample.json` the legacy
 * sample shape; the full dataset arrives as an operator-supplied path
 * resolved by the bench verb's literal-then-repo-fixture-dir logic. The
 * official download lives at https://github.com/xiaowu0162/LongMemEval
 * (the dataset ships as JSON: longmemeval_s.json, longmemeval_m.json,
 * longmemeval_oracle.json) - never fetched at run time.
 *
 * Both dataset shapes are accepted, official first: the official files
 * spell the keys `question_id` / `haystack_session_ids` /
 * `haystack_sessions`, mark abstention by a question id ending in
 * `_abs`, and use the six official question types; the legacy sample
 * shape uses `q_id` / `haystack_session_id` / `haystack_session`, an
 * explicit `abstention` question type, and the pre-wave type spellings.
 *
 * Two traps are handled HERE, at conversion, not downstream:
 *
 *  - `expected_text` is NEVER set. `evaluateQuestion` ignores it for
 *    retrieval categories, so a gold answer pinned there would silently
 *    score nothing. The gold answer rides the judge payload instead
 *    (judge.ts), and scoring is expected_paths containment at the
 *    {@link LONGMEMEVAL_RECALL_CUTOFF}.
 *  - Gold session ids join by direct set membership against the entry's
 *    own haystack ids; an unknown id is a named parse error. Ids are
 *    sanitized only when they become note paths.
 *  - Each entry's evidence is namespaced under its own question's
 *    subtree (`sessions/<question_id>/...`): LongMemEval scores every
 *    question against its OWN haystack (~50 sessions), and evidence
 *    pooled across entries would have every query search the union of
 *    all haystacks. Residual, stated honestly: the fixture schema has
 *    no per-question retrieval scope, so the disjoint subtrees keep
 *    the gold paths per-question but cannot stop the ranker from
 *    surfacing another scope's notes inside the cutoff - full
 *    per-question retrieval needs a scope field on the fixture
 *    question plus a rank-time filter in the retrieve phase.
 *
 * Abstention questions are dropped at conversion time with the exclusion
 * count recorded in the suite receipt - the fixture parser rejects
 * unknown categories, so pass-through is not an option, and the report
 * family surfaces the count rather than hiding it.
 */

import { readFileSync } from "node:fs";

import { parseBenchFixture } from "./fixture.ts";
import {
  isLongMemEvalAbstention,
  LONGMEMEVAL_ABSTENTION_TYPE,
  sampleLongMemEvalEntries,
} from "./longmemeval-sample.ts";
import type { LongMemEvalSampleStratum } from "./longmemeval-sample.ts";
import type { BenchSuiteLoadOptions } from "./types.ts";
import {
  RETRIEVAL_CATEGORIES,
  type BenchCategory,
  type BenchFixture,
  type BenchFixtureContinuity,
  type BenchFixtureNote,
  type BenchQuestion,
} from "./types.ts";

/** The suite's registered name in the bench verb's suite registry. */
export const LONGMEMEVAL_SUITE_NAME = "longmemeval";

/**
 * Provenance of the answer-judging prompts below, stated honestly: they
 * are SELF-WRITTEN in the spirit of the upstream evaluation - its
 * `evaluate_qa.py` judges a model's response against the reference
 * answer with per-question-type prompts - but they are NOT a
 * transcription of the upstream text, and nothing upstream pins their
 * wording. The dataset itself ships as JSON in the same repository and
 * is downloaded by operators - never at run time.
 */
export const LONGMEMEVAL_PROMPT_PROVENANCE =
  "self-written prompts following the evaluate_qa.py approach in github.com/xiaowu0162/LongMemEval (arXiv:2410.10813); not transcribed from upstream - the dataset ships as longmemeval_s.json / longmemeval_m.json / longmemeval_oracle.json";

/**
 * Answer-judging prompt, system half (self-written; see
 * {@link LONGMEMEVAL_PROMPT_PROVENANCE}). The external judge command
 * composes its model call from this and the per-question template, so
 * the command itself stays a dumb model call and the prompt text has
 * exactly one home in the tree.
 */
export const LONGMEMEVAL_JUDGE_SYSTEM_PROMPT = Object.freeze(
  "You are a meticulous evaluator of long-term conversational memory. " +
    "Judge the provided answer against the reference answer and the gold " +
    "evidence, using only what the evidence supports. Answer with strict " +
    'JSON: {"correct": true} when the answer is supported, {"correct": false} otherwise.',
);

/**
 * Answer-judging prompt, per-question half (self-written; see
 * {@link LONGMEMEVAL_PROMPT_PROVENANCE}). The placeholders are the judge
 * payload's own field names, so a judge command renders the prompt
 * straight from the stdin JSON. Gold evidence is carried as the gold
 * sessions' ids (the payload stays bounded); a judge that wants their
 * conversation text resolves the ids from the dataset it already has.
 */
export const LONGMEMEVAL_JUDGE_QUESTION_TEMPLATE = Object.freeze(
  "Question: {question}\n" +
    "Reference answer: {answer}\n" +
    "Gold evidence sessions: {gold}\n" +
    "Decide whether the answer is supported by the gold evidence.",
);

export interface LongMemEvalTurn {
  readonly role: string;
  readonly content: string;
}

/** One dataset entry, in the official LongMemEval question shape. */
export interface LongMemEvalEntry {
  readonly q_id: string;
  readonly question_type: string;
  readonly question: string;
  readonly answer?: string;
  /**
   * The date the question was asked (official `question_date`). An
   * entry that states none is dated by its last haystack session - the
   * question follows the sessions it was built from.
   */
  readonly question_date: string;
  readonly haystack_session_id: ReadonlyArray<string>;
  readonly haystack_session: ReadonlyArray<ReadonlyArray<LongMemEvalTurn>>;
  readonly haystack_dates: ReadonlyArray<string>;
  readonly answer_session_ids: ReadonlyArray<string>;
}

export interface LongMemEvalDataset {
  readonly name?: string;
  readonly entries: ReadonlyArray<LongMemEvalEntry>;
}

/** What conversion decided, surfaced beside the converted fixture. */
export interface LongMemEvalReceipt {
  /** Abstention questions dropped at conversion time. */
  readonly abstention_excluded: number;
  /**
   * Seeded-sampling bookkeeping, present only when the loader sampled:
   * the population it chose from, the questions it selected, and the
   * per-stratum breakdown. Mirrors the abstention precedent - a
   * whole-dataset run carries no sample fields rather than a
   * manufactured 0 - and surfaces through the suite receipt the run
   * reports ({@link BenchSuiteReceipt}).
   */
  readonly sample_population?: number;
  readonly sample_selected?: number;
  readonly sample_strata?: ReadonlyArray<LongMemEvalSampleStratum>;
}

export interface LongMemEvalConversion {
  readonly fixture: BenchFixture;
  readonly receipt: LongMemEvalReceipt;
}

class LongMemEvalParseError extends Error {}

/**
 * The abstention rule lives in the sampler module, whose exclusion of
 * abstention entries must precede selection and which must stay a leaf
 * (this adapter imports it); re-exported here so the suite's public
 * API is unchanged.
 */
export { isLongMemEvalAbstention, LONGMEMEVAL_ABSTENTION_TYPE };

/**
 * The recall cutoff the suite scores at: a question passes retrieval
 * when EVERY gold session note ranks inside the top k. Drives both the
 * converted questions' `top_k` and the strict report metric.
 */
export const LONGMEMEVAL_RECALL_CUTOFF = 5;

/**
 * Question types -> OSB's canonical retrieval categories. The official
 * dataset's six types come first (`single-session-*` all read as the
 * single-hop category - each asks about one session, only the turn
 * role or preference focus differs, and scoring is identical
 * expected_paths containment for all of them); the legacy sample
 * shape's spellings follow. Abstention entries map to nothing: they
 * are excluded, never converted.
 */
const QUESTION_TYPE_TO_CATEGORY: Readonly<Record<string, BenchCategory>> = Object.freeze({
  "single-session-user": "single_hop",
  "single-session-assistant": "single_hop",
  "single-session-preference": "single_hop",
  "multi-session": "multi_evidence",
  "temporal-reasoning": "temporal",
  "knowledge-update": "contradiction",
  "single-hop": "single_hop",
  "multi-hop": "multi_evidence",
  "open-domain": "single_hop",
});

/** Every question type the parser accepts, abstention included. */
export const LONGMEMEVAL_QUESTION_TYPES: ReadonlyArray<string> = Object.freeze([
  ...Object.keys(QUESTION_TYPE_TO_CATEGORY),
  LONGMEMEVAL_ABSTENTION_TYPE,
]);

/** Minutes between turns of one session when deriving turn timestamps. */
const TURN_TIMESTAMP_STEP_MS = 60_000;

/**
 * Parse and validate a raw LongMemEval dataset: either the official bare
 * array of question entries or a `{ name, questions }` wrapper naming
 * the dataset. Fail-fast with a named {@link LongMemEvalParseError}.
 *
 * Haystack sessions are merged across entries by id for validation:
 * entries share sessions, and one id must carry the same TURNS
 * everywhere - a content mismatch is a named error rather than a
 * silently doubled note. Dates are assigned per instance upstream, so
 * a date difference between entries is tolerated (first write wins).
 */
export function parseLongMemEvalDataset(raw: unknown): LongMemEvalDataset {
  let entriesRaw: ReadonlyArray<unknown>;
  let name: string | undefined;
  if (Array.isArray(raw)) {
    entriesRaw = raw;
  } else if (
    raw !== null &&
    typeof raw === "object" &&
    Array.isArray((raw as Record<string, unknown>)["questions"])
  ) {
    const wrapper = raw as Record<string, unknown>;
    if (typeof wrapper["name"] === "string" && wrapper["name"].trim() !== "") {
      name = wrapper["name"];
    }
    entriesRaw = wrapper["questions"] as ReadonlyArray<unknown>;
  } else {
    throw new LongMemEvalParseError(
      "LongMemEval dataset must be an array of questions or a { name, questions } object",
    );
  }
  if (entriesRaw.length === 0) {
    throw new LongMemEvalParseError("LongMemEval dataset needs a non-empty questions array");
  }
  const sessions = new Map<
    string,
    { date: string; turns: ReadonlyArray<LongMemEvalTurn>; entry: string }
  >();
  const entries = entriesRaw.map((entry, index) => parseEntry(entry, index, sessions));
  return { ...(name !== undefined ? { name } : {}), entries };
}

/**
 * Read a field under both dataset spellings: the official files spell
 * it `official`, the legacy sample shape `legacy`. Official first; a
 * legacy key is read only when the official one is absent.
 */
function readKey(r: Record<string, unknown>, official: string, legacy: string): unknown {
  const value = r[official];
  return value !== undefined ? value : r[legacy];
}

function parseEntry(
  raw: unknown,
  index: number,
  sessions: Map<string, { date: string; turns: ReadonlyArray<LongMemEvalTurn>; entry: string }>,
): LongMemEvalEntry {
  if (raw === null || typeof raw !== "object") {
    throw new LongMemEvalParseError(`LongMemEval dataset: questions[${index}] must be an object`);
  }
  const r = raw as Record<string, unknown>;
  const qId = asString(
    readKey(r, "question_id", "q_id"),
    `questions[${index}].question_id (legacy q_id)`,
  );
  const questionType = asString(r["question_type"], `questions[${index}] (${qId}).question_type`);
  if (!(LONGMEMEVAL_QUESTION_TYPES as ReadonlyArray<string>).includes(questionType)) {
    throw new LongMemEvalParseError(
      `LongMemEval dataset: question '${qId}' has an unknown question_type '${questionType}' ` +
        `(expected one of ${LONGMEMEVAL_QUESTION_TYPES.join(", ")})`,
    );
  }
  const abstention = isLongMemEvalAbstention({ q_id: qId, question_type: questionType });
  const question = asString(r["question"], `questions[${index}] (${qId}).question`);
  const answer = answerOf(r["answer"]);

  const sessionIds = stringArray(
    readKey(r, "haystack_session_ids", "haystack_session_id"),
    `questions[${index}] (${qId}).haystack_session_ids (legacy haystack_session_id)`,
    true,
  );
  const sessionsRaw = readKey(r, "haystack_sessions", "haystack_session");
  if (!Array.isArray(sessionsRaw) || sessionsRaw.length !== sessionIds.length) {
    throw new LongMemEvalParseError(
      `LongMemEval dataset: question '${qId}' haystack_sessions (legacy haystack_session) ` +
        `must be an array aligned with haystack_session_ids`,
    );
  }
  const dates = stringArray(
    r["haystack_dates"],
    `questions[${index}] (${qId}).haystack_dates`,
    true,
  );
  if (dates.length !== sessionIds.length) {
    throw new LongMemEvalParseError(
      `LongMemEval dataset: question '${qId}' haystack_dates must be an array aligned with haystack_session_ids`,
    );
  }
  const localIds = new Set<string>();
  sessionIds.forEach((sessionId, j) => {
    localIds.add(sessionId);
    const turns = parseTurns(
      sessionsRaw[j],
      `questions[${index}] (${qId}).haystack_sessions[${j}]`,
    );
    const date = asValidDate(dates[j]!, `questions[${index}] (${qId}).haystack_dates[${j}]`);
    mergeSession(sessions, sessionId, date, turns, qId);
  });

  let gold: ReadonlyArray<string> = [];
  if (!abstention) {
    gold = stringArray(
      r["answer_session_ids"],
      `questions[${index}] (${qId}).answer_session_ids`,
      true,
    );
    if (gold.length === 0) {
      throw new LongMemEvalParseError(
        `LongMemEval dataset: question '${qId}' (${questionType}) needs a non-empty answer_session_ids`,
      );
    }
    for (const id of gold) {
      if (!localIds.has(id)) {
        throw new LongMemEvalParseError(
          `LongMemEval dataset: question '${qId}' references unknown session '${id}'`,
        );
      }
    }
  }
  // The official entries carry question_date; an entry that states none
  // is dated by its last haystack session.
  const questionDateRaw = r["question_date"];
  const questionDate =
    questionDateRaw === undefined
      ? dates[dates.length - 1]!
      : asValidDate(
          asString(questionDateRaw, `questions[${index}] (${qId}).question_date`),
          `questions[${index}] (${qId}).question_date`,
        );
  return {
    q_id: qId,
    question_type: questionType,
    question,
    ...(answer !== undefined ? { answer } : {}),
    question_date: questionDate,
    haystack_session_id: sessionIds,
    haystack_session: sessionIds.map((_, j) =>
      parseTurns(sessionsRaw[j], `haystack_sessions[${j}]`),
    ),
    haystack_dates: dates,
    answer_session_ids: gold,
  };
}

function mergeSession(
  sessions: Map<string, { date: string; turns: ReadonlyArray<LongMemEvalTurn>; entry: string }>,
  sessionId: string,
  date: string,
  turns: ReadonlyArray<LongMemEvalTurn>,
  entryId: string,
): void {
  const existing = sessions.get(sessionId);
  if (existing !== undefined) {
    // Upstream assigns haystack dates PER INSTANCE, so the same session
    // id can legitimately carry a different date in each entry; the
    // first write wins. Only a content mismatch means the dataset is
    // inconsistent - one id whose turns disagree between entries.
    if (JSON.stringify(existing.turns) !== JSON.stringify(turns)) {
      throw new LongMemEvalParseError(
        `LongMemEval dataset: session '${sessionId}' is inconsistent between entries '${existing.entry}' and '${entryId}'`,
      );
    }
    return;
  }
  sessions.set(sessionId, { date, turns, entry: entryId });
}

function parseTurns(raw: unknown, field: string): ReadonlyArray<LongMemEvalTurn> {
  if (!Array.isArray(raw) || raw.length === 0) {
    throw new LongMemEvalParseError(`LongMemEval dataset: ${field} must be a non-empty turn array`);
  }
  return raw.map((turn, k) => {
    if (turn === null || typeof turn !== "object") {
      throw new LongMemEvalParseError(`LongMemEval dataset: ${field}[${k}] must be an object`);
    }
    const t = turn as Record<string, unknown>;
    return {
      role: asString(t["role"], `${field}[${k}].role`),
      content: asString(t["content"], `${field}[${k}].content`),
    };
  });
}

function asString(value: unknown, field: string): string {
  if (typeof value !== "string" || value.trim() === "") {
    throw new LongMemEvalParseError(`LongMemEval dataset: ${field} must be a non-empty string`);
  }
  return value;
}

/**
 * Parse a dataset date into epoch milliseconds, deterministically as
 * UTC. The official dates look like `2023/05/20 (Sat) 02:21`, which
 * JavaScript would read in the host's local zone - so the slash forms
 * are rebuilt as UTC by hand. A naive ISO date-time (no offset) is read
 * as UTC by the same rule, date-only ISO is UTC by spec, and
 * offset-explicit ISO forms parse as stated. Any other shape returns
 * NaN - the caller refuses it by name rather than letting the host
 * zone decide.
 */
function parseDatasetDateMs(value: string): number {
  const trimmed = value.trim();
  const slash = /^(\d{4})\/(\d{1,2})\/(\d{1,2})(?:\s+\([^()]*\))?(?:\s+(\d{1,2}):(\d{2}))?$/.exec(
    trimmed,
  );
  if (slash !== null) {
    return Date.UTC(
      Number(slash[1]),
      Number(slash[2]) - 1,
      Number(slash[3]),
      slash[4] !== undefined ? Number(slash[4]) : 0,
      slash[5] !== undefined ? Number(slash[5]) : 0,
    );
  }
  const naiveIso = /^(\d{4})-(\d{2})-(\d{2})T(\d{1,2}):(\d{2})(?::(\d{2})(?:\.(\d{1,3}))?)?$/.exec(
    trimmed,
  );
  if (naiveIso !== null) {
    return Date.UTC(
      Number(naiveIso[1]),
      Number(naiveIso[2]) - 1,
      Number(naiveIso[3]),
      Number(naiveIso[4]),
      Number(naiveIso[5]),
      naiveIso[6] !== undefined ? Number(naiveIso[6]) : 0,
      naiveIso[7] !== undefined ? Number(naiveIso[7]) : 0,
    );
  }
  if (
    /^\d{4}-\d{2}-\d{2}$/.test(trimmed) ||
    /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d{1,3})?)?(Z|[+-]\d{2}:?\d{2})$/.test(trimmed)
  ) {
    return Date.parse(trimmed);
  }
  return Number.NaN;
}

function asValidDate(value: string, field: string): string {
  if (Number.isNaN(parseDatasetDateMs(value))) {
    throw new LongMemEvalParseError(
      `LongMemEval dataset: ${field} must be a valid date, got '${value}'`,
    );
  }
  return value;
}

function stringArray(value: unknown, field: string, nonEmpty: boolean): ReadonlyArray<string> {
  if (!Array.isArray(value) || !value.every((item) => typeof item === "string" && item !== "")) {
    throw new LongMemEvalParseError(`LongMemEval dataset: ${field} must be an array of strings`);
  }
  if (nonEmpty && value.length === 0) {
    throw new LongMemEvalParseError(`LongMemEval dataset: ${field} must be a non-empty array`);
  }
  return Object.freeze([...value]);
}

/** Sanitize a dataset id into one safe note-path segment. */
function sanitizePathSegment(raw: string): string {
  return raw.replace(/[^A-Za-z0-9_-]/g, "-");
}

/**
 * The path scope of one question's evidence: every note, continuity
 * record and expected path of an entry lives under
 * `sessions/<question_id>/`, so one question's haystack never pools
 * with another's. Two distinct question ids that sanitize to the same
 * scope would silently merge haystacks, so that is a named error.
 */
function questionScope(qId: string, claimed: Map<string, string>): string {
  const scope = sanitizePathSegment(qId);
  const owner = claimed.get(scope);
  if (owner !== undefined) {
    throw new LongMemEvalParseError(
      owner === qId
        ? `LongMemEval dataset: question id '${qId}' appears twice`
        : `LongMemEval dataset: question ids '${owner}' and '${qId}' sanitize to the same path scope '${scope}'; their haystacks would merge`,
    );
  }
  claimed.set(scope, qId);
  return scope;
}

/** Vault-relative note path for a session inside its question's scope. */
function sessionNotePath(scope: string, sessionId: string): string {
  return `sessions/${scope}/${sanitizePathSegment(sessionId)}.md`;
}

/**
 * Note body for one scoped session copy. The header names the question
 * scope on purpose: a session shared by several entries materializes
 * once per scope, and bodies that stayed byte-identical would collide
 * in the indexer's content-hash skip - only the first scope's copy
 * would ever index, dangling every other scope's gold path.
 */
function sessionNoteBody(
  scope: string,
  sessionId: string,
  turns: ReadonlyArray<LongMemEvalTurn>,
): string {
  return (
    `# Session ${sessionId} (question ${scope})\n\n` +
    turns.map((t) => `${t.role}: ${t.content}`).join("\n")
  );
}

function turnTimestamp(sessionDate: string, turnIndex: number): string {
  return new Date(
    parseDatasetDateMs(sessionDate) + turnIndex * TURN_TIMESTAMP_STEP_MS,
  ).toISOString();
}

/**
 * The judge reference answer, as the payload may carry it: strings pass
 * through; upstream states some counting answers as numbers, so finite
 * numbers are coerced with String(). Any other non-string type stays
 * refused - an answer that cannot be a string is dropped, never guessed.
 */
function answerOf(value: unknown): string | undefined {
  if (typeof value === "string" && value !== "") return value;
  if (typeof value === "number" && Number.isFinite(value)) return String(value);
  return undefined;
}

/**
 * Convert a parsed LongMemEval dataset into an OSB {@link BenchFixture}
 * plus its receipt. Each entry's evidence is namespaced under its own
 * question's subtree - its sessions become evidence notes and
 * session_turn continuity records at `sessions/<question_id>/...` - and
 * every non-abstention question becomes a retrieval question whose
 * expected_paths point at its gold session notes inside that same
 * subtree and whose top_k is the {@link LONGMEMEVAL_RECALL_CUTOFF}.
 * Abstention entries are excluded with the count recorded.
 *
 * Scoping is what keeps the suite's task like-for-like: LongMemEval
 * scores every question against ITS OWN haystack (~50 sessions), so
 * evidence pooled across entries would have every query search the
 * union of all haystacks. A session shared by several entries is
 * materialized once per scope - the trade is cross-entry note dedupe
 * for per-question isolation, which is what the metric scores.
 */
export function longMemEvalToBenchFixture(dataset: LongMemEvalDataset): LongMemEvalConversion {
  let abstentionExcluded = 0;
  const notes: BenchFixtureNote[] = [];
  const continuity: BenchFixtureContinuity[] = [];
  const questions: BenchQuestion[] = [];
  const claimedScopes = new Map<string, string>();
  for (const entry of dataset.entries) {
    if (isLongMemEvalAbstention(entry)) {
      abstentionExcluded += 1;
      continue;
    }
    const scope = questionScope(entry.q_id, claimedScopes);
    const scopedPaths = new Map<string, string>();
    const claimedInScope = new Map<string, string>();
    entry.haystack_session_id.forEach((sessionId, j) => {
      if (scopedPaths.has(sessionId)) return;
      const path = sessionNotePath(scope, sessionId);
      // Distinct session ids can sanitize to one note path (a/b and
      // a-b); within one scope the evidence of both would silently
      // merge into one note, so the collision is a named parse error.
      const collisionOwner = claimedInScope.get(path);
      if (collisionOwner !== undefined) {
        throw new LongMemEvalParseError(
          `LongMemEval dataset: question '${entry.q_id}' has session ids '${collisionOwner}' and '${sessionId}' ` +
            `that sanitize to the same note path '${path}'; the evidence of both would merge`,
        );
      }
      claimedInScope.set(path, sessionId);
      const turns = entry.haystack_session[j]!;
      const date = entry.haystack_dates[j]!;
      notes.push({ path, body: sessionNoteBody(scope, sessionId, turns) });
      turns.forEach((turn, i) => {
        continuity.push({
          kind: "session_turn" as const,
          created_at: turnTimestamp(date, i),
          payload: {
            question_id: entry.q_id,
            session_id: sessionId,
            role: turn.role,
            text: turn.content,
          },
        });
      });
      scopedPaths.set(sessionId, path);
    });
    questions.push({
      id: entry.q_id,
      category: QUESTION_TYPE_TO_CATEGORY[entry.question_type]!,
      query: entry.question,
      top_k: LONGMEMEVAL_RECALL_CUTOFF,
      expected_paths: entry.answer_session_ids.map((id) => {
        const path = scopedPaths.get(id);
        if (path === undefined) {
          // The parser guarantees gold ids join the entry's own
          // haystack; this names the impossible instead of emitting a
          // dangling path.
          throw new LongMemEvalParseError(
            `LongMemEval dataset: question '${entry.q_id}' references unknown session '${id}'`,
          );
        }
        return path;
      }),
      judge_input: Object.freeze({
        question: entry.question,
        ...(entry.answer !== undefined ? { answer: entry.answer } : {}),
        gold: Object.freeze([...entry.answer_session_ids]),
      }),
    });
  }
  if (questions.length === 0) {
    // A dataset of only abstention entries converts to zero questions,
    // which parseBenchFixture refuses generically; name the real cause -
    // the abstention filter, not a missing dataset.
    throw new LongMemEvalParseError(
      `LongMemEval dataset: every one of the ${abstentionExcluded} question(s) is an abstention entry and was excluded from scoring; nothing remains to evaluate`,
    );
  }
  const fixture = parseBenchFixture({
    name: `longmemeval-${dataset.name ?? "suite"}`,
    description:
      "LongMemEval dataset converted to the OSB bench fixture; each question's haystack is scoped under its own subtree and abstention questions are excluded from scoring.",
    notes,
    continuity,
    questions,
  });
  return { fixture, receipt: { abstention_excluded: abstentionExcluded } };
}

/**
 * Load a LongMemEval dataset file and convert it to a suite load.
 *
 * The second parameter is OPTIONAL and carries the run's sampling
 * request: with one, abstention entries are excluded first, the seeded
 * sampler selects a type-proportional subset of the survivors, and only
 * the selected entries convert - sampling happens PRE-conversion, so
 * notes and continuity records materialize just for the questions a run
 * will score. Without one, the conversion is byte-identical to the
 * shipped whole-dataset loader.
 *
 * The receipt mirrors the abstention precedent: on a sampled run it
 * carries the sample population, the selected count and the per-stratum
 * breakdown beside the abstention count, which stays the SAMPLER's
 * count - the exclusion happened before selection, so the conversion of
 * the selected entries would otherwise read 0 and hide the dataset's
 * abstention questions from the report.
 */
export function loadLongMemEvalFixture(
  path: string,
  options?: BenchSuiteLoadOptions,
): LongMemEvalConversion {
  let raw: unknown;
  try {
    raw = JSON.parse(readFileSync(path, "utf8"));
  } catch (e) {
    throw new LongMemEvalParseError(
      `LongMemEval dataset not readable JSON at ${path}: ${e instanceof Error ? e.message : String(e)}`,
    );
  }
  const dataset = parseLongMemEvalDataset(raw);
  if (options?.sample === undefined) return longMemEvalToBenchFixture(dataset);
  const sample = sampleLongMemEvalEntries(dataset.entries, options.sample);
  const converted = longMemEvalToBenchFixture({
    ...(dataset.name !== undefined ? { name: dataset.name } : {}),
    entries: sample.entries,
  });
  return {
    fixture: converted.fixture,
    receipt: {
      abstention_excluded: sample.receipt.abstention_excluded,
      sample_population: sample.receipt.population,
      sample_selected: sample.receipt.selected,
      sample_strata: sample.receipt.strata,
    },
  };
}

/** One persisted retrieval result the strict metric reads. */
export interface RecallAllRetrieved {
  readonly id: string;
  /** Result paths in rank order. */
  readonly paths?: ReadonlyArray<string>;
}

export interface RecallAllScore {
  /** Questions scored: the non-abstention, retrieval-category ones. */
  readonly scored: number;
  /** Questions with EVERY gold session note inside the cutoff. */
  readonly passed: number;
}

/**
 * Strict recall_all@{@link LONGMEMEVAL_RECALL_CUTOFF}: the count of
 * scored questions whose EVERY gold session note ranks inside the
 * cutoff, over the PERSISTED retrieval results. This is the aggregate
 * roll-off of the same containment rule `evaluateQuestion` applies per
 * question - not a second scorer - kept as a named metric because the
 * field reports recall_all@5, the number the memory field compares.
 */
export function scoreLongMemEvalRecall(
  questions: ReadonlyArray<BenchQuestion>,
  retrieved: ReadonlyArray<RecallAllRetrieved>,
): RecallAllScore {
  const byId = new Map(retrieved.map((result) => [result.id, result]));
  let scored = 0;
  let passed = 0;
  for (const question of questions) {
    if (!RETRIEVAL_CATEGORIES.has(question.category)) continue;
    scored += 1;
    const top = new Set((byId.get(question.id)?.paths ?? []).slice(0, LONGMEMEVAL_RECALL_CUTOFF));
    if ((question.expected_paths ?? []).every((path) => top.has(path))) passed += 1;
  }
  return { scored, passed };
}