Skip to content

Local-first 🧠 memory for Hermes Agent that lives in your Obsidian vault and remembers project context. Nightly 😴 dream passes turn repeat corrections into confirmed preferences with measurable confidence. Adapters ship for Claude Code, Codex, and OpenClaw, with an MCP server for anything else.

TypeScript
16.3 KB
Raw Blame
/**
 * Census over the in-vault state population.
 *
 * `STATE_SURFACES` claims to be the complete list of durable locations
 * this tool keeps inside a vault. A hand-written list cannot vouch for its
 * own completeness, and the failure mode is silent: a new module joins a
 * path under `<vault>/.open-second-brain/`, ships, and the inventory an
 * operator reads before migrating their vault simply does not mention it.
 *
 * So the derived-store half of the population is DERIVED, from the source:
 * every module under `src/` that names {@link DERIVED_STORE_DIR} - by the
 * constant or by its value - has to be attributed to a row through
 * {@link StateSurface.sources} or excused in
 * {@link STATE_SURFACE_SWEEP_EXCLUSIONS} with a written reason. That is
 * the same instrument `OUT_OF_VAULT_SWEEP_EXCLUSIONS` already applies to
 * the outside-the-vault half, pointed at the other root.
 *
 * ## What the sweep can and cannot see, stated rather than implied
 *
 * It lexes through the shared lexer (`tests/helpers/source-lexer.ts`) and
 * matches on the CODE view, so a directory named in a docblock is not a
 * path builder and a `"{"` inside a value cannot move a block. Four
 * private copies of that lexer produced a real defect in v1.49.0; there is
 * no fifth here.
 *
 *   - `src/` only. `hooks/lib/session-state.ts` builds a real surface
 *     under the same root and is out of the swept tree, so its row names
 *     it explicitly and `tests/core/state/surfaces.test.ts` asserts that
 *     the file it names still exists.
 *   - The `Brain/` half of the population carries no single anchor token -
 *     `BRAIN_LOG_REL`, `brainDirs(...).log` and a bare `"Brain"` are all
 *     the same root - so it is held true by the resolver-binding block in
 *     the unit test rather than by this sweep. Saying which half is swept
 *     is the difference between a census and a claim about a census. That
 *     delegation is only worth what the block covers: it bound 26 of 40
 *     rows when this sentence first appeared, and a deliberately wrong
 *     value in one of the other fourteen changed no test. It now binds
 *     every row bar a counted, reasoned exception list, and the count is
 *     itself an assertion there.
 *   - This sweep NEVER checks that a row's declared path is the path its
 *     owner builds. `attributed()` asks only whether a module is claimed
 *     by some row's `sources`, which is a question about attribution and
 *     not about correctness.
 *   - A path whose root arrives as an argument carries no token to match
 *     on. `writer-lock.ts` and `session-focus.ts` are exactly that shape:
 *     both build under the store directory and neither names it.
 *
 * ## The alarm on the parser
 *
 * A block matcher that goes wrong reports a clean tree, so the population
 * is cross-checked against a second, dumber search that cannot mis-parse
 * because it does not parse - a line regex for a `join(` whose arguments
 * name the store directory. Every file it finds must be in the population.
 */

import { describe, expect, test } from "bun:test";
import { readdirSync, readFileSync } from "node:fs";
import { join, relative } from "node:path";

import { DERIVED_STORE_DIR } from "../../../src/core/brain/path-constants.ts";
import {
  STATE_SURFACE_SWEEP_EXCLUSIONS,
  STATE_SURFACES,
  STATE_SURFACE_ID,
} from "../../../src/core/state/surfaces.ts";
import { lexSource } from "../../helpers/source-lexer.ts";

const REPO_ROOT = join(import.meta.dir, "..", "..", "..");

/** The one root the sweep walks. See the docblock for what living outside costs. */
const SWEPT_ROOT = "src";

/** The identifier the constant is imported under everywhere it is used. */
const STORE_DIR_IDENTIFIER = "DERIVED_STORE_DIR";

/**
 * Modules in the swept tree today. An equality rather than a floor: a
 * glob that stopped matching would sweep an empty set clean and every
 * assertion below would pass over nothing.
 */
// 24: the bootstrap receipt writer joins the population with its row
// (write-side-trust Task 14), the same join install.lock.json made.
// 24 -> 25: the session mining ledger joins the population with its row
// (session-mining-economy), the same join the session import ledger made.
// 25 -> 26: the storage usage meter joins, excused (storage lifecycle
// external review, M3): it joins the directory to MEASURE the store and the
// state beside it as census surfaces, and writes nothing.
// 26 -> 27: the pending mining surveys join the population with their row
// (v1.86.1 review fixes), the same join the session mining ledger made.
// 27 -> 28, net, from the trust-and-responsiveness-hardening wave: the
// gitignore guard and its doctor tracked-state probe join the population,
// excused (one seeds the vault `.gitignore` that covers the directory, one
// probes git's view of it; neither creates state inside it), while the bench
// verb LEFT the population - its runs dir was hoisted to `BENCH_RUNS_REL` in
// `path-constants.ts`, so it no longer spells the directory - taking its now
// stale exclusion with it.
// 30 -> 32, net, from the agent-presence wave: the approvals ledger writer
// (`src/core/brain/lessons-promotion.ts`, t_1e952e43) spells the directory
// for `lessons-approvals.json` and is answered by that surface's row, and
// the instruction-block installer (`src/core/install/instruction-block.ts`,
// t_ba41a78a) spells it for `instruction-block.lock.json` and is answered
// by the `instruction_block_lock` row's `sources`.
// 32 -> 33: the lessons-block readiness check
// (`src/core/brain/doctor/lessons-block-check.ts`, t_1e952e43) joins with
// the same wave, answered by the `lesson_approvals` row's `sources`.
// 33 -> 32: the readiness check names the ledger through
// `lessonsApprovalsPath` (device-local once switched) instead of joining
// the directory itself.
// 32 -> 33: machine-state joins, excused: it copies a bare vault's own
// `.open-second-brain/` once into the device-local root (task 2.10 review).
const SWEPT_POPULATION_SIZE = 33;

/** Declared surfaces today. Pinned for the same reason. */
// 50: the source-archives row (answer-trust Task C1) joins the population,
// the same join the decisions ledger and the bootstrap receipt made.
// 50 -> 51: the session mining ledger joins the population with its row
// (session-mining-economy), the same join the session import ledger made.
// 51 -> 52: the pending mining surveys row (v1.86.1 review fixes).
// 52 -> 53: the cross-topic rule ledger row (t_642bc2b8 substrate), the
// same join the rollup ladder ledger made.
// 53 -> 55: the delivery-trust wave's attestation kernel joins the
// population with two rows - the signed attestation records (attestations)
// and the operator's Ed25519 signing keypair (attestation_keys), the same
// derived-store join the ingest checkpoints row made.
// 55 -> 56: the instruction-block lock row (t_ba41a78a Task B4) joins the
// catalogue - the ownership record uninstall reads to remove exactly the
// recorded pair, the same derived-store join the install manifest made.
// 56 -> 57: the skill-benchmarks row (t_635e3e72 Task A5) joins the
// catalogue - the preregistration a staged scored_rewrite proposal is
// audited against, the same vault-content join the skills tree made.
// 57 -> 58: the lesson_approvals row joins the catalogue with the lessons lane's final
// commit (t_1e952e43), the same join the session-mining ledger made.
// 58 -> 61: the embedding and rerank provider registries and the search
// activation history join the catalogue as device-local rows, so the
// migration moves them (task 2.10 review).
// 61 -> 64: the procedural usage counters, graph and hints join the
// catalogue as device-local rows, so a usage tick appends no store record
// and the migration moves the counters (task 3.10).
const DECLARED_SURFACE_COUNT = 64;

/** An exclusion reason has to be an argument, not a label. */
const MIN_REASON_LENGTH = 80;

interface SourceFile {
  readonly path: string;
  readonly text: string;
}

function readSourceTree(): ReadonlyArray<SourceFile> {
  const files: SourceFile[] = [];
  const walk = (dir: string): void => {
    for (const entry of readdirSync(dir, { withFileTypes: true })) {
      const abs = join(dir, entry.name);
      if (entry.isDirectory()) walk(abs);
      else if (entry.name.endsWith(".ts")) {
        files.push({
          path: relative(REPO_ROOT, abs).split("\\").join("/"),
          text: readFileSync(abs, "utf8"),
        });
      }
    }
  };
  walk(join(REPO_ROOT, SWEPT_ROOT));
  return files.toSorted((a, b) => (a.path < b.path ? -1 : a.path > b.path ? 1 : 0));
}

/**
 * Whether `file` names the derived-store directory in CODE.
 *
 * Two shapes, because both ship: the imported constant, and the bare
 * literal a dozen modules spell privately. The literal is read off the
 * comment-stripped view - which keeps string CONTENTS, so `".open-second-
 * brain"` is still readable - and confirmed against the code view, where a
 * quote delimiter survives but a comment does not. Reading it off the code
 * view alone would find nothing, because that view blanks literal
 * contents; reading it off the raw text would enrol every docblock that
 * mentions the directory by name.
 */
function namesStoreDir(file: SourceFile): boolean {
  const { withoutComments, code } = lexSource(file.text);
  if (new RegExp(`\\b${STORE_DIR_IDENTIFIER}\\b`).test(code)) return true;
  for (const quote of ['"', "'"]) {
    const needle = `${quote}${DERIVED_STORE_DIR}${quote}`;
    for (let at = withoutComments.indexOf(needle); at !== -1;) {
      if (code[at] === quote) return true;
      at = withoutComments.indexOf(needle, at + 1);
    }
  }
  return false;
}

/** Every swept module that names the store directory. */
function sweepPopulation(tree: ReadonlyArray<SourceFile> = TREE): ReadonlyArray<string> {
  return tree.filter(namesStoreDir).map((file) => file.path);
}

/**
 * The naive cross-check: a `join(` whose arguments name the store
 * directory, matched on raw text one line at a time. It cannot mis-parse
 * because it does not parse, and it is strictly narrower than the rule
 * above, so every file it finds must be in the population.
 */
const NAIVE_RE = new RegExp(
  `join\\([^)\\n]*(?:"${DERIVED_STORE_DIR.replace(/\./g, "\\.")}"|${STORE_DIR_IDENTIFIER})`,
);

function naivePathBuilders(tree: ReadonlyArray<SourceFile> = TREE): ReadonlyArray<string> {
  return tree.filter((file) => NAIVE_RE.test(file.text)).map((file) => file.path);
}

/**
 * Whether any declared row claims `path` through its `sources`.
 *
 * A trailing-slash source names the files IN that directory and not the
 * tree beneath it. Unbounded, it was an absolution: a row declaring
 * `sources: ["src/core/"]` would have answered for most of this
 * repository, and the "every declared source module exists" check in
 * `tests/core/state/surfaces.test.ts` would have waved it through
 * because `src/core` does exist. No row spells one today, which is
 * exactly when the bound is cheap to put on.
 */
function attributed(
  path: string,
  surfaces: ReadonlyArray<{ readonly sources: ReadonlyArray<string> }> = STATE_SURFACES,
): boolean {
  return surfaces.some((surface) =>
    surface.sources.some((source) =>
      source.endsWith("/")
        ? path.startsWith(source) && !path.slice(source.length).includes("/")
        : path === source,
    ),
  );
}

const TREE = readSourceTree();

describe("state surface census", () => {
  test("every module that names the derived store is attributed to a row or excused", () => {
    // ONE folded expectation whose value NAMES the work: asserting per
    // module would stop at the first and hide the rest.
    const unaccounted = sweepPopulation().filter(
      (path) => !attributed(path) && !STATE_SURFACE_SWEEP_EXCLUSIONS.has(path),
    );
    expect(unaccounted.toSorted().join("\n")).toBe("");
  });

  test("no exclusion outlives the module it excuses, and every reason says something", () => {
    const swept = new Set(sweepPopulation());
    const stale: string[] = [];
    const thin: string[] = [];
    for (const [path, reason] of STATE_SURFACE_SWEEP_EXCLUSIONS) {
      if (!swept.has(path)) stale.push(path);
      // A placeholder ("todo", "n/a", "") cannot satisfy this.
      if (reason.trim().length < MIN_REASON_LENGTH) thin.push(path);
      if (/\bTODO\b|\bfor now\b|\blater\b/i.test(reason)) thin.push(path);
    }
    expect(stale.toSorted().join("\n")).toBe("");
    expect(thin.toSorted().join("\n")).toBe("");
  });

  test("no module is both attributed and excused", () => {
    // Two answers to one question is how an exclusion survives the row
    // that replaced it, still excusing a module nobody reads it for.
    const both = sweepPopulation().filter(
      (path) => attributed(path) && STATE_SURFACE_SWEEP_EXCLUSIONS.has(path),
    );
    expect(both.toSorted().join("\n")).toBe("");
  });

  test("the naive cross-check finds nothing the population missed", () => {
    // The alarm on the parser itself. A lexer that blanked the wrong span
    // would report a clean sweep over a shrunken population; this search
    // cannot, because it never looks at structure.
    const population = new Set(sweepPopulation());
    const unseen = naivePathBuilders().filter((path) => !population.has(path));
    expect(unseen.toSorted().join("\n")).toBe("");
    // …and it has to have found something, or it is no alarm at all.
    expect(naivePathBuilders().length).toBeGreaterThan(5);
  });
  test("the lesson approvals row attributes the ledger writer (t_1e952e43)", () => {
    const row = STATE_SURFACES.find((r) => r.id === STATE_SURFACE_ID.lessonApprovals);
    expect(row).toBeDefined();
    expect(row?.sources).toContain("src/core/brain/lessons-promotion.ts");
    // ...and the writer sits inside the swept population it answers.
    expect(sweepPopulation()).toContain("src/core/brain/lessons-promotion.ts");
  });

  test("the census is not vacuous", () => {
    expect(TREE.length).toBeGreaterThan(850);
    expect(sweepPopulation().length).toBe(SWEPT_POPULATION_SIZE);
    expect(STATE_SURFACES.length).toBe(DECLARED_SURFACE_COUNT);
  });
});

describe("the census can fail", () => {
  /** The intruder joins the real tree, so the rule runs under real conditions. */
  const intruder = (path: string, text: string): ReadonlyArray<SourceFile> => [
    ...TREE,
    { path, text },
  ];

  /** Only the intruder's own row, so an unrelated module cannot answer for it. */
  const unaccountedFrom = (path: string, text: string): ReadonlyArray<string> =>
    sweepPopulation(intruder(path, text)).filter(
      (p) => p === path && !attributed(p) && !STATE_SURFACE_SWEEP_EXCLUSIONS.has(p),
    );

  test("a new path builder under the store directory is unaccounted for", () => {
    const path = "src/core/synthetic-store-writer.ts";
    const text =
      'import { join } from "node:path";\n' +
      "export function spoolPath(vault: string): string {\n" +
      `  return join(vault, "${DERIVED_STORE_DIR}", "synthetic-spool.jsonl");\n` +
      "}\n";
    expect(unaccountedFrom(path, text)).toEqual([path]);
  });

  test("a builder reaching the directory through the constant is caught too", () => {
    const path = "src/core/synthetic-constant-writer.ts";
    const text =
      'import { DERIVED_STORE_DIR } from "./brain/path-constants.ts";\n' +
      'export const dir = (vault: string): string => join(vault, DERIVED_STORE_DIR, "x");\n';
    expect(unaccountedFrom(path, text)).toEqual([path]);
    expect(naivePathBuilders(intruder(path, text))).toContain(path);
  });

  test("a directory source answers for its own files and not for the tree below it", () => {
    // Driven against a synthetic row rather than a real one, because the
    // point is what the RULE would allow: an unbounded prefix let one
    // source line absolve every module under it, and the population is
    // only as honest as the narrowest claim it accepts.
    const sweeping = [{ sources: ["src/core/"] }];
    expect(attributed("src/core/doctor.ts", sweeping)).toBe(true);
    expect(attributed("src/core/state/migrate.ts", sweeping)).toBe(false);
    expect(attributed("src/core/brain/anticipatory-cache.ts", sweeping)).toBe(false);
  });

  test("a docblock that merely names the directory is not a path builder", () => {
    // The false positive the lexing exists to prevent. Seven modules in
    // the real tree mention the directory only in prose.
    const path = "src/core/synthetic-prose.ts";
    const text = `/** Nothing here writes to ${DERIVED_STORE_DIR}. */\nexport const x = 1;\n`;
    expect(sweepPopulation(intruder(path, text))).not.toContain(path);
  });
});