#1 Persistent memory for AI coding agents based on real-world benchmarks
TypeScript agentmemoryagentsaiclaudeclaudecodecodexcopilotcursorgenaiharnesshermesmemoryompopenclawopencodepi
Preserve captured evidence and checkpoint new vectors earlier (#1448)
Rohit Ghumare3b328a3f3587
diff --git a/docs/reliability/source-and-vector-recovery.md b/docs/reliability/source-and-vector-recovery.mdnew file mode 100644index 00000000..4bfb8986--- /dev/null+++ b/docs/reliability/source-and-vector-recovery.md@@ -0,0 +1,32 @@+# Source retention and vector recovery++Compressed observations retain sanitized source fields for replay and export. The source lives in the same observation row and follows its deletion lifecycle. Search, context expansion, observation listings, shared feeds, and compressed live updates continue to return summaries without the retained source.++## Storage and token bounds++Each serialized source is capped at 16 KiB, including its metadata. Oversized fields retain a prefix and set `truncated: true`; `originalBytes` records the source size before truncation. Previously discarded source cannot be reconstructed. Replay reports source availability and truncation separately for every observation.++Live compression and imports allocate new source from an 8 MiB session budget after subtracting already-retained source. Available response space can reduce that budget further. Existing source above the allowance is preserved, with no additional source allocated; it is not retroactively capped. Imports report `sourceTruncated` and `sourceOmitted`; omission preserves the summary when even source metadata will not fit. A batch that already fits keeps its source unchanged. Live writes and imports share the existing per-session write lock while checking the budget and writing observations. Model calls happen outside the compression write lock. JSONL observation writes run in sequential batches of 20.++These locks coordinate one worker process. They do not provide distributed transactions. JSONL imports commit one file at a time, so a rejected later file can leave earlier files imported. Raising the ordinary capture limit of 500 observations does not raise the retained-source allowance; summary data and temporary raw observations have separate storage and response-size requirements.++Retention adds storage and serialization work. It does not enable model compression, inject extra source into ordinary recall, or add a queue. Explicit replay and export responses can be larger and can consume agent context if an agent requests them.++## First checkpoint and crash window++A fresh vector index schedules its first checkpoint within five seconds of completed vector work, or sooner when the configured interval requires it. Subsequent checkpoints use the configured interval, which defaults to 600,000 ms. Saves write changed vector entries with bounded concurrency rather than rewriting the full index per observation.++Five seconds is a scheduling bound, not a disk-durability guarantee. Completion depends on the storage service, and its persistence window still applies. A crash before a completed, persisted checkpoint can lose vectors. Later unsaved vectors remain subject to the normal batching window. Failed writes stay pending for retry at the normal interval.++Status distinguishes active vector backfill from backfill waiting for explicit opt-in. Missing vectors do not silently trigger unlimited embedding calls. Keyword recovery remains independent of vector coverage.++## Verification++Validation used main `ab3e4efd282659b87d31b36c8514c6bc6b871f0b` as the baseline and an installed package built from the fix branch. Synthetic records and a deterministic local embedding fixture exercised persistence, not semantic retrieval quality.++- The baseline lost long source tails and had zero recovered vectors before its first scheduled checkpoint. The fixed package retained all 20 source records and vectors after a completed checkpoint, a six-second storage-persistence opportunity, and forced worker/service termination.+- Embedding requests stayed at 21 across restart: 20 captures and one search. The ordinary search response stayed at 4,288 wire bytes on both builds; capture hooks emitted no context.+- A 1,000-record import remained readable with explicit truncation. Three subsequent live captures preserved their summaries, truncated one source to the remaining allowance, and omitted the other two sources. Two concurrent 600-record imports remained readable as a 1,200-record session, with source omissions reported for the exhausted budget.+- The full suite passed 2,099 tests with one skipped; build and skill validation passed. Type checking reported the same 29 existing diagnostics as the baseline, with none added.++The initial 20-record fixed fixture used 156,762 bytes of state files versus 41,682 bytes on the baseline. Combined worker/service RSS after restart was 181,488 KiB versus 180,032 KiB. These are point measurements, not capacity or CPU benchmarks; the disk difference includes both retained source and saved vectors.diff --git a/src/functions/compress-synthetic.ts b/src/functions/compress-synthetic.tsindex 6c224bf5..8c64e208 100644--- a/src/functions/compress-synthetic.ts+++ b/src/functions/compress-synthetic.ts@@ -3,6 +3,7 @@ import type {CompressedObservation,ObservationType,} from "../types.js";+import { createObservationSource } from "./observation-source.js";// Zero-LLM compression path. Converts a RawObservation into a// CompressedObservation using only heuristics — no Claude call, no token@@ -100,6 +101,7 @@ export function buildSyntheticCompression(files: extractFiles(raw.toolName, raw.toolInput),importance: 5,confidence: 0.3,+ source: createObservationSource(raw),};if (raw.modality) result.modality = raw.modality;if (raw.imageData) result.imageData = raw.imageData;diff --git a/src/functions/compress.ts b/src/functions/compress.tsindex b6a389a1..becfb766 100644--- a/src/functions/compress.ts+++ b/src/functions/compress.ts@@ -22,6 +22,9 @@ import { scoreCompression } from "../eval/quality.js";import { compressWithRetry } from "../eval/self-correct.js";import type { MetricsStore } from "../eval/metrics-store.js";import { logger } from "../logger.js";+import { createObservationSource, withoutObservationSource } from "./observation-source.js";+import { budgetLiveObservationSource } from "./observation-source-budget.js";+import { withKeyedLock } from "../state/keyed-mutex.js";const VALID_TYPES = new Set<string>(["file_read",@@ -77,6 +80,9 @@ export function registerCompressFunction(raw: RawObservation;}) => {const startMs = Date.now();+ if (!data?.raw || typeof data.raw.hookType !== "string" || !data.raw.hookType) {+ return { success: false, error: "invalid_raw_observation" };+ }let imageDescription: string | undefined;const hasImage = data.raw.modality === "image" || data.raw.modality === "mixed";@@ -156,12 +162,13 @@ export function registerCompressFunction(const qualityScore = scoreCompression(parsed);- const compressed: CompressedObservation = {+ const candidate: CompressedObservation = {id: data.observationId,sessionId: data.sessionId,timestamp: data.raw.timestamp,...parsed,confidence: qualityScore / 100,+ source: createObservationSource(data.raw),...(hasImage ? { modality: data.raw.modality } : {}),...(imageDescription ? { imageDescription } : {}),...(data.raw.imageData ? { imageRef: data.raw.imageData } : {}),@@ -169,11 +176,12 @@ export function registerCompressFunction(...(data.raw.origin ? { origin: data.raw.origin } : {}),};- await kv.set(- KV.observations(data.sessionId),- data.observationId,- compressed,- );+ const compressed = await withKeyedLock(`obs:${data.sessionId}`, async () => {+ const existing = await kv.list<CompressedObservation>(KV.observations(data.sessionId));+ const bounded = budgetLiveObservationSource(candidate, existing);+ await kv.set(KV.observations(data.sessionId), data.observationId, bounded);+ return bounded;+ });try {getSearchIndex().add(compressed);@@ -204,7 +212,7 @@ export function registerCompressFunction(type: "compressed_observation",data: {type: "compressed",- observation: compressed,+ observation: withoutObservationSource(compressed),sessionId: data.sessionId,},},@@ -242,7 +250,7 @@ export function registerCompressFunction(retried,});- return { success: true, compressed, qualityScore };+ return { success: true, compressed: withoutObservationSource(compressed), qualityScore };} catch (err) {const msg = err instanceof Error ? err.message : String(err);const latencyMs = Date.now() - startMs;diff --git a/src/functions/export-import.ts b/src/functions/export-import.tsindex bef2b9ef..1cdd977c 100644--- a/src/functions/export-import.ts+++ b/src/functions/export-import.ts@@ -37,12 +37,13 @@ import {indexObservationSession,unindexObservationSession,} from "../state/obs-index.js";-import { withKeyedLock } from "../state/keyed-mutex.js";+import { withKeyedLock, withKeyedLocks } from "../state/keyed-mutex.js";import { VERSION } from "../version.js";import { recordAudit } from "./audit.js";import { indexRecords } from "./search.js";import { resetLessonIndex } from "./lessons.js";import { logger } from "../logger.js";+import { budgetImportedObservationSources } from "./observation-source-budget.js";// Bounded-concurrency chunk size for the import delete/write loops. A// "replace" or "merge" of a large export (up to MAX_TOTAL_OBSERVATIONS,@@ -310,128 +311,136 @@ export function registerExportImportFunction(sdk: IIIClient, kv: StateKV): voidmemories: 0,summaries: 0,skipped: 0,+ sourceTruncated: 0,+ sourceOmitted: 0,};- if (strategy === "replace") {- const existing = await kv.list<Session>(KV.sessions);- // Collect observation deletes across all sessions, then run them in- // one bounded pass: a runChunked nested inside a runChunked callback- // multiplies in-flight deletes to chunk-size squared.- const obsDeletes: Array<{ sessionId: string; obsId: string }> = [];- await runChunked(existing, async (session) => {- await kv.delete(KV.sessions, session.id);- await removeSessionFromProjectIndex(- kv,- session.project,- session.id,- ).catch(() => {});- const obs = await kv- .list<CompressedObservation>(KV.observations(session.id))- .catch(() => []);- for (const o of obs) {- obsDeletes.push({ sessionId: session.id, obsId: o.id });- }- });- await runChunked(obsDeletes, async (d) => {- await kv.delete(KV.observations(d.sessionId), d.obsId);- await unindexObservationSession(kv, d.obsId).catch(() => {});- });- await runChunked(await kv.list<Memory>(KV.memories), (m) =>- kv.delete(KV.memories, m.id),- );- await runChunked(- await kv.list<SessionSummary>(KV.summaries),- (s) => kv.delete(KV.summaries, s.sessionId),- );- await runChunked(await kv.list<Action>(KV.actions).catch(() => []), (a) =>- kv.delete(KV.actions, a.id),- );- await runChunked(- await kv.list<ActionEdge>(KV.actionEdges).catch(() => []),- (e) => kv.delete(KV.actionEdges, e.id),- );- await runChunked(- await kv.list<Routine>(KV.routines).catch(() => []),- (r) => kv.delete(KV.routines, r.id),- );- await runChunked(- await kv.list<Signal>(KV.signals).catch(() => []),- (s) => kv.delete(KV.signals, s.id),- );- await runChunked(- await kv.list<Checkpoint>(KV.checkpoints).catch(() => []),- (c) => kv.delete(KV.checkpoints, c.id),- );- await runChunked(- await kv.list<Sentinel>(KV.sentinels).catch(() => []),- (s) => kv.delete(KV.sentinels, s.id),- );- await runChunked(- await kv.list<Sketch>(KV.sketches).catch(() => []),- (s) => kv.delete(KV.sketches, s.id),- );- await runChunked(- await kv.list<Crystal>(KV.crystals).catch(() => []),- (c) => kv.delete(KV.crystals, c.id),- );- await runChunked(- await kv.list<Facet>(KV.facets).catch(() => []),- (f) => kv.delete(KV.facets, f.id),- );- await runChunked(- await kv.list<Lesson>(KV.lessons).catch(() => []),- (l) => kv.delete(KV.lessons, l.id),- );- resetLessonIndex();- await runChunked(- await kv.list<Insight>(KV.insights).catch(() => []),- (i) => kv.delete(KV.insights, i.id),- );- await withKeyedLock("graph:persist", async () =>- runChunked(- await kv.list<{ id: string }>(KV.graphNodes).catch(() => []),- (n) => kv.delete(KV.graphNodes, n.id),- ),- );- await withKeyedLock("graph:persist", async () =>- runChunked(- await kv.list<{ id: string }>(KV.graphEdges).catch(() => []),- (e) => kv.delete(KV.graphEdges, e.id),- ),- );- await runChunked(- await kv.list<{ id: string }>(KV.semantic).catch(() => []),- (s) => kv.delete(KV.semantic, s.id),- );- await runChunked(- await kv.list<{ id: string }>(KV.procedural).catch(() => []),- (p) => kv.delete(KV.procedural, p.id),- );- await runChunked(- await kv.list<ProjectProfile>(KV.profiles).catch(() => []),- (profile) => kv.delete(KV.profiles, profile.project),- );- await runChunked(- await kv.list<AccessLogExport>(KV.accessLog).catch(() => []),- (a) => kv.delete(KV.accessLog, a.memoryId),- );- }-- // Records actually written this run, accumulated for search- // indexing after the KV writes settle. Skipped (already-present)- // and merge-overwritten rows are already in the index or will be- // re-added below, so re-indexing them is harmless; we only skip the- // ones the "skip" strategy declined to write.+ const replaceSessions = strategy === "replace" ? await kv.list<Session>(KV.sessions) : [];+ const sessionKeys = [...obsBuckets, ...importData.sessions.map((session) => session.id), ...replaceSessions.map((session) => session.id)]+ .map((id) => `obs:${id}`);const indexObs: CompressedObservation[] = [];const indexMems: Memory[] = [];+ const preflightError = await withKeyedLocks(sessionKeys, async () => {+ const plannedObservations = new Map<string, CompressedObservation[]>();+ for (const [sessionId, observations] of Object.entries(importData.observations)) {+ const existing = strategy === "replace" ? [] : await kv.list<CompressedObservation>(KV.observations(sessionId));+ const planned = budgetImportedObservationSources(observations.map((row) => ({+ ...row, origin: importOrigin(row.origin, row.timestamp),+ })), existing, strategy);+ if (!planned.success) return planned;+ plannedObservations.set(sessionId, planned.observations);+ stats.sourceTruncated += planned.sourceTruncated;+ stats.sourceOmitted += planned.sourceOmitted;+ }++ if (strategy === "replace") {+ const existing = replaceSessions;+ // Collect observation deletes across all sessions, then run them in+ // one bounded pass: a runChunked nested inside a runChunked callback+ // multiplies in-flight deletes to chunk-size squared.+ const obsDeletes: Array<{ sessionId: string; obsId: string }> = [];+ await runChunked(existing, async (session) => {+ await kv.delete(KV.sessions, session.id);+ await removeSessionFromProjectIndex(+ kv,+ session.project,+ session.id,+ ).catch(() => {});+ const obs = await kv+ .list<CompressedObservation>(KV.observations(session.id))+ .catch(() => []);+ for (const o of obs) {+ obsDeletes.push({ sessionId: session.id, obsId: o.id });+ }+ });+ await runChunked(obsDeletes, async (d) => {+ await kv.delete(KV.observations(d.sessionId), d.obsId);+ await unindexObservationSession(kv, d.obsId).catch(() => {});+ });+ await runChunked(await kv.list<Memory>(KV.memories), (m) =>+ kv.delete(KV.memories, m.id),+ );+ await runChunked(+ await kv.list<SessionSummary>(KV.summaries),+ (s) => kv.delete(KV.summaries, s.sessionId),+ );+ await runChunked(await kv.list<Action>(KV.actions).catch(() => []), (a) =>+ kv.delete(KV.actions, a.id),+ );+ await runChunked(+ await kv.list<ActionEdge>(KV.actionEdges).catch(() => []),+ (e) => kv.delete(KV.actionEdges, e.id),+ );+ await runChunked(+ await kv.list<Routine>(KV.routines).catch(() => []),+ (r) => kv.delete(KV.routines, r.id),+ );+ await runChunked(+ await kv.list<Signal>(KV.signals).catch(() => []),+ (s) => kv.delete(KV.signals, s.id),+ );+ await runChunked(+ await kv.list<Checkpoint>(KV.checkpoints).catch(() => []),+ (c) => kv.delete(KV.checkpoints, c.id),+ );+ await runChunked(+ await kv.list<Sentinel>(KV.sentinels).catch(() => []),+ (s) => kv.delete(KV.sentinels, s.id),+ );+ await runChunked(+ await kv.list<Sketch>(KV.sketches).catch(() => []),+ (s) => kv.delete(KV.sketches, s.id),+ );+ await runChunked(+ await kv.list<Crystal>(KV.crystals).catch(() => []),+ (c) => kv.delete(KV.crystals, c.id),+ );+ await runChunked(+ await kv.list<Facet>(KV.facets).catch(() => []),+ (f) => kv.delete(KV.facets, f.id),+ );+ await runChunked(+ await kv.list<Lesson>(KV.lessons).catch(() => []),+ (l) => kv.delete(KV.lessons, l.id),+ );+ resetLessonIndex();+ await runChunked(+ await kv.list<Insight>(KV.insights).catch(() => []),+ (i) => kv.delete(KV.insights, i.id),+ );+ await withKeyedLock("graph:persist", async () =>+ runChunked(+ await kv.list<{ id: string }>(KV.graphNodes).catch(() => []),+ (n) => kv.delete(KV.graphNodes, n.id),+ ),+ );+ await withKeyedLock("graph:persist", async () =>+ runChunked(+ await kv.list<{ id: string }>(KV.graphEdges).catch(() => []),+ (e) => kv.delete(KV.graphEdges, e.id),+ ),+ );+ await runChunked(+ await kv.list<{ id: string }>(KV.semantic).catch(() => []),+ (s) => kv.delete(KV.semantic, s.id),+ );+ await runChunked(+ await kv.list<{ id: string }>(KV.procedural).catch(() => []),+ (p) => kv.delete(KV.procedural, p.id),+ );+ await runChunked(+ await kv.list<ProjectProfile>(KV.profiles).catch(() => []),+ (profile) => kv.delete(KV.profiles, profile.project),+ );+ await runChunked(+ await kv.list<AccessLogExport>(KV.accessLog).catch(() => []),+ (a) => kv.delete(KV.accessLog, a.memoryId),+ );+ }- await runChunked(importData.sessions, async (session) => {- let wrote = false;- await withKeyedLock(`obs:${session.id}`, async () => {+ await runChunked(importData.sessions, async (session) => {if (strategy === "skip") {- const existing = await kv- .get<Session>(KV.sessions, session.id)- .catch(() => null);+ const existing = await kv.get<Session>(KV.sessions, session.id).catch(() => null);if (existing) {stats.skipped++;return;@@ -439,35 +448,34 @@ export function registerExportImportFunction(sdk: IIIClient, kv: StateKV): void}await kv.set(KV.sessions, session.id, session);stats.sessions++;- wrote = true;- });- if (wrote) {await addSessionToProjectIndex(kv, session.project, {id: session.id,startedAt: session.startedAt,...(session.agentId ? { agentId: session.agentId } : {}),}).catch(() => {});- }- });+ });- for (const [sessionId, obs] of Object.entries(importData.observations)) {- await runChunked(obs, async (o) => {- if (strategy === "skip") {- const existing = await kv- .get<CompressedObservation>(KV.observations(sessionId), o.id)- .catch(() => null);- if (existing) {- stats.skipped++;- return;+ for (const [sessionId, obs] of plannedObservations) {+ await runChunked(obs, async (o) => {+ if (strategy === "skip") {+ const existing = await kv+ .get<CompressedObservation>(KV.observations(sessionId), o.id)+ .catch(() => null);+ if (existing) {+ stats.skipped++;+ return;+ }}- }- o.origin = importOrigin(o.origin, o.timestamp);- await kv.set(KV.observations(sessionId), o.id, o);- await indexObservationSession(kv, o.id, sessionId).catch(() => {});- stats.observations++;- indexObs.push(o);- });- }+ await kv.set(KV.observations(sessionId), o.id, o);+ await indexObservationSession(kv, o.id, sessionId).catch(() => {});+ stats.observations++;+ indexObs.push(o);+ });+ }++ return null;+ });+ if (preflightError) return preflightError;await runChunked(importData.memories, async (memory) => {if (strategy === "skip") {diff --git a/src/functions/observation-source-budget.ts b/src/functions/observation-source-budget.tsnew file mode 100644index 00000000..e16748c3--- /dev/null+++ b/src/functions/observation-source-budget.ts@@ -0,0 +1,54 @@+import type { CompressedObservation } from "../types.js";+import { checkPayloadFrameSize, payloadByteLength, SAFE_PAYLOAD_BYTES, type OversizedPayload } from "../state/frame-guard.js";+import { normalizeObservationSource, OBSERVATION_SOURCE_MAX_BYTES, withoutObservationSource } from "./observation-source.js";++export const SESSION_SOURCE_MAX_BYTES = 8 * 1024 * 1024;++export function budgetLiveObservationSource(+ observation: CompressedObservation,+ existing: CompressedObservation[],+): CompressedObservation {+ const planned = budgetImportedObservationSources([observation], existing);+ return planned.success ? planned.observations[0]! : withoutObservationSource(observation);+}++export function budgetImportedObservationSources(+ incoming: CompressedObservation[],+ existing: CompressedObservation[] = [],+ strategy: "merge" | "replace" | "skip" = "merge",+): { success: true; observations: CompressedObservation[]; sourceTruncated: number; sourceOmitted: number } | OversizedPayload {+ const existingById = new Map(existing.map((row) => [row.id, row]));+ const rows = incoming.map((row) => ({ ...row, source: normalizeObservationSource(row.source) }));+ const written = rows.filter((row) => strategy !== "skip" || !existingById.has(row.id));+ const replacedIds = new Set(written.map((row) => row.id));+ const kept = strategy === "replace" ? [] : existing.filter((row) => !replacedIds.has(row.id));+ const base = [...kept, ...written.map(withoutObservationSource)];+ const envelope = { message_type: "invocation_result", invocation_id: "00000000-0000-0000-0000-000000000000", result: base };+ const oversized = checkPayloadFrameSize(envelope, "observation summaries already exceed one session response; split the import into smaller sessions");+ if (oversized) return oversized;++ const existingSourceBytes = kept.reduce((sum, row) => sum + (row.source ? payloadByteLength(row.source) + 10 : 0), 0);+ const sourceRows = written.filter((row) => row.source !== undefined);+ const available = Math.max(0, Math.min(+ SESSION_SOURCE_MAX_BYTES - existingSourceBytes,+ SAFE_PAYLOAD_BYTES - payloadByteLength(envelope) - 64 * 1024,+ ));+ const incomingSourceBytes = sourceRows.reduce((sum, row) => sum + payloadByteLength(row.source) + 10, 0);+ const perSource = incomingSourceBytes <= available+ ? OBSERVATION_SOURCE_MAX_BYTES+ : Math.min(OBSERVATION_SOURCE_MAX_BYTES, Math.floor(available / Math.max(1, sourceRows.length)) - 10);+ let sourceTruncated = 0;+ let sourceOmitted = 0;+ for (const row of sourceRows) {+ const source = row.source!;+ const metadata = { hookType: source.hookType, originalBytes: source.originalBytes, truncated: true };+ if (payloadByteLength(metadata) > perSource) {+ delete row.source;+ sourceOmitted++;+ continue;+ }+ row.source = normalizeObservationSource(source, perSource);+ if (row.source?.truncated) sourceTruncated++;+ }+ return { success: true, observations: rows, sourceTruncated, sourceOmitted };+}diff --git a/src/functions/observation-source.ts b/src/functions/observation-source.tsnew file mode 100644index 00000000..aff6fb0a--- /dev/null+++ b/src/functions/observation-source.ts@@ -0,0 +1,112 @@+import type { CompressedObservation, HookType, ObservationSource, RawObservation } from "../types.js";+import { stripPrivateData } from "./privacy.js";++export const OBSERVATION_SOURCE_MAX_BYTES = 16 * 1024;++function sanitize(value: unknown): unknown {+ if (typeof value === "string") return stripPrivateData(value);+ if (Array.isArray(value)) return value.map(sanitize);+ if (value && typeof value === "object") {+ return Object.fromEntries(Object.entries(value).map(([key, entry]) => [stripPrivateData(key), sanitize(entry)]));+ }+ return value;+}++function bytes(value: unknown): number {+ return Buffer.byteLength(JSON.stringify(value), "utf8");+}++function prefixWithinBytes(text: string, budget: number): string {+ let low = 0;+ let high = text.length;+ while (low < high) {+ const mid = Math.ceil((low + high) / 2);+ if (bytes(text.slice(0, mid)) <= budget) low = mid;+ else high = mid - 1;+ }+ if (low > 0 && /[\uD800-\uDBFF]/.test(text[low - 1])) low--;+ return text.slice(0, low);+}++export function createObservationSource(+ raw: RawObservation,+ previous?: Pick<ObservationSource, "originalBytes" | "truncated">,+ maxBytes = OBSERVATION_SOURCE_MAX_BYTES,+): ObservationSource {+ const fields: Record<string, unknown> = {+ toolName: raw.toolName,+ toolInput: raw.toolInput,+ toolOutput: raw.toolOutput,+ userPrompt: raw.userPrompt,+ assistantResponse: raw.assistantResponse,+ };+ if (Object.values(fields).every((value) => value === undefined)) fields.payload = raw.raw;+ const retained = sanitize(fields) as Record<string, unknown>;+ const hookType = prefixWithinBytes(stripPrivateData(raw.hookType), 128) as HookType;+ const originalBytes = Math.max(bytes({ hookType: raw.hookType, ...retained }), previous?.originalBytes ?? 0);+ const truncated = previous?.truncated === true || hookType !== raw.hookType;+ const full = { hookType, originalBytes, truncated, ...retained };+ if (bytes(full) <= maxBytes) return full;++ const bounded: ObservationSource = { hookType, originalBytes, truncated: true };+ for (const [key, value] of Object.entries(retained)) {+ if (value === undefined) continue;+ if (bytes({ ...bounded, [key]: value }) <= maxBytes) {+ Object.assign(bounded, { [key]: value });+ continue;+ }+ const budget = maxBytes - bytes({ ...bounded, [key]: "" }) + 2;+ if (budget >= 2) {+ const text = typeof value === "string" ? value : JSON.stringify(value);+ Object.assign(bounded, { [key]: prefixWithinBytes(text, budget) });+ }+ break;+ }+ return bounded;+}++export function normalizeObservationSource(value: unknown, maxBytes = OBSERVATION_SOURCE_MAX_BYTES): ObservationSource | undefined {+ if (!value || typeof value !== "object" || Array.isArray(value)) return undefined;+ const source = value as Record<string, unknown>;+ if (typeof source.hookType !== "string" || !source.hookType) return undefined;+ return createObservationSource({+ id: "", sessionId: "", timestamp: "",+ hookType: source.hookType as HookType,+ toolName: typeof source.toolName === "string" ? source.toolName : undefined,+ toolInput: source.toolInput,+ toolOutput: source.toolOutput,+ userPrompt: typeof source.userPrompt === "string" ? source.userPrompt : undefined,+ assistantResponse: typeof source.assistantResponse === "string" ? source.assistantResponse : undefined,+ raw: source.payload,+ }, {+ originalBytes: Number.isSafeInteger(source.originalBytes) && (source.originalBytes as number) >= 0+ ? source.originalBytes as number : 0,+ truncated: source.truncated === true,+ }, maxBytes);+}++export function withoutObservationSource<T extends CompressedObservation>(observation: T): Omit<T, "source"> {+ const { source: _source, ...summary } = observation;+ return summary;+}++export function rawFromObservationSource(observation: CompressedObservation): RawObservation | null {+ const source = observation.source;+ if (!source || typeof source.hookType !== "string") return null;+ return {+ id: observation.id,+ sessionId: observation.sessionId,+ timestamp: observation.timestamp,+ hookType: source.hookType,+ toolName: source.toolName,+ toolInput: source.toolInput,+ toolOutput: source.toolOutput,+ userPrompt: source.userPrompt,+ assistantResponse: source.assistantResponse,+ raw: source.payload ?? {},+ modality: observation.modality,+ imageData: observation.imageRef ?? observation.imageData,+ agentId: observation.agentId,+ origin: observation.origin,+ };+}diff --git a/src/functions/observe.ts b/src/functions/observe.tsindex eb2ed345..e2d3c3f3 100644--- a/src/functions/observe.ts+++ b/src/functions/observe.ts@@ -1,5 +1,5 @@import { TriggerAction, type IIIClient } from "iii-sdk";-import type { RawObservation, HookPayload, Origin } from "../types.js";+import type { RawObservation, CompressedObservation, HookPayload, Origin } from "../types.js";const TOOL_HOOKS = new Set(["pre_tool_use", "post_tool_use", "post_tool_failure"]);import { KV, STREAM, generateId } from "../state/schema.js";@@ -16,6 +16,8 @@ import { getSearchIndex, scheduleIndexSave, vectorIndexAddGuarded } from "./searimport { getAgentId } from "../config.js";import { logger } from "../logger.js";import { saveImageToDisk } from "../utils/image-store.js";+import { withoutObservationSource } from "./observation-source.js";+import { budgetLiveObservationSource } from "./observation-source-budget.js";export function extractImage(d: unknown): string | undefined {if (!d) return undefined;@@ -120,18 +122,19 @@ export function registerObserveFunction(if (typeof sanitizedRaw === "object" && sanitizedRaw !== null) {const d = sanitizedRaw as Record<string, unknown>;- if (- payload.hookType === "post_tool_use" ||- payload.hookType === "post_tool_failure"- ) {+ if (TOOL_HOOKS.has(payload.hookType)) {raw.toolName = d["tool_name"] as string | undefined;raw.toolInput = d["tool_input"];- raw.toolOutput = d["tool_output"] || d["error"];+ raw.toolOutput = d["tool_output"] ?? d["error"];if (raw.origin && raw.toolName) raw.origin.detail = raw.toolName;}if (payload.hookType === "prompt_submit") {raw.userPrompt = d["prompt"] as string | undefined;}+ if (payload.hookType === "stop") {+ const response = d["assistant_response"] ?? d["last_assistant_message"] ?? d["response"];+ if (typeof response === "string") raw.assistantResponse = response;+ }extractedImage = extractImage(sanitizedRaw);if (extractedImage) {@@ -147,8 +150,8 @@ export function registerObserveFunction(const pendingImageData = extractedImage;return withKeyedLock(`obs:${payload.sessionId}`, async () => {+ const existing = await kv.list<CompressedObservation>(KV.observations(payload.sessionId));if (maxObservationsPerSession && maxObservationsPerSession > 0) {- const existing = await kv.list(KV.observations(payload.sessionId));if (existing.length >= maxObservationsPerSession) {return {success: false,@@ -327,7 +330,7 @@ export function registerObserveFunction(action: TriggerAction.Void(),});} else {- const synthetic = buildSyntheticCompression(raw);+ const synthetic = budgetLiveObservationSource(buildSyntheticCompression(raw), existing);await kv.set(KV.observations(payload.sessionId),obsId,@@ -349,7 +352,7 @@ export function registerObserveFunction(item_id: obsId,data: {type: "compressed",- observation: synthetic,+ observation: withoutObservationSource(synthetic),sessionId: payload.sessionId,},},diff --git a/src/functions/replay.ts b/src/functions/replay.tsindex 4197e374..92ff5959 100644--- a/src/functions/replay.ts+++ b/src/functions/replay.ts@@ -15,12 +15,15 @@ import { KV, generateId, fingerprintId } from "../state/schema.js";import { addSessionToProjectIndex } from "../state/session-index.js";import { withKeyedLock } from "../state/keyed-mutex.js";import { parseJsonlText } from "../replay/jsonl-parser.js";+import { checkPayloadFrameSize } from "../state/frame-guard.js";import { resetLessonIndex } from "./lessons.js";import { projectTimeline, type Timeline } from "../replay/timeline.js";import { safeAudit } from "./audit.js";import { buildSyntheticCompression } from "./compress-synthetic.js";import { indexRecords } from "./search.js";import { logger } from "../logger.js";+import { rawFromObservationSource } from "./observation-source.js";+import { budgetImportedObservationSources } from "./observation-source-budget.js";export const MAX_FILES_DEFAULT = 200;export const MAX_FILES_UPPER_BOUND = 1000;@@ -67,6 +70,8 @@ function splitSyntheticNarrative(narrative: string): { input: unknown; output: s}export function rawFromCompressed(obs: CompressedObservation): RawObservation {+ const retained = rawFromObservationSource(obs);+ if (retained) return retained;const isPrompt = obs.type === "conversation";const narrative = obs.narrative || "";const split = isPrompt ? null : splitSyntheticNarrative(narrative);@@ -220,11 +225,28 @@ function isRawShape(o: unknown): o is RawObservation {async function loadObservations(kv: StateKV,sessionId: string,-): Promise<RawObservation[]> {+): Promise<{ observations: RawObservation[]; sourceRetention: SourceRetention[] }> {const rows = await kv.list<RawObservation | CompressedObservation>(KV.observations(sessionId),);- return rows.map((r) => (isRawShape(r) ? r : rawFromCompressed(r as CompressedObservation)));+ return {+ observations: rows.map((r) => (isRawShape(r) ? r : rawFromCompressed(r as CompressedObservation))),+ sourceRetention: rows.map((row) => {+ const source = !isRawShape(row) ? row.source : undefined;+ return {+ observationId: row.id,+ retained: isRawShape(row) || source !== undefined,+ ...(source ? { truncated: source.truncated, originalBytes: source.originalBytes } : {}),+ };+ }),+ };+}++interface SourceRetention {+ observationId: string;+ retained: boolean;+ truncated?: boolean;+ originalBytes?: number;}async function findJsonlFiles(@@ -286,16 +308,17 @@ export function registerReplayFunctions(sdk: IIIClient, kv: StateKV): void {sdk.registerFunction("mem::replay::load",async (data: { sessionId: string }): Promise<- | { success: true; timeline: Timeline; session: Session | null }+ | { success: true; timeline: Timeline; session: Session | null; sourceRetention: SourceRetention[] }| { success: false; error: string }> => {if (!data?.sessionId || typeof data.sessionId !== "string") {return { success: false, error: "sessionId is required" };}const session = await kv.get<Session>(KV.sessions, data.sessionId);- const observations = await loadObservations(kv, data.sessionId);+ const { observations, sourceRetention } = await loadObservations(kv, data.sessionId);const timeline = projectTimeline(observations);- return { success: true, timeline, session };+ const result = { success: true as const, timeline, session, sourceRetention };+ return checkPayloadFrameSize(result, "this session is too large to replay in one response") ?? result;},);@@ -323,8 +346,10 @@ export function registerReplayFunctions(sdk: IIIClient, kv: StateKV): void {traversalCapped: boolean;maxFiles: number;maxFilesUpperBound: number;+ sourceTruncated: number;+ sourceOmitted: number;}- | { success: false; error: string }+ | { success: false; error: string; imported?: number; sessionIds?: string[]; observations?: number }> => {const defaultRoot = join(homedir(), ".claude", "projects");const rawPath = data.path || defaultRoot;@@ -385,11 +410,15 @@ export function registerReplayFunctions(sdk: IIIClient, kv: StateKV): void {traversalCapped,maxFiles,maxFilesUpperBound: MAX_FILES_UPPER_BOUND,+ sourceTruncated: 0,+ sourceOmitted: 0,};}const sessionIds: string[] = [];let observationCount = 0;+ let sourceTruncated = 0;+ let sourceOmitted = 0;for (const file of files) {if (isSensitive(file)) continue;@@ -415,70 +444,52 @@ export function registerReplayFunctions(sdk: IIIClient, kv: StateKV): void {? firstPromptObs.userPrompt.replace(/\s+/g, " ").trim().slice(0, 200): undefined;- const sessionRow = await withKeyedLock(`obs:${parsed.sessionId}`, async (): Promise<Session> => {+ const incoming = parsed.observations.map((obs) => {+ const synthetic = buildSyntheticCompression(obs);+ synthetic.origin = importOrigin(synthetic.origin, synthetic.timestamp, "jsonl");+ return synthetic;+ });+ const committed = await withKeyedLock(`obs:${parsed.sessionId}`, async () => {+ const existingRows = await kv.list<CompressedObservation>(KV.observations(parsed.sessionId));+ const planned = budgetImportedObservationSources(incoming, existingRows);+ if (!planned.success) return planned;const existing = await kv.get<Session>(KV.sessions, parsed.sessionId);- if (existing) {- existing.observationCount =- (existing.observationCount || 0) + parsed.observations.length;- if (parsed.endedAt > (existing.endedAt || "")) {- existing.endedAt = parsed.endedAt;- }- if (existing.status === "active") existing.status = "completed";- const existingTags = existing.tags || [];- if (!existingTags.includes("jsonl-import")) {- existing.tags = [...existingTags, "jsonl-import"];- }- if (!existing.firstPrompt && firstPrompt) {- existing.firstPrompt = firstPrompt;- }- // #775: re-key on parsed.sessionId, not existing.id. Older- // session rows may be missing the `id` field; existing.id- // would then be undefined, JSON.stringify would drop the- // `key` from the state::set payload, and the engine would- // reject the call with `missing field \`key\``. Because the- // rejection aborts the whole import handler, a single- // legacy row killed the entire batch. parsed.sessionId is- // always populated (parseJsonlText has a three-level- // fallback) and is what we just used to read the row.- if (!existing.id) existing.id = parsed.sessionId;- await kv.set(KV.sessions, parsed.sessionId, existing);- return existing;- } else {- const session: Session = {- id: parsed.sessionId,- project: parsed.project,- cwd: parsed.cwd,- startedAt: parsed.startedAt,- endedAt: parsed.endedAt,- status: "completed",- observationCount: parsed.observations.length,- tags: ["jsonl-import"],- firstPrompt,- };- await kv.set(KV.sessions, session.id, session);- return session;+ const sessionRow: Session = existing ?? {+ id: parsed.sessionId,+ project: parsed.project,+ cwd: parsed.cwd,+ startedAt: parsed.startedAt,+ endedAt: parsed.endedAt,+ status: "completed",+ observationCount: 0,+ tags: ["jsonl-import"],+ firstPrompt,+ };+ sessionRow.id ||= parsed.sessionId;+ sessionRow.observationCount = (sessionRow.observationCount || 0) + parsed.observations.length;+ if (parsed.endedAt > (sessionRow.endedAt || "")) sessionRow.endedAt = parsed.endedAt;+ if (sessionRow.status === "active") sessionRow.status = "completed";+ if (!sessionRow.tags?.includes("jsonl-import")) sessionRow.tags = [...(sessionRow.tags || []), "jsonl-import"];+ if (!sessionRow.firstPrompt && firstPrompt) sessionRow.firstPrompt = firstPrompt;+ await kv.set(KV.sessions, parsed.sessionId, sessionRow);+ for (let i = 0; i < planned.observations.length; i += 20) {+ const batch = planned.observations.slice(i, i + 20);+ const results = await Promise.allSettled(batch.map((obs) => kv.set(KV.observations(parsed.sessionId), obs.id, obs)));+ const failure = results.find((result) => result.status === "rejected");+ if (failure) throw failure.reason;}+ return { ...planned, sessionRow };});-+ if (!committed.success) return { ...committed, imported: sessionIds.length, sessionIds, observations: observationCount };+ const { sessionRow, observations: compressed } = committed;+ sourceTruncated += committed.sourceTruncated;+ sourceOmitted += committed.sourceOmitted;await addSessionToProjectIndex(kv, sessionRow.project, {id: sessionRow.id,startedAt: sessionRow.startedAt,...(sessionRow.agentId ? { agentId: sessionRow.agentId } : {}),}).catch(() => {});- const compressed: CompressedObservation[] = [];- await Promise.all(- parsed.observations.map(async (obs) => {- const synthetic = buildSyntheticCompression(obs);- synthetic.origin = importOrigin(- synthetic.origin,- synthetic.timestamp,- "jsonl",- );- compressed.push(synthetic);- await kv.set(KV.observations(parsed.sessionId), obs.id, synthetic);- }),- );// BM25 + vector in one path so jsonl-imported observations are// reachable by semantic search, not just keyword.try {@@ -518,6 +529,8 @@ export function registerReplayFunctions(sdk: IIIClient, kv: StateKV): void {traversalCapped,maxFiles,maxFilesUpperBound: MAX_FILES_UPPER_BOUND,+ sourceTruncated,+ sourceOmitted,};},);diff --git a/src/functions/search.ts b/src/functions/search.tsindex 7be779b6..0493e23a 100644--- a/src/functions/search.ts+++ b/src/functions/search.ts@@ -9,6 +9,7 @@ import type { EmbeddingProvider } from '../types.js'import { memoryToObservation } from '../state/memory-utils.js'import { recordAccessBatch } from './access-tracker.js'import { logger } from "../logger.js";+import { withoutObservationSource } from "./observation-source.js";import {getAgentId,getVectorBackfillMax,@@ -56,6 +57,14 @@ export function isBm25RebuildIncomplete(): boolean {}let pendingVectorBackfill = 0+export type VectorBackfillState = "idle" | "running" | "waiting-for-opt-in"+let vectorBackfillState: VectorBackfillState = "idle"+export function getVectorBackfillState(): VectorBackfillState {+ return vectorBackfillState+}+export function setVectorBackfillState(state: VectorBackfillState): void {+ vectorBackfillState = state+}export function getPendingVectorBackfillCount(): number {return pendingVectorBackfill}@@ -218,7 +227,9 @@ export async function vectorIndexAddGuarded(}if (commit) {try {- return await commit(embedding)+ const committed = await commit(embedding)+ if (committed) scheduleIndexSave()+ return committed} catch (err) {logger.warn("vector-index add: commit failed — skipping", {kind: context.kind,@@ -230,6 +241,7 @@ export async function vectorIndexAddGuarded(}}vi.add(id, sessionId, embedding)+ scheduleIndexSave()return true}@@ -841,7 +853,7 @@ export function registerSearchFunction(sdk: IIIClient, kv: StateKV): void {if (filterAgentId !== undefined && obs.agentId !== filterAgentId) continueif (enriched.length >= effectiveLimit) breakenriched.push({- observation: obs,+ observation: withoutObservationSource(obs),score: candidates[i].score,sessionId: candidates[i].sessionId,})diff --git a/src/functions/smart-search.ts b/src/functions/smart-search.tsindex c8c74d49..a58f8553 100644--- a/src/functions/smart-search.ts+++ b/src/functions/smart-search.ts@@ -20,6 +20,7 @@ import {getFollowupWindowSeconds,} from "../config.js";import { logger } from "../logger.js";+import { withoutObservationSource } from "./observation-source.js";import { getCounters } from "../telemetry/setup.js";// #771: smart-search followup-rate diagnostic. Stored per session as@@ -152,7 +153,7 @@ export function registerSmartSearchFunction(const results = await Promise.all(items.map(({ obsId, sessionId }) =>findObservation(kv, obsId, sessionId).then((obs) =>- obs ? { obsId, sessionId: obs.sessionId, observation: obs } : null,+ obs ? { obsId, sessionId: obs.sessionId, observation: withoutObservationSource(obs) } : null,),),);diff --git a/src/functions/status.ts b/src/functions/status.tsindex fbc24f24..19fb07d7 100644--- a/src/functions/status.ts+++ b/src/functions/status.ts@@ -1,4 +1,5 @@import type { IndexLegStatus, IndexPersistenceStatus } from "../state/index-persistence.js";+import type { VectorBackfillState } from "./search.js";export type StatusLevel = "ok" | "info" | "warn" | "error";@@ -65,6 +66,7 @@ export interface StatusInputs {sessions: number | null;bm25Incomplete: boolean;pendingVectorBackfill: number;+ vectorBackfillState?: VectorBackfillState;};graph: GraphStatsInput | null;graphExtractionEnabled: boolean;@@ -199,6 +201,16 @@ function secondsBetween(later: Date, earlierIso: string | undefined): number | nreturn Math.max(0, Math.round((later.getTime() - earlier) / 1000));}+function vectorBackfillFix(state: VectorBackfillState | undefined): string {+ if (state === "waiting-for-opt-in") {+ return "Backfill is paused. Set AGENTMEMORY_VECTOR_BACKFILL=all and restart to opt in. This calls the embedding provider and is capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX.";+ }+ if (state === "running") {+ return "Backfill is running in the background, capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX. Check the server log for embedding provider errors if progress stops.";+ }+ return "Backfill is not running. Check the server log and embedding provider configuration. A full backfill requires AGENTMEMORY_VECTOR_BACKFILL=all and a restart, and can consume embedding provider tokens.";+}+export function evaluateStatus(input: StatusInputs): StatusReport {const problems: StatusProblem[] = [];@@ -300,7 +312,7 @@ export function evaluateStatus(input: StatusInputs): StatusReport {level: "info",code: "index-vector-backfill-pending",message: `${input.index.pendingVectorBackfill} documents are waiting for a vector embedding.`,- fix: "This runs in the background and is capped per boot by AGENTMEMORY_VECTOR_BACKFILL_MAX. If it is stuck at a nonzero count with no embedding provider errors, set AGENTMEMORY_VECTOR_BACKFILL=all to run a full backfill.",+ fix: vectorBackfillFix(input.index.vectorBackfillState),});}@@ -321,7 +333,7 @@ export function evaluateStatus(input: StatusInputs): StatusReport {level: "warn",code: "index-save-stale",message: `The vector index has unsaved changes from ${formatDuration(dirtyAge)} ago.`,- fix: "Saves run at most once per AGENTMEMORY_INDEX_SAVE_INTERVAL_MS. Check the server log for save errors or a save that never finishes.",+ fix: "After the first checkpoint, saves are batched by AGENTMEMORY_INDEX_SAVE_INTERVAL_MS. Check the server log for save errors or a save that never finishes.",});}}@@ -330,8 +342,8 @@ export function evaluateStatus(input: StatusInputs): StatusReport {problems.push({level: "warn",code: "index-vector-count-shortfall",- message: `Only ${loaded} of ${expected} vectors loaded from the last save; a bounded backfill re-embeds the rest.`,- fix: "No action needed — missing vectors are queued for backfill up to the per-boot cap.",+ message: `Only ${loaded} of ${expected} vectors loaded from the last save.`,+ fix: vectorBackfillFix(input.index.vectorBackfillState),});}@@ -657,6 +669,7 @@ ${row("Missing from index", escapeHtml(idx.missingObservations ?? "not checked")${row("Sessions", escapeHtml(idx.sessions ?? "unknown"))}${row("Keyword index rebuild", idx.bm25Incomplete ? '<span class="warn">incomplete</span>' : "complete")}${row("Pending vector backfill", escapeHtml(idx.pendingVectorBackfill))}+${idx.vectorBackfillState ? row("Vector backfill", escapeHtml(idx.vectorBackfillState === "waiting-for-opt-in" ? "paused, waiting for opt-in" : idx.vectorBackfillState)) : ""}${indexPersistenceRows(report)}</table><h2>Knowledge graph</h2><table>diff --git a/src/functions/team.ts b/src/functions/team.tsindex 532e8b28..351a6bdb 100644--- a/src/functions/team.ts+++ b/src/functions/team.ts@@ -4,11 +4,13 @@ import type {TeamSharedItem,TeamProfile,Memory,+ CompressedObservation,} from "../types.js";import { KV, generateId } from "../state/schema.js";import type { StateKV } from "../state/kv.js";import { recordAudit } from "./audit.js";import { logger } from "../logger.js";+import { withoutObservationSource } from "./observation-source.js";const VALID_ITEM_TYPES = new Set(["memory", "pattern", "observation"]);@@ -42,7 +44,8 @@ export function registerTeamFunction(error: "sessionId is required for observations",};}- content = await kv.get(KV.observations(data.sessionId), data.itemId);+ const observation = await kv.get<CompressedObservation>(KV.observations(data.sessionId), data.itemId);+ content = observation ? withoutObservationSource(observation) : null;} else {content = await kv.get<Memory>(KV.memories, data.itemId);}@@ -89,7 +92,12 @@ export function registerTeamFunction().slice(0, limit);- return { items: sorted, total: filtered.length };+ return {+ items: sorted.map((item) => item.type === "observation" && item.content && typeof item.content === "object"+ ? { ...item, content: withoutObservationSource(item.content as CompressedObservation) }+ : item),+ total: filtered.length,+ };},);diff --git a/src/functions/timeline.ts b/src/functions/timeline.tsindex 2d83876a..231bb967 100644--- a/src/functions/timeline.ts+++ b/src/functions/timeline.ts@@ -8,6 +8,7 @@ import { KV } from "../state/schema.js";import { StateKV } from "../state/kv.js";import { recordAccessBatch } from "./access-tracker.js";import { logger } from "../logger.js";+import { withoutObservationSource } from "./observation-source.js";export function registerTimelineFunction(sdk: IIIClient, kv: StateKV): void {sdk.registerFunction("mem::timeline",@@ -85,7 +86,7 @@ export function registerTimelineFunction(sdk: IIIClient, kv: StateKV): void {const obs = allObs[i];const { sid, ...observation } = obs;entries.push({- observation,+ observation: withoutObservationSource(observation),sessionId: sid,relativePosition: i - anchorIdx,});diff --git a/src/index.ts b/src/index.tsindex e85f1621..16dfd396 100644--- a/src/index.ts+++ b/src/index.ts@@ -46,6 +46,7 @@ import {setIndexPersistence,setHybridRanker,setPendingVectorBackfillCount,+ setVectorBackfillState,} from "./functions/search.js";import { registerContextFunction } from "./functions/context.js";import { registerSessionIndexMaintenanceFunction } from "./functions/session-index-maintenance.js";@@ -507,6 +508,7 @@ async function main() {`Rebuilt BM25 index from stored content (${keyword.documents} docs in ${Date.now() - keywordStart} ms)`,);setPendingVectorBackfillCount(keyword.vectorJobs.length + keyword.fullBackfillPending);+ setVectorBackfillState(keyword.fullBackfillPending > 0 ? "waiting-for-opt-in" : "idle");if (keyword.fullBackfillPending > 0) {bootLog(`Vector backfill needs ${keyword.fullBackfillPending} embeddings but a full backfill was not started ` +@@ -514,13 +516,16 @@ async function main() {);}if (keyword.vectorJobs.length > 0) {+ setVectorBackfillState("running");bootLog(`Backfilling ${keyword.vectorJobs.length} missing vectors in the background`);void backfillVectors(keyword.vectorJobs).then((count) => {setPendingVectorBackfillCount(keyword.fullBackfillPending);+ setVectorBackfillState(keyword.fullBackfillPending > 0 ? "waiting-for-opt-in" : "idle");if (count > 0) bootLog(`Vector index backfilled: ${count} entries`);}).catch((err) => {+ setVectorBackfillState("idle");console.warn(`[agentmemory] Failed to backfill vectors:`, err);});}diff --git a/src/replay/timeline.ts b/src/replay/timeline.tsindex 03e8051d..e706605c 100644--- a/src/replay/timeline.ts+++ b/src/replay/timeline.ts@@ -87,6 +87,9 @@ function truncate(text: string, max: number): string {function bodyFor(obs: RawObservation, kind: TimelineEventKind): string | undefined {if (kind === "prompt") return obs.userPrompt;if (kind === "response") return obs.assistantResponse;+ if (kind === "hook" && obs.raw != null) {+ return typeof obs.raw === "string" ? obs.raw : JSON.stringify(obs.raw);+ }return undefined;}diff --git a/src/state/index-persistence.ts b/src/state/index-persistence.tsindex d046d0dd..7313c109 100644--- a/src/state/index-persistence.ts+++ b/src/state/index-persistence.ts@@ -15,6 +15,7 @@ const VECTOR_META_KEY = "vectors:meta";const VECTOR_BUCKET_SCOPE_PREFIX = `${KV.bm25Index}:vec:`;const WRITE_CONCURRENCY = 32;const LOAD_CONCURRENCY = 8;+const FIRST_CHECKPOINT_DELAY_MS = 5_000;function auditIndexPersistEnabled(): boolean {const raw = process.env.AGENTMEMORY_AUDIT_INDEX_PERSIST;@@ -63,6 +64,8 @@ export interface VectorCountShortfall {export interface IndexPersistenceStatus {saveIntervalMs: number;+ firstCheckpointPending?: boolean;+ nextSaveAt?: string | null;saving: boolean;buckets: number;pendingChanges: number;@@ -144,6 +147,8 @@ export class IndexPersistence {private dirtyEpoch = 0;private markedDuringRunAt: number | null = null;private metaWritten = false;+ private firstSaveAttempted = false;+ private nextSaveAt: number | null = null;private leg: IndexLegStatus = emptyLegStatus();private readonly saveIntervalMs: number;private readonly bucketSize: number;@@ -174,17 +179,12 @@ export class IndexPersistence {}scheduleSave(): void {- if (this.stopped || !this.vector) return;+ if (this.stopped || !this.vector?.pendingChanges) return;const now = this.now();this.dirtyEpoch++;if (this.running && this.markedDuringRunAt === null) this.markedDuringRunAt = now;if (this.leg.dirtySince === null) this.leg.dirtySince = new Date(now).toISOString();- if (this.timer) return;- const delay = Math.max(0, this.lastSaveAt + this.saveIntervalMs - now);- this.timer = setTimeout(() => {- this.timer = null;- this.save().catch((err) => this.logFailure(err));- }, delay);+ this.queueSave();}save(): Promise<void> {@@ -204,6 +204,8 @@ export class IndexPersistence {status(): IndexPersistenceStatus {return {saveIntervalMs: this.saveIntervalMs,+ firstCheckpointPending: Boolean(this.vector && !this.metaWritten && (this.vector.pendingChanges > 0 || this.running)),+ nextSaveAt: this.nextSaveAt === null ? null : new Date(this.nextSaveAt).toISOString(),saving: this.running !== null,buckets: this.bucketCountInUse(),pendingChanges: this.vector?.pendingChanges ?? 0,@@ -285,11 +287,28 @@ export class IndexPersistence {clearTimeout(this.timer);this.timer = null;}+ this.nextSaveAt = null;+ }++ private queueSave(): void {+ if (this.stopped || this.timer || !this.vector?.pendingChanges) return;+ const now = this.now();+ const remaining = Math.max(0, this.lastSaveAt + this.saveIntervalMs - now);+ const delay = !this.metaWritten && !this.firstSaveAttempted+ ? Math.min(FIRST_CHECKPOINT_DELAY_MS, remaining)+ : remaining;+ this.nextSaveAt = now + delay;+ this.timer = setTimeout(() => {+ this.timer = null;+ this.nextSaveAt = null;+ this.save().catch((err) => this.logFailure(err));+ }, delay);}private startRun(): Promise<void> {const run = this.runSave().finally(() => {if (this.running === run) this.running = null;+ if (!this.queued) this.queueSave();});this.running = run;return run;@@ -297,10 +316,11 @@ export class IndexPersistence {private async runSave(): Promise<void> {const vector = this.vector;- if (!vector) return;+ if (!vector?.pendingChanges) return;const epoch = this.dirtyEpoch;this.markedDuringRunAt = null;this.lastSaveAt = this.now();+ this.firstSaveAttempted = true;try {await this.writeChanges(vector);this.leg.lastSavedAt = new Date(this.now()).toISOString();@@ -315,13 +335,15 @@ export class IndexPersistence {this.leg.lastError = errorMessage(err);this.leg.lastErrorAt = new Date(this.now()).toISOString();if (this.leg.dirtySince === null) this.leg.dirtySince = this.leg.lastErrorAt;+ this.lastSaveAt = this.now();+ this.clearTimer();this.logFailure(err);}}private async writeChanges(vector: VectorIndex): Promise<void> {const changes = vector.takeChanges();- if (changes.size === 0 && this.metaWritten) return;+ if (changes.size === 0) return;const failed = new Map<string, boolean>();const failures = await inBatches([...changes], WRITE_CONCURRENCY, async ([id, present]) => {const entry = present ? vector.get(id) : undefined;@@ -351,12 +373,17 @@ export class IndexPersistence {`${failures.length} of ${changes.size} vector writes failed: ${errorMessage(failures[0])}`,);}- await this.kv.set<VectorMeta>(KV.bm25Index, VECTOR_META_KEY, {- v: 3,- bucketCount: this.bucketCountInUse(),- savedAt: new Date(this.now()).toISOString(),- count: vector.size,- });+ try {+ await this.kv.set<VectorMeta>(KV.bm25Index, VECTOR_META_KEY, {+ v: 3,+ bucketCount: this.bucketCountInUse(),+ savedAt: new Date(this.now()).toISOString(),+ count: vector.size,+ });+ } catch (err) {+ vector.returnChanges(changes);+ throw err;+ }this.metaWritten = true;}diff --git a/src/state/keyed-mutex.ts b/src/state/keyed-mutex.tsindex b28531a6..28d9ceac 100644--- a/src/state/keyed-mutex.ts+++ b/src/state/keyed-mutex.ts@@ -16,3 +16,11 @@ export function withKeyedLock<T>(});return next;}++export function withKeyedLocks<T>(keys: string[], fn: () => Promise<T>): Promise<T> {+ const ordered = [...new Set(keys)].sort();+ const acquire = (index: number): Promise<T> => index === ordered.length+ ? fn()+ : withKeyedLock(ordered[index], () => acquire(index + 1));+ return acquire(0);+}diff --git a/src/triggers/api.ts b/src/triggers/api.tsindex 8ae72d43..55f2c599 100644--- a/src/triggers/api.ts+++ b/src/triggers/api.ts@@ -19,6 +19,7 @@ import {getIndexPersistenceStatus,getEmbeddingProvider,getPendingVectorBackfillCount,+ getVectorBackfillState,getSearchIndex,getVectorIndex,isBm25RebuildIncomplete,@@ -41,6 +42,7 @@ import { MAX_FILES_UPPER_BOUND } from "../functions/replay.js";import { describeGraphNode } from "../functions/graph-node.js";import { LESSON_SOURCE_IDS_MAX, normalizeLessonSourceIds } from "../functions/lessons.js";import { logger } from "../logger.js";+import { withoutObservationSource } from "../functions/observation-source.js";import {isGraphExtractionEnabled,isConsolidationEnabled,@@ -390,6 +392,7 @@ export function createStatusReporter(sdk: IIIClient, kv: StateKV, deps: StatusResessions: unindexed ? unindexed.sessions : null,bm25Incomplete: isBm25RebuildIncomplete(),pendingVectorBackfill: getPendingVectorBackfillCount(),+ vectorBackfillState: getVectorBackfillState(),},graph,graphExtractionEnabled: isGraphExtractionEnabled(),@@ -1283,7 +1286,7 @@ export function registerApiTriggers(filtered = filtered.filter((o) => (o.importance ?? 0) >= minImportance);}if (listQuery.limit === undefined && !listQuery.cursor) {- return { status_code: 200, body: { observations: filtered, total: filtered.length, nextCursor: null } };+ return { status_code: 200, body: { observations: filtered.map(withoutObservationSource), total: filtered.length, nextCursor: null } };}const total = filtered.length;const paged = pageAfterCursor(@@ -1295,7 +1298,7 @@ export function registerApiTriggers();return {status_code: 200,- body: { observations: paged.page, total, nextCursor: paged.nextCursor },+ body: { observations: paged.page.map(withoutObservationSource), total, nextCursor: paged.nextCursor },};},);diff --git a/src/types.ts b/src/types.tsindex 38af3d2d..a8674e2f 100644--- a/src/types.ts+++ b/src/types.ts@@ -67,6 +67,18 @@ export interface RawObservation {origin?: Origin;}+export interface ObservationSource {+ hookType: HookType;+ originalBytes: number;+ truncated: boolean;+ toolName?: string;+ toolInput?: unknown;+ toolOutput?: unknown;+ userPrompt?: string;+ assistantResponse?: string;+ payload?: unknown;+}+export interface CompressedObservation {id: string;sessionId: string;@@ -80,6 +92,7 @@ export interface CompressedObservation {files: string[];importance: number;confidence?: number;+ source?: ObservationSource;imageRef?: string;imageData?: string;imageDescription?: string;diff --git a/test/index-first-checkpoint.test.ts b/test/index-first-checkpoint.test.tsnew file mode 100644index 00000000..b0e02eaa--- /dev/null+++ b/test/index-first-checkpoint.test.ts@@ -0,0 +1,194 @@+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import { IndexPersistence, vectorBucketScope } from "../src/state/index-persistence.js";+import { VectorIndex } from "../src/state/vector-index.js";+import { scheduleIndexSave, setEmbeddingProvider, setIndexPersistence, setVectorIndex, vectorIndexAddGuarded } from "../src/functions/search.js";++const INDEX_SCOPE = "mem:index:bm25";+const META_KEY = "vectors:meta";++function mockKV() {+ const store = new Map<string, Map<string, unknown>>();+ const writes: Array<{ scope: string; key: string }> = [];+ return {+ store,+ writes,+ get: async <T>(scope: string, key: string): Promise<T | null> => (store.get(scope)?.get(key) as T) ?? null,+ set: async <T>(scope: string, key: string, data: T): Promise<T> => {+ writes.push({ scope, key });+ if (!store.has(scope)) store.set(scope, new Map());+ store.get(scope)!.set(key, data);+ return data;+ },+ delete: async (scope: string, key: string): Promise<void> => { store.get(scope)?.delete(key); },+ list: async <T>(scope: string): Promise<T[]> => Array.from(store.get(scope)?.values() ?? []) as T[],+ };+}++function add(vector: VectorIndex, id: string): void {+ vector.add(id, "ses_1", new Float32Array([1, 0.5, 0.25]));+}++describe("first vector checkpoint", () => {+ beforeEach(() => { vi.useFakeTimers(); });+ afterEach(() => {+ setIndexPersistence(null);+ setEmbeddingProvider(null);+ setVectorIndex(null);+ vi.useRealTimers();+ });++ it.each([false, true])("schedules a lone vector only after its slow embedding commits (callback=%s)", async (withCommit) => {+ const kv = mockKV();+ const vector = new VectorIndex();+ const persistence = new IndexPersistence(kv as never, vector, { saveIntervalMs: 600_000 });+ let finish!: (embedding: Float32Array) => void;+ const embed = vi.fn(() => new Promise<Float32Array>((resolve) => { finish = resolve; }));+ setVectorIndex(vector);+ setIndexPersistence(persistence);+ setEmbeddingProvider({ name: "test", dimensions: 3, embed, embedBatch: vi.fn() });+ scheduleIndexSave();+ const insertion = vectorIndexAddGuarded("obs_1", "ses_1", "one captured observation", { kind: "observation", logId: "obs_1" }, withCommit ? (embedding) => {+ vector.add("obs_1", "ses_1", embedding);+ return true;+ } : undefined);+ await vi.advanceTimersByTimeAsync(6_000);+ expect(kv.writes).toHaveLength(0);+ finish(new Float32Array([1, 0.5, 0.25]));+ await expect(insertion).resolves.toBe(true);+ expect(persistence.status().firstCheckpointPending).toBe(true);+ await vi.advanceTimersByTimeAsync(4_999);+ expect(kv.writes).toHaveLength(0);+ await vi.advanceTimersByTimeAsync(1);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(1);+ expect(embed).toHaveBeenCalledTimes(1);+ });++ it("recovers the first burst after its early checkpoint, leaving later changes batched", async () => {+ const kv = mockKV();+ const vector = new VectorIndex();+ const persistence = new IndexPersistence(kv as never, vector, { saveIntervalMs: 600_000 });+ for (let i = 0; i < 100; i++) {+ add(vector, `obs_${i}`);+ persistence.scheduleSave();+ }+ expect(persistence.status().firstCheckpointPending).toBe(true);+ expect(Date.parse(persistence.status().nextSaveAt!) - Date.now()).toBe(5_000);+ await vi.advanceTimersByTimeAsync(4_999);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).state).toBe("none");+ expect(kv.writes).toHaveLength(0);++ await vi.advanceTimersByTimeAsync(1);+ const loaded = await new IndexPersistence(kv as never, new VectorIndex()).load();+ expect(loaded.vector?.size).toBe(100);+ expect(loaded.vector?.get("obs_99")?.embedding).toEqual(vector.get("obs_99")?.embedding);+ expect(persistence.status()).toMatchObject({ firstCheckpointPending: false, pendingChanges: 0, nextSaveAt: null });+ expect(kv.writes).toHaveLength(101);++ add(vector, "obs_later");+ persistence.scheduleSave();+ await vi.advanceTimersByTimeAsync(599_999);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(100);+ expect(persistence.status().pendingChanges).toBe(1);+ await vi.advanceTimersByTimeAsync(1);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(101);+ expect(kv.writes).toHaveLength(103);+ });++ it("respects a save interval shorter than five seconds", async () => {+ const kv = mockKV();+ const vector = new VectorIndex();+ const persistence = new IndexPersistence(kv as never, vector, { saveIntervalMs: 1_000 });+ add(vector, "obs_1");+ persistence.scheduleSave();+ await vi.advanceTimersByTimeAsync(999);+ expect(kv.writes).toHaveLength(0);+ await vi.advanceTimersByTimeAsync(1);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(1);+ });++ it("retains the configured interval when a checkpoint was loaded", async () => {+ const kv = mockKV();+ const previous = new VectorIndex();+ add(previous, "obs_old");+ await new IndexPersistence(kv as never, previous).save();+ const vector = new VectorIndex();+ const persistence = new IndexPersistence(kv as never, vector, { saveIntervalMs: 60_000 });+ vector.restoreFrom((await persistence.load()).vector!);+ kv.writes.length = 0;+ add(vector, "obs_new");+ persistence.scheduleSave();+ expect(persistence.status().firstCheckpointPending).toBe(false);+ await vi.advanceTimersByTimeAsync(59_999);+ expect(kv.writes).toHaveLength(0);+ await vi.advanceTimersByTimeAsync(1);+ expect(kv.writes.map((write) => write.key)).toEqual(["obs_new", META_KEY]);+ });++ it("does not treat an empty flush as the first vector checkpoint", async () => {+ const kv = mockKV();+ const vector = new VectorIndex();+ const persistence = new IndexPersistence(kv as never, vector, { saveIntervalMs: 60_000 });+ await persistence.save();+ expect(kv.writes).toHaveLength(0);+ expect(persistence.status().vector?.lastSavedAt).toBeNull();+ add(vector, "obs_1");+ persistence.scheduleSave();+ await vi.advanceTimersByTimeAsync(5_000);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(1);+ });++ it.each(["entry", "metadata"])("retries a failed first %s write at the normal interval", async (failure) => {+ const kv = mockKV();+ let fail = true;+ const failingKV = {+ ...kv,+ set: async <T>(scope: string, key: string, data: T): Promise<T> => {+ if (fail && (failure === "metadata" ? key === META_KEY : scope === vectorBucketScope(0))) {+ throw new Error("state write unavailable");+ }+ return kv.set(scope, key, data);+ },+ };+ const vector = new VectorIndex();+ add(vector, "obs_1");+ const persistence = new IndexPersistence(failingKV as never, vector, { saveIntervalMs: 60_000 });+ persistence.scheduleSave();+ await vi.advanceTimersByTimeAsync(5_000);+ expect(persistence.status()).toMatchObject({ firstCheckpointPending: true, pendingChanges: 1 });+ expect(persistence.status().vector?.lastSavedAt).toBeNull();+ expect(await kv.get(INDEX_SCOPE, META_KEY)).toBeNull();+ const writesAfterFailure = kv.writes.length;+ fail = false;+ await vi.advanceTimersByTimeAsync(59_999);+ expect(kv.writes).toHaveLength(writesAfterFailure);+ await vi.advanceTimersByTimeAsync(1);+ expect(persistence.status()).toMatchObject({ firstCheckpointPending: false, pendingChanges: 0 });+ expect(persistence.status().vector?.lastError).toBeNull();+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(1);+ });++ it("keeps a later metadata failure pending even with an existing checkpoint", async () => {+ const kv = mockKV();+ let fail = false;+ const failingKV = {+ ...kv,+ set: async <T>(scope: string, key: string, data: T): Promise<T> => {+ if (fail && key === META_KEY) throw new Error("metadata unavailable");+ return kv.set(scope, key, data);+ },+ };+ const vector = new VectorIndex();+ const persistence = new IndexPersistence(failingKV as never, vector, { bucketSize: 1, saveIntervalMs: 60_000 });+ add(vector, "obs_1");+ await persistence.save();+ add(vector, "obs_2");+ fail = true;+ await persistence.save();+ expect(persistence.status().pendingChanges).toBe(1);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(1);+ fail = false;+ await vi.advanceTimersByTimeAsync(60_000);+ expect((await new IndexPersistence(kv as never, new VectorIndex()).load()).vector?.size).toBe(2);+ expect(persistence.status().pendingChanges).toBe(0);+ });+});diff --git a/test/index-persistence-throttle.test.ts b/test/index-persistence-throttle.test.tsindex bc03834d..9a28a668 100644--- a/test/index-persistence-throttle.test.ts+++ b/test/index-persistence-throttle.test.ts@@ -44,7 +44,7 @@ describe("IndexPersistence save throttling", () => {vi.useRealTimers();});- it("saves at most once per interval however often changes are scheduled", async () => {+ it("checkpoints the first burst early, then saves at most once per interval", async () => {const vector = new VectorIndex();const persistence = new IndexPersistence(kv as never, vector, { saveIntervalMs: 60_000, bucketSize: 16 });@@ -52,7 +52,7 @@ describe("IndexPersistence save throttling", () => {touch(vector, `obs_${i}`);persistence.scheduleSave();}- await vi.advanceTimersByTimeAsync(59_000);+ await vi.advanceTimersByTimeAsync(4_000);expect(metaSaves(kv)).toBe(0);await vi.advanceTimersByTimeAsync(1_000);diff --git a/test/live-source-budget.test.ts b/test/live-source-budget.test.tsnew file mode 100644index 00000000..7dfc85ac--- /dev/null+++ b/test/live-source-budget.test.ts@@ -0,0 +1,131 @@+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import type { CompressedObservation, RawObservation } from "../src/types.js";+import { registerObserveFunction } from "../src/functions/observe.js";+import { registerCompressFunction } from "../src/functions/compress.js";+import { buildSyntheticCompression } from "../src/functions/compress-synthetic.js";+import { withoutObservationSource } from "../src/functions/observation-source.js";+import { SESSION_SOURCE_MAX_BYTES } from "../src/functions/observation-source-budget.js";+import { getSearchIndex, setEmbeddingProvider, setVectorIndex } from "../src/functions/search.js";+import { payloadByteLength, SAFE_PAYLOAD_BYTES } from "../src/state/frame-guard.js";+import { KV } from "../src/state/schema.js";++vi.mock("../src/logger.js", () => ({ logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() } }));++const sessionId = "live-source-budget";+const scope = KV.observations(sessionId);+const output = "captured evidence ".repeat(1500);+const xml = "<type>file_read</type><title>Read configuration</title><facts><fact>Port configured</fact></facts><narrative>Read the configuration file.</narrative><concepts><concept>config</concept></concepts><files></files><importance>5</importance>";++function raw(id: string): RawObservation {+ return { id, sessionId, timestamp: "2026-09-30T00:00:00Z", hookType: "post_tool_use", toolName: "Read", toolOutput: output, raw: {} };+}++function rig(mode: "observe" | "compress") {+ const store = new Map<string, Map<string, any>>();+ const handlers = new Map<string, (data: any) => Promise<any>>();+ const kv = {+ get: async (key: string, id: string) => store.get(key)?.get(id) ?? null,+ list: async (key: string) => {+ const rows = [...(store.get(key)?.values() ?? [])];+ await Promise.resolve();+ return rows;+ },+ set: async (key: string, id: string, value: unknown) => {+ if (!store.has(key)) store.set(key, new Map());+ store.get(key)!.set(id, value);+ return value;+ },+ update: vi.fn(),+ };+ const sdk = {+ registerFunction: (id: string, handler: (data: any) => Promise<any>) => handlers.set(id, handler),+ trigger: vi.fn(async () => ({})),+ };+ registerObserveFunction(sdk as never, kv as never);+ registerCompressFunction(sdk as never, kv as never, { name: "test", compress: vi.fn(async () => xml), summarize: vi.fn() });+ let nextId = 0;+ async function write(id = `live-${nextId++}`): Promise<CompressedObservation> {+ if (mode === "compress") {+ const result = await handlers.get("mem::compress")!({ observationId: id, sessionId, raw: raw(id) });+ expect(result.success).toBe(true);+ } else {+ const result = await handlers.get("mem::observe")!({ sessionId, timestamp: raw(id).timestamp, hookType: "post_tool_use", data: { tool_name: "Read", tool_output: output } });+ id = result.observationId;+ }+ return kv.get(scope, id);+ }+ async function seedSource() {+ const template = buildSyntheticCompression(raw("seed"));+ for (let i = 0; i < 511; i++) await kv.set(scope, `seed-${i}`, { ...template, id: `seed-${i}` });+ }+ return { kv, write, seedSource };+}++function sourceBytes(rows: CompressedObservation[]): number {+ return rows.reduce((sum, row) => sum + (row.source ? payloadByteLength(row.source) + 10 : 0), 0);+}++beforeEach(() => {+ vi.stubEnv("AGENTMEMORY_AUTO_COMPRESS", "false");+ getSearchIndex().clear();+ setVectorIndex(null);+ setEmbeddingProvider(null);+});+afterEach(() => vi.unstubAllEnvs());++describe.each(["observe", "compress"] as const)("live %s source budget", (mode) => {+ it("truncates to the remaining session allowance, then omits source while preserving summaries", async () => {+ const expected = withoutObservationSource(await rig(mode).write());+ const r = rig(mode);+ await r.seedSource();+ const first = await r.write();+ expect(first.source?.truncated).toBe(true);+ expect(payloadByteLength(first.source)).toBeLessThan(payloadByteLength(buildSyntheticCompression(raw("unbounded")).source));+ const second = await r.write();+ expect(second.source).toBeUndefined();+ expect(withoutObservationSource(second)).toEqual({ ...expected, id: second.id });+ const rows = await r.kv.list(scope);+ expect(sourceBytes(rows)).toBeLessThanOrEqual(SESSION_SOURCE_MAX_BYTES);+ expect(payloadByteLength({ result: rows })).toBeLessThan(SAFE_PAYLOAD_BYTES);+ });++ it("reduces source for transport headroom before the source allowance is exhausted", async () => {+ const r = rig(mode);+ const seed = withoutObservationSource(buildSyntheticCompression(raw("summary-seed")));+ seed.narrative = "s".repeat(SAFE_PAYLOAD_BYTES - 64 * 1024 - 6000);+ await r.kv.set(scope, seed.id, seed);+ const saved = await r.write();+ expect(saved.source?.truncated).toBe(true);+ expect(payloadByteLength(saved.source)).toBeLessThan(6000);+ expect(saved.source?.toolName).toBe("Read");+ expect(payloadByteLength({ result: await r.kv.list(scope) })).toBeLessThan(SAFE_PAYLOAD_BYTES);+ });++ it("preserves a completed summary when existing summaries leave no frame allowance", async () => {+ const expected = withoutObservationSource(await rig(mode).write());+ const r = rig(mode);+ const seed = withoutObservationSource(buildSyntheticCompression(raw("oversized-summary")));+ seed.narrative = "s".repeat(SAFE_PAYLOAD_BYTES);+ await r.kv.set(scope, seed.id, seed);+ const saved = await r.write();+ expect(saved.source).toBeUndefined();+ expect(withoutObservationSource(saved)).toEqual({ ...expected, id: saved.id });+ });+});++it("serializes concurrent model-compression writes against the remaining source budget", async () => {+ const r = rig("compress");+ await r.seedSource();+ const saved = await Promise.all([r.write(), r.write()]);+ expect(saved.filter((row) => row.source)).toHaveLength(1);+ expect(sourceBytes(await r.kv.list(scope))).toBeLessThanOrEqual(SESSION_SOURCE_MAX_BYTES);+});++it("credits a replaced observation's retained source when model compression rewrites it", async () => {+ const r = rig("compress");+ await r.seedSource();+ const first = await r.write("replaced");+ const rewritten = await r.write("replaced");+ expect(rewritten.source).toEqual(first.source);+ expect(sourceBytes(await r.kv.list(scope))).toBeLessThanOrEqual(SESSION_SOURCE_MAX_BYTES);+});diff --git a/test/multimodal.test.ts b/test/multimodal.test.tsindex b2792db1..5c53cbcd 100644--- a/test/multimodal.test.ts+++ b/test/multimodal.test.ts@@ -1,4 +1,4 @@-import { describe, it, expect, vi, afterAll, beforeEach } from "vitest";+import { describe, it, expect, vi, afterAll, afterEach, beforeEach } from "vitest";import { existsSync, rmSync } from "node:fs";vi.mock("iii-sdk", async (importOriginal) => {@@ -71,8 +71,11 @@ describe("End-to-End Multimodal Flow", () => {beforeEach(() => {mockTrigger.mockClear();+ vi.stubEnv("AGENTMEMORY_AUTO_COMPRESS", "true");});+ afterEach(() => vi.unstubAllEnvs());+it("Step 1: Agent image should be successfully saved to hard drive", async () => {let observeCallback: any = null;const sdkMocker = { ...mockSdk, registerFunction: vi.fn((id, cb) => { if (id === "mem::observe") observeCallback = cb; }) };diff --git a/test/observation-source.test.ts b/test/observation-source.test.tsnew file mode 100644index 00000000..f7641f3e--- /dev/null+++ b/test/observation-source.test.ts@@ -0,0 +1,386 @@+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { join } from "node:path";+import { tmpdir } from "node:os";+import type { CompressedObservation, ExportData, HookType, MemoryProvider, RawObservation } from "../src/types.js";+import { KV } from "../src/state/schema.js";+import { registerObserveFunction } from "../src/functions/observe.js";+import { registerCompressFunction } from "../src/functions/compress.js";+import { buildSyntheticCompression } from "../src/functions/compress-synthetic.js";+import { registerReplayFunctions } from "../src/functions/replay.js";+import { registerExportImportFunction } from "../src/functions/export-import.js";+import { registerSmartSearchFunction } from "../src/functions/smart-search.js";+import { registerTimelineFunction } from "../src/functions/timeline.js";+import { registerTeamFunction } from "../src/functions/team.js";+import { registerApiTriggers } from "../src/triggers/api.js";+import { getSearchIndex, registerSearchFunction, setEmbeddingProvider, setVectorIndex } from "../src/functions/search.js";+import { createObservationSource, OBSERVATION_SOURCE_MAX_BYTES } from "../src/functions/observation-source.js";+import { buildSummaryPrompt } from "../src/prompts/summary.js";+import { budgetImportedObservationSources, SESSION_SOURCE_MAX_BYTES } from "../src/functions/observation-source-budget.js";+import { payloadByteLength, SAFE_PAYLOAD_BYTES } from "../src/state/frame-guard.js";++vi.mock("../src/logger.js", () => ({ logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() } }));++type Handler = (data: any) => Promise<any>;++function rig() {+ const store = new Map<string, Map<string, any>>();+ const handlers = new Map<string, Handler>();+ const pendingCompression: Promise<unknown>[] = [];+ const clone = <T>(value: T): T => JSON.parse(JSON.stringify(value));+ const kv = {+ get: async (scope: string, key: string) => clone(store.get(scope)?.get(key) ?? null),+ set: async (scope: string, key: string, value: unknown) => {+ if (!store.has(scope)) store.set(scope, new Map());+ store.get(scope)!.set(key, clone(value));+ return value;+ },+ update: async (scope: string, key: string, updates: Array<{ path: string; value: unknown }>) => {+ const value = store.get(scope)?.get(key);+ for (const update of updates) value[update.path] = clone(update.value);+ return value;+ },+ list: async (scope: string) => clone(Array.from(store.get(scope)?.values() ?? [])),+ delete: async (scope: string, key: string) => { store.get(scope)?.delete(key); },+ };+ const sdk = {+ registerFunction: (id: string, handler: Handler) => handlers.set(id, handler),+ registerTrigger: () => {},+ on: () => {},+ trigger: vi.fn(async (input: { function_id: string; payload: unknown }) => {+ const result = handlers.get(input.function_id)?.(input.payload);+ if (input.function_id === "mem::compress") {+ if (result) pendingCompression.push(result);+ return;+ }+ return result;+ }),+ };+ registerObserveFunction(sdk as never, kv as never);+ registerReplayFunctions(sdk as never, kv as never);+ registerExportImportFunction(sdk as never, kv as never);+ registerSearchFunction(sdk as never, kv as never);+ registerSmartSearchFunction(sdk as never, kv as never, async () => []);+ registerTimelineFunction(sdk as never, kv as never);+ registerTeamFunction(sdk as never, kv as never, { teamId: "source-team", userId: "source-user", mode: "shared" });+ registerApiTriggers(sdk as never, kv as never);+ const call = (id: string, payload: unknown) => handlers.get(id)!(payload);+ const observe = async (hookType: HookType, data: unknown) => {+ const result = await call("mem::observe", {+ sessionId: "source-session", project: "source-test", cwd: "/test/project",+ timestamp: "2026-09-30T00:00:00Z", hookType, data,+ });+ await Promise.all(pendingCompression.splice(0));+ return kv.get(KV.observations("source-session"), result.observationId) as Promise<CompressedObservation>;+ };+ return { kv, sdk, call, observe };+}++const raw = (overrides: Partial<RawObservation>): RawObservation => ({+ id: "obs-source", sessionId: "source-session", timestamp: "2026-09-30T00:00:00Z",+ hookType: "post_tool_use", raw: {}, ...overrides,+});++beforeEach(() => {+ vi.stubEnv("AGENTMEMORY_AUTO_COMPRESS", "false");+ vi.stubEnv("AGENTMEMORY_AGENT_SCOPE", "shared");+ getSearchIndex().clear();+ setVectorIndex(null);+ setEmbeddingProvider(null);+});+afterEach(() => vi.unstubAllEnvs());++describe("retained observation source", () => {+ it("preserves sanitized long prompts, tool results and responses through export/import and replay", async () => {+ const { observe, call } = rig();+ const tail = "deployment-port-19091";+ const long = `prefix ${"detail ".repeat(180)} ${tail}`;+ const secret = "sk-proj-abcdefghijklmnopqrstuvwxyz0123456789";+ const prompt = await observe("prompt_submit", { prompt: long });+ const tool = await observe("post_tool_use", {+ tool_name: "Read", tool_input: { file_path: "config.ts" },+ tool_output: `${long} <private>private-canary</private> ${secret}`,+ });+ const response = await observe("stop", { assistant_response: long });+ expect(prompt.narrative).toHaveLength(400);+ expect(prompt.source?.userPrompt).toBe(long);+ expect(tool.source?.toolOutput).toContain(tail);+ expect(response.source?.assistantResponse).toBe(long);+ expect(tool.source).not.toHaveProperty("raw");+ expect(tool.source).not.toHaveProperty("payload");+ expect(JSON.stringify(tool).split(tail)).toHaveLength(2);+ expect(JSON.stringify(tool)).not.toContain("private-canary");+ expect(JSON.stringify(tool)).not.toContain(secret);+ const exported = JSON.parse(JSON.stringify(await call("mem::export", {}))) as ExportData;+ expect(exported.observations["source-session"]).toHaveLength(3);+ const fresh = rig();+ expect(await fresh.call("mem::import", { exportData: exported })).toMatchObject({ success: true, observations: 3 });+ const replay = await fresh.call("mem::replay::load", { sessionId: "source-session" });+ expect(replay.timeline.events.map((event: any) => event.kind)).toEqual(["prompt", "tool_result", "response"]);+ expect(replay.timeline.events[0].body).toBe(long);+ expect(replay.timeline.events[1].toolOutput).toBe(tool.source?.toolOutput);+ expect(replay.timeline.events[2].body).toBe(long);+ expect(replay.sourceRetention.every((entry: any) => entry.retained && !entry.truncated)).toBe(true);+ });++ it("keeps retained source out of search, expansion, listings, timeline and summary prompts", async () => {+ const { observe, call, sdk } = rig();+ const tail = "SOURCE_ONLY_TAIL_951";+ const observation = await observe("post_tool_use", {+ tool_name: "Read", tool_input: { file_path: "config.ts" }, tool_output: `${"config ".repeat(200)}${tail}`,+ });+ const responses = [+ await call("mem::search", { query: "Read" }),+ await call("mem::smart-search", { expandIds: [{ obsId: observation.id, sessionId: observation.sessionId }] }),+ await call("mem::timeline", { anchor: observation.timestamp }),+ await call("api::observations", { headers: {}, query_params: { sessionId: observation.sessionId } }),+ await call("api::observations", { headers: {}, query_params: { sessionId: observation.sessionId, limit: "1" } }),+ ];+ for (const response of responses) {+ expect(JSON.stringify(response)).toContain(observation.id);+ expect(JSON.stringify(response)).not.toContain('"source":');+ expect(JSON.stringify(response)).not.toContain(tail);+ }+ expect(buildSummaryPrompt([observation])).not.toContain(tail);+ const updates = sdk.trigger.mock.calls.map(([input]) => input)+ .filter((input) => input.function_id === "stream::set");+ expect(updates.length).toBeGreaterThan(0);+ expect(JSON.stringify(updates)).not.toContain(tail);+ });++ it("preserves source when model compression succeeds without returning it to the model caller or viewer", async () => {+ const r = rig();+ vi.stubEnv("AGENTMEMORY_AUTO_COMPRESS", "true");+ const provider: MemoryProvider = {+ name: "test", summarize: vi.fn(),+ compress: vi.fn(async () => "<type>file_read</type><title>Read configuration</title><facts><fact>Port configured</fact></facts><narrative>Read the configuration file.</narrative><concepts><concept>config</concept></concepts><files></files><importance>5</importance>"),+ };+ registerCompressFunction(r.sdk as never, r.kv as never, provider);+ const tail = "MODEL_SOURCE_ONLY_TAIL";+ const saved = await r.observe("post_tool_use", {+ tool_name: "Read", tool_input: { file_path: "config.ts" }, tool_output: `${"detail ".repeat(1000)}${tail}`,+ });+ expect(saved.source?.toolOutput).toContain(tail);+ expect(saved.narrative).not.toContain(tail);+ expect(provider.compress).toHaveBeenCalledOnce();+ expect(JSON.stringify(vi.mocked(provider.compress).mock.calls)).not.toContain(tail);+ const updates = r.sdk.trigger.mock.calls.map(([input]) => input)+ .filter((input) => input.function_id === "stream::send" && (input.payload as any).type === "compressed_observation");+ expect(updates.length).toBe(1);+ expect(JSON.stringify(updates)).not.toContain(tail);+ });++ it("leaves sanitized stored source intact when model compression fails", async () => {+ const r = rig();+ vi.stubEnv("AGENTMEMORY_AUTO_COMPRESS", "true");+ const provider: MemoryProvider = {+ name: "test", summarize: vi.fn(), compress: vi.fn(async () => { throw new Error("provider unavailable"); }),+ };+ registerCompressFunction(r.sdk as never, r.kv as never, provider);+ const output = `${"detail ".repeat(200)}retained-after-failure`;+ await r.observe("post_tool_use", { tool_name: "Read", tool_output: output });+ const replay = await r.call("mem::replay::load", { sessionId: "source-session" });+ expect(replay.timeline.events[0].toolOutput).toBe(output);+ expect(replay.sourceRetention[0].retained).toBe(true);+ });++ it("rejects compressed rows passed as raw without calling a model or replacing retained evidence", async () => {+ const r = rig();+ const saved = await r.observe("prompt_submit", { prompt: "preserved evidence" });+ const provider: MemoryProvider = { name: "test", compress: vi.fn(), summarize: vi.fn() };+ registerCompressFunction(r.sdk as never, r.kv as never, provider);+ expect(await r.call("mem::compress", { observationId: saved.id, sessionId: saved.sessionId, raw: saved }))+ .toEqual({ success: false, error: "invalid_raw_observation" });+ expect(provider.compress).not.toHaveBeenCalled();+ expect(await r.kv.get(KV.observations(saved.sessionId), saved.id)).toEqual(saved);+ });++ it("bounds oversized Unicode evidence by serialized bytes and marks the truncation", () => {+ const input = raw({ toolName: "Read", toolOutput: '界😀"\\\n'.repeat(20000) });+ const source = createObservationSource(input);+ const encoded = JSON.stringify(source);+ expect(Buffer.byteLength(encoded, "utf8")).toBeLessThanOrEqual(OBSERVATION_SOURCE_MAX_BYTES);+ expect(source.originalBytes).toBeGreaterThan(OBSERVATION_SOURCE_MAX_BYTES);+ expect(source.truncated).toBe(true);+ expect(JSON.parse(encoded)).toEqual(source);+ expect(createObservationSource(input)).toEqual(source);+ expect(source.toolOutput).not.toMatch(/[\uD800-\uDBFF]$/u);+ });++ it("keeps structured evidence intact within the cap and marks oversized objects as text prefixes", () => {+ const small = { command: "read configuration", count: 0 };+ expect(createObservationSource(raw({ toolInput: small })).toolInput).toEqual(small);+ const large = createObservationSource(raw({ toolInput: { data: "a".repeat(100000) } }));+ expect(large.truncated).toBe(true);+ expect(typeof large.toolInput).toBe("string");+ expect(Buffer.byteLength(JSON.stringify(large), "utf8")).toBeLessThanOrEqual(OBSERVATION_SOURCE_MAX_BYTES);+ });++ it.each(["x".repeat(40000), '界😀"\\\n'.repeat(10000)])("keeps 500 near-cap sources plus summaries inside a state response frame", (toolOutput) => {+ const template = buildSyntheticCompression(raw({ toolName: "Read", toolInput: { file_path: "config.ts" }, toolOutput }));+ expect(payloadByteLength(template.source)).toBeGreaterThan(OBSERVATION_SOURCE_MAX_BYTES - 16);+ const rows = Array.from({ length: 500 }, (_, i) => ({ ...template, id: `obs-source-${i}` }));+ const planned = budgetImportedObservationSources(rows);+ expect(planned.success).toBe(true);+ if (!planned.success) return;+ expect(planned.observations[0].source).toEqual(template.source);+ const envelope = { message_type: "invocation_result", invocation_id: "00000000-0000-0000-0000-000000000000", result: planned.observations };+ expect(payloadByteLength(envelope)).toBeLessThan(SAFE_PAYLOAD_BYTES);+ });++ it("budgets 1000 imported records and accounts for retained evidence when merging another batch", async () => {+ const r = rig();+ await r.observe("prompt_submit", { prompt: "seed" });+ const exported = await r.call("mem::export", {}) as ExportData;+ const template = buildSyntheticCompression(raw({ toolName: "Read", toolOutput: "detail ".repeat(5000) }));+ exported.observations[template.sessionId] = Array.from({ length: 1000 }, (_, i) => ({ ...template, id: `bulk-${i}` }));+ const fresh = rig();+ expect(await fresh.call("mem::import", { exportData: exported })).toMatchObject({ success: true, observations: 1000, sourceTruncated: 1000, sourceOmitted: 0 });+ const first = await fresh.kv.list(KV.observations(template.sessionId));+ expect(first.reduce((sum, row) => sum + payloadByteLength(row.source) + 10, 0)).toBeLessThanOrEqual(SESSION_SOURCE_MAX_BYTES);+ expect(payloadByteLength({ result: first })).toBeLessThan(SAFE_PAYLOAD_BYTES);+ exported.observations[template.sessionId] = exported.observations[template.sessionId].map((row) => ({ ...row, id: `next-${row.id}` }));+ expect(await fresh.call("mem::import", { exportData: exported })).toMatchObject({ success: true, observations: 1000, sourceOmitted: 1000 });+ expect(await fresh.kv.get(KV.observations(template.sessionId), first[0].id)).toEqual(first[0]);+ expect(payloadByteLength({ result: await fresh.kv.list(KV.observations(template.sessionId)) })).toBeLessThan(SAFE_PAYLOAD_BYTES);+ });++ it("rejects an oversized replacement before deleting or writing any stored records", async () => {+ const r = rig();+ const saved = await r.observe("prompt_submit", { prompt: "keep original" });+ const exported = await r.call("mem::export", {}) as ExportData;+ exported.observations[saved.sessionId][0].narrative = "x".repeat(SAFE_PAYLOAD_BYTES);+ const writes = vi.spyOn(r.kv, "set");+ const deletes = vi.spyOn(r.kv, "delete");+ expect(await r.call("mem::import", { exportData: exported, strategy: "replace" })).toMatchObject({ success: false, oversized: true });+ expect(writes).not.toHaveBeenCalled();+ expect(deletes).not.toHaveBeenCalled();+ expect(await r.kv.get(KV.observations(saved.sessionId), saved.id)).toEqual(saved);+ });++ it("keeps unequal source sizes intact when their total fits the session budget", () => {+ const short = buildSyntheticCompression(raw({ hookType: "prompt_submit", userPrompt: "short" }));+ const long = buildSyntheticCompression(raw({ hookType: "prompt_submit", userPrompt: "x".repeat(15000) }));+ const rows = [...Array.from({ length: 2999 }, (_, i) => ({ ...short, id: `short-${i}` })), { ...long, id: "long" }];+ const planned = budgetImportedObservationSources(rows);+ expect(planned.success).toBe(true);+ if (!planned.success) return;+ expect(planned.observations[2999].source).toEqual(long.source);+ expect(planned.sourceTruncated).toBe(0);+ });++ it("serializes concurrent imports so both cannot spend the same session source budget", async () => {+ const r = rig();+ await r.observe("prompt_submit", { prompt: "seed" });+ const exported = await r.call("mem::export", {}) as ExportData;+ const template = buildSyntheticCompression(raw({ toolName: "Read", toolOutput: "x".repeat(20000) }));+ const fresh = rig();+ const results = await Promise.all(["first", "second"].map((prefix) => fresh.call("mem::import", {+ exportData: { ...exported, observations: { [template.sessionId]: Array.from({ length: 600 }, (_, i) => ({ ...template, id: `${prefix}-${i}` })) } },+ })));+ expect(results.every((result) => result.success)).toBe(true);+ const rows = await fresh.kv.list(KV.observations(template.sessionId));+ expect(rows).toHaveLength(1200);+ expect(rows.reduce((sum, row) => sum + (row.source ? payloadByteLength(row.source) + 10 : 0), 0)).toBeLessThanOrEqual(SESSION_SOURCE_MAX_BYTES);+ expect(payloadByteLength({ result: rows })).toBeLessThan(SAFE_PAYLOAD_BYTES);+ });++ it("refuses a replay result that expands beyond the transport frame", async () => {+ const r = rig();+ const template = buildSyntheticCompression(raw({ toolName: "tool".repeat(10000) }));+ for (let i = 0; i < 500; i++) {+ const row = { ...template, id: `large-label-${i}` };+ await r.kv.set(KV.observations(row.sessionId), row.id, row);+ }+ expect(await r.call("mem::replay::load", { sessionId: template.sessionId })).toMatchObject({ success: false, oversized: true });+ });++ it("bounds unrecognized hook metadata as well as the retained payload", () => {+ const hookType = "😀".repeat(30000) as HookType;+ const source = createObservationSource(raw({ hookType, raw: { event: "retained" } }));+ expect(Buffer.byteLength(JSON.stringify(source), "utf8")).toBeLessThanOrEqual(OBSERVATION_SOURCE_MAX_BYTES);+ expect(Buffer.byteLength(source.hookType, "utf8")).toBeLessThanOrEqual(128);+ expect(source.originalBytes).toBeGreaterThan(100000);+ expect(source.truncated).toBe(true);+ expect(source.payload).toEqual({ event: "retained" });+ });++ it("reapplies privacy and byte bounds to imported evidence and preserves truncation metadata", async () => {+ const r = rig();+ const saved = await r.observe("prompt_submit", { prompt: "summary" });+ const exported = await r.call("mem::export", {}) as ExportData;+ exported.observations[saved.sessionId][0].source = {+ hookType: "prompt_submit", originalBytes: 999999, truncated: true,+ userPrompt: `<private>import-private</private>${"界".repeat(100000)}`,+ };+ const fresh = rig();+ await fresh.call("mem::import", { exportData: exported });+ const imported = await fresh.kv.get(KV.observations(saved.sessionId), saved.id);+ expect(imported.source.originalBytes).toBe(999999);+ expect(imported.source.truncated).toBe(true);+ expect(JSON.stringify(imported.source)).not.toContain("import-private");+ expect(Buffer.byteLength(JSON.stringify(imported.source), "utf8")).toBeLessThanOrEqual(OBSERVATION_SOURCE_MAX_BYTES);+ const replay = await fresh.call("mem::replay::load", { sessionId: saved.sessionId });+ expect(replay.sourceRetention[0]).toMatchObject({ retained: true, truncated: true, originalBytes: 999999 });+ });++ it("shares observation summaries without retaining or returning their source in the team feed", async () => {+ const r = rig();+ const saved = await r.observe("post_tool_use", { tool_name: "Read", tool_output: `${"config ".repeat(200)}team-source-tail` });+ const shared = await r.call("mem::team-share", { itemId: saved.id, itemType: "observation", sessionId: saved.sessionId });+ expect(shared.sharedItem.content.source).toBeUndefined();+ expect((await r.kv.get(KV.teamShared("source-team"), shared.sharedItem.id)).content.source).toBeUndefined();+ await r.kv.set(KV.teamShared("source-team"), "old-share", { ...shared.sharedItem, id: "old-share", content: saved });+ const memory = { id: "mem-source", title: "memory", content: "untouched", source: "user-authored metadata" };+ await r.kv.set(KV.memories, memory.id, memory);+ const sharedMemory = await r.call("mem::team-share", { itemId: memory.id, itemType: "memory" });+ expect(sharedMemory.sharedItem.content).toEqual(memory);+ const feed = await r.call("mem::team-feed", {});+ expect(feed.items.find((item: any) => item.type === "memory").content).toEqual(memory);+ expect(feed.items.filter((item: any) => item.type === "observation").every((item: any) => item.content.source === undefined)).toBe(true);+ expect(JSON.stringify(feed)).not.toContain("team-source-tail");+ });++ it("retains sanitized unknown-hook payloads without inventing tool or prompt fields", async () => {+ const { observe, call } = rig();+ const saved = await observe("notification", { message: "visible <private>hidden</private>", status: "ready" });+ expect(saved.source?.payload).toEqual({ message: "visible [REDACTED]", status: "ready" });+ expect(saved.source?.hookType).toBe("notification");+ expect(saved.source?.toolInput).toBeUndefined();+ const replay = await call("mem::replay::load", { sessionId: saved.sessionId });+ expect(replay.timeline.events[0].body).toContain("visible [REDACTED]");+ });++ it("retains imported prompt and response evidence once and sanitizes it", async () => {+ const folder = mkdtempSync(join(tmpdir(), "source-retention-"));+ try {+ const text = `${"message ".repeat(150)}import-tail <private>hidden-import</private>`;+ const path = join(folder, "session.jsonl");+ writeFileSync(path, ["user", "assistant"].map((role) => JSON.stringify({+ type: role, sessionId: "import-source", timestamp: "2026-09-30T00:00:00Z",+ message: { role, content: text },+ })).join("\n"));+ const r = rig();+ expect(await r.call("mem::replay::import-jsonl", { path })).toMatchObject({ success: true, observations: 2 });+ const rows = await r.kv.list(KV.observations("import-source"));+ for (const row of rows) {+ expect(JSON.stringify(row.source)).not.toContain("hidden-import");+ expect(JSON.stringify(row.source).split("import-tail")).toHaveLength(2);+ }+ const replay = await r.call("mem::replay::load", { sessionId: "import-source" });+ expect(replay.timeline.events.map((event: any) => event.kind)).toEqual(["prompt", "response"]);+ expect(replay.timeline.events[1].body).toContain("import-tail");+ expect(replay.timeline.events[1].body).not.toContain("hidden-import");+ } finally { rmSync(folder, { recursive: true, force: true }); }+ });++ it("keeps legacy summaries replayable without claiming their original source was retained", async () => {+ const r = rig();+ const legacy = buildSyntheticCompression(raw({ toolName: "Read", toolInput: { file_path: "config.ts" }, toolOutput: "legacy-summary" }));+ delete legacy.source;+ await r.kv.set(KV.observations(legacy.sessionId), legacy.id, legacy);+ const replay = await r.call("mem::replay::load", { sessionId: legacy.sessionId });+ expect(replay.timeline.events[0].toolOutput).toBe("legacy-summary");+ expect(replay.sourceRetention[0]).toEqual({ observationId: legacy.id, retained: false });+ });+});diff --git a/test/replay-write-batching.test.ts b/test/replay-write-batching.test.tsnew file mode 100644index 00000000..867fea58--- /dev/null+++ b/test/replay-write-batching.test.ts@@ -0,0 +1,185 @@+import { describe, expect, it, vi } from "vitest";+import { mkdtempSync, rmSync, writeFileSync } from "node:fs";+import { join } from "node:path";+import type { CompressedObservation } from "../src/types.js";+import { registerReplayFunctions } from "../src/functions/replay.js";+import { KV } from "../src/state/schema.js";+import * as sessionLocks from "../src/state/keyed-mutex.js";++vi.mock("../src/logger.js", () => ({+ logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() },+}));+vi.mock("../src/functions/search.js", () => ({ indexRecords: vi.fn() }));++describe("JSONL observation write batching", () => {+ it("limits writes to 20 and waits for the entire batch before starting the next", async () => {+ const folder = mkdtempSync(join(__dirname, "replay-write-batching-"));+ const path = join(folder, "session.jsonl");+ const sessionId = "write-batching-session";+ const scope = KV.observations(sessionId);+ const prompts = Array.from({ length: 45 }, (_, i) => `record ${i}`);+ writeFileSync(path, prompts.map((content, i) => JSON.stringify({+ type: "user", uuid: `entry-${i}`, sessionId, timestamp: "2026-09-30T00:00:00Z",+ message: { role: "user", content },+ })).join("\n"));++ const store = new Map<string, Map<string, unknown>>();+ const writes: Array<{ scope: string; key: string; value: CompressedObservation }> = [];+ const release: Array<() => void> = [];+ let active = 0;+ let peak = 0;+ let draining = false;+ const kv = {+ get: async (bucket: string, key: string) => store.get(bucket)?.get(key) ?? null,+ list: async (bucket: string) => Array.from(store.get(bucket)?.values() ?? []),+ set: async (bucket: string, key: string, value: unknown) => {+ if (bucket === scope) {+ writes.push({ scope: bucket, key, value: value as CompressedObservation });+ active++;+ peak = Math.max(peak, active);+ if (!draining) await new Promise<void>((resolve) => release.push(resolve));+ active--;+ }+ if (!store.has(bucket)) store.set(bucket, new Map());+ store.get(bucket)!.set(key, value);+ return value;+ },+ };+ let importJsonl: (data: { path: string }) => Promise<unknown> = async () => undefined;+ const sdk = {+ registerFunction: (id: string, handler: typeof importJsonl) => {+ if (id === "mem::replay::import-jsonl") importJsonl = handler;+ },+ };+ registerReplayFunctions(sdk as never, kv as never);+ const importing = importJsonl({ path });+ const flush = () => new Promise<void>((resolve) => setImmediate(resolve));+ try {+ await vi.waitFor(() => expect(writes).toHaveLength(20));+ for (const finish of release.slice(0, 19)) finish();+ await flush();+ expect(active).toBe(1);+ expect(writes).toHaveLength(20);++ release[19]();+ await vi.waitFor(() => expect(writes).toHaveLength(40));+ for (const finish of release.slice(20, 39)) finish();+ await flush();+ expect(active).toBe(1);+ expect(writes).toHaveLength(40);++ release[39]();+ await vi.waitFor(() => expect(writes).toHaveLength(45));+ for (const finish of release.slice(40)) finish();+ expect(await importing).toMatchObject({ success: true, observations: 45 });+ expect(peak).toBe(20);+ expect(active).toBe(0);+ expect(writes.every((write) => write.scope === scope && write.key === write.value.id && write.value.sessionId === sessionId)).toBe(true);+ expect(new Set(writes.map((write) => write.key)).size).toBe(45);+ expect(writes.map((write) => write.value.source?.userPrompt)).toEqual(prompts);+ expect(store.get(scope)?.size).toBe(45);+ } finally {+ draining = true;+ for (const finish of release) finish();+ await importing.catch(() => {});+ rmSync(folder, { recursive: true, force: true });+ }+ });++ it("drains a failed batch before a competing import acquires the session lock", async () => {+ const folder = mkdtempSync(join(__dirname, "replay-write-batching-"));+ const sessionId = "failed-batch-session";+ const scope = KV.observations(sessionId);+ const fixture = (name: string, count: number) => {+ const path = join(folder, `${name}.jsonl`);+ writeFileSync(path, Array.from({ length: count }, (_, i) => JSON.stringify({+ type: "user", uuid: `${name}-${i}`, sessionId, timestamp: "2026-09-30T00:00:00Z",+ message: { role: "user", content: `${name}-${i}` },+ })).join("\n"));+ return path;+ };+ const firstPath = fixture("first", 45);+ const secondPath = fixture("second", 1);+ const store = new Map<string, Map<string, unknown>>();+ const gates: Array<{ resolve: () => void; reject: (error: Error) => void }> = [];+ const observationsAtRead: number[] = [];+ const activeAtSecondWrite: number[] = [];+ let active = 0;+ let firstWrites = 0;+ let draining = false;+ const kv = {+ get: async (bucket: string, key: string) => store.get(bucket)?.get(key) ?? null,+ list: async (bucket: string) => {+ const rows = Array.from(store.get(bucket)?.values() ?? []);+ if (bucket === scope) observationsAtRead.push(rows.length);+ return rows;+ },+ set: async (bucket: string, key: string, value: unknown) => {+ if (bucket === scope) {+ const observation = value as CompressedObservation;+ if (observation.source?.userPrompt?.startsWith("first-")) {+ firstWrites++;+ active++;+ try {+ if (!draining) await new Promise<void>((resolve, reject) => gates.push({ resolve, reject }));+ } finally {+ active--;+ }+ } else {+ activeAtSecondWrite.push(active);+ }+ }+ if (!store.has(bucket)) store.set(bucket, new Map());+ store.get(bucket)!.set(key, value);+ return value;+ },+ };+ let importJsonl: (data: { path: string }) => Promise<unknown> = async () => undefined;+ const sdk = {+ registerFunction: (id: string, handler: typeof importJsonl) => {+ if (id === "mem::replay::import-jsonl") importJsonl = handler;+ },+ };+ registerReplayFunctions(sdk as never, kv as never);+ const lockCalls = vi.spyOn(sessionLocks, "withKeyedLock");+ let firstFinished = false;+ const first = importJsonl({ path: firstPath }).then(+ (value) => { firstFinished = true; return { value }; },+ (error) => { firstFinished = true; return { error }; },+ );+ let second = Promise.resolve<unknown>(undefined);+ const flush = () => new Promise<void>((resolve) => setImmediate(resolve));+ try {+ await vi.waitFor(() => expect(gates).toHaveLength(20));+ second = importJsonl({ path: secondPath });+ await vi.waitFor(() => expect(lockCalls.mock.calls.filter(([key]) => key === `obs:${sessionId}`)).toHaveLength(2));+ const failure = new Error("observation write failed");+ gates[0].reject(failure);+ await flush();+ expect(firstFinished).toBe(false);+ expect(active).toBe(19);+ expect(observationsAtRead).toEqual([0]);++ for (const gate of gates.slice(1, 19)) gate.resolve();+ await flush();+ expect(firstFinished).toBe(false);+ expect(active).toBe(1);+ expect(observationsAtRead).toEqual([0]);+ expect(firstWrites).toBe(20);++ gates[19].resolve();+ expect(await first).toEqual({ error: failure });+ expect(await second).toMatchObject({ success: true, observations: 1 });+ expect(firstWrites).toBe(20);+ expect(observationsAtRead).toEqual([0, 19]);+ expect(activeAtSecondWrite).toEqual([0]);+ expect(store.get(scope)?.size).toBe(20);+ } finally {+ draining = true;+ for (const gate of gates) gate.resolve();+ await Promise.allSettled([first, second]);+ lockCalls.mockRestore();+ rmSync(folder, { recursive: true, force: true });+ }+ });+});diff --git a/test/status.test.ts b/test/status.test.tsindex a5cad34e..7e621de5 100644--- a/test/status.test.ts+++ b/test/status.test.ts@@ -129,6 +129,28 @@ describe("evaluateStatus", () => {expect(problem?.message).toContain("40 of 100 vectors");});+ it("reports absent and partial vector recovery as paused until explicitly opted in", () => {+ const report = evaluateStatus(inputs({+ index: { ...inputs().index, pendingVectorBackfill: 60, vectorBackfillState: "waiting-for-opt-in" },+ indexPersistence: { saveIntervalMs: 600_000, saving: false, buckets: 3, pendingChanges: 0, vector: null, vectorCountShortfall: { expected: 100, loaded: 40 } },+ }));+ for (const problem of report.problems) {+ expect(problem.fix).toContain("Backfill is paused.");+ expect(problem.fix).toContain("AGENTMEMORY_VECTOR_BACKFILL=all");+ expect(problem.fix).not.toContain("running in the background");+ expect(problem.message).not.toContain("re-embeds the rest");+ }+ expect(report.index.vectorBackfillState).toBe("waiting-for-opt-in");+ expect(renderStatusHtml(report, "n")).toContain("paused, waiting for opt-in");+ });++ it("only describes vector recovery as running when the worker reports an active backfill", () => {+ const active = evaluateStatus(inputs({ index: { ...inputs().index, pendingVectorBackfill: 42, vectorBackfillState: "running" } }));+ expect(active.problems[0].fix).toContain("Backfill is running in the background");+ const idle = evaluateStatus(inputs({ index: { ...inputs().index, pendingVectorBackfill: 42, vectorBackfillState: "idle" } }));+ expect(idle.problems[0].fix).toContain("Backfill is not running");+ });+it("checks snapshot presence when extraction is on, and snapshot age whenever a snapshot exists", () => {const noSnapshot = { totalNodes: 0, totalEdges: 0, fromSnapshot: false };expect(codes(evaluateStatus(inputs({ graph: noSnapshot })))).toEqual(["graph-no-snapshot"]);