Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

fix(openclaw): declare transcriptSemantics and implement commitTurn (#3442)

Radhakrishnan Pachyappana28dd4eab70a
diff --git a/plugins/openclaw/src/advancement-key-store.ts b/plugins/openclaw/src/advancement-key-store.ts
new file mode 100644
index 000000000..e9801e6d6
--- /dev/null
+++ b/plugins/openclaw/src/advancement-key-store.ts
@@ -0,0 +1,240 @@
+/**
+ * Durable, atomic, restart-safe, cross-process-safe record of committed
+ * turn advancements.
+ *
+ * OpenClaw's `atomic-idempotent-v1` transcriptSemantics contract requires
+ * that a retried `commitTurn` call report "duplicate" for a key that was
+ * already committed -- including after this process (or the host) restarts
+ * -- and that "the accepted turn and key advance together" (the committed
+ * messages are persisted atomically with the key, not discarded).
+ *
+ * The default commit-log path is shared by every gateway process using the
+ * same Headroom workspace, not just every store instance in one process --
+ * an in-process mutex (a JS `Map`/promise queue) does nothing to stop two
+ * separate OS processes from each reading the same file, adding a different
+ * key, and racing their renames, with the loser's accepted turn silently
+ * discarded. The read/check/write transaction is therefore protected by a
+ * real inter-process lock: an exclusive lock FILE (`<path>.lock`), acquired
+ * via atomic exclusive creation (`open(..., "wx")`, which is atomic at the
+ * OS level on POSIX and Windows alike) so only one process -- in this one or
+ * any other -- can be inside the critical section at a time.
+ *
+ * The lock is never force-broken by age. An age-based "the holder must have
+ * crashed" heuristic cannot tell a dead holder apart from one that is simply
+ * slow (a large write, a GC pause, disk contention): a second process that
+ * reclaims a lock the first still holds can commit and release before the
+ * first resumes and overwrites that commit with its own now-stale snapshot
+ * -- both processes report "committed", but only the second process's write
+ * survives, and the first process's genuinely-accepted turn silently
+ * vanishes. Node has no built-in binding for a real OS-mediated lock
+ * (POSIX `flock`/Windows `LockFileEx`, where the kernel itself releases the
+ * lock when the holder's process exits, crash included, so no staleness
+ * guess is needed at all), so this fails closed instead: a lock that is
+ * never released is never reclaimed, and a caller that can't acquire it
+ * within the timeout gets a clear error rather than a silent lost write. A
+ * lock file orphaned by a genuinely crashed process requires a human (or an
+ * operator script) to remove `<path>.lock` before commits can resume --
+ * that operational cost is the trade for never silently dropping an
+ * acknowledged commit. Lock ownership is also verified before release (see
+ * `withFileLock`): a process only ever deletes the lock file it created.
+ *
+ * Every `tryCommit` re-reads the file fresh from disk inside the lock (no
+ * long-lived in-memory cache), and the in-memory return value is only ever
+ * "committed" once the atomic (temp file + rename) write recording the key
+ * has actually succeeded -- a failed write is never remembered as seen, so
+ * retrying the same key attempts the commit again instead of returning a
+ * phantom "duplicate" for a commit that never reached disk.
+ *
+ * Keys are never evicted. OpenClaw may retry at any point after a restart or
+ * a long stall, so pruning by count or age would risk re-accepting an
+ * already-committed turn -- exactly the failure this store exists to avoid.
+ */
+
+import { mkdir, open, readFile, rename, stat, unlink, writeFile } from "node:fs/promises";
+import { dirname, join } from "node:path";
+import * as os from "node:os";
+
+/**
+ * True if `p` exists and is something other than a directory (e.g. a file
+ * sitting where a directory needs to be). Some platforms (observed on
+ * Windows) report `ENOENT` -- not a distinct "not a directory" code -- for
+ * `readFile` when a path component like this blocks the rest of the path,
+ * which is indistinguishable from "the commit log just doesn't exist yet"
+ * by error code alone. Checking the parent directly is what tells the two
+ * apart: silently treating a broken storage path as an empty store would
+ * make every future commit re-accept an already-committed key.
+ */
+async function isBlockingNonDirectory(p: string): Promise<boolean> {
+ try {
+ const info = await stat(p);
+ return !info.isDirectory();
+ } catch {
+ return false; // Doesn't exist -- not blocking anything.
+ }
+}
+
+/** Default commit-log location, honoring the same workspace override the
+ * rest of the Headroom workspace convention uses (`HEADROOM_WORKSPACE_DIR`). */
+export function defaultCommitLogPath(): string {
+ const workspaceDir = process.env.HEADROOM_WORKSPACE_DIR?.trim() || join(os.homedir(), ".headroom");
+ return join(workspaceDir, "openclaw", "commit-log.json");
+}
+
+/** One durably-committed turn: the accepted messages plus when they landed. */
+export interface CommittedTurn {
+ messages: unknown;
+ committedAt: string;
+}
+
+type CommitLog = Record<string, CommittedTurn>;
+
+/** Give up (rather than poll forever) if the lock still can't be acquired after this long. */
+const LOCK_ACQUIRE_TIMEOUT_MS = 60_000;
+const LOCK_POLL_BASE_MS = 20;
+
+function sleep(ms: number): Promise<void> {
+ return new Promise((resolve) => setTimeout(resolve, ms));
+}
+
+/**
+ * Run `fn` while holding an exclusive lock on `path` (a sibling `<path>.lock`
+ * file). Serializes every caller -- this process or another -- against the
+ * same file, not just calls on one object or one process.
+ *
+ * The lock is held until `fn` returns, however long that takes, and is
+ * never reclaimed by another caller on a staleness guess -- see the module
+ * docstring for why an age-based heuristic can silently drop a genuinely
+ * accepted commit. A caller that can't acquire the lock within
+ * `LOCK_ACQUIRE_TIMEOUT_MS` gets a clear timeout error instead of the
+ * transaction being allowed to proceed unsafely.
+ */
+async function withFileLock<T>(path: string, fn: () => Promise<T>): Promise<T> {
+ const lockPath = `${path}.lock`;
+ await mkdir(dirname(path), { recursive: true });
+ // Unique per acquisition, so release only ever removes the lock file this
+ // exact call created -- never one a different (later) holder created.
+ const token = `${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}`;
+ const deadline = Date.now() + LOCK_ACQUIRE_TIMEOUT_MS;
+ let attempt = 0;
+ for (;;) {
+ try {
+ const handle = await open(lockPath, "wx");
+ try {
+ await handle.writeFile(`${token}\n${process.pid}\n${new Date().toISOString()}\n`);
+ } finally {
+ await handle.close();
+ }
+ break;
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException)?.code !== "EEXIST") {
+ throw error;
+ }
+ if (Date.now() > deadline) {
+ throw new Error(
+ `Timed out waiting for the advancement-key-store lock on ${path}. If the ` +
+ `process that created ${lockPath} is no longer running, remove it manually.`,
+ );
+ }
+ attempt += 1;
+ await sleep(LOCK_POLL_BASE_MS + Math.random() * LOCK_POLL_BASE_MS * Math.min(attempt, 10));
+ }
+ }
+ try {
+ return await fn();
+ } finally {
+ // Verify the lock file is still the one this call created before
+ // removing it -- if it doesn't start with our token, some other process
+ // (or a human) replaced or is using it, and it must not be touched.
+ try {
+ const current = await readFile(lockPath, "utf8");
+ if (current.startsWith(`${token}\n`)) {
+ await unlink(lockPath);
+ }
+ } catch (error) {
+ if ((error as NodeJS.ErrnoException)?.code !== "ENOENT") {
+ throw error;
+ }
+ }
+ }
+}
+
+export class DurableAdvancementKeyStore {
+ constructor(private readonly path: string) {}
+
+ private async readAll(): Promise<CommitLog> {
+ let raw: string;
+ try {
+ raw = await readFile(this.path, "utf8");
+ } catch (error) {
+ const code = (error as NodeJS.ErrnoException)?.code;
+ if (code !== "ENOENT" && code !== "ENOTDIR") {
+ throw error;
+ }
+ if (await isBlockingNonDirectory(dirname(this.path))) {
+ throw error;
+ }
+ return {};
+ }
+ const parsed = JSON.parse(raw) as unknown;
+ if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
+ return parsed as CommitLog;
+ }
+ if (Array.isArray(parsed)) {
+ // Migrate the earlier key-only array format (no message payload) so
+ // upgrading from it never re-accepts an already-committed key.
+ const migrated: CommitLog = {};
+ for (const entry of parsed) {
+ if (typeof entry === "string") {
+ migrated[entry] = { messages: null, committedAt: "" };
+ }
+ }
+ return migrated;
+ }
+ return {};
+ }
+
+ private async writeAll(entries: CommitLog): Promise<void> {
+ await mkdir(dirname(this.path), { recursive: true });
+ const tmpPath = `${this.path}.${process.pid}.${Math.random().toString(36).slice(2)}.tmp`;
+ await writeFile(tmpPath, JSON.stringify(entries), "utf8");
+ await rename(tmpPath, this.path);
+ }
+
+ /**
+ * Atomically check-and-record `key` together with the `messages` it
+ * advances. Holds the cross-process file lock (see `withFileLock`) for
+ * the full read/check/write transaction, so two processes -- or two
+ * instances in this one -- can't each read the same starting state and
+ * race their writes.
+ *
+ * Throws if the write fails -- the caller sees a rejected commit, not a
+ * false "committed"/"duplicate", and a retry re-reads the (still
+ * unchanged) on-disk state and attempts the write again.
+ */
+ async tryCommit(key: string, messages: unknown): Promise<"committed" | "duplicate"> {
+ return withFileLock(this.path, async () => {
+ const entries = await this.readAll();
+ if (key in entries) {
+ return "duplicate";
+ }
+ const next: CommitLog = {
+ ...entries,
+ [key]: { messages, committedAt: new Date().toISOString() },
+ };
+ await this.writeAll(next);
+ return "committed";
+ });
+ }
+
+ /** Test-only: check membership without recording. */
+ async has(key: string): Promise<boolean> {
+ const entries = await this.readAll();
+ return key in entries;
+ }
+
+ /** Test-only: read back a committed entry (messages + commit time) without recording. */
+ async get(key: string): Promise<CommittedTurn | undefined> {
+ const entries = await this.readAll();
+ return entries[key];
+ }
+}
diff --git a/plugins/openclaw/src/engine.ts b/plugins/openclaw/src/engine.ts
index 71a3836fb..7b2a8f5ed 100644
--- a/plugins/openclaw/src/engine.ts
+++ b/plugins/openclaw/src/engine.ts
@@ -10,6 +10,7 @@
import { compress } from "headroom-ai";
import { ProxyManager, defaultLogger, type ProxyManagerConfig, type ProxyManagerLogger } from "./proxy-manager.js";
import { agentToOpenAI, normalizeAgentMessages, openAIToAgent } from "./convert.js";
+import { DurableAdvancementKeyStore, defaultCommitLogPath } from "./advancement-key-store.js";
import {
delegateCompactionToRuntime,
type OpenClawCompactParams,
@@ -32,6 +33,9 @@ export interface HeadroomEngineConfig extends ProxyManagerConfig {
requestTimeoutMs?: number;
circuitBreakerThreshold?: number;
circuitBreakerCooldownMs?: number;
+ /** Where to durably record committed turn-advancement keys (see
+ * `DurableAdvancementKeyStore`). Defaults to `defaultCommitLogPath()`. */
+ commitLogPath?: string;
}
export class HeadroomContextEngine {
@@ -40,8 +44,17 @@ export class HeadroomContextEngine {
name: "Headroom Context Compression",
version: "0.1.0",
ownsCompaction: false,
+ transcriptSemantics: {
+ currentTurnFence: "before-current-turn-entry-v1",
+ turnAdvancementIdempotency: "atomic-idempotent-v1",
+ },
};
+ // Durable, restart-safe record of committed advancement keys, for
+ // commitTurn's idempotent-retry check. See advancement-key-store.ts for
+ // why this must survive a process restart and must not evict entries.
+ private advancementKeyStore: DurableAdvancementKeyStore;
+
private proxyManager: ProxyManager;
private proxyUrl: string | null = null;
private config: HeadroomEngineConfig;
@@ -61,6 +74,9 @@ export class HeadroomContextEngine {
this.config = config;
this.logger = logger ?? defaultLogger;
this.proxyManager = new ProxyManager(config, this.logger);
+ this.advancementKeyStore = new DurableAdvancementKeyStore(
+ config.commitLogPath ?? defaultCommitLogPath(),
+ );
}
// === ContextEngine Lifecycle ===
@@ -189,6 +205,23 @@ export class HeadroomContextEngine {
return result;
}
+ /**
+ * Durable turn-advancement commit — required by OpenClaw's transcriptSemantics
+ * contract. Called only for the accepted, successful turn; failed or aborted
+ * turns never reach here. Must be an atomic, idempotent write of the
+ * accepted `messages` keyed by `advancementKey` so a host retry with the
+ * same key reports "duplicate" instead of re-applying the advancement —
+ * including a retry that arrives after this process restarted, which is
+ * why the record lives on disk (see `DurableAdvancementKeyStore`) rather
+ * than in memory, and includes the messages rather than just the key.
+ */
+ async commitTurn(params: { advancementKey: string; messages: any[] }): Promise<{
+ status: "committed" | "duplicate";
+ }> {
+ const status = await this.advancementKeyStore.tryCommit(params.advancementKey, params.messages);
+ return { status };
+ }
+
async afterTurn?(params: {
sessionId: string;
messages: any[];
diff --git a/plugins/openclaw/test/advancement-key-store.test.ts b/plugins/openclaw/test/advancement-key-store.test.ts
new file mode 100644
index 000000000..765e66bb1
--- /dev/null
+++ b/plugins/openclaw/test/advancement-key-store.test.ts
@@ -0,0 +1,350 @@
+import { execFile } from "node:child_process";
+import * as fs from "node:fs/promises";
+import * as os from "node:os";
+import * as path from "node:path";
+import { fileURLToPath } from "node:url";
+import { promisify } from "node:util";
+
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+
+import { DurableAdvancementKeyStore, defaultCommitLogPath } from "../src/advancement-key-store.js";
+
+const execFileAsync = promisify(execFile);
+
+// Node's native TypeScript support (needed to run the worker fixture, which
+// imports the real DurableAdvancementKeyStore, as a plain `node` child
+// process with no build step) requires `--experimental-transform-types`,
+// added in Node 22.6. Skip the cross-process tests rather than fail on an
+// older Node -- the failure mode being tested (a lost write across two OS
+// processes) is orthogonal to which Node version happens to run the test.
+const NODE_SUPPORTS_TRANSFORM_TYPES = (() => {
+ const [major, minor] = process.versions.node.split(".").map(Number);
+ return major > 22 || (major === 22 && minor >= 6);
+})();
+
+const WORKER_PATH = fileURLToPath(new URL("./fixtures/commit-turn-worker.ts", import.meta.url));
+const HOLD_LOCK_WORKER_PATH = fileURLToPath(
+ new URL("./fixtures/hold-lock-worker.ts", import.meta.url),
+);
+
+/** Run the commit-turn worker as a real child process; returns its parsed JSON result. */
+async function runCommitWorker(
+ storePath: string,
+ key: string,
+ goPath: string,
+ resultPath: string,
+): Promise<{ status?: string; error?: string }> {
+ await execFileAsync(process.execPath, [
+ "--experimental-transform-types",
+ WORKER_PATH,
+ storePath,
+ key,
+ goPath,
+ resultPath,
+ ]);
+ return JSON.parse(await fs.readFile(resultPath, "utf8"));
+}
+
+async function waitForFile(p: string): Promise<void> {
+ for (;;) {
+ try {
+ await fs.access(p);
+ return;
+ } catch {
+ await new Promise((r) => setTimeout(r, 5));
+ }
+ }
+}
+
+describe("DurableAdvancementKeyStore", () => {
+ let dir: string;
+ let storePath: string;
+
+ beforeEach(async () => {
+ dir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-advancement-key-store-"));
+ storePath = path.join(dir, "nested", "commit-log.json");
+ });
+
+ afterEach(async () => {
+ await fs.rm(dir, { recursive: true, force: true });
+ });
+
+ it("commits a new key and creates the parent directory", async () => {
+ const store = new DurableAdvancementKeyStore(storePath);
+
+ await expect(store.tryCommit("turn-1", [{ role: "user", content: "hi" }])).resolves.toBe(
+ "committed",
+ );
+ await expect(fs.stat(storePath)).resolves.toBeTruthy();
+ });
+
+ it("reports duplicate for an already-committed key", async () => {
+ const store = new DurableAdvancementKeyStore(storePath);
+
+ await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
+ await expect(store.tryCommit("turn-1", [])).resolves.toBe("duplicate");
+ });
+
+ it("persists across a simulated restart (a fresh store instance, same path)", async () => {
+ const before = new DurableAdvancementKeyStore(storePath);
+ await expect(before.tryCommit("turn-1", [])).resolves.toBe("committed");
+
+ const after = new DurableAdvancementKeyStore(storePath);
+ await expect(after.tryCommit("turn-1", [])).resolves.toBe("duplicate");
+ // A key that was never committed is still accepted normally.
+ await expect(after.tryCommit("turn-2", [])).resolves.toBe("committed");
+ });
+
+ it("treats a missing commit-log file as an empty store rather than an error", async () => {
+ const store = new DurableAdvancementKeyStore(storePath);
+
+ await expect(store.has("turn-1")).resolves.toBe(false);
+ await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
+ });
+
+ it("fails loudly on a corrupt commit-log file instead of silently forgetting its contents", async () => {
+ await fs.mkdir(path.dirname(storePath), { recursive: true });
+ await fs.writeFile(storePath, "{not valid json", "utf8");
+ const store = new DurableAdvancementKeyStore(storePath);
+
+ await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
+ });
+
+ it("serializes concurrent commits of the same key so exactly one wins", async () => {
+ const store = new DurableAdvancementKeyStore(storePath);
+
+ const results = await Promise.all([
+ store.tryCommit("turn-race", []),
+ store.tryCommit("turn-race", []),
+ store.tryCommit("turn-race", []),
+ ]);
+
+ expect(results.filter((r) => r === "committed")).toHaveLength(1);
+ expect(results.filter((r) => r === "duplicate")).toHaveLength(2);
+ });
+
+ it(
+ "never evicts a key regardless of how many others were committed since",
+ async () => {
+ const store = new DurableAdvancementKeyStore(storePath);
+
+ for (let i = 0; i < 600; i++) {
+ await store.tryCommit(`turn-${i}`, []);
+ }
+
+ await expect(store.tryCommit("turn-0", [])).resolves.toBe("duplicate");
+ },
+ 20_000,
+ );
+
+ // ---------------------------------------------------------------------
+ // PR #3442 review, round 2: three additional atomic-commit failures.
+ // ---------------------------------------------------------------------
+
+ it("persists the accepted messages together with the key, not just the key", async () => {
+ const store = new DurableAdvancementKeyStore(storePath);
+ const messages = [
+ { role: "user", content: "what's the weather?" },
+ { role: "assistant", content: "Let me check." },
+ ];
+
+ await store.tryCommit("turn-1", messages);
+
+ const entry = await store.get("turn-1");
+ expect(entry).toBeDefined();
+ expect(entry?.messages).toEqual(messages);
+ expect(typeof entry?.committedAt).toBe("string");
+ expect(entry?.committedAt.length).toBeGreaterThan(0);
+
+ // Survives a restart too -- the messages aren't an in-memory-only echo.
+ const after = new DurableAdvancementKeyStore(storePath);
+ const reloaded = await after.get("turn-1");
+ expect(reloaded?.messages).toEqual(messages);
+ });
+
+ it("does not remember a key when storage is inaccessible, so a retry can still succeed", async () => {
+ // Block the parent directory with a file. Both the commit's initial
+ // read and a membership lookup must propagate the filesystem error.
+ const blockedParent = path.join(dir, "blocked");
+ await fs.writeFile(blockedParent, "not a directory", "utf8");
+ const blockedPath = path.join(blockedParent, "commit-log.json");
+ const store = new DurableAdvancementKeyStore(blockedPath);
+
+ await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
+ // The failed write must not be remembered in memory: retrying the same
+ // key must attempt the commit again (and fail again, for the same
+ // reason) rather than silently reporting "duplicate" for a commit that
+ // never reached disk.
+ await expect(store.tryCommit("turn-1", [])).rejects.toThrow();
+ await expect(store.has("turn-1")).rejects.toThrow();
+
+ // Once the underlying problem is fixed, the key is still absent and
+ // the same key commits normally.
+ await fs.rm(blockedParent, { force: true });
+ await expect(store.has("turn-1")).resolves.toBe(false);
+ await expect(store.tryCommit("turn-1", [])).resolves.toBe("committed");
+ await expect(store.has("turn-1")).resolves.toBe(true);
+ });
+
+ it("serializes across store instances sharing the same file, not only within one object", async () => {
+ const storeA = new DurableAdvancementKeyStore(storePath);
+ const storeB = new DurableAdvancementKeyStore(storePath);
+
+ // Interleaved commits through two independent instances against the
+ // same file must not lose either one to a read-modify-write race.
+ const [resultA, resultB] = await Promise.all([
+ storeA.tryCommit("turn-a", ["from a"]),
+ storeB.tryCommit("turn-b", ["from b"]),
+ ]);
+
+ expect(resultA).toBe("committed");
+ expect(resultB).toBe("committed");
+
+ // A third, fresh instance (simulating a restart) must see both.
+ const after = new DurableAdvancementKeyStore(storePath);
+ await expect(after.has("turn-a")).resolves.toBe(true);
+ await expect(after.has("turn-b")).resolves.toBe(true);
+ await expect(after.tryCommit("turn-a", [])).resolves.toBe("duplicate");
+ await expect(after.tryCommit("turn-b", [])).resolves.toBe("duplicate");
+ });
+
+ // -------------------------------------------------------------------
+ // PR #3442 review, round 3: an in-process lock (a JS Map/promise queue)
+ // does nothing to stop two separate OS processes -- e.g. two gateway
+ // processes sharing one Headroom workspace -- from racing the same
+ // commit-log file. These spawn the real DurableAdvancementKeyStore in
+ // real child processes (not Worker threads, which share the parent's
+ // memory and wouldn't exercise inter-process file locking) and release
+ // them via a shared "go" file so their read/check/write transactions
+ // contend for the file lock as close to simultaneously as possible.
+ // -------------------------------------------------------------------
+
+ it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
+ "serializes commits from two separate OS processes sharing the same file",
+ async () => {
+ const goPath = path.join(dir, "go");
+ const resultAPath = path.join(dir, "result-a.json");
+ const resultBPath = path.join(dir, "result-b.json");
+
+ const workerA = runCommitWorker(storePath, "turn-a", goPath, resultAPath);
+ const workerB = runCommitWorker(storePath, "turn-b", goPath, resultBPath);
+ // Give both worker processes time to spawn and reach their poll loop
+ // before releasing them together.
+ await new Promise((r) => setTimeout(r, 300));
+ await fs.writeFile(goPath, "go", "utf8");
+
+ const [resultA, resultB] = await Promise.all([workerA, workerB]);
+
+ expect(resultA).toEqual({ status: "committed" });
+ expect(resultB).toEqual({ status: "committed" });
+
+ // Neither commit was lost to a cross-process race.
+ const after = new DurableAdvancementKeyStore(storePath);
+ await expect(after.has("turn-a")).resolves.toBe(true);
+ await expect(after.has("turn-b")).resolves.toBe(true);
+ },
+ 30_000,
+ );
+
+ it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
+ "reports duplicate for the same key committed from two separate OS processes",
+ async () => {
+ const goPath = path.join(dir, "go");
+ const resultAPath = path.join(dir, "result-a.json");
+ const resultBPath = path.join(dir, "result-b.json");
+
+ const workerA = runCommitWorker(storePath, "turn-same", goPath, resultAPath);
+ const workerB = runCommitWorker(storePath, "turn-same", goPath, resultBPath);
+ await new Promise((r) => setTimeout(r, 300));
+ await fs.writeFile(goPath, "go", "utf8");
+
+ const [resultA, resultB] = await Promise.all([workerA, workerB]);
+
+ // Exactly one process's transaction should win the race and see
+ // "committed"; the other must see "duplicate", never both
+ // "committed" (which would mean the file lock let them interleave).
+ const statuses = [resultA.status, resultB.status].sort();
+ expect(statuses).toEqual(["committed", "duplicate"]);
+ },
+ 30_000,
+ );
+
+ it.skipIf(!NODE_SUPPORTS_TRANSFORM_TYPES)(
+ "never lets a second writer commit while a live holder still holds the lock",
+ async () => {
+ // Regression for PR #3442 review, round 4: a lock reclaimed on an
+ // age-based staleness guess let a second process commit and release
+ // while the first (merely slow, not dead) holder still believed it
+ // owned the lock -- the first then resumed and overwrote the
+ // second's accepted commit with its own stale snapshot. This proves
+ // the invariant that replaces that heuristic: a second writer must
+ // never proceed while ANY holder -- however long it takes -- still
+ // holds the lock. Deterministic and fast: the "holder" is a worker
+ // that occupies the real lock file and only releases it when told
+ // to, rather than relying on a real multi-second sleep.
+ const readyPath = path.join(dir, "holder-ready");
+ const releasePath = path.join(dir, "holder-release");
+
+ const holder = execFileAsync(process.execPath, [
+ "--experimental-transform-types",
+ HOLD_LOCK_WORKER_PATH,
+ storePath,
+ readyPath,
+ releasePath,
+ ]);
+
+ // Wait until the holder has actually acquired the lock before racing
+ // a commit against it.
+ await waitForFile(readyPath);
+
+ const store = new DurableAdvancementKeyStore(storePath);
+ let secondCommitSettled = false;
+ const secondCommit = store.tryCommit("turn-b", ["b"]).finally(() => {
+ secondCommitSettled = true;
+ });
+
+ // Give the second commit ample opportunity to (wrongly) proceed
+ // while the holder is still alive and has not released.
+ try {
+ await new Promise((r) => setTimeout(r, 500));
+ expect(secondCommitSettled).toBe(false);
+ } finally {
+ // Release even if the assertion fails, so the child cannot leak.
+ await fs.writeFile(releasePath, "go", "utf8");
+ await holder;
+ }
+ // The second commit must now complete successfully, with nothing lost.
+ await expect(secondCommit).resolves.toBe("committed");
+ await expect(store.has("turn-b")).resolves.toBe(true);
+ },
+ 15_000,
+ );
+});
+
+describe("defaultCommitLogPath", () => {
+ const originalWorkspaceDir = process.env.HEADROOM_WORKSPACE_DIR;
+
+ afterEach(() => {
+ if (originalWorkspaceDir === undefined) {
+ delete process.env.HEADROOM_WORKSPACE_DIR;
+ } else {
+ process.env.HEADROOM_WORKSPACE_DIR = originalWorkspaceDir;
+ }
+ });
+
+ it("honors HEADROOM_WORKSPACE_DIR when set", () => {
+ process.env.HEADROOM_WORKSPACE_DIR = "/custom/workspace";
+
+ expect(defaultCommitLogPath()).toBe(
+ path.join("/custom/workspace", "openclaw", "commit-log.json"),
+ );
+ });
+
+ it("falls back to ~/.headroom when unset", () => {
+ delete process.env.HEADROOM_WORKSPACE_DIR;
+
+ expect(defaultCommitLogPath()).toBe(
+ path.join(os.homedir(), ".headroom", "openclaw", "commit-log.json"),
+ );
+ });
+});
diff --git a/plugins/openclaw/test/engine.test.ts b/plugins/openclaw/test/engine.test.ts
index d9886f707..786717727 100644
--- a/plugins/openclaw/test/engine.test.ts
+++ b/plugins/openclaw/test/engine.test.ts
@@ -1,4 +1,8 @@
-import { afterEach, describe, expect, it, vi } from "vitest";
+import * as fs from "node:fs/promises";
+import * as os from "node:os";
+import * as path from "node:path";
+
+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const mocked = vi.hoisted(() => ({
compress: vi.fn(),
@@ -325,3 +329,102 @@ describe("HeadroomContextEngine proxy startup helpers", () => {
);
});
});
+
+describe("HeadroomContextEngine transcriptSemantics contract", () => {
+ let commitLogDir: string;
+ let commitLogPath: string;
+
+ beforeEach(async () => {
+ commitLogDir = await fs.mkdtemp(path.join(os.tmpdir(), "headroom-commit-log-"));
+ commitLogPath = path.join(commitLogDir, "commit-log.json");
+ });
+
+ afterEach(async () => {
+ await fs.rm(commitLogDir, { recursive: true, force: true });
+ });
+
+ it("declares the durable-commit transcript semantics OpenClaw requires", () => {
+ const engine = new HeadroomContextEngine({ commitLogPath });
+
+ expect(engine.info.transcriptSemantics).toEqual({
+ currentTurnFence: "before-current-turn-entry-v1",
+ turnAdvancementIdempotency: "atomic-idempotent-v1",
+ });
+ });
+
+ it("commits a new advancement key", async () => {
+ const engine = new HeadroomContextEngine({ commitLogPath });
+
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
+ ).resolves.toEqual({ status: "committed" });
+ });
+
+ it("reports duplicate on a retried advancement key", async () => {
+ const engine = new HeadroomContextEngine({ commitLogPath });
+
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
+ ).resolves.toEqual({ status: "committed" });
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
+ ).resolves.toEqual({ status: "duplicate" });
+ });
+
+ it("treats distinct advancement keys independently", async () => {
+ const engine = new HeadroomContextEngine({ commitLogPath });
+
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-1", messages: [] }),
+ ).resolves.toEqual({ status: "committed" });
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-2", messages: [] }),
+ ).resolves.toEqual({ status: "committed" });
+ });
+
+ it("reports duplicate for a key committed before a process restart", async () => {
+ // Simulate a restart: a brand new engine instance (no shared in-memory
+ // state) pointed at the same durable commit-log path.
+ const before = new HeadroomContextEngine({ commitLogPath });
+ await expect(
+ before.commitTurn({ advancementKey: "turn-restart", messages: [] }),
+ ).resolves.toEqual({ status: "committed" });
+
+ const after = new HeadroomContextEngine({ commitLogPath });
+ await expect(
+ after.commitTurn({ advancementKey: "turn-restart", messages: [] }),
+ ).resolves.toEqual({ status: "duplicate" });
+ });
+
+ it(
+ "never forgets a key regardless of how many other keys were committed since",
+ async () => {
+ // Regression: the old implementation evicted the oldest key past a
+ // 512-entry cap, so a retry of an early key was wrongly re-accepted as
+ // new instead of reported as a duplicate. There is no such cap now.
+ const engine = new HeadroomContextEngine({ commitLogPath });
+
+ for (let i = 0; i < 600; i++) {
+ await engine.commitTurn({ advancementKey: `turn-${i}`, messages: [] });
+ }
+
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-0", messages: [] }),
+ ).resolves.toEqual({ status: "duplicate" });
+ },
+ 20_000,
+ );
+
+ it("persists the accepted messages together with the advancement key", async () => {
+ const engine = new HeadroomContextEngine({ commitLogPath });
+ const messages = [{ role: "user", content: "hello" }];
+
+ await expect(
+ engine.commitTurn({ advancementKey: "turn-1", messages }),
+ ).resolves.toEqual({ status: "committed" });
+
+ const raw = await fs.readFile(commitLogPath, "utf8");
+ const entries = JSON.parse(raw) as Record<string, { messages: unknown }>;
+ expect(entries["turn-1"].messages).toEqual(messages);
+ });
+});
diff --git a/plugins/openclaw/test/fixtures/commit-turn-worker.ts b/plugins/openclaw/test/fixtures/commit-turn-worker.ts
new file mode 100644
index 000000000..440d8c10b
--- /dev/null
+++ b/plugins/openclaw/test/fixtures/commit-turn-worker.ts
@@ -0,0 +1,59 @@
+/**
+ * Child-process worker for the cross-process DurableAdvancementKeyStore
+ * regression tests (PR #3442 review, round 3).
+ *
+ * Run as a real, separate OS process (not a Worker thread, which shares the
+ * parent's memory and wouldn't exercise inter-process file locking) via
+ * `node --experimental-transform-types commit-turn-worker.ts <storePath>
+ * <key> <goPath> <resultPath>`.
+ *
+ * Blocks until `goPath` exists (the parent test creates it only after every
+ * worker has been spawned and had time to reach this poll loop), then
+ * commits `key` through the real `DurableAdvancementKeyStore` and writes
+ * the outcome to `resultPath` as JSON -- this is what lets two independently
+ * launched processes race their read/check/write transactions against the
+ * same commit-log file as closely to simultaneously as possible.
+ */
+
+import { access, writeFile } from "node:fs/promises";
+
+// Explicit `.ts` extension: this file runs directly under Node's native
+// TypeScript support (`--experimental-transform-types`), which -- unlike
+// the TypeScript compiler / bundler used to build the plugin itself --
+// resolves imports literally and does not map a `.js` specifier onto a
+// sibling `.ts` file.
+import { DurableAdvancementKeyStore } from "../../src/advancement-key-store.ts";
+
+async function waitForGo(goPath: string): Promise<void> {
+ for (;;) {
+ try {
+ await access(goPath);
+ return;
+ } catch {
+ await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
+ }
+ }
+}
+
+async function main(): Promise<void> {
+ const [, , storePath, key, goPath, resultPath] = process.argv;
+ if (!storePath || !key || !goPath || !resultPath) {
+ throw new Error("usage: commit-turn-worker.ts <storePath> <key> <goPath> <resultPath>");
+ }
+
+ await waitForGo(goPath);
+
+ const store = new DurableAdvancementKeyStore(storePath);
+ try {
+ const status = await store.tryCommit(key, [key]);
+ await writeFile(resultPath, JSON.stringify({ status }), "utf8");
+ } catch (error) {
+ await writeFile(
+ resultPath,
+ JSON.stringify({ error: error instanceof Error ? error.message : String(error) }),
+ "utf8",
+ );
+ }
+}
+
+main();
diff --git a/plugins/openclaw/test/fixtures/hold-lock-worker.ts b/plugins/openclaw/test/fixtures/hold-lock-worker.ts
new file mode 100644
index 000000000..9845ce110
--- /dev/null
+++ b/plugins/openclaw/test/fixtures/hold-lock-worker.ts
@@ -0,0 +1,52 @@
+/**
+ * Simulates a live-but-slow lock holder for the DurableAdvancementKeyStore
+ * cross-process regression tests (PR #3442 review, round 4): acquires the
+ * exact same lock file the real store uses, signals readiness, then holds
+ * it until told to release -- proving a real, still-alive holder is never
+ * force-preempted no matter how long its critical section runs, unlike the
+ * round-3 age-based staleness heuristic this replaces.
+ *
+ * Deliberately does not import `DurableAdvancementKeyStore`: it only needs
+ * to occupy the lock file at the path the real class would use, using the
+ * same `open(path, "wx")` protocol, so the *other* worker's real
+ * `tryCommit` call is what gets exercised and observed.
+ *
+ * Usage: node --experimental-transform-types hold-lock-worker.ts
+ * <storePath> <readyPath> <releasePath>
+ */
+
+import { access, mkdir, open, unlink, utimes, writeFile } from "node:fs/promises";
+import { dirname } from "node:path";
+
+async function waitFor(p: string): Promise<void> {
+ for (;;) {
+ try {
+ await access(p);
+ return;
+ } catch {
+ await new Promise((resolveDelay) => setTimeout(resolveDelay, 5));
+ }
+ }
+}
+
+async function main(): Promise<void> {
+ const [, , storePath, readyPath, releasePath] = process.argv;
+ if (!storePath || !readyPath || !releasePath) {
+ throw new Error("usage: hold-lock-worker.ts <storePath> <readyPath> <releasePath>");
+ }
+
+ const lockPath = `${storePath}.lock`;
+ await mkdir(dirname(storePath), { recursive: true });
+ const handle = await open(lockPath, "wx");
+ await handle.close();
+ // Exercise the former 30-second reclaim threshold without a slow test.
+ const oldTime = new Date(Date.now() - 60_000);
+ await utimes(lockPath, oldTime, oldTime);
+
+ await writeFile(readyPath, "ready", "utf8");
+
+ await waitFor(releasePath);
+ await unlink(lockPath).catch(() => undefined);
+}
+
+main();