Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(storage): stall bound, sync write failures and rollback window
Alfredo Lopez3f2ae594df90
diff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.mdindex 31b03f322..60f929598 100644--- a/docs/architecture/host-storage-layout.md+++ b/docs/architecture/host-storage-layout.md@@ -49,7 +49,9 @@ workspace root as its working directory (`mcp-host/src/core/tools/shell.ts`),so a command such as `rm -f *`, `find . -type f -delete` or `git clean -fdx`run there deletes `state.db` and its journals, and a hard link(`ln state.db notes`) exposes its contents under a name the prefix rule-allows. Stateful Hosts had this exposure before #825; stateless Hosts gain it,+allows. A repository or archive operation run there, such as+`git add -A && git push` or a `tar` of the workspace, copies the database out+with the agent's files; neither prefix check sees it. Stateful Hosts had this exposure before #825; stateless Hosts gain it,because their database moves from the separate `state/` mount to the root.Isolating the database from the agent tree is a follow-up.@@ -157,7 +159,11 @@ HCC renders `workspace-layout-merge` only while this flag is enabled(`host-context-controller/src/config.ts`, `parseWorkspaceLayoutMerge`). On astateful Host the PVC root is the agent workspace, so an agent can create a`state/state.db*` entry and trigger the merge; the merge is needed once per-Host, and the flag closes that window once every Host is merged.+Host, and the flag closes that window once every Host is merged. While the+flag is on, the init runs on every Host at every start, so such an entry can+also make it refuse (F2 next to a root `state.db`, F1 for a symlinked+`state.db*` entry) and leave a stateful Host in `Init:Error` until an operator+renames the entry.- Unset or empty means `true`. Only the exact strings `true` and `false` areaccepted; any other value fails HCC config load with@@ -187,6 +193,18 @@ Host, and the flag closes that window once every Host is merged.- If the database worker exits, or an asynchronous write fails, the persistqueue rejects later writes and the process exits with code 1. The kubeletrestarts the container; a persistent cause shows as `CrashLoopBackOff`.+- A slow write is not a failure. If the worker has operations in flight and+ completes none of them for `CLERUM_DB_WORKER_STALL_TIMEOUT_MS` (default+ `30000`, at least `1000` and above `CLERUM_DB_PERSIST_SYNC_TIMEOUT_MS`), the+ queue fails the same way and the worker is terminated.+- At shutdown mcp-host waits for every queued write, bounded only by that+ stall timeout and the Pod's termination grace period. A persistence failure+ during shutdown exits 1 instead of 0.+- A failed synchronous write, such as `SQLITE_FULL` on the first write of a+ turn, rejects only that request; the queue stays usable and the process+ keeps running. A full PVC therefore shows either as turns that fail while+ the Pod stays Ready, or as `CrashLoopBackOff` once an asynchronous write+ fails.- mcp-host never creates `state.db` next to an orphan `state.db-wal` or`state.db-shm`; opening the store fails with `ORPHAN_WAL` instead(`mcp-host/src/db/worker/openDatabase.ts`).@@ -253,7 +271,11 @@ forced by a deploy.- The deploy workflows roll HCC back automatically when its rollout gate fails(`keyper-labs/evenfire-infra`, `deploy-clerum-dev.yml:715-742`,`deploy-clerum-prod.yml:643-670`). This is safe for a Host in `Init:Error`,- because every refusal exits before the first move.+ because every refusal exits before the first move. The workflow reads no+ Host state, so it can also roll back while a Host Pod is still running+ `workspace-layout-merge`, the window the operator rule below forbids, and+ the case above of a merge cut between the journals and the main database+ can then occur.- For an operator-initiated rollback: do not roll back while any Host is in`Init:Error` in `workspace-layout-merge` or any Host Pod is still runningthat init (Pending, init not finished), and do not change a Host'sdiff --git a/mcp-host/package-lock.json b/mcp-host/package-lock.jsonindex 5ae4f745d..2c029dab8 100644--- a/mcp-host/package-lock.json+++ b/mcp-host/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-mcp-host",- "version": "0.1.575",+ "version": "0.1.576","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-mcp-host",- "version": "0.1.575",+ "version": "0.1.576","license": "SEE LICENSE IN LICENSE","dependencies": {"@anthropic-ai/sdk": "^0.32.0",diff --git a/mcp-host/package.json b/mcp-host/package.jsonindex 7c1cbc29d..3e0f48a8e 100644--- a/mcp-host/package.json+++ b/mcp-host/package.json@@ -1,6 +1,6 @@{"name": "clerum-mcp-host",- "version": "0.1.575",+ "version": "0.1.576","description": "MCP Host - reads Host CRD and provides LLM access via OpenAI/Claude","main": "dist/main.js","engines": {diff --git a/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts b/mcp-host/src/core/conversation/persistence/conversationStoreFactory.tsindex e0079a46e..8edbb643a 100644--- a/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts+++ b/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts@@ -109,9 +109,10 @@ export function createConversationStore(// The worker answers only after it has opened and migrated the database, so// the path is logged once that is true. One line per process names the// database this Host reads and writes, so a Pod replacement that changes the- // store is visible in the Pod logs. On a worker exit, `onFailure` ends the- // process in production before this rejection handler runs; the worker's- // fatal log and the exit log name the cause there.+ // store is visible in the Pod logs. A worker exit before the answer rejects+ // the ping and fires `onFailure`; in production the process then exits 1+ // once the pre-exit flush ends, so this handler still logs, next to the+ // worker's fatal log and the exit log that name the cause.persistQueue.awaitOpen().then(() => logger.info({ dbPath: opts.dbPath, mode: opts.mode }, '[ConversationStore] opened'),(err: unknown) => logger.error({ err }, '[ConversationStore] database did not open')