Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(storage): stall bound, sync write failures and rollback window

Alfredo Lopez3f2ae594df90
diff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.md
index 31b03f322..60f929598 100644
--- a/docs/architecture/host-storage-layout.md
+++ b/docs/architecture/host-storage-layout.md
@@ -49,7 +49,9 @@ workspace root as its working directory (`mcp-host/src/core/tools/shell.ts`),
so a command such as `rm -f *`, `find . -type f -delete` or `git clean -fdx`
run there deletes `state.db` and its journals, and a hard link
(`ln state.db notes`) exposes its contents under a name the prefix rule
-allows. Stateful Hosts had this exposure before #825; stateless Hosts gain it,
+allows. A repository or archive operation run there, such as
+`git add -A && git push` or a `tar` of the workspace, copies the database out
+with the agent's files; neither prefix check sees it. Stateful Hosts had this exposure before #825; stateless Hosts gain it,
because their database moves from the separate `state/` mount to the root.
Isolating the database from the agent tree is a follow-up.
@@ -157,7 +159,11 @@ HCC renders `workspace-layout-merge` only while this flag is enabled
(`host-context-controller/src/config.ts`, `parseWorkspaceLayoutMerge`). On a
stateful Host the PVC root is the agent workspace, so an agent can create a
`state/state.db*` entry and trigger the merge; the merge is needed once per
-Host, and the flag closes that window once every Host is merged.
+Host, and the flag closes that window once every Host is merged. While the
+flag is on, the init runs on every Host at every start, so such an entry can
+also make it refuse (F2 next to a root `state.db`, F1 for a symlinked
+`state.db*` entry) and leave a stateful Host in `Init:Error` until an operator
+renames the entry.
- Unset or empty means `true`. Only the exact strings `true` and `false` are
accepted; any other value fails HCC config load with
@@ -187,6 +193,18 @@ Host, and the flag closes that window once every Host is merged.
- If the database worker exits, or an asynchronous write fails, the persist
queue rejects later writes and the process exits with code 1. The kubelet
restarts the container; a persistent cause shows as `CrashLoopBackOff`.
+- A slow write is not a failure. If the worker has operations in flight and
+ completes none of them for `CLERUM_DB_WORKER_STALL_TIMEOUT_MS` (default
+ `30000`, at least `1000` and above `CLERUM_DB_PERSIST_SYNC_TIMEOUT_MS`), the
+ queue fails the same way and the worker is terminated.
+- At shutdown mcp-host waits for every queued write, bounded only by that
+ stall timeout and the Pod's termination grace period. A persistence failure
+ during shutdown exits 1 instead of 0.
+- A failed synchronous write, such as `SQLITE_FULL` on the first write of a
+ turn, rejects only that request; the queue stays usable and the process
+ keeps running. A full PVC therefore shows either as turns that fail while
+ the Pod stays Ready, or as `CrashLoopBackOff` once an asynchronous write
+ fails.
- mcp-host never creates `state.db` next to an orphan `state.db-wal` or
`state.db-shm`; opening the store fails with `ORPHAN_WAL` instead
(`mcp-host/src/db/worker/openDatabase.ts`).
@@ -253,7 +271,11 @@ forced by a deploy.
- The deploy workflows roll HCC back automatically when its rollout gate fails
(`keyper-labs/evenfire-infra`, `deploy-clerum-dev.yml:715-742`,
`deploy-clerum-prod.yml:643-670`). This is safe for a Host in `Init:Error`,
- because every refusal exits before the first move.
+ because every refusal exits before the first move. The workflow reads no
+ Host state, so it can also roll back while a Host Pod is still running
+ `workspace-layout-merge`, the window the operator rule below forbids, and
+ the case above of a merge cut between the journals and the main database
+ can then occur.
- For an operator-initiated rollback: do not roll back while any Host is in
`Init:Error` in `workspace-layout-merge` or any Host Pod is still running
that init (Pending, init not finished), and do not change a Host's
diff --git a/mcp-host/package-lock.json b/mcp-host/package-lock.json
index 5ae4f745d..2c029dab8 100644
--- a/mcp-host/package-lock.json
+++ b/mcp-host/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-mcp-host",
- "version": "0.1.575",
+ "version": "0.1.576",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-mcp-host",
- "version": "0.1.575",
+ "version": "0.1.576",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@anthropic-ai/sdk": "^0.32.0",
diff --git a/mcp-host/package.json b/mcp-host/package.json
index 7c1cbc29d..3e0f48a8e 100644
--- a/mcp-host/package.json
+++ b/mcp-host/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-mcp-host",
- "version": "0.1.575",
+ "version": "0.1.576",
"description": "MCP Host - reads Host CRD and provides LLM access via OpenAI/Claude",
"main": "dist/main.js",
"engines": {
diff --git a/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts b/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts
index e0079a46e..8edbb643a 100644
--- a/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts
+++ b/mcp-host/src/core/conversation/persistence/conversationStoreFactory.ts
@@ -109,9 +109,10 @@ export function createConversationStore(
// The worker answers only after it has opened and migrated the database, so
// the path is logged once that is true. One line per process names the
// database this Host reads and writes, so a Pod replacement that changes the
- // store is visible in the Pod logs. On a worker exit, `onFailure` ends the
- // process in production before this rejection handler runs; the worker's
- // fatal log and the exit log name the cause there.
+ // store is visible in the Pod logs. A worker exit before the answer rejects
+ // the ping and fires `onFailure`; in production the process then exits 1
+ // once the pre-exit flush ends, so this handler still logs, next to the
+ // worker's fatal log and the exit log that name the cause.
persistQueue.awaitOpen().then(
() => logger.info({ dbPath: opts.dbPath, mode: opts.mode }, '[ConversationStore] opened'),
(err: unknown) => logger.error({ err }, '[ConversationStore] database did not open')