Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(storage): document the PVC layout, merge init and rollback rule

Alfredo Lopez61ee787f3888
diff --git a/docs/README.md b/docs/README.md
index 4610e7c7d..c1f1af323 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -32,6 +32,7 @@ Use this index for long-form docs.
| [Architecture overview](architecture/overview.md) | Services, CRDs, message lifecycle, NetworkPolicy model |
| [Platform topology](architecture/platform-topology.md) | Namespaces, controller split, deny-all baseline |
| [Non-MCP services](architecture/non-mcp-services.md) | Namespace splitting and L0–L3 policy layers |
+| [Host storage layout](architecture/host-storage-layout.md) | Workspace PVC layout, merge init, database failures, rollback rule |
| [Diagrams](architecture/diagrams/) | Excalidraw sources |
## Surfaces (the UIs)
diff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.md
new file mode 100644
index 000000000..e2c4bcc3c
--- /dev/null
+++ b/docs/architecture/host-storage-layout.md
@@ -0,0 +1,139 @@
+# Host Storage Layout
+
+Every Host keeps its files and its conversation database on one workspace
+PVC, in the same place, whatever its lifecycle. Switching a Host between
+stateful and stateless changes how its Pod runs (replicas, priority class,
+probes, `CLERUM_STATELESS_LIFECYCLE`); it never changes where data lives
+(#825).
+
+## Layout
+
+HCC renders the same storage for every Host Deployment
+(`host-context-controller/src/hostReconciler.ts`):
+
+| Item | Value |
+| --------------------- | ---------------------------------------------------------------------- |
+| Volume mount | the PVC `<host>-workspace` root at `workspacePath`, no `subPath` |
+| `workspacePath` | `/workspace`; `/config/workspace` for a `spec.desktop` Host |
+| Conversation database | `<workspacePath>/state.db` (with `-wal` and `-shm`) |
+| Store env | `CLERUM_SESSION_STORE=sqlite`, `CLERUM_SESSION_DB_DIR=<workspacePath>` |
+| Init container | `workspace-layout-merge` |
+| Deployment strategy | `Recreate` |
+
+`Recreate` stops the old Pod before the new one starts. A rolling update can
+run both Pods on the same node at once, and a `ReadWriteOnce` PVC does not
+prevent that, so two processes could write the same SQLite file.
+
+Agent file tools cannot read or write any file whose basename starts with
+`state.db` (database, journals and `.pre-<ts>.bak` backups), checked on the
+requested and on the resolved path
+(`mcp-host/src/workspace/protectedPaths.ts`). The shell gate applies the same
+prefix lexically and is not a security boundary.
+
+## The previous split layout
+
+Before #825, stateless Hosts mounted `PVC/workspace` at `/workspace` and
+`PVC/state` at `/var/lib/clerum/state`, and an init container
+(`workspace-layout-init`) moved root content into those two directories. A
+switch back to stateful mounted the PVC root again, did not find
+`state.db` there, and started an empty database. Repeated switches split the
+history across databases and duplicated personalization files.
+
+## `workspace-layout-merge`
+
+The init container runs before `mcp-host` on every start, with the Host
+image, the main container's `securityContext` and pull policy, the PVC root
+mounted at `/mnt/workspace-root`, and requests of 10m CPU / 32Mi memory
+(`host-context-controller/src/statelessDeployment.ts`). It never opens
+SQLite.
+
+**Trigger.** It acts only on the split-layout signature: `state/` is a real
+directory (not a symlink) holding at least one `state.db*` entry. Every other
+tree, including an agent folder named `workspace/` or `state/` without a
+database, exits 0 unchanged. Stateful and desktop Hosts therefore never
+change.
+
+**Moves**, in this order:
+
+1. every entry of `workspace/`, dotfiles and symlinks included, to the root
+ (`mv` renames a symlink and never follows it);
+2. every entry of `state/` except `state.db`, including journals and
+ `state.db*.pre-<ts>.bak` backups;
+3. `state/state.db` last, so an interrupted merge still carries the
+ signature and completes on the next start;
+4. remove the empty `state/` and `workspace/`, then `sync`.
+
+A crash after step 3 and before step 4 leaves empty `state/` and `workspace/`
+directories at the root. The signature is gone, so later starts exit 0 and
+keep them; the database and its journals are already at the root.
+
+**Refusals.** Each prints one `workspace layout merge refused: ...` line to
+stderr and exits 1 before the first move, leaving the tree unchanged. The Pod
+stays in `Init:Error`.
+
+| Case | Line contains |
+| ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
+| F1: `state`, `workspace` or the PVC root is a symlink, not a directory, or escapes the root; a `state.db*` entry is a symlink | `directory is a symlink`, `is not a directory`, `escapes the workspace PVC root`, `SQLite file is a symlink` |
+| F2: more than one live database family across the root, `state/` and `workspace/` | `more than one live SQLite database` |
+| F3: a name to move already exists at the root, or under both `state/` and `workspace/`; `workspace/lost+found` exists | `destination exists`, `exists under both state/ and workspace/`, `lost+found would remain` |
+| F4: the PVC root, `state/` or `workspace/` is not writable by the container uid | `not writable by uid` |
+
+The PVC root checks of F1 run on every start, before the signature check.
+
+Two exceptions to F2/F3 cover the earlier init stopped half-way:
+
+- root orphan journals with `state/state.db` as the only main file. A journal
+ with the same name both at the root and in `state/` still refuses, because
+ the merge cannot tell which one belongs to the database;
+- an empty source directory whose destination is a real directory.
+
+**Operator action on `Init:Error`.** Read the line with
+`kubectl --context <ctx> -n mcp-host logs <pod> -c workspace-layout-merge`,
+inspect the PVC from a helper Pod, and rename or remove the conflicting entry.
+Keep every database file until the history is confirmed.
+
+## Database failures in `mcp-host`
+
+- If the database worker exits, or an asynchronous write fails, the persist
+ queue rejects later writes and the process exits with code 1. The kubelet
+ restarts the container; a persistent cause shows as `CrashLoopBackOff`.
+- mcp-host never creates `state.db` next to an orphan `state.db-wal` or
+ `state.db-shm`; opening the store fails with `ORPHAN_WAL` instead
+ (`mcp-host/src/db/worker/openDatabase.ts`).
+- A PVC with files and no database starts normally and mcp-host creates
+ `state.db`.
+
+## Suspended stateless Hosts
+
+A suspended Host has no Pod, so its PVC is merged on its next wake. No wake is
+forced by a deploy.
+
+## Rollback
+
+- Rolling HCC back is safe for stateful and stateless Hosts. The previous
+ `workspace-layout-init` re-splits stateless trees, and `workspace-layout-merge`
+ merges them again on roll-forward.
+- The deploy workflows roll HCC back automatically when its rollout gate fails
+ (`keyper-labs/evenfire-infra`, `deploy-clerum-dev.yml:715-742`,
+ `deploy-clerum-prod.yml:643-670`). This is safe for a Host in `Init:Error`,
+ because every refusal exits before the first move.
+- For an operator-initiated rollback: do not roll back while any Host is in
+ `Init:Error` in `workspace-layout-merge`, and do not change a Host's
+ lifecycle while rolled back.
+- Never roll back the Host image alone; HCC's Deployment carries both
+ (`CONTEXT_MAPPER_HOST_IMAGE`, `CONTEXT_MAPPER_DESKTOP_IMAGE`).
+- A desktop Host's history from before #825 lived in the container, outside
+ the PVC, and is lost at the first rollout unless it is copied out of the
+ live Pod first (`kubectl --context <ctx> cp` of `/workspace/state.db*`).
+
+## Tests
+
+- Unit: `host-context-controller/test/statelessDeployment.workspaceLayoutMerge.test.ts`
+ runs the script under `sh` on real trees, including an uncheckpointed WAL,
+ each refusal, re-runs after an interrupted merge and a round trip through
+ the previous init (`test/__fixtures__/base-workspace-layout-init.sh`).
+- Live: `make test-e2e-hcc-storage-layout-switch`
+ (`scripts/e2e/e2e-hcc-storage-layout-switch.sh`) on an owned branch profile.
+- Desktop: `make test-e2e-825-desktop-continuity`
+ (`desktop-app/test/e2e-playwright/stateful-stateless-switch-continuity.spec.ts`)
+ keeps one conversation across stateful → stateless → stateful switches.