Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(storage): document the PVC layout, merge init and rollback rule
Alfredo Lopez61ee787f3888
diff --git a/docs/README.md b/docs/README.mdindex 4610e7c7d..c1f1af323 100644--- a/docs/README.md+++ b/docs/README.md@@ -32,6 +32,7 @@ Use this index for long-form docs.| [Architecture overview](architecture/overview.md) | Services, CRDs, message lifecycle, NetworkPolicy model || [Platform topology](architecture/platform-topology.md) | Namespaces, controller split, deny-all baseline || [Non-MCP services](architecture/non-mcp-services.md) | Namespace splitting and L0–L3 policy layers |+| [Host storage layout](architecture/host-storage-layout.md) | Workspace PVC layout, merge init, database failures, rollback rule || [Diagrams](architecture/diagrams/) | Excalidraw sources |## Surfaces (the UIs)diff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.mdnew file mode 100644index 000000000..e2c4bcc3c--- /dev/null+++ b/docs/architecture/host-storage-layout.md@@ -0,0 +1,139 @@+# Host Storage Layout++Every Host keeps its files and its conversation database on one workspace+PVC, in the same place, whatever its lifecycle. Switching a Host between+stateful and stateless changes how its Pod runs (replicas, priority class,+probes, `CLERUM_STATELESS_LIFECYCLE`); it never changes where data lives+(#825).++## Layout++HCC renders the same storage for every Host Deployment+(`host-context-controller/src/hostReconciler.ts`):++| Item | Value |+| --------------------- | ---------------------------------------------------------------------- |+| Volume mount | the PVC `<host>-workspace` root at `workspacePath`, no `subPath` |+| `workspacePath` | `/workspace`; `/config/workspace` for a `spec.desktop` Host |+| Conversation database | `<workspacePath>/state.db` (with `-wal` and `-shm`) |+| Store env | `CLERUM_SESSION_STORE=sqlite`, `CLERUM_SESSION_DB_DIR=<workspacePath>` |+| Init container | `workspace-layout-merge` |+| Deployment strategy | `Recreate` |++`Recreate` stops the old Pod before the new one starts. A rolling update can+run both Pods on the same node at once, and a `ReadWriteOnce` PVC does not+prevent that, so two processes could write the same SQLite file.++Agent file tools cannot read or write any file whose basename starts with+`state.db` (database, journals and `.pre-<ts>.bak` backups), checked on the+requested and on the resolved path+(`mcp-host/src/workspace/protectedPaths.ts`). The shell gate applies the same+prefix lexically and is not a security boundary.++## The previous split layout++Before #825, stateless Hosts mounted `PVC/workspace` at `/workspace` and+`PVC/state` at `/var/lib/clerum/state`, and an init container+(`workspace-layout-init`) moved root content into those two directories. A+switch back to stateful mounted the PVC root again, did not find+`state.db` there, and started an empty database. Repeated switches split the+history across databases and duplicated personalization files.++## `workspace-layout-merge`++The init container runs before `mcp-host` on every start, with the Host+image, the main container's `securityContext` and pull policy, the PVC root+mounted at `/mnt/workspace-root`, and requests of 10m CPU / 32Mi memory+(`host-context-controller/src/statelessDeployment.ts`). It never opens+SQLite.++**Trigger.** It acts only on the split-layout signature: `state/` is a real+directory (not a symlink) holding at least one `state.db*` entry. Every other+tree, including an agent folder named `workspace/` or `state/` without a+database, exits 0 unchanged. Stateful and desktop Hosts therefore never+change.++**Moves**, in this order:++1. every entry of `workspace/`, dotfiles and symlinks included, to the root+ (`mv` renames a symlink and never follows it);+2. every entry of `state/` except `state.db`, including journals and+ `state.db*.pre-<ts>.bak` backups;+3. `state/state.db` last, so an interrupted merge still carries the+ signature and completes on the next start;+4. remove the empty `state/` and `workspace/`, then `sync`.++A crash after step 3 and before step 4 leaves empty `state/` and `workspace/`+directories at the root. The signature is gone, so later starts exit 0 and+keep them; the database and its journals are already at the root.++**Refusals.** Each prints one `workspace layout merge refused: ...` line to+stderr and exits 1 before the first move, leaving the tree unchanged. The Pod+stays in `Init:Error`.++| Case | Line contains |+| ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |+| F1: `state`, `workspace` or the PVC root is a symlink, not a directory, or escapes the root; a `state.db*` entry is a symlink | `directory is a symlink`, `is not a directory`, `escapes the workspace PVC root`, `SQLite file is a symlink` |+| F2: more than one live database family across the root, `state/` and `workspace/` | `more than one live SQLite database` |+| F3: a name to move already exists at the root, or under both `state/` and `workspace/`; `workspace/lost+found` exists | `destination exists`, `exists under both state/ and workspace/`, `lost+found would remain` |+| F4: the PVC root, `state/` or `workspace/` is not writable by the container uid | `not writable by uid` |++The PVC root checks of F1 run on every start, before the signature check.++Two exceptions to F2/F3 cover the earlier init stopped half-way:++- root orphan journals with `state/state.db` as the only main file. A journal+ with the same name both at the root and in `state/` still refuses, because+ the merge cannot tell which one belongs to the database;+- an empty source directory whose destination is a real directory.++**Operator action on `Init:Error`.** Read the line with+`kubectl --context <ctx> -n mcp-host logs <pod> -c workspace-layout-merge`,+inspect the PVC from a helper Pod, and rename or remove the conflicting entry.+Keep every database file until the history is confirmed.++## Database failures in `mcp-host`++- If the database worker exits, or an asynchronous write fails, the persist+ queue rejects later writes and the process exits with code 1. The kubelet+ restarts the container; a persistent cause shows as `CrashLoopBackOff`.+- mcp-host never creates `state.db` next to an orphan `state.db-wal` or+ `state.db-shm`; opening the store fails with `ORPHAN_WAL` instead+ (`mcp-host/src/db/worker/openDatabase.ts`).+- A PVC with files and no database starts normally and mcp-host creates+ `state.db`.++## Suspended stateless Hosts++A suspended Host has no Pod, so its PVC is merged on its next wake. No wake is+forced by a deploy.++## Rollback++- Rolling HCC back is safe for stateful and stateless Hosts. The previous+ `workspace-layout-init` re-splits stateless trees, and `workspace-layout-merge`+ merges them again on roll-forward.+- The deploy workflows roll HCC back automatically when its rollout gate fails+ (`keyper-labs/evenfire-infra`, `deploy-clerum-dev.yml:715-742`,+ `deploy-clerum-prod.yml:643-670`). This is safe for a Host in `Init:Error`,+ because every refusal exits before the first move.+- For an operator-initiated rollback: do not roll back while any Host is in+ `Init:Error` in `workspace-layout-merge`, and do not change a Host's+ lifecycle while rolled back.+- Never roll back the Host image alone; HCC's Deployment carries both+ (`CONTEXT_MAPPER_HOST_IMAGE`, `CONTEXT_MAPPER_DESKTOP_IMAGE`).+- A desktop Host's history from before #825 lived in the container, outside+ the PVC, and is lost at the first rollout unless it is copied out of the+ live Pod first (`kubectl --context <ctx> cp` of `/workspace/state.db*`).++## Tests++- Unit: `host-context-controller/test/statelessDeployment.workspaceLayoutMerge.test.ts`+ runs the script under `sh` on real trees, including an uncheckpointed WAL,+ each refusal, re-runs after an interrupted merge and a round trip through+ the previous init (`test/__fixtures__/base-workspace-layout-init.sh`).+- Live: `make test-e2e-hcc-storage-layout-switch`+ (`scripts/e2e/e2e-hcc-storage-layout-switch.sh`) on an owned branch profile.+- Desktop: `make test-e2e-825-desktop-continuity`+ (`desktop-app/test/e2e-playwright/stateful-stateless-switch-continuity.spec.ts`)+ keeps one conversation across stateful → stateless → stateful switches.