Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(deploy): host PVC layout upgrade note

Alfredo Lopez24aa8d56e5b5
diff --git a/docs/README.md b/docs/README.md
index c1f1af323..e290b42c3 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -64,6 +64,7 @@ Use this index for long-form docs.
| [Shared & global files](how-to/shared-and-global-files.md) | Team workspaces (SFS) and the brokered drive (GFS) |
| [Minikube full stack](deploy/minikube.md) | Local Kubernetes platform |
| [Production notes](deploy/production.md) | Checklist and in-repo deploy assets |
+| [Host PVC layout upgrade](deploy/host-pvc-layout-upgrade.md) | Upgrading stateless Hosts from v0.10.x / v0.11.x |
| [WorkflowRecipes operations](deploy/workflow-recipes-guide.md) | Recipe ops, RBAC, debugging |
## Reference
diff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.md
index cafe9c06a..454135ebc 100644
--- a/docs/architecture/host-storage-layout.md
+++ b/docs/architecture/host-storage-layout.md
@@ -92,7 +92,11 @@ Two agent-made folder pairs on a stateful Host do stop it from starting:
- a `state/` holding `state.db` or one of its journals beside the root
database, next to a `workspace/` folder. The signature holds and F2
refuses, because the merge cannot tell the agent's file from a second
- database;
+ database. The earlier release produces the same tree without any agent: a
+ Host created stateless and switched to stateful on v0.10 or v0.11 keeps its
+ old history in `state/state.db` and writes new chats to a root `state.db`.
+ `host-layout-merge.sh inventory` reports it as `WILL_REFUSE_F2` before the
+ upgrade, and the upgrade note describes the repair;
- a `state/` the container uid cannot list and search, next to a
`workspace/` folder. F1 refuses before the database check, because the
merge cannot tell whether it holds a database.
@@ -180,12 +184,18 @@ renames the entry.
and no `workspace-layout-init`;
2. every running Host Pod's `workspace-layout-merge` exited 0;
3. no Host is in `Init:Error` in `workspace-layout-merge`;
- 4. every Host that was suspended at deploy time has woken once with that
- init exiting 0.
-- The turn-off is an overlay patch in `keyper-labs/evenfire-infra`
- (`patches/hcc-workspace-layout-merge.yaml`, value `"false"`), never in
- `dynamic-images.yaml`. Turning it off changes the pod template, so every
- running Host restarts once.
+ 4. no Host PVC still carries the split-layout signature.
+
+ [`deploy/scripts/host-layout-merge.sh verify`](../../deploy/scripts/host-layout-merge.sh)
+ checks all four and reads each PVC without a Pod from a read-only check
+ Pod; it prints `HOST_LAYOUT_VERIFY=PASS` only when they hold. The procedure
+ for any installation is in
+ [Host PVC Layout Upgrade Note](../deploy/host-pvc-layout-upgrade.md).
+
+- The turn-off is a patch in the installation's own overlay (clerum:
+ `keyper-labs/evenfire-infra`, `patches/hcc-workspace-layout-merge.yaml`,
+ value `"false"`), never in `dynamic-images.yaml`. Turning it off changes the
+ pod template, so every running Host restarts once.
- Never turn it off in the deploy that first introduces this HCC.
- Before any manual HCC undo to a revision older than #825's, set it back to
`true`. The older init re-splits stateless trees, and only a roll-forward
@@ -300,8 +310,15 @@ forced by a deploy.
at least one case of each refusal family (F1–F4), re-runs after an
interrupted merge and a round trip through the previous init
(`test/__fixtures__/base-workspace-layout-init.sh`).
+- Script: `deploy/scripts/host-layout-merge.test.mjs` runs
+ `host-layout-merge.sh` against a fake `kubectl`, and runs its check script
+ on real trees. The HCC merge suite above also checks that the check script
+ predicts the merge's outcome (merge, no-op or the refusal line) on every
+ tree it tests.
- Live: `make test-e2e-hcc-storage-layout-switch`
(`scripts/e2e/e2e-hcc-storage-layout-switch.sh`) on an owned branch profile.
+ S5 and S6 also run `host-layout-merge.sh inventory` and `verify` on their
+ Hosts.
- Desktop: `make test-e2e-825-desktop-continuity`
(`desktop-app/test/e2e-playwright/stateful-stateless-switch-continuity.spec.ts`)
keeps one conversation across stateful → stateless → stateful switches.
diff --git a/docs/deploy/host-pvc-layout-upgrade.md b/docs/deploy/host-pvc-layout-upgrade.md
new file mode 100644
index 000000000..cea246b3e
--- /dev/null
+++ b/docs/deploy/host-pvc-layout-upgrade.md
@@ -0,0 +1,228 @@
+# Host PVC Layout Upgrade Note
+
+This upgrade gives every Host one storage layout on its workspace PVC (#825):
+the PVC root is mounted at the workspace path and the conversation database
+is `<workspace>/state.db`, whatever the Host's lifecycle. The layout and the
+`workspace-layout-merge` init container are described in
+[Host Storage Layout](../architecture/host-storage-layout.md).
+
+## Who is affected
+
+Installations that run v0.10.x or v0.11.x and have created stateless Hosts.
+Those releases render `workspace-layout-init`, which splits a stateless
+Host's PVC into `state/` (the database) and `workspace/` (the agent files).
+After the upgrade, `workspace-layout-merge` moves a split tree back to the
+PVC root on the Host's next start. A Host that was always stateful keeps its
+tree unchanged.
+
+Desktop Hosts kept their history inside the container, outside the PVC.
+Copy `/workspace/state.db*` out of each desktop Host Pod
+(`kubectl --context <ctx> cp`) before the first rollout of this release, or
+that history is lost.
+
+No flag needs to be set for the upgrade: `CONTEXT_MAPPER_WORKSPACE_LAYOUT_MERGE`
+defaults to `true`, and none of the overlays in this repository set it.
+
+## The script
+
+[`deploy/scripts/host-layout-merge.sh`](../../deploy/scripts/host-layout-merge.sh)
+reports what the merge does or did to every Host PVC. It needs `bash`, `jq`
+and `kubectl`, and a context that can read the HCC Deployment, Hosts,
+Deployments, Pods and PVCs, and create and delete Pods in the Host namespace.
+Every call names `--context`; the current context is never used.
+
+- `inventory`: what the merge will do to each PVC. Run it before the upgrade.
+- `verify`: whether the flag can be turned off. Run it after the upgrade.
+
+Each PVC is read by a short-lived Pod that runs the merge script with the
+moves removed: Host image, uid/gid 1001, claim mounted read-only, no
+service-account token, deleted after its log is read. For a running Host the
+Pod is pinned to the Host Pod's node, because the claim is ReadWriteOnce. The
+script never opens a database and never execs into a Host. `--no-pvc-pods`
+skips the check Pods and reports every PVC as `UNVERIFIED`.
+
+Exit codes: `0` pass, `1` an observed blocker, `2` usage error, read failure,
+no Host and no Host PVC in the namespace, or a check Pod that could not be
+deleted.
+
+## Before the upgrade
+
+With the old release still running:
+
+```bash
+deploy/scripts/host-layout-merge.sh inventory --context <ctx>
+```
+
+`PRE_825=yes` is expected: the old Host Deployments render
+`workspace-layout-init` and `subPath` mounts. Each `PVC` line ends with a
+result:
+
+| Result | Meaning | Action |
+| ------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
+| `NO_SPLIT_SIGNATURE` | The tree does not carry the split-layout signature; the merge exits 0 and changes nothing | None |
+| `WILL_MERGE` | The merge moves the tree to the PVC root on the Host's next start | None |
+| `WILL_REFUSE_F<n> <line>` | The merge will refuse with that line and the Host will stay in `Init:Error` after the upgrade | See [Repairs](#repairs); counted in `UPGRADE_BLOCKERS` |
+| `UNREADABLE <reason>` | The check Pod could not read the PVC (pull error, timeout, unexpected output) | Fix the reason and run the command again |
+
+The command exits 1 while `UPGRADE_BLOCKERS` is above 0, while a PVC is
+`UNREADABLE`, or when the flag is set to an invalid value or to `false` with a
+split tree present. `DESKTOP_HOSTS` repeats the export warning above.
+
+A `NO_SPLIT_SIGNATURE` result says only that the merge will not touch the
+tree. It does not prove where the history is; see the trees without a
+signature in [Host Storage Layout](../architecture/host-storage-layout.md).
+
+## The upgrade
+
+Deploy the release with the flag unset or `"true"`. **Never set it to
+`"false"` in the upgrade deploy**: without the init, a split Host starts on an
+empty database at the PVC root and reports no error. `inventory` and `verify`
+print `FLAG_FALSE_WITH_SPLIT_TREES=<n>` and exit 1 in that state; set the
+flag back to `"true"` and the next start merges the tree.
+
+Running Hosts restart with the new template and merge on that start.
+Suspended Hosts merge on their next wake; the upgrade does not wake them.
+
+## After the upgrade
+
+```bash
+deploy/scripts/host-layout-merge.sh verify --context <ctx>
+```
+
+| Result | Meaning | Action |
+| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
+| `POD_OK` | The running Host Pod's merge init exited 0 | None |
+| `NO_SPLIT_SIGNATURE` | A Host without a Pod whose tree the merge will not touch | None |
+| `SPLIT` | A Host without a Pod whose tree is still split | Wake the Host (open one of its chats, or `POST /api/v1/rpc/hosts/<host>/wake` on rpc-proxy) |
+| `WILL_REFUSE_F<n> <line>` | A Host without a Pod whose next start will refuse | See [Repairs](#repairs) |
+| `MERGE_REFUSED <line>` | The running Host Pod is in `Init:Error` with that line | See [Repairs](#repairs) |
+| `MERGE_RUNNING` | The merge init has not finished | Run `verify` again |
+| `UNKNOWN`, `UNREADABLE`, `UNVERIFIED` | The script could not decide | Fix the reason and run `verify` again |
+
+`HOST_LAYOUT_VERIFY=PRE_825` (exit 1) means a Host Deployment still renders
+`workspace-layout-init` or a `subPath` mount: the upgrade has not reached it.
+A Host Deployment that does not render `workspace-layout-merge` counts in
+`template_stale` and fails the verdict; it means HCC has not reconciled that
+Host yet.
+
+`HOST_LAYOUT_VERIFY=PASS` (exit 0) when every Host Deployment renders the
+merge init and no Host PVC is split, refused, running, or undecided.
+`PVC_CONTENTS=NOT_INSPECTED` is a reminder that the script checks the layout,
+not the database contents.
+
+Orphan PVCs, whose `clerum.io/host` label names no Host, are listed with
+`kind=ORPHAN` and counted in `ORPHAN_SPLIT` when split. They do not block the
+verdict, because no Host will start on them. A Host created later with the
+same name would start on that PVC: with the flag off, on an empty database.
+Delete the orphan PVC, or recreate its Host before turning the flag off.
+
+## Repairs
+
+Every refusal exits before the first move, so the tree is as the old release
+left it. The Host Pod in `Init:Error` runs no `mcp-host`, so nothing writes to
+the PVC while you repair it. Work from a helper Pod on the Host Pod's node
+(the claim is ReadWriteOnce), with the Host image and the PVC mounted
+read-write:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: layout-repair-<host>
+ namespace: mcp-host
+spec:
+ nodeName: <node of the Host Pod>
+ restartPolicy: Never
+ automountServiceAccountToken: false
+ securityContext: { runAsUser: 1001, runAsGroup: 1001, runAsNonRoot: true }
+ containers:
+ - name: repair
+ image: <image of the Host's mcp-host container>
+ command: ['sleep', '3600']
+ volumeMounts: [{ name: pvc, mountPath: /mnt/pvc }]
+ volumes:
+ - name: pvc
+ persistentVolumeClaim: { claimName: <host>-workspace }
+```
+
+Rename, never delete, database files. Keep each database family together:
+rename `state.db`, `state.db-wal` and `state.db-shm` to
+`state.db.<label>.bak`, `state.db.<label>.bak-wal` and
+`state.db.<label>.bak-shm`, so SQLite still finds the journals when the
+backup is opened. Do not use `pre-` in the label: `state.db.pre-*.bak` names
+are part of the split-layout signature. Delete the helper Pod when done; the
+Host Pod retries the init on its own, or delete the Host Pod to retry at once.
+
+**A Host switched from stateless to stateful on v0.10 or v0.11
+(`WILL_REFUSE_F2`).** A Host created stateless and later switched to stateful
+(through the admin API, or by HCC when a CommunicationChannel, a desktop or a
+SharedFileSystem co-location rejected stateless) keeps its old history in
+`state/state.db` and writes new chats to a root `state.db`. The merge cannot
+choose between two live databases. Decide which history the Host serves and
+rename the other family:
+
+- keep the old history: rename the root family. The signature still holds and
+ the merge moves `state/` and `workspace/` to the root on the next start. A
+ file written at the root after the switch with the same name as one under
+ `workspace/` (for example `IDENTITY.md`) then refuses with F3; rename one of
+ the two;
+- keep the chats since the switch: rename the family under `state/`. Unless
+ `state/` also holds a `state.db*.pre-<ts>.bak` file, the tree no longer
+ carries the signature: the merge exits 0 and `state/` and `workspace/`
+ stay as folders in the agent's workspace, with the old files under
+ `workspace/`. Move them to the root yourself if the Host needs them.
+
+The renamed family stays on the PVC; agent file tools cannot read any
+`state.db*` name.
+
+**A name in `workspace/` that already exists at the root (`WILL_REFUSE_F3`).**
+This includes Hosts the old release already left in `Init:Error` with a
+personalization-file collision (#1101). Rename one of the two files.
+
+**A `state/` without a database next to `workspace/`.** A stateless Host that
+never started `mcp-host` has an empty `state/`. The tree has no signature, so
+the merge exits 0 and the files stay under `workspace/`, which the Host then
+sees as `<workspace>/workspace/`. Nothing is lost; move them to the root from
+a helper Pod if the Host needs them at their old paths.
+
+Every other refusal line is listed in
+[Host Storage Layout](../architecture/host-storage-layout.md#workspace-layout-merge).
+
+## Turning the flag off
+
+The merge is needed once per Host. Once `verify` passes, set
+`CONTEXT_MAPPER_WORKSPACE_LAYOUT_MERGE` to `"false"` in your own overlay
+(clerum: `keyper-labs/evenfire-infra`), as a strategic-merge patch on the HCC
+Deployment:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: host-context-controller
+ namespace: control-plane
+spec:
+ template:
+ spec:
+ containers:
+ - name: host-context-controller
+ env:
+ - name: CONTEXT_MAPPER_WORKSPACE_LAYOUT_MERGE
+ value: 'false'
+```
+
+- Turning it off changes HCC's Host template, so every running Host restarts
+ once.
+- `kubectl set env` works only until the next apply of the overlay.
+- On Minikube the overlays in this repository do not pin the flag, so a new
+ setup returns it to `true`.
+- After the turn-off, `verify` reports `HOST_LAYOUT_VERIFY=ALREADY_OFF`, or
+ `FLAG_FALSE_WITH_SPLIT_TREES` if a split tree appeared since.
+
+## Rollback
+
+Before rolling HCC back to v0.11.x or older, set the flag back to `"true"`
+(or remove the patch). The old init re-splits stateless trees, and only a
+roll-forward with the merge rendered brings them back to the root. The other
+rollback rules are in
+[Host Storage Layout](../architecture/host-storage-layout.md#rollback).
diff --git a/docs/deploy/production.md b/docs/deploy/production.md
index 097dc2d74..fba84a8c3 100644
--- a/docs/deploy/production.md
+++ b/docs/deploy/production.md
@@ -17,15 +17,16 @@ assets.
## Building blocks in this repo
-| Asset | Path |
-| -------------------------- | ------------------------------------------------------------------ |
-| CRDs (Helm) | [charts/clerum-crds](../../charts/clerum-crds/README.md) |
-| Base manifests | [deploy/base](../../deploy/base) |
-| Overlays | [deploy/overlays](../../deploy/overlays) |
-| Bootstrap scripts | [scripts/](../../scripts/), [deploy/scripts](../../deploy/scripts) |
-| Local full-stack reference | [minikube.md](minikube.md) |
-| Plugin Workload SDK upgrade | [plugin-workload-sdk-upgrade.md](plugin-workload-sdk-upgrade.md) |
-| Observability starter | [monitoring/](../../monitoring/README.md) |
+| Asset | Path |
+| --------------------------- | ------------------------------------------------------------------ |
+| CRDs (Helm) | [charts/clerum-crds](../../charts/clerum-crds/README.md) |
+| Base manifests | [deploy/base](../../deploy/base) |
+| Overlays | [deploy/overlays](../../deploy/overlays) |
+| Bootstrap scripts | [scripts/](../../scripts/), [deploy/scripts](../../deploy/scripts) |
+| Local full-stack reference | [minikube.md](minikube.md) |
+| Plugin Workload SDK upgrade | [plugin-workload-sdk-upgrade.md](plugin-workload-sdk-upgrade.md) |
+| Host PVC layout upgrade | [host-pvc-layout-upgrade.md](host-pvc-layout-upgrade.md) |
+| Observability starter | [monitoring/](../../monitoring/README.md) |
Install CRDs:
@@ -90,6 +91,7 @@ modified MPL-licensed files. See [LICENSE](../../LICENSE).
- [WorkflowRecipes operations](workflow-recipes-guide.md)
- [Plugin Workload SDK upgrade and policy migration](plugin-workload-sdk-upgrade.md)
+- [Host PVC layout upgrade from v0.10.x / v0.11.x](host-pvc-layout-upgrade.md)
- [Member invitations on self-hosted](../how-to/member-invitations-self-hosted.md)
- [Architecture / topology](../architecture/platform-topology.md)
- [FAQ](../faq.md)