Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
docs(deploy): host PVC layout upgrade note
Alfredo Lopez24aa8d56e5b5
diff --git a/docs/README.md b/docs/README.mdindex c1f1af323..e290b42c3 100644--- a/docs/README.md+++ b/docs/README.md@@ -64,6 +64,7 @@ Use this index for long-form docs.| [Shared & global files](how-to/shared-and-global-files.md) | Team workspaces (SFS) and the brokered drive (GFS) || [Minikube full stack](deploy/minikube.md) | Local Kubernetes platform || [Production notes](deploy/production.md) | Checklist and in-repo deploy assets |+| [Host PVC layout upgrade](deploy/host-pvc-layout-upgrade.md) | Upgrading stateless Hosts from v0.10.x / v0.11.x || [WorkflowRecipes operations](deploy/workflow-recipes-guide.md) | Recipe ops, RBAC, debugging |## Referencediff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.mdindex cafe9c06a..454135ebc 100644--- a/docs/architecture/host-storage-layout.md+++ b/docs/architecture/host-storage-layout.md@@ -92,7 +92,11 @@ Two agent-made folder pairs on a stateful Host do stop it from starting:- a `state/` holding `state.db` or one of its journals beside the rootdatabase, next to a `workspace/` folder. The signature holds and F2refuses, because the merge cannot tell the agent's file from a second- database;+ database. The earlier release produces the same tree without any agent: a+ Host created stateless and switched to stateful on v0.10 or v0.11 keeps its+ old history in `state/state.db` and writes new chats to a root `state.db`.+ `host-layout-merge.sh inventory` reports it as `WILL_REFUSE_F2` before the+ upgrade, and the upgrade note describes the repair;- a `state/` the container uid cannot list and search, next to a`workspace/` folder. F1 refuses before the database check, because themerge cannot tell whether it holds a database.@@ -180,12 +184,18 @@ renames the entry.and no `workspace-layout-init`;2. every running Host Pod's `workspace-layout-merge` exited 0;3. no Host is in `Init:Error` in `workspace-layout-merge`;- 4. every Host that was suspended at deploy time has woken once with that- init exiting 0.-- The turn-off is an overlay patch in `keyper-labs/evenfire-infra`- (`patches/hcc-workspace-layout-merge.yaml`, value `"false"`), never in- `dynamic-images.yaml`. Turning it off changes the pod template, so every- running Host restarts once.+ 4. no Host PVC still carries the split-layout signature.++ [`deploy/scripts/host-layout-merge.sh verify`](../../deploy/scripts/host-layout-merge.sh)+ checks all four and reads each PVC without a Pod from a read-only check+ Pod; it prints `HOST_LAYOUT_VERIFY=PASS` only when they hold. The procedure+ for any installation is in+ [Host PVC Layout Upgrade Note](../deploy/host-pvc-layout-upgrade.md).++- The turn-off is a patch in the installation's own overlay (clerum:+ `keyper-labs/evenfire-infra`, `patches/hcc-workspace-layout-merge.yaml`,+ value `"false"`), never in `dynamic-images.yaml`. Turning it off changes the+ pod template, so every running Host restarts once.- Never turn it off in the deploy that first introduces this HCC.- Before any manual HCC undo to a revision older than #825's, set it back to`true`. The older init re-splits stateless trees, and only a roll-forward@@ -300,8 +310,15 @@ forced by a deploy.at least one case of each refusal family (F1–F4), re-runs after aninterrupted merge and a round trip through the previous init(`test/__fixtures__/base-workspace-layout-init.sh`).+- Script: `deploy/scripts/host-layout-merge.test.mjs` runs+ `host-layout-merge.sh` against a fake `kubectl`, and runs its check script+ on real trees. The HCC merge suite above also checks that the check script+ predicts the merge's outcome (merge, no-op or the refusal line) on every+ tree it tests.- Live: `make test-e2e-hcc-storage-layout-switch`(`scripts/e2e/e2e-hcc-storage-layout-switch.sh`) on an owned branch profile.+ S5 and S6 also run `host-layout-merge.sh inventory` and `verify` on their+ Hosts.- Desktop: `make test-e2e-825-desktop-continuity`(`desktop-app/test/e2e-playwright/stateful-stateless-switch-continuity.spec.ts`)keeps one conversation across stateful → stateless → stateful switches.diff --git a/docs/deploy/host-pvc-layout-upgrade.md b/docs/deploy/host-pvc-layout-upgrade.mdnew file mode 100644index 000000000..cea246b3e--- /dev/null+++ b/docs/deploy/host-pvc-layout-upgrade.md@@ -0,0 +1,228 @@+# Host PVC Layout Upgrade Note++This upgrade gives every Host one storage layout on its workspace PVC (#825):+the PVC root is mounted at the workspace path and the conversation database+is `<workspace>/state.db`, whatever the Host's lifecycle. The layout and the+`workspace-layout-merge` init container are described in+[Host Storage Layout](../architecture/host-storage-layout.md).++## Who is affected++Installations that run v0.10.x or v0.11.x and have created stateless Hosts.+Those releases render `workspace-layout-init`, which splits a stateless+Host's PVC into `state/` (the database) and `workspace/` (the agent files).+After the upgrade, `workspace-layout-merge` moves a split tree back to the+PVC root on the Host's next start. A Host that was always stateful keeps its+tree unchanged.++Desktop Hosts kept their history inside the container, outside the PVC.+Copy `/workspace/state.db*` out of each desktop Host Pod+(`kubectl --context <ctx> cp`) before the first rollout of this release, or+that history is lost.++No flag needs to be set for the upgrade: `CONTEXT_MAPPER_WORKSPACE_LAYOUT_MERGE`+defaults to `true`, and none of the overlays in this repository set it.++## The script++[`deploy/scripts/host-layout-merge.sh`](../../deploy/scripts/host-layout-merge.sh)+reports what the merge does or did to every Host PVC. It needs `bash`, `jq`+and `kubectl`, and a context that can read the HCC Deployment, Hosts,+Deployments, Pods and PVCs, and create and delete Pods in the Host namespace.+Every call names `--context`; the current context is never used.++- `inventory`: what the merge will do to each PVC. Run it before the upgrade.+- `verify`: whether the flag can be turned off. Run it after the upgrade.++Each PVC is read by a short-lived Pod that runs the merge script with the+moves removed: Host image, uid/gid 1001, claim mounted read-only, no+service-account token, deleted after its log is read. For a running Host the+Pod is pinned to the Host Pod's node, because the claim is ReadWriteOnce. The+script never opens a database and never execs into a Host. `--no-pvc-pods`+skips the check Pods and reports every PVC as `UNVERIFIED`.++Exit codes: `0` pass, `1` an observed blocker, `2` usage error, read failure,+no Host and no Host PVC in the namespace, or a check Pod that could not be+deleted.++## Before the upgrade++With the old release still running:++```bash+deploy/scripts/host-layout-merge.sh inventory --context <ctx>+```++`PRE_825=yes` is expected: the old Host Deployments render+`workspace-layout-init` and `subPath` mounts. Each `PVC` line ends with a+result:++| Result | Meaning | Action |+| ------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------ |+| `NO_SPLIT_SIGNATURE` | The tree does not carry the split-layout signature; the merge exits 0 and changes nothing | None |+| `WILL_MERGE` | The merge moves the tree to the PVC root on the Host's next start | None |+| `WILL_REFUSE_F<n> <line>` | The merge will refuse with that line and the Host will stay in `Init:Error` after the upgrade | See [Repairs](#repairs); counted in `UPGRADE_BLOCKERS` |+| `UNREADABLE <reason>` | The check Pod could not read the PVC (pull error, timeout, unexpected output) | Fix the reason and run the command again |++The command exits 1 while `UPGRADE_BLOCKERS` is above 0, while a PVC is+`UNREADABLE`, or when the flag is set to an invalid value or to `false` with a+split tree present. `DESKTOP_HOSTS` repeats the export warning above.++A `NO_SPLIT_SIGNATURE` result says only that the merge will not touch the+tree. It does not prove where the history is; see the trees without a+signature in [Host Storage Layout](../architecture/host-storage-layout.md).++## The upgrade++Deploy the release with the flag unset or `"true"`. **Never set it to+`"false"` in the upgrade deploy**: without the init, a split Host starts on an+empty database at the PVC root and reports no error. `inventory` and `verify`+print `FLAG_FALSE_WITH_SPLIT_TREES=<n>` and exit 1 in that state; set the+flag back to `"true"` and the next start merges the tree.++Running Hosts restart with the new template and merge on that start.+Suspended Hosts merge on their next wake; the upgrade does not wake them.++## After the upgrade++```bash+deploy/scripts/host-layout-merge.sh verify --context <ctx>+```++| Result | Meaning | Action |+| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------- |+| `POD_OK` | The running Host Pod's merge init exited 0 | None |+| `NO_SPLIT_SIGNATURE` | A Host without a Pod whose tree the merge will not touch | None |+| `SPLIT` | A Host without a Pod whose tree is still split | Wake the Host (open one of its chats, or `POST /api/v1/rpc/hosts/<host>/wake` on rpc-proxy) |+| `WILL_REFUSE_F<n> <line>` | A Host without a Pod whose next start will refuse | See [Repairs](#repairs) |+| `MERGE_REFUSED <line>` | The running Host Pod is in `Init:Error` with that line | See [Repairs](#repairs) |+| `MERGE_RUNNING` | The merge init has not finished | Run `verify` again |+| `UNKNOWN`, `UNREADABLE`, `UNVERIFIED` | The script could not decide | Fix the reason and run `verify` again |++`HOST_LAYOUT_VERIFY=PRE_825` (exit 1) means a Host Deployment still renders+`workspace-layout-init` or a `subPath` mount: the upgrade has not reached it.+A Host Deployment that does not render `workspace-layout-merge` counts in+`template_stale` and fails the verdict; it means HCC has not reconciled that+Host yet.++`HOST_LAYOUT_VERIFY=PASS` (exit 0) when every Host Deployment renders the+merge init and no Host PVC is split, refused, running, or undecided.+`PVC_CONTENTS=NOT_INSPECTED` is a reminder that the script checks the layout,+not the database contents.++Orphan PVCs, whose `clerum.io/host` label names no Host, are listed with+`kind=ORPHAN` and counted in `ORPHAN_SPLIT` when split. They do not block the+verdict, because no Host will start on them. A Host created later with the+same name would start on that PVC: with the flag off, on an empty database.+Delete the orphan PVC, or recreate its Host before turning the flag off.++## Repairs++Every refusal exits before the first move, so the tree is as the old release+left it. The Host Pod in `Init:Error` runs no `mcp-host`, so nothing writes to+the PVC while you repair it. Work from a helper Pod on the Host Pod's node+(the claim is ReadWriteOnce), with the Host image and the PVC mounted+read-write:++```yaml+apiVersion: v1+kind: Pod+metadata:+ name: layout-repair-<host>+ namespace: mcp-host+spec:+ nodeName: <node of the Host Pod>+ restartPolicy: Never+ automountServiceAccountToken: false+ securityContext: { runAsUser: 1001, runAsGroup: 1001, runAsNonRoot: true }+ containers:+ - name: repair+ image: <image of the Host's mcp-host container>+ command: ['sleep', '3600']+ volumeMounts: [{ name: pvc, mountPath: /mnt/pvc }]+ volumes:+ - name: pvc+ persistentVolumeClaim: { claimName: <host>-workspace }+```++Rename, never delete, database files. Keep each database family together:+rename `state.db`, `state.db-wal` and `state.db-shm` to+`state.db.<label>.bak`, `state.db.<label>.bak-wal` and+`state.db.<label>.bak-shm`, so SQLite still finds the journals when the+backup is opened. Do not use `pre-` in the label: `state.db.pre-*.bak` names+are part of the split-layout signature. Delete the helper Pod when done; the+Host Pod retries the init on its own, or delete the Host Pod to retry at once.++**A Host switched from stateless to stateful on v0.10 or v0.11+(`WILL_REFUSE_F2`).** A Host created stateless and later switched to stateful+(through the admin API, or by HCC when a CommunicationChannel, a desktop or a+SharedFileSystem co-location rejected stateless) keeps its old history in+`state/state.db` and writes new chats to a root `state.db`. The merge cannot+choose between two live databases. Decide which history the Host serves and+rename the other family:++- keep the old history: rename the root family. The signature still holds and+ the merge moves `state/` and `workspace/` to the root on the next start. A+ file written at the root after the switch with the same name as one under+ `workspace/` (for example `IDENTITY.md`) then refuses with F3; rename one of+ the two;+- keep the chats since the switch: rename the family under `state/`. Unless+ `state/` also holds a `state.db*.pre-<ts>.bak` file, the tree no longer+ carries the signature: the merge exits 0 and `state/` and `workspace/`+ stay as folders in the agent's workspace, with the old files under+ `workspace/`. Move them to the root yourself if the Host needs them.++The renamed family stays on the PVC; agent file tools cannot read any+`state.db*` name.++**A name in `workspace/` that already exists at the root (`WILL_REFUSE_F3`).**+This includes Hosts the old release already left in `Init:Error` with a+personalization-file collision (#1101). Rename one of the two files.++**A `state/` without a database next to `workspace/`.** A stateless Host that+never started `mcp-host` has an empty `state/`. The tree has no signature, so+the merge exits 0 and the files stay under `workspace/`, which the Host then+sees as `<workspace>/workspace/`. Nothing is lost; move them to the root from+a helper Pod if the Host needs them at their old paths.++Every other refusal line is listed in+[Host Storage Layout](../architecture/host-storage-layout.md#workspace-layout-merge).++## Turning the flag off++The merge is needed once per Host. Once `verify` passes, set+`CONTEXT_MAPPER_WORKSPACE_LAYOUT_MERGE` to `"false"` in your own overlay+(clerum: `keyper-labs/evenfire-infra`), as a strategic-merge patch on the HCC+Deployment:++```yaml+apiVersion: apps/v1+kind: Deployment+metadata:+ name: host-context-controller+ namespace: control-plane+spec:+ template:+ spec:+ containers:+ - name: host-context-controller+ env:+ - name: CONTEXT_MAPPER_WORKSPACE_LAYOUT_MERGE+ value: 'false'+```++- Turning it off changes HCC's Host template, so every running Host restarts+ once.+- `kubectl set env` works only until the next apply of the overlay.+- On Minikube the overlays in this repository do not pin the flag, so a new+ setup returns it to `true`.+- After the turn-off, `verify` reports `HOST_LAYOUT_VERIFY=ALREADY_OFF`, or+ `FLAG_FALSE_WITH_SPLIT_TREES` if a split tree appeared since.++## Rollback++Before rolling HCC back to v0.11.x or older, set the flag back to `"true"`+(or remove the patch). The old init re-splits stateless trees, and only a+roll-forward with the merge rendered brings them back to the root. The other+rollback rules are in+[Host Storage Layout](../architecture/host-storage-layout.md#rollback).diff --git a/docs/deploy/production.md b/docs/deploy/production.mdindex 097dc2d74..fba84a8c3 100644--- a/docs/deploy/production.md+++ b/docs/deploy/production.md@@ -17,15 +17,16 @@ assets.## Building blocks in this repo-| Asset | Path |-| -------------------------- | ------------------------------------------------------------------ |-| CRDs (Helm) | [charts/clerum-crds](../../charts/clerum-crds/README.md) |-| Base manifests | [deploy/base](../../deploy/base) |-| Overlays | [deploy/overlays](../../deploy/overlays) |-| Bootstrap scripts | [scripts/](../../scripts/), [deploy/scripts](../../deploy/scripts) |-| Local full-stack reference | [minikube.md](minikube.md) |-| Plugin Workload SDK upgrade | [plugin-workload-sdk-upgrade.md](plugin-workload-sdk-upgrade.md) |-| Observability starter | [monitoring/](../../monitoring/README.md) |+| Asset | Path |+| --------------------------- | ------------------------------------------------------------------ |+| CRDs (Helm) | [charts/clerum-crds](../../charts/clerum-crds/README.md) |+| Base manifests | [deploy/base](../../deploy/base) |+| Overlays | [deploy/overlays](../../deploy/overlays) |+| Bootstrap scripts | [scripts/](../../scripts/), [deploy/scripts](../../deploy/scripts) |+| Local full-stack reference | [minikube.md](minikube.md) |+| Plugin Workload SDK upgrade | [plugin-workload-sdk-upgrade.md](plugin-workload-sdk-upgrade.md) |+| Host PVC layout upgrade | [host-pvc-layout-upgrade.md](host-pvc-layout-upgrade.md) |+| Observability starter | [monitoring/](../../monitoring/README.md) |Install CRDs:@@ -90,6 +91,7 @@ modified MPL-licensed files. See [LICENSE](../../LICENSE).- [WorkflowRecipes operations](workflow-recipes-guide.md)- [Plugin Workload SDK upgrade and policy migration](plugin-workload-sdk-upgrade.md)+- [Host PVC layout upgrade from v0.10.x / v0.11.x](host-pvc-layout-upgrade.md)- [Member invitations on self-hosted](../how-to/member-invitations-self-hosted.md)- [Architecture / topology](../architecture/platform-topology.md)- [FAQ](../faq.md)