Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

docs(storage): flag-off recovery, F3 listing refusal, verify coverage

Alfredo Lopezf2c4c4db946e
diff --git a/docs/architecture/host-storage-layout.md b/docs/architecture/host-storage-layout.md
index 454135ebc..657285399 100644
--- a/docs/architecture/host-storage-layout.md
+++ b/docs/architecture/host-storage-layout.md
@@ -138,12 +138,12 @@ keep them; the database and its journals are already at the root.
stderr and exits 1 before the first move, leaving the tree unchanged. The Pod
stays in `Init:Error`.
-| Case | Line contains |
-| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| F1: the PVC root is a symlink, not a directory, or resolves to `/`; `state/` or `workspace/` escapes the root or cannot be listed and searched; a database entry (`state.db`, `-wal`, `-shm`, `.pre-*.bak`) is a symlink or not a regular file; a directory that decides a collision cannot be listed | `directory is a symlink`, `is not a directory`, `escapes the workspace PVC root`, `state directory is not readable and searchable`, `workspace directory is not readable and searchable`, `SQLite file is a symlink`, `SQLite file is not a regular file`, `cannot list directory` |
-| F2: more than one live database family across the root, `state/` and `workspace/` | `more than one live SQLite database` |
-| F3: a name to move already exists at the root, or under both `state/` and `workspace/`; `workspace/lost+found` exists | `destination exists`, `exists under both state/ and workspace/`, `lost+found would remain` |
-| F4: the PVC root, `state/`, `workspace/`, or a directory to be moved out of them is not writable by the container uid | `not writable by uid` |
+| Case | Line contains |
+| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| F1: the PVC root is a symlink, not a directory, or resolves to `/`; `state/` or `workspace/` escapes the root or cannot be listed and searched; a database entry (`state.db`, `-wal`, `-shm`, `.pre-*.bak`) is a symlink or not a regular file | `directory is a symlink`, `is not a directory`, `escapes the workspace PVC root`, `state directory is not readable and searchable`, `workspace directory is not readable and searchable`, `SQLite file is a symlink`, `SQLite file is not a regular file` |
+| F2: more than one live database family across the root, `state/` and `workspace/` | `more than one live SQLite database` |
+| F3: a name to move already exists at the root, or under both `state/` and `workspace/`; a directory that decides such a collision cannot be listed; `workspace/lost+found` exists | `destination exists`, `exists under both state/ and workspace/`, `lost+found would remain`, `cannot list directory` |
+| F4: the PVC root, `state/`, `workspace/`, or a directory to be moved out of them is not writable by the container uid | `not writable by uid` |
The PVC root checks of F1 run on every start, before the signature check.
diff --git a/docs/deploy/host-pvc-layout-upgrade.md b/docs/deploy/host-pvc-layout-upgrade.md
index cea246b3e..16cd07794 100644
--- a/docs/deploy/host-pvc-layout-upgrade.md
+++ b/docs/deploy/host-pvc-layout-upgrade.md
@@ -57,15 +57,16 @@ deploy/scripts/host-layout-merge.sh inventory --context <ctx>
`workspace-layout-init` and `subPath` mounts. Each `PVC` line ends with a
result:
-| Result | Meaning | Action |
-| ------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
-| `NO_SPLIT_SIGNATURE` | The tree does not carry the split-layout signature; the merge exits 0 and changes nothing | None |
-| `WILL_MERGE` | The merge moves the tree to the PVC root on the Host's next start | None |
-| `WILL_REFUSE_F<n> <line>` | The merge will refuse with that line and the Host will stay in `Init:Error` after the upgrade | See [Repairs](#repairs); counted in `UPGRADE_BLOCKERS` |
-| `UNREADABLE <reason>` | The check Pod could not read the PVC (pull error, timeout, unexpected output) | Fix the reason and run the command again |
+| Result | Meaning | Action |
+| ------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
+| `NO_SPLIT_SIGNATURE` | The tree does not carry the split-layout signature; the merge exits 0 and changes nothing | None |
+| `WILL_MERGE` | The merge moves the tree to the PVC root on the Host's next start | None |
+| `WILL_REFUSE_F<n> <line>` | The merge will refuse with that line and the Host will stay in `Init:Error` after the upgrade | See [Repairs](#repairs); counted in `UPGRADE_BLOCKERS` |
+| `UNREADABLE <reason>` | The check Pod could not read the PVC (pull error, timeout, unexpected output) | Fix the reason and run the command again |
+| `UNKNOWN <reason>` | A Host's claim is missing from the PVC list, or a PVC labelled for a Host has another name | Find the claim the Host mounts and run the command again |
The command exits 1 while `UPGRADE_BLOCKERS` is above 0, while a PVC is
-`UNREADABLE`, or when the flag is set to an invalid value or to `false` with a
+`UNREADABLE` or `UNKNOWN`, when Hosts exist but no PVC was inspected, or when the flag is set to an invalid value or to `false` with a
split tree present. `DESKTOP_HOSTS` repeats the export warning above.
A `NO_SPLIT_SIGNATURE` result says only that the merge will not touch the
@@ -77,8 +78,13 @@ signature in [Host Storage Layout](../architecture/host-storage-layout.md).
Deploy the release with the flag unset or `"true"`. **Never set it to
`"false"` in the upgrade deploy**: without the init, a split Host starts on an
empty database at the PVC root and reports no error. `inventory` and `verify`
-print `FLAG_FALSE_WITH_SPLIT_TREES=<n>` and exit 1 in that state; set the
-flag back to `"true"` and the next start merges the tree.
+print `FLAG_FALSE_WITH_SPLIT_TREES=<n>` and exit 1 in that state. Set the
+flag back to `"true"` before the split Host starts: the next start then
+merges the tree. If the Host already started with the flag off, it created a
+root `state.db` next to `state/state.db`; with the flag back on, that start
+refuses with F2 (`more than one live SQLite database`). Repair it as a Host
+switched from stateless to stateful (see [Repairs](#repairs)): the root
+family holds only the chats since the flag-off start.
Running Hosts restart with the new template and merge on that start.
Suspended Hosts merge on their next wake; the upgrade does not wake them.
@@ -89,15 +95,15 @@ Suspended Hosts merge on their next wake; the upgrade does not wake them.
deploy/scripts/host-layout-merge.sh verify --context <ctx>
```
-| Result | Meaning | Action |
-| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
-| `POD_OK` | The running Host Pod's merge init exited 0 | None |
-| `NO_SPLIT_SIGNATURE` | A Host without a Pod whose tree the merge will not touch | None |
-| `SPLIT` | A Host without a Pod whose tree is still split | Wake the Host (open one of its chats, or `POST /api/v1/rpc/hosts/<host>/wake` on rpc-proxy) |
-| `WILL_REFUSE_F<n> <line>` | A Host without a Pod whose next start will refuse | See [Repairs](#repairs) |
-| `MERGE_REFUSED <line>` | The running Host Pod is in `Init:Error` with that line | See [Repairs](#repairs) |
-| `MERGE_RUNNING` | The merge init has not finished | Run `verify` again |
-| `UNKNOWN`, `UNREADABLE`, `UNVERIFIED` | The script could not decide | Fix the reason and run `verify` again |
+| Result | Meaning | Action |
+| ------------------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
+| `POD_OK` | The running Host Pod's merge init exited 0 and its tree carries no signature | None |
+| `NO_SPLIT_SIGNATURE` | A Host without a Pod whose tree the merge will not touch | None |
+| `SPLIT` | A Host whose tree is still split (with a running Pod, the tree was split again after its start) | Wake the Host (open one of its chats, or `POST /api/v1/rpc/hosts/<host>/wake` on rpc-proxy) |
+| `WILL_REFUSE_F<n> <line>` | A Host whose next start will refuse | See [Repairs](#repairs) |
+| `MERGE_REFUSED <line>` | The running Host Pod is in `Init:Error` with that line | See [Repairs](#repairs) |
+| `MERGE_RUNNING` | The merge init has not finished | Run `verify` again |
+| `UNKNOWN`, `UNREADABLE`, `UNVERIFIED` | The script could not decide | Fix the reason and run `verify` again |
`HOST_LAYOUT_VERIFY=PRE_825` (exit 1) means a Host Deployment still renders
`workspace-layout-init` or a `subPath` mount: the upgrade has not reached it.
@@ -110,9 +116,13 @@ merge init and no Host PVC is split, refused, running, or undecided.
`PVC_CONTENTS=NOT_INSPECTED` is a reminder that the script checks the layout,
not the database contents.
+`verify` reads the PVC of every Host, running or not; for a running Host the
+init exit code and the tree must both agree.
+
Orphan PVCs, whose `clerum.io/host` label names no Host, are listed with
-`kind=ORPHAN` and counted in `ORPHAN_SPLIT` when split. They do not block the
-verdict, because no Host will start on them. A Host created later with the
+`kind=ORPHAN` and counted in `ORPHAN_SPLIT` when split. A split orphan does not
+block the verdict, because no Host will start on it; an orphan the script
+could not read is counted in `ORPHAN_UNDECIDED` and blocks it. A Host created later with the
same name would start on that PVC: with the flag off, on an empty database.
Delete the orphan PVC, or recreate its Host before turning the flag off.
@@ -216,8 +226,9 @@ spec:
- `kubectl set env` works only until the next apply of the overlay.
- On Minikube the overlays in this repository do not pin the flag, so a new
setup returns it to `true`.
-- After the turn-off, `verify` reports `HOST_LAYOUT_VERIFY=ALREADY_OFF`, or
- `FLAG_FALSE_WITH_SPLIT_TREES` if a split tree appeared since.
+- After the turn-off, `verify` reports `HOST_LAYOUT_VERIFY=ALREADY_OFF`,
+ `FLAG_FALSE_WITH_SPLIT_TREES` if a split tree appeared since, or `FAIL` if
+ a PVC could not be read.
## Rollback
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index 83acb0740..07fa8d82b 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.398",
+ "version": "0.1.399",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.398",
+ "version": "0.1.399",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/codex-catalog-projection": "file:../packages/codex-catalog-projection",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index 8d8feed1c..608899981 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.398",
+ "version": "0.1.399",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/statelessDeployment.ts b/host-context-controller/src/statelessDeployment.ts
index 3641a75ec..30d57b64b 100644
--- a/host-context-controller/src/statelessDeployment.ts
+++ b/host-context-controller/src/statelessDeployment.ts
@@ -39,15 +39,15 @@ function shellSingleQuote(value: string): string {
* so they must not stop it).
*
* Every refusal prints one line and exits 1 before the first move:
- * F1 `state/` or `workspace/` cannot be listed and searched, a database
- * entry is a symlink or not a regular file, or a directory that
- * decides a collision cannot be listed;
+ * F1 `state/` or `workspace/` cannot be listed and searched, or a database
+ * entry is a symlink or not a regular file;
* F2 more than one live database (a `state.db` main file or its journals)
* across the root, `state/` and `workspace/`, except the root journals
* a stopped merge leaves next to the remaining `state/state.db`;
* F3 a name to move already exists at the root, or exists under both
* `state/` and `workspace/` (an empty source directory is removed
- * instead), or `workspace/lost+found` would keep `workspace/` alive;
+ * instead), a directory that decides such a collision cannot be listed,
+ * or `workspace/lost+found` would keep `workspace/` alive;
* F4 the root, `state/`, `workspace/`, or a directory to be moved out of
* them is not writable (renaming a directory rewrites its `..`).
* Moves: `workspace/*`, then `state/*` except `state.db`, then