Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
test(e2e): repair T5 before the upgrade in the upgrade lane
Alfredo Lopezf028661664fa
diff --git a/docs/deploy/host-pvc-layout-upgrade.md b/docs/deploy/host-pvc-layout-upgrade.mdindex 16cd07794..854881011 100644--- a/docs/deploy/host-pvc-layout-upgrade.md+++ b/docs/deploy/host-pvc-layout-upgrade.md@@ -69,6 +69,12 @@ The command exits 1 while `UPGRADE_BLOCKERS` is above 0, while a PVC is`UNREADABLE` or `UNKNOWN`, when Hosts exist but no PVC was inspected, or when the flag is set to an invalid value or to `false` with asplit tree present. `DESKTOP_HOSTS` repeats the export warning above.+Repair every `WILL_REFUSE_*` before the upgrade and run `inventory` again+until `UPGRADE_BLOCKERS=0`. A refused Host stays in `Init:Error` after the+upgrade, and a deploy that waits for Host rollouts fails on it (the minikube+setup does: `scripts/minikube/sync-auth-key.sh` waits for every Host+Deployment that is not suspended).+A `NO_SPLIT_SIGNATURE` result says only that the merge will not touch thetree. It does not prove where the history is; see the trees without asignature in [Host Storage Layout](../architecture/host-storage-layout.md).@@ -129,8 +135,10 @@ Delete the orphan PVC, or recreate its Host before turning the flag off.## RepairsEvery refusal exits before the first move, so the tree is as the old release-left it. The Host Pod in `Init:Error` runs no `mcp-host`, so nothing writes to-the PVC while you repair it. Work from a helper Pod on the Host Pod's node+left it. Repair before the upgrade when the files you rename are not the+database the running Host opens; otherwise repair after the upgrade, when the+Host Pod is in `Init:Error` and runs no `mcp-host`, so nothing writes to the+PVC. Work from a helper Pod on the Host Pod's node(the claim is ReadWriteOnce), with the Host image and the PVC mountedread-write:@@ -171,12 +179,15 @@ SharedFileSystem co-location rejected stateless) keeps its old history inchoose between two live databases. Decide which history the Host serves andrename the other family:-- keep the old history: rename the root family. The signature still holds and+- keep the old history: rename the root family. On the old release the Host+ runs stateful on that family, so do this after the upgrade, on the+ `Init:Error` Pod. The signature still holds andthe merge moves `state/` and `workspace/` to the root on the next start. Afile written at the root after the switch with the same name as one under`workspace/` (for example `IDENTITY.md`) then refuses with F3; rename one ofthe two;-- keep the chats since the switch: rename the family under `state/`. Unless+- keep the chats since the switch: rename the family under `state/`. The+ running Host does not open it, so do this before the upgrade. Unless`state/` also holds a `state.db*.pre-<ts>.bak` file, the tree no longercarries the signature: the merge exits 0 and `state/` and `workspace/`stay as folders in the agent's workspace, with the old files underdiff --git a/scripts/e2e/e2e-hcc-layout-upgrade.sh b/scripts/e2e/e2e-hcc-layout-upgrade.shindex 07897e277..f354db9f6 100755--- a/scripts/e2e/e2e-hcc-layout-upgrade.sh+++ b/scripts/e2e/e2e-hcc-layout-upgrade.sh@@ -7,9 +7,9 @@# the profile between the two modes.## main installs v0.11.0 (published images), seeds one Host per row, runs-# deploy/scripts/host-layout-merge.sh inventory, upgrades to this-# head (images built from this tree, database kept), then checks-# verify, repairs T5 with the upgrade-note procedure and checks the+# deploy/scripts/host-layout-merge.sh inventory, repairs T5 with the+# upgrade-note procedure (plan A7.44), upgrades to this head (images+# built from this tree, database kept), then checks verify and the# history of every row. Rows:# T1 stateful;# T2 stateless, active at the upgrade;@@ -35,8 +35,8 @@## Cluster writes besides the setups and the admin API: the named fixtures of# e2e-hcc-storage-layout-switch.sh (hcc-cadence, pvc-fixture, channel,-# host-layout-merge), the T5 repair (a pvc-fixture rename plus the deletion of-# the refused Host Pod, as the upgrade note says) and, in control mode,+# host-layout-merge), the T5 repair (a pvc-fixture rename on the release, as the+# upgrade note says) and, in control mode,# FAULT:flag-false (kubectl set env on the HCC Deployment, undone in the# trap). No kubectl exec into Host Pods, no SQLite reads or writes.set -euo pipefail@@ -341,20 +341,6 @@ wait_merge_templates() {return 1}-# wait_refused_pod HOST → prints the Pod whose merge init exited 1.-wait_refused_pod() {- local host=$1 elapsed=0 pod- while [ "$elapsed" -lt "$POD_READY_TIMEOUT" ]; do- pod="$(host_pods "$host" | jq -r '[.items[] | select(any(.status.initContainerStatuses[]?;- .name == "workspace-layout-merge" and ((.state.terminated.exitCode // 0) == 1 or (.lastState.terminated.exitCode // 0) == 1)))][0].metadata.name // ""')"- [ -z "$pod" ] || { printf '%s' "$pod"; return 0; }- sleep 3- elapsed=$((elapsed + 3))- done- echo "${host}: no Pod with workspace-layout-merge exit 1 within ${POD_READY_TIMEOUT}s" >&2- return 1-}-# run_layout CMD: one run of the operator script; output in LAYOUT_OUT, exit in LAYOUT_RC.run_layout() {LAYOUT_OUT="${SCRATCH}/layout-$1-${SECONDS}.out"@@ -420,39 +406,45 @@ predict_rows() {expect_row "$HOST_T1" NO_SPLIT_SIGNATURE 'inventory T1'}+# The upgrade note's T5 repair, keeping the chats since the switch, on the+# release: the running stateful Pod opens only the root database, so the+# state/ family is renamed while it serves. Repaired before the upgrade+# because make minikube-setup waits for every Host rollout, and a Host left in+# Init:Error by the merge fails it (sync-auth-key.sh).+repair_blockers() {+ log "REPAIR:${HOST_T5} rename the state/ database family (upgrade note)"+ pvc_fixture "$HOST_T5" rw repair "$REPAIR_STATE_FAMILY_SCRIPT" >/dev/null || return 1+ run_layout inventory+ if [ "$LAYOUT_RC" != 0 ] || ! grep -qx 'UPGRADE_BLOCKERS=0' "$LAYOUT_OUT"; then+ fail "inventory after the repair exited ${LAYOUT_RC}: $(grep UPGRADE_BLOCKERS "$LAYOUT_OUT")"+ return 1+ fi+ expect_row "$HOST_T5" NO_SPLIT_SIGNATURE 'inventory after repair T5'+}+check_upgraded() {- local pod logs host refused_uid+ local hostwait_merge_templates "$HOST_T1" "$HOST_T2" "$HOST_T3" "$HOST_T5" "$HOST_T6" || return 1wait_upgraded_pod "$HOST_T1" && assert_pod "$HOST_T1" false /workspace || return 1wait_upgraded_pod "$HOST_T2" && assert_pod "$HOST_T2" true /workspace || return 1+ wait_upgraded_pod "$HOST_T5" && assert_pod "$HOST_T5" false /workspace || return 1wait_upgraded_pod "$HOST_T6" && assert_pod "$HOST_T6" true /workspace || return 1- pod="$(wait_refused_pod "$HOST_T5")" || { fail 'T5: the merge did not refuse'; return 1; }- logs="$(kctl logs "$pod" -n "$HOST_NS" -c workspace-layout-merge --previous 2>/dev/null ||- kctl logs "$pod" -n "$HOST_NS" -c workspace-layout-merge)"- grep -q "$F2" <<<"$logs" || { fail "T5: merge refused without the F2 line: ${logs}"; return 1; }[ "$(kctl get host "$HOST_T3" -n "$HOST_NS" -o jsonpath='{.status.lifecycle.state}')" = suspended ] ||{ fail 'T3: no longer suspended before verify'; return 1; }run_layout verify- [ "$LAYOUT_RC" = 1 ] || { fail "verify exited ${LAYOUT_RC}, expected 1 (T3 split, T5 refused)"; return 1; }- for host in "$HOST_T1" "$HOST_T2" "$HOST_T6"; do expect_row "$host" POD_OK 'verify' || return 1; done- expect_row "$HOST_T3" SPLIT 'verify T3' && expect_row "$HOST_T5" "MERGE_REFUSED ${F2}*" 'verify T5' || return 1+ [ "$LAYOUT_RC" = 1 ] || { fail "verify exited ${LAYOUT_RC}, expected 1 (T3 split)"; return 1; }+ for host in "$HOST_T1" "$HOST_T2" "$HOST_T5" "$HOST_T6"; do expect_row "$host" POD_OK 'verify' || return 1; done+ expect_row "$HOST_T3" SPLIT 'verify T3' || return 1# T3 merges on its natural wake.post_wake "$HOST_T3" && wait_new_ready_pod "$HOST_T3" '' "$POD_READY_TIMEOUT" && assert_pod "$HOST_T3" true /workspace || return 1- # T5: the upgrade note's repair, keeping the chats since the switch.- log "REPAIR:${HOST_T5} rename the state/ database family (upgrade note)"- pvc_fixture "$HOST_T5" rw repair "$REPAIR_STATE_FAMILY_SCRIPT" >/dev/null || return 1- refused_uid="$(kctl get pod "$pod" -n "$HOST_NS" -o jsonpath='{.metadata.uid}')" || { fail "T5: could not read refused Pod ${pod}"; return 1; }- kctl delete pod "$pod" -n "$HOST_NS" --wait=true --timeout=60s >/dev/null || { fail "T5: delete of refused Pod ${pod} failed"; return 1; }- wait_new_ready_pod "$HOST_T5" "$refused_uid" "$POD_READY_TIMEOUT" &&- assert_pod "$HOST_T5" false /workspace || return 1run_layout verifyif [ "$LAYOUT_RC" != 0 ] || ! grep -qx 'HOST_LAYOUT_VERIFY=PASS' "$LAYOUT_OUT"; then- fail "verify after the repairs exited ${LAYOUT_RC}: $(tail -n 3 "$LAYOUT_OUT" | tr '\n' ' ')"+ fail "verify after the T3 wake exited ${LAYOUT_RC}: $(tail -n 3 "$LAYOUT_OUT" | tr '\n' ' ')"return 1fi- for host in "$HOST_T1" "$HOST_T2" "$HOST_T3" "$HOST_T5" "$HOST_T6"; do expect_row "$host" POD_OK 'verify after repair' || return 1; done+ for host in "$HOST_T1" "$HOST_T2" "$HOST_T3" "$HOST_T5" "$HOST_T6"; do expect_row "$host" POD_OK 'verify after wake' || return 1; done}check_history() {@@ -465,7 +457,7 @@ check_history() {ROWS_DONE=$((ROWS_DONE + 1))# T5 serves the root family it served before the upgrade; the old family is# kept as a backup, and its old files stay under workspace/ (no signature).- assert_history "$HOST_T5" "upgrade(two-databases)->repair" || return 1+ assert_history "$HOST_T5" "repair(two-databases)->upgrade" || return 1listing="$(pvc_fixture "$HOST_T5" ro list-repaired "$LIST_SCRIPT")" || return 1grep -qx 'f ./state/state.db.upgrade-lane.bak' <<<"$listing" ||{ fail "T5: the renamed family is missing: ${listing}"; return 1; }@@ -481,8 +473,9 @@ run_main() {header "Install ${RELEASE}" && install_release &&header 'Seed rows on the release' && seed_rows &&header 'Predict with inventory' && predict_rows &&+ header 'Repair blockers on the release' && repair_blockers &&header 'Upgrade to this head' && upgrade_to_head &&- header 'Verify and repair' && check_upgraded &&+ header 'Verify and wake' && check_upgraded &&header 'History of every row' && check_history}diff --git a/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs b/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjsindex 289ee4958..7ae63d249 100644--- a/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs+++ b/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs@@ -1,6 +1,6 @@import assert from 'node:assert/strict'import { spawnSync } from 'node:child_process'-import { mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'+import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'import { tmpdir } from 'node:os'import path from 'node:path'import test from 'node:test'@@ -414,3 +414,41 @@ test('the upgrade lane installs the release deploy tree and checks the CRDs', ()// The manifest-diff refusal is gone (A7.43).assert.doesNotMatch(SOURCE, /this tree no longer installs the release/)})++// A7.44: make minikube-setup waits for every Host rollout, so a Host left in+// Init:Error by the upgrade fails it. The lane repairs T5 on the release.+test('the upgrade lane repairs blockers before it upgrades', () => {+ const main = helper('run_main')+ const repair = main.indexOf('repair_blockers')+ const upgrade = main.indexOf('upgrade_to_head')+ assert.notEqual(repair, -1, 'run_main does not call repair_blockers')+ assert.notEqual(upgrade, -1, 'run_main does not call upgrade_to_head')+ assert.ok(repair < upgrade, 'repair_blockers runs after upgrade_to_head')+ assert.match(helper('repair_blockers'), /grep -qx 'UPGRADE_BLOCKERS=0'/)+ assert.doesNotMatch(helper('check_upgraded'), /wait_refused_pod|REPAIR_STATE_FAMILY_SCRIPT/)+})++test('the lane T5 repair leaves a tree the merge does not touch', () => {+ const repairScript = SOURCE.match(/^REPAIR_STATE_FAMILY_SCRIPT='([^']*)'/m)+ assert.ok(repairScript, 'REPAIR_STATE_FAMILY_SCRIPT not found')+ const layoutScript = fileURLToPath(new URL('../../deploy/scripts/host-layout-merge.sh', import.meta.url))+ const root = mkdtempSync(path.join(tmpdir(), 'e2e-825-t5-repair-'))+ try {+ for (const p of ['state', 'workspace']) mkdirSync(path.join(root, p))+ for (const p of ['state.db', 'state/state.db', 'state/state.db-wal', 'workspace/seed.txt']) writeFileSync(path.join(root, p), p)+ const classify = () => {+ const printed = spawnSync('bash', [layoutScript, 'check-script', '--root', root], { encoding: 'utf8' })+ assert.equal(printed.status, 0, printed.stderr)+ return spawnSync('/bin/sh', ['-c', printed.stdout], { encoding: 'utf8' }).stdout+ }+ // Witness: the seeded tree is the T5 blocker.+ assert.match(classify(), /^WILL_REFUSE_F2 /)+ const repaired = spawnSync('/bin/sh', ['-c', repairScript[1], 'repair', root], { encoding: 'utf8' })+ assert.equal(repaired.status, 0, repaired.stderr)+ assert.equal(classify(), 'NO_SPLIT_SIGNATURE\n')+ assert.deepEqual(readdirSync(path.join(root, 'state')).sort(), ['state.db.upgrade-lane.bak', 'state.db.upgrade-lane.bak-wal'])+ assert.equal(readFileSync(path.join(root, 'state.db'), 'utf8'), 'state.db')+ } finally {+ rmSync(root, { recursive: true, force: true })+ }+})