Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

test(e2e): repair T5 before the upgrade in the upgrade lane

Alfredo Lopezf028661664fa
diff --git a/docs/deploy/host-pvc-layout-upgrade.md b/docs/deploy/host-pvc-layout-upgrade.md
index 16cd07794..854881011 100644
--- a/docs/deploy/host-pvc-layout-upgrade.md
+++ b/docs/deploy/host-pvc-layout-upgrade.md
@@ -69,6 +69,12 @@ The command exits 1 while `UPGRADE_BLOCKERS` is above 0, while a PVC is
`UNREADABLE` or `UNKNOWN`, when Hosts exist but no PVC was inspected, or when the flag is set to an invalid value or to `false` with a
split tree present. `DESKTOP_HOSTS` repeats the export warning above.
+Repair every `WILL_REFUSE_*` before the upgrade and run `inventory` again
+until `UPGRADE_BLOCKERS=0`. A refused Host stays in `Init:Error` after the
+upgrade, and a deploy that waits for Host rollouts fails on it (the minikube
+setup does: `scripts/minikube/sync-auth-key.sh` waits for every Host
+Deployment that is not suspended).
+
A `NO_SPLIT_SIGNATURE` result says only that the merge will not touch the
tree. It does not prove where the history is; see the trees without a
signature in [Host Storage Layout](../architecture/host-storage-layout.md).
@@ -129,8 +135,10 @@ Delete the orphan PVC, or recreate its Host before turning the flag off.
## Repairs
Every refusal exits before the first move, so the tree is as the old release
-left it. The Host Pod in `Init:Error` runs no `mcp-host`, so nothing writes to
-the PVC while you repair it. Work from a helper Pod on the Host Pod's node
+left it. Repair before the upgrade when the files you rename are not the
+database the running Host opens; otherwise repair after the upgrade, when the
+Host Pod is in `Init:Error` and runs no `mcp-host`, so nothing writes to the
+PVC. Work from a helper Pod on the Host Pod's node
(the claim is ReadWriteOnce), with the Host image and the PVC mounted
read-write:
@@ -171,12 +179,15 @@ SharedFileSystem co-location rejected stateless) keeps its old history in
choose between two live databases. Decide which history the Host serves and
rename the other family:
-- keep the old history: rename the root family. The signature still holds and
+- keep the old history: rename the root family. On the old release the Host
+ runs stateful on that family, so do this after the upgrade, on the
+ `Init:Error` Pod. The signature still holds and
the merge moves `state/` and `workspace/` to the root on the next start. A
file written at the root after the switch with the same name as one under
`workspace/` (for example `IDENTITY.md`) then refuses with F3; rename one of
the two;
-- keep the chats since the switch: rename the family under `state/`. Unless
+- keep the chats since the switch: rename the family under `state/`. The
+ running Host does not open it, so do this before the upgrade. Unless
`state/` also holds a `state.db*.pre-<ts>.bak` file, the tree no longer
carries the signature: the merge exits 0 and `state/` and `workspace/`
stay as folders in the agent's workspace, with the old files under
diff --git a/scripts/e2e/e2e-hcc-layout-upgrade.sh b/scripts/e2e/e2e-hcc-layout-upgrade.sh
index 07897e277..f354db9f6 100755
--- a/scripts/e2e/e2e-hcc-layout-upgrade.sh
+++ b/scripts/e2e/e2e-hcc-layout-upgrade.sh
@@ -7,9 +7,9 @@
# the profile between the two modes.
#
# main installs v0.11.0 (published images), seeds one Host per row, runs
-# deploy/scripts/host-layout-merge.sh inventory, upgrades to this
-# head (images built from this tree, database kept), then checks
-# verify, repairs T5 with the upgrade-note procedure and checks the
+# deploy/scripts/host-layout-merge.sh inventory, repairs T5 with the
+# upgrade-note procedure (plan A7.44), upgrades to this head (images
+# built from this tree, database kept), then checks verify and the
# history of every row. Rows:
# T1 stateful;
# T2 stateless, active at the upgrade;
@@ -35,8 +35,8 @@
#
# Cluster writes besides the setups and the admin API: the named fixtures of
# e2e-hcc-storage-layout-switch.sh (hcc-cadence, pvc-fixture, channel,
-# host-layout-merge), the T5 repair (a pvc-fixture rename plus the deletion of
-# the refused Host Pod, as the upgrade note says) and, in control mode,
+# host-layout-merge), the T5 repair (a pvc-fixture rename on the release, as the
+# upgrade note says) and, in control mode,
# FAULT:flag-false (kubectl set env on the HCC Deployment, undone in the
# trap). No kubectl exec into Host Pods, no SQLite reads or writes.
set -euo pipefail
@@ -341,20 +341,6 @@ wait_merge_templates() {
return 1
}
-# wait_refused_pod HOST → prints the Pod whose merge init exited 1.
-wait_refused_pod() {
- local host=$1 elapsed=0 pod
- while [ "$elapsed" -lt "$POD_READY_TIMEOUT" ]; do
- pod="$(host_pods "$host" | jq -r '[.items[] | select(any(.status.initContainerStatuses[]?;
- .name == "workspace-layout-merge" and ((.state.terminated.exitCode // 0) == 1 or (.lastState.terminated.exitCode // 0) == 1)))][0].metadata.name // ""')"
- [ -z "$pod" ] || { printf '%s' "$pod"; return 0; }
- sleep 3
- elapsed=$((elapsed + 3))
- done
- echo "${host}: no Pod with workspace-layout-merge exit 1 within ${POD_READY_TIMEOUT}s" >&2
- return 1
-}
-
# run_layout CMD: one run of the operator script; output in LAYOUT_OUT, exit in LAYOUT_RC.
run_layout() {
LAYOUT_OUT="${SCRATCH}/layout-$1-${SECONDS}.out"
@@ -420,39 +406,45 @@ predict_rows() {
expect_row "$HOST_T1" NO_SPLIT_SIGNATURE 'inventory T1'
}
+# The upgrade note's T5 repair, keeping the chats since the switch, on the
+# release: the running stateful Pod opens only the root database, so the
+# state/ family is renamed while it serves. Repaired before the upgrade
+# because make minikube-setup waits for every Host rollout, and a Host left in
+# Init:Error by the merge fails it (sync-auth-key.sh).
+repair_blockers() {
+ log "REPAIR:${HOST_T5} rename the state/ database family (upgrade note)"
+ pvc_fixture "$HOST_T5" rw repair "$REPAIR_STATE_FAMILY_SCRIPT" >/dev/null || return 1
+ run_layout inventory
+ if [ "$LAYOUT_RC" != 0 ] || ! grep -qx 'UPGRADE_BLOCKERS=0' "$LAYOUT_OUT"; then
+ fail "inventory after the repair exited ${LAYOUT_RC}: $(grep UPGRADE_BLOCKERS "$LAYOUT_OUT")"
+ return 1
+ fi
+ expect_row "$HOST_T5" NO_SPLIT_SIGNATURE 'inventory after repair T5'
+}
+
check_upgraded() {
- local pod logs host refused_uid
+ local host
wait_merge_templates "$HOST_T1" "$HOST_T2" "$HOST_T3" "$HOST_T5" "$HOST_T6" || return 1
wait_upgraded_pod "$HOST_T1" && assert_pod "$HOST_T1" false /workspace || return 1
wait_upgraded_pod "$HOST_T2" && assert_pod "$HOST_T2" true /workspace || return 1
+ wait_upgraded_pod "$HOST_T5" && assert_pod "$HOST_T5" false /workspace || return 1
wait_upgraded_pod "$HOST_T6" && assert_pod "$HOST_T6" true /workspace || return 1
- pod="$(wait_refused_pod "$HOST_T5")" || { fail 'T5: the merge did not refuse'; return 1; }
- logs="$(kctl logs "$pod" -n "$HOST_NS" -c workspace-layout-merge --previous 2>/dev/null ||
- kctl logs "$pod" -n "$HOST_NS" -c workspace-layout-merge)"
- grep -q "$F2" <<<"$logs" || { fail "T5: merge refused without the F2 line: ${logs}"; return 1; }
[ "$(kctl get host "$HOST_T3" -n "$HOST_NS" -o jsonpath='{.status.lifecycle.state}')" = suspended ] ||
{ fail 'T3: no longer suspended before verify'; return 1; }
run_layout verify
- [ "$LAYOUT_RC" = 1 ] || { fail "verify exited ${LAYOUT_RC}, expected 1 (T3 split, T5 refused)"; return 1; }
- for host in "$HOST_T1" "$HOST_T2" "$HOST_T6"; do expect_row "$host" POD_OK 'verify' || return 1; done
- expect_row "$HOST_T3" SPLIT 'verify T3' && expect_row "$HOST_T5" "MERGE_REFUSED ${F2}*" 'verify T5' || return 1
+ [ "$LAYOUT_RC" = 1 ] || { fail "verify exited ${LAYOUT_RC}, expected 1 (T3 split)"; return 1; }
+ for host in "$HOST_T1" "$HOST_T2" "$HOST_T5" "$HOST_T6"; do expect_row "$host" POD_OK 'verify' || return 1; done
+ expect_row "$HOST_T3" SPLIT 'verify T3' || return 1
# T3 merges on its natural wake.
post_wake "$HOST_T3" && wait_new_ready_pod "$HOST_T3" '' "$POD_READY_TIMEOUT" && assert_pod "$HOST_T3" true /workspace || return 1
- # T5: the upgrade note's repair, keeping the chats since the switch.
- log "REPAIR:${HOST_T5} rename the state/ database family (upgrade note)"
- pvc_fixture "$HOST_T5" rw repair "$REPAIR_STATE_FAMILY_SCRIPT" >/dev/null || return 1
- refused_uid="$(kctl get pod "$pod" -n "$HOST_NS" -o jsonpath='{.metadata.uid}')" || { fail "T5: could not read refused Pod ${pod}"; return 1; }
- kctl delete pod "$pod" -n "$HOST_NS" --wait=true --timeout=60s >/dev/null || { fail "T5: delete of refused Pod ${pod} failed"; return 1; }
- wait_new_ready_pod "$HOST_T5" "$refused_uid" "$POD_READY_TIMEOUT" &&
- assert_pod "$HOST_T5" false /workspace || return 1
run_layout verify
if [ "$LAYOUT_RC" != 0 ] || ! grep -qx 'HOST_LAYOUT_VERIFY=PASS' "$LAYOUT_OUT"; then
- fail "verify after the repairs exited ${LAYOUT_RC}: $(tail -n 3 "$LAYOUT_OUT" | tr '\n' ' ')"
+ fail "verify after the T3 wake exited ${LAYOUT_RC}: $(tail -n 3 "$LAYOUT_OUT" | tr '\n' ' ')"
return 1
fi
- for host in "$HOST_T1" "$HOST_T2" "$HOST_T3" "$HOST_T5" "$HOST_T6"; do expect_row "$host" POD_OK 'verify after repair' || return 1; done
+ for host in "$HOST_T1" "$HOST_T2" "$HOST_T3" "$HOST_T5" "$HOST_T6"; do expect_row "$host" POD_OK 'verify after wake' || return 1; done
}
check_history() {
@@ -465,7 +457,7 @@ check_history() {
ROWS_DONE=$((ROWS_DONE + 1))
# T5 serves the root family it served before the upgrade; the old family is
# kept as a backup, and its old files stay under workspace/ (no signature).
- assert_history "$HOST_T5" "upgrade(two-databases)->repair" || return 1
+ assert_history "$HOST_T5" "repair(two-databases)->upgrade" || return 1
listing="$(pvc_fixture "$HOST_T5" ro list-repaired "$LIST_SCRIPT")" || return 1
grep -qx 'f ./state/state.db.upgrade-lane.bak' <<<"$listing" ||
{ fail "T5: the renamed family is missing: ${listing}"; return 1; }
@@ -481,8 +473,9 @@ run_main() {
header "Install ${RELEASE}" && install_release &&
header 'Seed rows on the release' && seed_rows &&
header 'Predict with inventory' && predict_rows &&
+ header 'Repair blockers on the release' && repair_blockers &&
header 'Upgrade to this head' && upgrade_to_head &&
- header 'Verify and repair' && check_upgraded &&
+ header 'Verify and wake' && check_upgraded &&
header 'History of every row' && check_history
}
diff --git a/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs b/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs
index 289ee4958..7ae63d249 100644
--- a/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs
+++ b/scripts/e2e/e2e-hcc-storage-layout-switch.test.mjs
@@ -1,6 +1,6 @@
import assert from 'node:assert/strict'
import { spawnSync } from 'node:child_process'
-import { mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'
+import { mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import path from 'node:path'
import test from 'node:test'
@@ -414,3 +414,41 @@ test('the upgrade lane installs the release deploy tree and checks the CRDs', ()
// The manifest-diff refusal is gone (A7.43).
assert.doesNotMatch(SOURCE, /this tree no longer installs the release/)
})
+
+// A7.44: make minikube-setup waits for every Host rollout, so a Host left in
+// Init:Error by the upgrade fails it. The lane repairs T5 on the release.
+test('the upgrade lane repairs blockers before it upgrades', () => {
+ const main = helper('run_main')
+ const repair = main.indexOf('repair_blockers')
+ const upgrade = main.indexOf('upgrade_to_head')
+ assert.notEqual(repair, -1, 'run_main does not call repair_blockers')
+ assert.notEqual(upgrade, -1, 'run_main does not call upgrade_to_head')
+ assert.ok(repair < upgrade, 'repair_blockers runs after upgrade_to_head')
+ assert.match(helper('repair_blockers'), /grep -qx 'UPGRADE_BLOCKERS=0'/)
+ assert.doesNotMatch(helper('check_upgraded'), /wait_refused_pod|REPAIR_STATE_FAMILY_SCRIPT/)
+})
+
+test('the lane T5 repair leaves a tree the merge does not touch', () => {
+ const repairScript = SOURCE.match(/^REPAIR_STATE_FAMILY_SCRIPT='([^']*)'/m)
+ assert.ok(repairScript, 'REPAIR_STATE_FAMILY_SCRIPT not found')
+ const layoutScript = fileURLToPath(new URL('../../deploy/scripts/host-layout-merge.sh', import.meta.url))
+ const root = mkdtempSync(path.join(tmpdir(), 'e2e-825-t5-repair-'))
+ try {
+ for (const p of ['state', 'workspace']) mkdirSync(path.join(root, p))
+ for (const p of ['state.db', 'state/state.db', 'state/state.db-wal', 'workspace/seed.txt']) writeFileSync(path.join(root, p), p)
+ const classify = () => {
+ const printed = spawnSync('bash', [layoutScript, 'check-script', '--root', root], { encoding: 'utf8' })
+ assert.equal(printed.status, 0, printed.stderr)
+ return spawnSync('/bin/sh', ['-c', printed.stdout], { encoding: 'utf8' }).stdout
+ }
+ // Witness: the seeded tree is the T5 blocker.
+ assert.match(classify(), /^WILL_REFUSE_F2 /)
+ const repaired = spawnSync('/bin/sh', ['-c', repairScript[1], 'repair', root], { encoding: 'utf8' })
+ assert.equal(repaired.status, 0, repaired.stderr)
+ assert.equal(classify(), 'NO_SPLIT_SIGNATURE\n')
+ assert.deepEqual(readdirSync(path.join(root, 'state')).sort(), ['state.db.upgrade-lane.bak', 'state.db.upgrade-lane.bak-wal'])
+ assert.equal(readFileSync(path.join(root, 'state.db'), 'utf8'), 'state.db')
+ } finally {
+ rmSync(root, { recursive: true, force: true })
+ }
+})