Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(hcc): count denied GFS applies as D3 skips
Alfredo Lopezb32cccbeabc9
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdcindex 7fc35ff30..bfb46439b 100644--- a/.cursor/rules/minikube-t0-t1-t2.mdc+++ b/.cursor/rules/minikube-t0-t1-t2.mdc@@ -82,7 +82,8 @@ Host-side hold for Control UI / Desktop. Profile-owned random ports onlyReady-reader leftovers (claim, stale ReplicaSets, CrashLoopBackOff pods)first, and the `gfs-rollout-shim` PATH prefix replaces the reader`rollout status` wait with a readiness poll because HCC's gfsReconciler- strips `restartedAt` and makes generation waits time out.+ now preserves `restartedAt`; leftover ReplicaSets can still poison a+ generation-based wait, so the harness still judges reader readiness.- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdcindex 861f583d8..f3a7bbe09 100644--- a/.cursor/rules/minikube-t2-harness.mdc+++ b/.cursor/rules/minikube-t2-harness.mdc@@ -48,10 +48,11 @@ Invariants that must never regress:marks that leftover claim ready before reconcile, scales leftovernon-current unready reader ReplicaSets to 0, and deletes CrashLoopBackOffreader pods so a second `rollout restart` is not triggered and cannot race- HCC's gfsReconciler. HCC strips the `restartedAt` annotation, so every- harness GFS reconcile runs with the `scripts/minikube/gfs-rollout-shim`- PATH prefix: the reader `rollout status` wait becomes the readiness poll- in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and+ HCC's gfsReconciler. HCC now preserves the `restartedAt` annotation; leftover+ ReplicaSets can still poison a generation wait, so every harness GFS+ reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:+ the reader `rollout status` wait becomes the readiness poll in+ `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready andno live non-terminating unready reader pod), never a generation wait.`pre-gate-sync` reader convergence uses the same readiness contract.`scripts/tests/test-minikube-gfs-provision-order.sh` exercises the composeddiff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.mdindex 9f586d493..d963426d7 100644--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md@@ -143,8 +143,9 @@ make minikube-t2-real-postgres`.(`settle-gfs-reader-rollout.sh`: leftover claim, stale non-currentReplicaSets, CrashLoopBackOff pods) and runs with the `gfs-rollout-shim`PATH prefix, which replaces the reader `rollout status` wait with the- readiness poll in `wait-gfs-reader-ready.sh` — HCC's gfsReconciler strips- the `restartedAt` annotation, so a generation-based wait times out.+ readiness poll in `wait-gfs-reader-ready.sh` — HCC's gfsReconciler now+ preserves `restartedAt`, so leftover ReplicaSets still require a readiness+ wait instead of a generation-based one.## Step 4 — Verdict and evidence reportingdiff --git a/AGENTS.md b/AGENTS.mdindex ed1d48236..adca324d9 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -192,8 +192,9 @@ security gates do not mutate GFS and no manual repair script belongs betweenplan and verdict. Harness GFS reconciles settleReady-reader leftovers first (`settle-gfs-reader-rollout.sh`) and wait onreader readiness through the `gfs-rollout-shim` PATH prefix instead of a-generation-based `rollout status`, because HCC's gfsReconciler strips the-`restartedAt` annotation and makes that wait time out. The standalone preflight and+generation-based `rollout status`, because HCC's gfsReconciler now preserves+the `restartedAt` annotation and leftover ReplicaSets can still poison a+generation wait. The standalone preflight andthe final exact-head T2 check stay fail-loud on an unready deployment.When REUSE_DB recovery is needed, the fence covers all four database writers:HCC, workflow-recipes, trace-maintenance-worker, and control-api. The durablediff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex 85c4ed718..175038ace 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.337",+ "version": "0.1.338","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.337",+ "version": "0.1.338","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/codex-catalog-projection": "file:../packages/codex-catalog-projection",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex 1c0ad4de2..350629bc4 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.337",+ "version": "0.1.338","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/gfsReconciler.test.ts b/host-context-controller/src/gfsReconciler.test.tsindex 6cb69d8f0..208f66bbc 100644--- a/host-context-controller/src/gfsReconciler.test.ts+++ b/host-context-controller/src/gfsReconciler.test.ts@@ -51,8 +51,8 @@ class FakeApi implements GfsK8sApi {writerAvailabilitySequence: boolean[] = []failPvc = falsestatusFailuresRemaining = 0- applyOutcome: 'created' | 'replaced' | 'up_to_date' = 'replaced'- readerApplyOutcome?: 'created' | 'replaced' | 'up_to_date'+ applyOutcome: ResourceApplyResult = 'replaced'+ readerApplyOutcome?: ResourceApplyResultasync applyPvc(pvc: k8s.V1PersistentVolumeClaim): Promise<ResourceApplyResult> {if (this.failPvc) throw new Error('simulated PVC apply failure')@@ -460,6 +460,22 @@ describe('GfsReconciler resync pass log (D3)', () => {expect(fields.objects).toBeGreaterThan(0)})+ it('D3: missing and not_allowed results count as skips', async () => {+ for (const outcome of ['missing', 'not_allowed'] as const) {+ const api = new FakeApi()+ api.writerNeedsUpdate = false+ api.applyOutcome = outcome+ const info = vi.spyOn(hccLogger, 'info').mockImplementation(() => {})+ await new GfsReconciler(api, config).fullReconcile([gfs])+ const line = resyncCalls().at(-1)+ const fields = line?.[1] as { objects: number; skips: number; writes: number }+ expect(fields.writes).toBe(0)+ expect(fields.skips).toBe(fields.objects)+ expect(fields.objects).toBeGreaterThan(0)+ info.mockRestore()+ }+ })+it('D3: a drifted reader logs writes 1', async () => {const api = new FakeApi()api.writerNeedsUpdate = falsediff --git a/host-context-controller/src/gfsReconciler.ts b/host-context-controller/src/gfsReconciler.tsindex b6adece9a..b91e16291 100644--- a/host-context-controller/src/gfsReconciler.ts+++ b/host-context-controller/src/gfsReconciler.ts@@ -59,8 +59,22 @@ function recordGfsApply(result: ResourceApplyResult | 'writer_hash_skip'): void {stats.objects += 1- if (result === 'created' || result === 'replaced') stats.writes += 1- else if (result === 'up_to_date' || result === 'writer_hash_skip') stats.skips += 1+ switch (result) {+ case 'created':+ case 'replaced':+ stats.writes += 1+ return+ case 'up_to_date':+ case 'writer_hash_skip':+ case 'missing':+ case 'not_allowed':+ stats.skips += 1+ return+ default: {+ const exhaustive: never = result+ throw new Error(`unhandled GFS apply result: ${String(exhaustive)}`)+ }+ }}export interface GfsK8sApi {diff --git a/host-context-controller/src/llmHookReconciler.ts b/host-context-controller/src/llmHookReconciler.tsindex 8540cde7e..303239214 100644--- a/host-context-controller/src/llmHookReconciler.ts+++ b/host-context-controller/src/llmHookReconciler.ts@@ -63,8 +63,21 @@ function recordLlmHookApply(result: ResourceApplyResult): void {const acc = llmHookResyncPass.getStore()if (!acc) returnacc.objects += 1- if (result === 'created' || result === 'replaced') acc.writes += 1- else if (result === 'up_to_date') acc.skips += 1+ switch (result) {+ case 'created':+ case 'replaced':+ acc.writes += 1+ return+ case 'up_to_date':+ case 'missing':+ case 'not_allowed':+ acc.skips += 1+ return+ default: {+ const exhaustive: never = result+ throw new Error(`unhandled LlmHook apply result: ${String(exhaustive)}`)+ }+ }}const GROUP = 'clerum.io'diff --git a/scripts/minikube/pre-gate-incremental.sh b/scripts/minikube/pre-gate-incremental.shindex def390395..5e88eb151 100755--- a/scripts/minikube/pre-gate-incremental.sh+++ b/scripts/minikube/pre-gate-incremental.sh@@ -531,9 +531,9 @@ incremental_restart_targets() {${KC} rollout restart "deployment/${deployment}" -n "${namespace}" >/dev/nulllog "Restarted ${namespace}/${deployment} for image selector ${selector}"if [[ "${namespace}/${deployment}" == "gfs/gfsc-reader" ]]; then- # HCC owns the reader template and strips kubectl's restartedAt- # annotation. The canonical shim judges Ready replicas/pods instead of- # waiting forever on a deployment generation HCC can rewrite.+ # HCC now preserves kubectl's restartedAt. The canonical shim still+ # judges Ready replicas/pods because leftover ReplicaSets can poison a+ # generation-based wait.PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \CONTEXT="${PROFILE}" ${KC} rollout status "deployment/${deployment}" \-n "${namespace}" --timeout=120s >/dev/nulldiff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.shindex d1008ffdb..56f8d7fca 100755--- a/scripts/minikube/pre-gate-sync.sh+++ b/scripts/minikube/pre-gate-sync.sh@@ -553,8 +553,7 @@ provision_gfs_serving() {# Settle a Ready reader first so reconcile does not restart it and# race HCC's gfsReconciler during kubectl rollout status. If reconcile# still needs a reader rollout, the gfs-rollout-shim PATH prefix makes- # that wait judge readiness instead of the template generation HCC keeps- # rewriting.+ # that wait judge readiness instead of a leftover ReplicaSet generation.if ! settle_gfs_reader_rollout; thenlog "ERROR: unable to settle the branch-owned GFS reader rollout before credential reconciliation"exit 1