Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(hcc): count denied GFS applies as D3 skips

Alfredo Lopezb32cccbeabc9
diff --git a/.cursor/rules/minikube-t0-t1-t2.mdc b/.cursor/rules/minikube-t0-t1-t2.mdc
index 7fc35ff30..bfb46439b 100644
--- a/.cursor/rules/minikube-t0-t1-t2.mdc
+++ b/.cursor/rules/minikube-t0-t1-t2.mdc
@@ -82,7 +82,8 @@ Host-side hold for Control UI / Desktop. Profile-owned random ports only
Ready-reader leftovers (claim, stale ReplicaSets, CrashLoopBackOff pods)
first, and the `gfs-rollout-shim` PATH prefix replaces the reader
`rollout status` wait with a readiness poll because HCC's gfsReconciler
- strips `restartedAt` and makes generation waits time out.
+ now preserves `restartedAt`; leftover ReplicaSets can still poison a
+ generation-based wait, so the harness still judges reader readiness.
- Evidence lives under ignored `.local-notes/infra/runs/`; never commit it.
Full workflow: skill `.cursor/skills/minikube-t0-t1-t2/SKILL.md`.
diff --git a/.cursor/rules/minikube-t2-harness.mdc b/.cursor/rules/minikube-t2-harness.mdc
index 861f583d8..f3a7bbe09 100644
--- a/.cursor/rules/minikube-t2-harness.mdc
+++ b/.cursor/rules/minikube-t2-harness.mdc
@@ -48,10 +48,11 @@ Invariants that must never regress:
marks that leftover claim ready before reconcile, scales leftover
non-current unready reader ReplicaSets to 0, and deletes CrashLoopBackOff
reader pods so a second `rollout restart` is not triggered and cannot race
- HCC's gfsReconciler. HCC strips the `restartedAt` annotation, so every
- harness GFS reconcile runs with the `scripts/minikube/gfs-rollout-shim`
- PATH prefix: the reader `rollout status` wait becomes the readiness poll
- in `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and
+ HCC's gfsReconciler. HCC now preserves the `restartedAt` annotation; leftover
+ ReplicaSets can still poison a generation wait, so every harness GFS
+ reconcile runs with the `scripts/minikube/gfs-rollout-shim` PATH prefix:
+ the reader `rollout status` wait becomes the readiness poll in
+ `scripts/minikube/wait-gfs-reader-ready.sh` (desired replicas Ready and
no live non-terminating unready reader pod), never a generation wait.
`pre-gate-sync` reader convergence uses the same readiness contract.
`scripts/tests/test-minikube-gfs-provision-order.sh` exercises the composed
diff --git a/.cursor/skills/minikube-t0-t1-t2/SKILL.md b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
index 9f586d493..d963426d7 100644
--- a/.cursor/skills/minikube-t0-t1-t2/SKILL.md
+++ b/.cursor/skills/minikube-t0-t1-t2/SKILL.md
@@ -143,8 +143,9 @@ make minikube-t2-real-postgres`.
(`settle-gfs-reader-rollout.sh`: leftover claim, stale non-current
ReplicaSets, CrashLoopBackOff pods) and runs with the `gfs-rollout-shim`
PATH prefix, which replaces the reader `rollout status` wait with the
- readiness poll in `wait-gfs-reader-ready.sh` — HCC's gfsReconciler strips
- the `restartedAt` annotation, so a generation-based wait times out.
+ readiness poll in `wait-gfs-reader-ready.sh` — HCC's gfsReconciler now
+ preserves `restartedAt`, so leftover ReplicaSets still require a readiness
+ wait instead of a generation-based one.
## Step 4 — Verdict and evidence reporting
diff --git a/AGENTS.md b/AGENTS.md
index ed1d48236..adca324d9 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -192,8 +192,9 @@ security gates do not mutate GFS and no manual repair script belongs between
plan and verdict. Harness GFS reconciles settle
Ready-reader leftovers first (`settle-gfs-reader-rollout.sh`) and wait on
reader readiness through the `gfs-rollout-shim` PATH prefix instead of a
-generation-based `rollout status`, because HCC's gfsReconciler strips the
-`restartedAt` annotation and makes that wait time out. The standalone preflight and
+generation-based `rollout status`, because HCC's gfsReconciler now preserves
+the `restartedAt` annotation and leftover ReplicaSets can still poison a
+generation wait. The standalone preflight and
the final exact-head T2 check stay fail-loud on an unready deployment.
When REUSE_DB recovery is needed, the fence covers all four database writers:
HCC, workflow-recipes, trace-maintenance-worker, and control-api. The durable
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index 85c4ed718..175038ace 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.337",
+ "version": "0.1.338",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.337",
+ "version": "0.1.338",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/codex-catalog-projection": "file:../packages/codex-catalog-projection",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index 1c0ad4de2..350629bc4 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.337",
+ "version": "0.1.338",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/gfsReconciler.test.ts b/host-context-controller/src/gfsReconciler.test.ts
index 6cb69d8f0..208f66bbc 100644
--- a/host-context-controller/src/gfsReconciler.test.ts
+++ b/host-context-controller/src/gfsReconciler.test.ts
@@ -51,8 +51,8 @@ class FakeApi implements GfsK8sApi {
writerAvailabilitySequence: boolean[] = []
failPvc = false
statusFailuresRemaining = 0
- applyOutcome: 'created' | 'replaced' | 'up_to_date' = 'replaced'
- readerApplyOutcome?: 'created' | 'replaced' | 'up_to_date'
+ applyOutcome: ResourceApplyResult = 'replaced'
+ readerApplyOutcome?: ResourceApplyResult
async applyPvc(pvc: k8s.V1PersistentVolumeClaim): Promise<ResourceApplyResult> {
if (this.failPvc) throw new Error('simulated PVC apply failure')
@@ -460,6 +460,22 @@ describe('GfsReconciler resync pass log (D3)', () => {
expect(fields.objects).toBeGreaterThan(0)
})
+ it('D3: missing and not_allowed results count as skips', async () => {
+ for (const outcome of ['missing', 'not_allowed'] as const) {
+ const api = new FakeApi()
+ api.writerNeedsUpdate = false
+ api.applyOutcome = outcome
+ const info = vi.spyOn(hccLogger, 'info').mockImplementation(() => {})
+ await new GfsReconciler(api, config).fullReconcile([gfs])
+ const line = resyncCalls().at(-1)
+ const fields = line?.[1] as { objects: number; skips: number; writes: number }
+ expect(fields.writes).toBe(0)
+ expect(fields.skips).toBe(fields.objects)
+ expect(fields.objects).toBeGreaterThan(0)
+ info.mockRestore()
+ }
+ })
+
it('D3: a drifted reader logs writes 1', async () => {
const api = new FakeApi()
api.writerNeedsUpdate = false
diff --git a/host-context-controller/src/gfsReconciler.ts b/host-context-controller/src/gfsReconciler.ts
index b6adece9a..b91e16291 100644
--- a/host-context-controller/src/gfsReconciler.ts
+++ b/host-context-controller/src/gfsReconciler.ts
@@ -59,8 +59,22 @@ function recordGfsApply(
result: ResourceApplyResult | 'writer_hash_skip'
): void {
stats.objects += 1
- if (result === 'created' || result === 'replaced') stats.writes += 1
- else if (result === 'up_to_date' || result === 'writer_hash_skip') stats.skips += 1
+ switch (result) {
+ case 'created':
+ case 'replaced':
+ stats.writes += 1
+ return
+ case 'up_to_date':
+ case 'writer_hash_skip':
+ case 'missing':
+ case 'not_allowed':
+ stats.skips += 1
+ return
+ default: {
+ const exhaustive: never = result
+ throw new Error(`unhandled GFS apply result: ${String(exhaustive)}`)
+ }
+ }
}
export interface GfsK8sApi {
diff --git a/host-context-controller/src/llmHookReconciler.ts b/host-context-controller/src/llmHookReconciler.ts
index 8540cde7e..303239214 100644
--- a/host-context-controller/src/llmHookReconciler.ts
+++ b/host-context-controller/src/llmHookReconciler.ts
@@ -63,8 +63,21 @@ function recordLlmHookApply(result: ResourceApplyResult): void {
const acc = llmHookResyncPass.getStore()
if (!acc) return
acc.objects += 1
- if (result === 'created' || result === 'replaced') acc.writes += 1
- else if (result === 'up_to_date') acc.skips += 1
+ switch (result) {
+ case 'created':
+ case 'replaced':
+ acc.writes += 1
+ return
+ case 'up_to_date':
+ case 'missing':
+ case 'not_allowed':
+ acc.skips += 1
+ return
+ default: {
+ const exhaustive: never = result
+ throw new Error(`unhandled LlmHook apply result: ${String(exhaustive)}`)
+ }
+ }
}
const GROUP = 'clerum.io'
diff --git a/scripts/minikube/pre-gate-incremental.sh b/scripts/minikube/pre-gate-incremental.sh
index def390395..5e88eb151 100755
--- a/scripts/minikube/pre-gate-incremental.sh
+++ b/scripts/minikube/pre-gate-incremental.sh
@@ -531,9 +531,9 @@ incremental_restart_targets() {
${KC} rollout restart "deployment/${deployment}" -n "${namespace}" >/dev/null
log "Restarted ${namespace}/${deployment} for image selector ${selector}"
if [[ "${namespace}/${deployment}" == "gfs/gfsc-reader" ]]; then
- # HCC owns the reader template and strips kubectl's restartedAt
- # annotation. The canonical shim judges Ready replicas/pods instead of
- # waiting forever on a deployment generation HCC can rewrite.
+ # HCC now preserves kubectl's restartedAt. The canonical shim still
+ # judges Ready replicas/pods because leftover ReplicaSets can poison a
+ # generation-based wait.
PATH="${PROJECT_DIR}/scripts/minikube/gfs-rollout-shim:${PATH}" \
CONTEXT="${PROFILE}" ${KC} rollout status "deployment/${deployment}" \
-n "${namespace}" --timeout=120s >/dev/null
diff --git a/scripts/minikube/pre-gate-sync.sh b/scripts/minikube/pre-gate-sync.sh
index d1008ffdb..56f8d7fca 100755
--- a/scripts/minikube/pre-gate-sync.sh
+++ b/scripts/minikube/pre-gate-sync.sh
@@ -553,8 +553,7 @@ provision_gfs_serving() {
# Settle a Ready reader first so reconcile does not restart it and
# race HCC's gfsReconciler during kubectl rollout status. If reconcile
# still needs a reader rollout, the gfs-rollout-shim PATH prefix makes
- # that wait judge readiness instead of the template generation HCC keeps
- # rewriting.
+ # that wait judge readiness instead of a leftover ReplicaSet generation.
if ! settle_gfs_reader_rollout; then
log "ERROR: unable to settle the branch-owned GFS reader rollout before credential reconciliation"
exit 1