Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(wrc): close review findings on the NetworkPolicy spec-hash gate
Alfredo Lopez223162a75334
diff --git a/docs/deploy/workflow-recipes-guide.md b/docs/deploy/workflow-recipes-guide.mdindex 97bd36ae3..0e6a3e717 100644--- a/docs/deploy/workflow-recipes-guide.md+++ b/docs/deploy/workflow-recipes-guide.md@@ -301,16 +301,27 @@ kubectl delete workflowrecipe my-nginx -n sandbox-recipes## Reconcile idempotency & drift behavior-WRC reconciles every recipe on a periodic resync. Each managed object (Deployment,-StatefulSet, headless Service, CronJob, Job, DaemonSet, the McpServer CRD, and every-WRC-authored NetworkPolicy applied through `applyNetworkPolicy` — `ui-ingress-*`,-`wl-ingress-*`, `wr-intdep-*`, `wf-*-oauth-broker-egress`, the three webhook-gateway-policies, and `wl-egress-*` once issue #299 has decided a write is due) is-applied through a **spec-hash idempotency gate**: WRC stamps a `clerum.io/spec-hash`-annotation (a hash of the _desired_ manifest) and, on the already-exists path, **skips-the write entirely when the existing object's stamped hash matches**. This keeps-`metadata.generation` from climbing on no-op reconciles (which previously produced a-`degraded↔active` status flap and a downstream HCC NetworkPolicy no-op write storm).+WRC reconciles every recipe on a periodic resync. Most managed objects (the recipe+Deployment, StatefulSet, headless Service, CronJob, Job, DaemonSet, the McpServer CRD,+and the NetworkPolicies `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*`,+`wf-*-oauth-broker-egress` and the three webhook-gateway policies) are applied through a+**spec-hash idempotency gate**: WRC stamps a `clerum.io/spec-hash` annotation (a hash of+the _desired_ manifest) and, on the already-exists path, **skips the write entirely when+the existing object's stamped hash matches**. This keeps `metadata.generation` from+climbing on no-op reconciles (which previously produced a `degraded↔active` status flap+and a downstream HCC NetworkPolicy no-op write storm).++**What the gate does _not_ cover.** Do not read the list above as "everything":++- `wl-egress-*` routes through the same method, but its manifest carries a+ `clerum.io/egress-fqdn-resolved-at` timestamp that is regenerated on every pass, so its+ desired hash never matches and **the gate never closes for it**. What actually keeps it+ quiet is the upstream issue #299 check (`egressWriteNeeded`), which compares the+ enforced egress rules. If that check ever regresses, the spec-hash gate will not catch it.+- `ui-egress-*` and the `coordinator-to-gfs` policy do not use the spec-hash at all; they+ have their own ad-hoc comparisons (#299 and #579 respectively).+- The **webhook gateway's own** ConfigMap, Deployment and Service are still ungated and+ rewritten on every pass — only its three NetworkPolicies were migrated.**Operational consequences:**@@ -319,9 +330,19 @@ the write entirely when the existing object's stamped hash matches**. This keepsany alert keyed on it.- **Drift is no longer auto-reverted.** If an operator hand-edits a WRC-managed object(e.g. `kubectl edit`/`scale` on a recipe Deployment), WRC will **not** revert it until- the recipe spec changes (which flips the hash). The change is still **visible**: the- recipe `status` reports the workload `degraded` when replica health drops. To force a- re-apply, change the recipe spec (or delete the object and let WRC recreate it).+ the recipe spec changes (which flips the hash). To force a re-apply, change the recipe+ spec (or delete the object and let WRC recreate it).+- **For a Deployment the drift is still visible**: the recipe `status` reports the+ workload `degraded` when replica health drops.+- **For a NetworkPolicy it is not visible by any path.** There is no replica health to+ degrade: the recipe stays `active` while the enforcement object diverges from the+ desired one, and nothing sweeps for it — `pruneStaleInternalDependencyPolicies`+ compares names, not content, and `deploy/scripts/verify-networkpolicies.sh` only checks+ that the overlay's policies exist. An edit that preserves the `clerum.io/spec-hash`+ annotation while changing the `spec` (widening `ingress`/`egress`, emptying+ `podSelector`, dropping a `policyTypes` entry) is **permanent and silent**. Treat any+ `kubectl edit` on a `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*` or gateway policy as+ permanent until a bounded resync exists (issue #581).- **First deploy after this change** does one stamping write per pre-existing managedobject (they have no hash yet), then steady-state is quiet.diff --git a/workflow-recipes/package-lock.json b/workflow-recipes/package-lock.jsonindex fbee3b101..8549a3f01 100644--- a/workflow-recipes/package-lock.json+++ b/workflow-recipes/package-lock.json@@ -1,12 +1,12 @@{"name": "workload-recipes-controller",- "version": "0.1.340",+ "version": "0.1.341","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "workload-recipes-controller",- "version": "0.1.340",+ "version": "0.1.341","license": "SEE LICENSE IN LICENSE","dependencies": {"@aws-sdk/client-s3": "^3.1009.0",diff --git a/workflow-recipes/package.json b/workflow-recipes/package.jsonindex 7747bce78..0e09abecd 100644--- a/workflow-recipes/package.json+++ b/workflow-recipes/package.json@@ -1,6 +1,6 @@{"name": "workload-recipes-controller",- "version": "0.1.340",+ "version": "0.1.341","description": "Workload Recipes Controller (WRC) - WorkloadRecipe lifecycle management and MCP server interface","main": "dist/main.js","engines": {diff --git a/workflow-recipes/src/reconciler/reconciler.test.ts b/workflow-recipes/src/reconciler/reconciler.test.tsindex cfccea8ec..83ca946a9 100644--- a/workflow-recipes/src/reconciler/reconciler.test.ts+++ b/workflow-recipes/src/reconciler/reconciler.test.ts@@ -18,7 +18,7 @@ import {resolveScopedWorkloadResourceName,resolveWorkloadRuntimeResourceName,} from './resourceBuilder'-import { stampSpecHash } from './specHash'+import { computeSpecHash, stampSpecHash } from './specHash'import {TRANSIENT_REQUEUE_BASE_MS,WORKFLOW_PROGRESS_REQUEUE_BASE_MS,@@ -839,7 +839,9 @@ describe('WorkflowRecipeReconciler', () => {}).body)- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBeTruthy()+ // The exact hash, not merely "some annotation": a stamp that is present but+ // wrong would still skip the write, and would skip it for the wrong reason.+ expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({@@ -878,7 +880,7 @@ describe('WorkflowRecipeReconciler', () => {}).bodyexpect(body.metadata?.resourceVersion).toBe('9')- expect(body.metadata?.annotations?.[SPEC_HASH]).toBeTruthy()+ expect(body.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))// One read serves both the gate and the replace, so the replace cannot act// on a different resourceVersion than the one the gate evaluated.expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)@@ -897,7 +899,7 @@ describe('WorkflowRecipeReconciler', () => {}).body)- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBeTruthy()+ expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(sealed)@@ -920,9 +922,19 @@ describe('WorkflowRecipeReconciler', () => {},})- await expect(apply(desired)).rejects.toThrow(- /Refusing to replace NetworkPolicy "wr-intdep-egress-r-w"/+ const rejection = await apply(desired).then(+ () => {+ throw new Error('expected the ownership assertion to reject')+ },+ (e: unknown) => e as Error)+ // Pin the CLASS, not just the message. `reconcileInternalDependencyPolicies`+ // keys the CRD condition `OwnershipConflict` off `instanceof`, so a bare+ // Error carrying the same text would set the wrong condition while passing a+ // message-only assertion. The class is module-local — exporting it just for+ // this test would be production code shaped by a test — so pin its `name`.+ expect(rejection.name).toBe('NetworkPolicyOwnershipConflictError')+ expect(rejection.message).toMatch(/Refusing to replace NetworkPolicy "wr-intdep-egress-r-w"/)expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()// Two reads: the gate's read raises the veto, `applyIsNoop` treats any// throw as "cannot prove this is a no-op" and falls through, and the replace@@ -933,6 +945,32 @@ describe('WorkflowRecipeReconciler', () => {expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)})+ it('T4b lets an internal-dependency policy WRC already owns through the gate', async () => {+ // T4 pins the veto; nothing pinned the other side. An assertion that rejected+ // EVERY internal-dependency policy would leave T4 green and silently stop the+ // whole intdep family from converging — the gate would never be reached and+ // every pass would set an OwnershipConflict condition. This is the test that+ // makes T4 mean "vetoes foreign owners" rather than "always vetoes".+ const desired = intdepPolicy()+ const ownHash = stampSpecHash(clone(desired))+ mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({+ metadata: {+ name: 'wr-intdep-egress-r-w',+ resourceVersion: '9',+ annotations: { [SPEC_HASH]: ownHash },+ labels: desired.metadata!.labels,+ },+ })++ await expect(apply(desired)).resolves.toBeUndefined()++ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ // One read: the assertion passed inside the gate's own read, so the replace+ // path is never entered and never re-reads. Contrast T4's two.+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ })+it('T5 falls through to the replace when the gate read fails with an HTTP status', async () => {mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })mockNetworkingApi.readNamespacedNetworkPolicy@@ -945,16 +983,10 @@ describe('WorkflowRecipeReconciler', () => {expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)})- it('T5c falls through on a codeless read error, the shape `:1354` pins for Deployment', async () => {- // The rule cannot key on the shape of the error. A bare Error with no code- // is a legitimate read failure — the existing Deployment test injects- // exactly `new Error('transient read timeout')` and asserts the write still- // happens — so `applyIsNoop` swallows everything and falls open. A version- // of this gate that classified errors instead broke that test; this pins the- // behaviour for NetworkPolicy so the same idea cannot come back quietly.+ it('T5b falls through to the replace on a retryable transport error', async () => {mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })mockNetworkingApi.readNamespacedNetworkPolicy- .mockRejectedValueOnce(new Error('transient read timeout'))+ .mockRejectedValueOnce({ code: 'ECONNRESET' }).mockResolvedValueOnce(ownedExisting())await apply(plainPolicy())@@ -963,10 +995,20 @@ describe('WorkflowRecipeReconciler', () => {expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)})- it('T5b falls through to the replace on a retryable transport error', async () => {+ it('T5c falls through on a codeless read error, the shape `:1354` pins for Deployment', async () => {+ // The rule cannot key on the shape of the error. A bare Error with no code+ // is a legitimate read failure — the existing Deployment test injects+ // exactly `new Error('transient read timeout')` and asserts the write still+ // happens — so `applyIsNoop` swallows everything and falls open. A version+ // of this gate that classified errors instead broke that test; this pins the+ // behaviour for NetworkPolicy so the same idea cannot come back quietly.+ //+ // Not redundant with T5b despite the total catch: T5b's error carries a+ // `code`, and every earlier draft that regressed this did so by keying on+ // `code`. The codeless shape is the one that broke the Deployment test.mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })mockNetworkingApi.readNamespacedNetworkPolicy- .mockRejectedValueOnce({ code: 'ECONNRESET' })+ .mockRejectedValueOnce(new Error('transient read timeout')).mockResolvedValueOnce(ownedExisting())await apply(plainPolicy())@@ -9415,7 +9457,13 @@ describe('WorkflowRecipeReconciler', () => {const body = (call[0] as { body: k8s.V1NetworkPolicy }).bodysealed.set(body.metadata!.name!, JSON.parse(JSON.stringify(body)))}- expect(sealed.size).toBe(3)+ // Which three, not just how many: cardinality alone stays green if a refactor+ // stops emitting gateway policies and three arrive from another family.+ expect([...sealed.keys()].sort()).toEqual([+ 'allow-gateway-egress-to-handler-wf-wh-recipe',+ 'allow-gateway-ingress-to-handler-wf-wh-recipe',+ 'allow-webhook-proxy-ingress-wf-wh-recipe',+ ])mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(@@ -9429,13 +9477,31 @@ describe('WorkflowRecipeReconciler', () => {})mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()+ mockNetworkingApi.readNamespacedNetworkPolicy.mockClear()+ const logSpy = vi.spyOn(console, 'log').mockImplementation(() => undefined)- await reconciler.reconcile(recipeWithWebhook())-- for (const name of sealed.keys()) {- expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalledWith(- expect.objectContaining({ name })- )+ try {+ await reconciler.reconcile(recipeWithWebhook())++ const logged = logSpy.mock.calls.flat().join('\n')+ for (const name of sealed.keys()) {+ // Negative half: the gate closed, no write went out.+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalledWith(+ expect.objectContaining({ name })+ )+ // Positive half, and the reason this test is not vacuous: prove the second+ // pass actually REACHED this policy. A negative assertion alone cannot tell+ // "the gate closed" from "the code never ran" — patching the gateway loop to+ // run only on the first pass leaves the negative half green. Both halves are+ // required of any convergence test built on this shape (issue #581).+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledWith(+ expect.objectContaining({ name })+ )+ expect(logged).toContain(`NetworkPolicy "${name}" in`)+ expect(logged).toContain('unchanged (spec-hash match); skipping update')+ }+ } finally {+ logSpy.mockRestore()}})diff --git a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts b/workflow-recipes/src/reconciler/workflowRecipeReconciler.tsindex adcb5a37c..dbb541400 100644--- a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts+++ b/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts@@ -3734,7 +3734,7 @@ export class WorkflowRecipeReconciler {console.log(`[WR-Reconciler] Created ${label}`)} catch (error: unknown) {if (getErrorCode(error) === 409) {- if (idempotency && (await this.applyIsNoop(idempotency))) {+ if (idempotency && (await this.applyIsNoop(idempotency, label))) {console.log(`[WR-Reconciler] ${label} unchanged (spec-hash match); skipping update`)return}@@ -3761,14 +3761,27 @@ export class WorkflowRecipeReconciler {* their own read in `replaceFn`: this catch is total by design (see* `applyNetworkPolicy`).*/- private async applyIsNoop(idempotency: {- manifest: { metadata?: { annotations?: { [key: string]: string } } }- readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>- }): Promise<boolean> {+ private async applyIsNoop(+ idempotency: {+ manifest: { metadata?: { annotations?: { [key: string]: string } } }+ readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>+ },+ label: string+ ): Promise<boolean> {try {const existing = await idempotency.readExisting()return specHashUnchanged(idempotency.manifest, existing ?? null)- } catch {+ } catch (error) {+ // Falling through to the replace is correct — never skip an update we cannot+ // prove unnecessary — but discarding the error left no trace of WHY: revoked+ // RBAC or a persistently failing apiserver looked exactly like a healthy+ // cache miss, and the gate silently stopped saving anything. The write still+ // happens, so this is a diagnostic line, not a failure path; a genuine veto+ // reaches the caller through `replaceFn`, loud.+ console.warn(+ `[WR-Reconciler] ${label}: idempotency pre-read failed, falling through to update:`,+ error+ )return false}}@@ -4892,14 +4905,17 @@ export class WorkflowRecipeReconciler {*/private async applyNetworkPolicy(policy: k8s.V1NetworkPolicy, namespace: string): Promise<void> {const policyName = policy.metadata!.name!- let existing: k8s.V1NetworkPolicy | undefined+ // Set only by a read whose ownership assertion PASSED, which is what makes it+ // safe to reuse in the replace path. Unset therefore means "no verified object+ // in hand" — the read failed, or its assertion vetoed.+ let assertedExisting: k8s.V1NetworkPolicy | undefinedconst readExistingAndAssertOwnership = async (): Promise<k8s.V1NetworkPolicy> => {const fetched = await this.networkingApi.readNamespacedNetworkPolicy({name: policyName,namespace,})this.assertInternalDependencyPolicyOwnership(policy, fetched, namespace)- existing = fetched+ assertedExisting = fetchedreturn fetched}await this.createOrReplace(@@ -4909,9 +4925,9 @@ export class WorkflowRecipeReconciler {body: policy,}),async () => {- // `existing` is unset when the gate's read failed OR its assertion vetoed- // (applyIsNoop swallows both); re-read so the veto surfaces from here, loud.- const current = existing ?? (await readExistingAndAssertOwnership())+ // Unset means the gate's read failed OR its assertion vetoed (applyIsNoop+ // swallows both); re-read so the veto surfaces from here, loud.+ const current = assertedExisting ?? (await readExistingAndAssertOwnership())policy.metadata!.resourceVersion = current.metadata?.resourceVersionreturn this.networkingApi.replaceNamespacedNetworkPolicy({name: policyName,