Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(wrc): close review findings on the NetworkPolicy spec-hash gate

Alfredo Lopez223162a75334
diff --git a/docs/deploy/workflow-recipes-guide.md b/docs/deploy/workflow-recipes-guide.md
index 97bd36ae3..0e6a3e717 100644
--- a/docs/deploy/workflow-recipes-guide.md
+++ b/docs/deploy/workflow-recipes-guide.md
@@ -301,16 +301,27 @@ kubectl delete workflowrecipe my-nginx -n sandbox-recipes
## Reconcile idempotency & drift behavior
-WRC reconciles every recipe on a periodic resync. Each managed object (Deployment,
-StatefulSet, headless Service, CronJob, Job, DaemonSet, the McpServer CRD, and every
-WRC-authored NetworkPolicy applied through `applyNetworkPolicy` — `ui-ingress-*`,
-`wl-ingress-*`, `wr-intdep-*`, `wf-*-oauth-broker-egress`, the three webhook-gateway
-policies, and `wl-egress-*` once issue #299 has decided a write is due) is
-applied through a **spec-hash idempotency gate**: WRC stamps a `clerum.io/spec-hash`
-annotation (a hash of the _desired_ manifest) and, on the already-exists path, **skips
-the write entirely when the existing object's stamped hash matches**. This keeps
-`metadata.generation` from climbing on no-op reconciles (which previously produced a
-`degraded↔active` status flap and a downstream HCC NetworkPolicy no-op write storm).
+WRC reconciles every recipe on a periodic resync. Most managed objects (the recipe
+Deployment, StatefulSet, headless Service, CronJob, Job, DaemonSet, the McpServer CRD,
+and the NetworkPolicies `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*`,
+`wf-*-oauth-broker-egress` and the three webhook-gateway policies) are applied through a
+**spec-hash idempotency gate**: WRC stamps a `clerum.io/spec-hash` annotation (a hash of
+the _desired_ manifest) and, on the already-exists path, **skips the write entirely when
+the existing object's stamped hash matches**. This keeps `metadata.generation` from
+climbing on no-op reconciles (which previously produced a `degraded↔active` status flap
+and a downstream HCC NetworkPolicy no-op write storm).
+
+**What the gate does _not_ cover.** Do not read the list above as "everything":
+
+- `wl-egress-*` routes through the same method, but its manifest carries a
+ `clerum.io/egress-fqdn-resolved-at` timestamp that is regenerated on every pass, so its
+ desired hash never matches and **the gate never closes for it**. What actually keeps it
+ quiet is the upstream issue #299 check (`egressWriteNeeded`), which compares the
+ enforced egress rules. If that check ever regresses, the spec-hash gate will not catch it.
+- `ui-egress-*` and the `coordinator-to-gfs` policy do not use the spec-hash at all; they
+ have their own ad-hoc comparisons (#299 and #579 respectively).
+- The **webhook gateway's own** ConfigMap, Deployment and Service are still ungated and
+ rewritten on every pass — only its three NetworkPolicies were migrated.
**Operational consequences:**
@@ -319,9 +330,19 @@ the write entirely when the existing object's stamped hash matches**. This keeps
any alert keyed on it.
- **Drift is no longer auto-reverted.** If an operator hand-edits a WRC-managed object
(e.g. `kubectl edit`/`scale` on a recipe Deployment), WRC will **not** revert it until
- the recipe spec changes (which flips the hash). The change is still **visible**: the
- recipe `status` reports the workload `degraded` when replica health drops. To force a
- re-apply, change the recipe spec (or delete the object and let WRC recreate it).
+ the recipe spec changes (which flips the hash). To force a re-apply, change the recipe
+ spec (or delete the object and let WRC recreate it).
+- **For a Deployment the drift is still visible**: the recipe `status` reports the
+ workload `degraded` when replica health drops.
+- **For a NetworkPolicy it is not visible by any path.** There is no replica health to
+ degrade: the recipe stays `active` while the enforcement object diverges from the
+ desired one, and nothing sweeps for it — `pruneStaleInternalDependencyPolicies`
+ compares names, not content, and `deploy/scripts/verify-networkpolicies.sh` only checks
+ that the overlay's policies exist. An edit that preserves the `clerum.io/spec-hash`
+ annotation while changing the `spec` (widening `ingress`/`egress`, emptying
+ `podSelector`, dropping a `policyTypes` entry) is **permanent and silent**. Treat any
+ `kubectl edit` on a `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*` or gateway policy as
+ permanent until a bounded resync exists (issue #581).
- **First deploy after this change** does one stamping write per pre-existing managed
object (they have no hash yet), then steady-state is quiet.
diff --git a/workflow-recipes/package-lock.json b/workflow-recipes/package-lock.json
index fbee3b101..8549a3f01 100644
--- a/workflow-recipes/package-lock.json
+++ b/workflow-recipes/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "workload-recipes-controller",
- "version": "0.1.340",
+ "version": "0.1.341",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "workload-recipes-controller",
- "version": "0.1.340",
+ "version": "0.1.341",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@aws-sdk/client-s3": "^3.1009.0",
diff --git a/workflow-recipes/package.json b/workflow-recipes/package.json
index 7747bce78..0e09abecd 100644
--- a/workflow-recipes/package.json
+++ b/workflow-recipes/package.json
@@ -1,6 +1,6 @@
{
"name": "workload-recipes-controller",
- "version": "0.1.340",
+ "version": "0.1.341",
"description": "Workload Recipes Controller (WRC) - WorkloadRecipe lifecycle management and MCP server interface",
"main": "dist/main.js",
"engines": {
diff --git a/workflow-recipes/src/reconciler/reconciler.test.ts b/workflow-recipes/src/reconciler/reconciler.test.ts
index cfccea8ec..83ca946a9 100644
--- a/workflow-recipes/src/reconciler/reconciler.test.ts
+++ b/workflow-recipes/src/reconciler/reconciler.test.ts
@@ -18,7 +18,7 @@ import {
resolveScopedWorkloadResourceName,
resolveWorkloadRuntimeResourceName,
} from './resourceBuilder'
-import { stampSpecHash } from './specHash'
+import { computeSpecHash, stampSpecHash } from './specHash'
import {
TRANSIENT_REQUEUE_BASE_MS,
WORKFLOW_PROGRESS_REQUEUE_BASE_MS,
@@ -839,7 +839,9 @@ describe('WorkflowRecipeReconciler', () => {
}
).body
)
- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBeTruthy()
+ // The exact hash, not merely "some annotation": a stamp that is present but
+ // wrong would still skip the write, and would skip it for the wrong reason.
+ expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))
mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({
@@ -878,7 +880,7 @@ describe('WorkflowRecipeReconciler', () => {
}
).body
expect(body.metadata?.resourceVersion).toBe('9')
- expect(body.metadata?.annotations?.[SPEC_HASH]).toBeTruthy()
+ expect(body.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))
// One read serves both the gate and the replace, so the replace cannot act
// on a different resourceVersion than the one the gate evaluated.
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
@@ -897,7 +899,7 @@ describe('WorkflowRecipeReconciler', () => {
}
).body
)
- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBeTruthy()
+ expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))
mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()
mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(sealed)
@@ -920,9 +922,19 @@ describe('WorkflowRecipeReconciler', () => {
},
})
- await expect(apply(desired)).rejects.toThrow(
- /Refusing to replace NetworkPolicy "wr-intdep-egress-r-w"/
+ const rejection = await apply(desired).then(
+ () => {
+ throw new Error('expected the ownership assertion to reject')
+ },
+ (e: unknown) => e as Error
)
+ // Pin the CLASS, not just the message. `reconcileInternalDependencyPolicies`
+ // keys the CRD condition `OwnershipConflict` off `instanceof`, so a bare
+ // Error carrying the same text would set the wrong condition while passing a
+ // message-only assertion. The class is module-local — exporting it just for
+ // this test would be production code shaped by a test — so pin its `name`.
+ expect(rejection.name).toBe('NetworkPolicyOwnershipConflictError')
+ expect(rejection.message).toMatch(/Refusing to replace NetworkPolicy "wr-intdep-egress-r-w"/)
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
// Two reads: the gate's read raises the veto, `applyIsNoop` treats any
// throw as "cannot prove this is a no-op" and falls through, and the replace
@@ -933,6 +945,32 @@ describe('WorkflowRecipeReconciler', () => {
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
})
+ it('T4b lets an internal-dependency policy WRC already owns through the gate', async () => {
+ // T4 pins the veto; nothing pinned the other side. An assertion that rejected
+ // EVERY internal-dependency policy would leave T4 green and silently stop the
+ // whole intdep family from converging — the gate would never be reached and
+ // every pass would set an OwnershipConflict condition. This is the test that
+ // makes T4 mean "vetoes foreign owners" rather than "always vetoes".
+ const desired = intdepPolicy()
+ const ownHash = stampSpecHash(clone(desired))
+ mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({
+ metadata: {
+ name: 'wr-intdep-egress-r-w',
+ resourceVersion: '9',
+ annotations: { [SPEC_HASH]: ownHash },
+ labels: desired.metadata!.labels,
+ },
+ })
+
+ await expect(apply(desired)).resolves.toBeUndefined()
+
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ // One read: the assertion passed inside the gate's own read, so the replace
+ // path is never entered and never re-reads. Contrast T4's two.
+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ })
+
it('T5 falls through to the replace when the gate read fails with an HTTP status', async () => {
mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
mockNetworkingApi.readNamespacedNetworkPolicy
@@ -945,16 +983,10 @@ describe('WorkflowRecipeReconciler', () => {
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
})
- it('T5c falls through on a codeless read error, the shape `:1354` pins for Deployment', async () => {
- // The rule cannot key on the shape of the error. A bare Error with no code
- // is a legitimate read failure — the existing Deployment test injects
- // exactly `new Error('transient read timeout')` and asserts the write still
- // happens — so `applyIsNoop` swallows everything and falls open. A version
- // of this gate that classified errors instead broke that test; this pins the
- // behaviour for NetworkPolicy so the same idea cannot come back quietly.
+ it('T5b falls through to the replace on a retryable transport error', async () => {
mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
mockNetworkingApi.readNamespacedNetworkPolicy
- .mockRejectedValueOnce(new Error('transient read timeout'))
+ .mockRejectedValueOnce({ code: 'ECONNRESET' })
.mockResolvedValueOnce(ownedExisting())
await apply(plainPolicy())
@@ -963,10 +995,20 @@ describe('WorkflowRecipeReconciler', () => {
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
})
- it('T5b falls through to the replace on a retryable transport error', async () => {
+ it('T5c falls through on a codeless read error, the shape `:1354` pins for Deployment', async () => {
+ // The rule cannot key on the shape of the error. A bare Error with no code
+ // is a legitimate read failure — the existing Deployment test injects
+ // exactly `new Error('transient read timeout')` and asserts the write still
+ // happens — so `applyIsNoop` swallows everything and falls open. A version
+ // of this gate that classified errors instead broke that test; this pins the
+ // behaviour for NetworkPolicy so the same idea cannot come back quietly.
+ //
+ // Not redundant with T5b despite the total catch: T5b's error carries a
+ // `code`, and every earlier draft that regressed this did so by keying on
+ // `code`. The codeless shape is the one that broke the Deployment test.
mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
mockNetworkingApi.readNamespacedNetworkPolicy
- .mockRejectedValueOnce({ code: 'ECONNRESET' })
+ .mockRejectedValueOnce(new Error('transient read timeout'))
.mockResolvedValueOnce(ownedExisting())
await apply(plainPolicy())
@@ -9415,7 +9457,13 @@ describe('WorkflowRecipeReconciler', () => {
const body = (call[0] as { body: k8s.V1NetworkPolicy }).body
sealed.set(body.metadata!.name!, JSON.parse(JSON.stringify(body)))
}
- expect(sealed.size).toBe(3)
+ // Which three, not just how many: cardinality alone stays green if a refactor
+ // stops emitting gateway policies and three arrive from another family.
+ expect([...sealed.keys()].sort()).toEqual([
+ 'allow-gateway-egress-to-handler-wf-wh-recipe',
+ 'allow-gateway-ingress-to-handler-wf-wh-recipe',
+ 'allow-webhook-proxy-ingress-wf-wh-recipe',
+ ])
mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
@@ -9429,13 +9477,31 @@ describe('WorkflowRecipeReconciler', () => {
}
)
mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockClear()
+ const logSpy = vi.spyOn(console, 'log').mockImplementation(() => undefined)
- await reconciler.reconcile(recipeWithWebhook())
-
- for (const name of sealed.keys()) {
- expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalledWith(
- expect.objectContaining({ name })
- )
+ try {
+ await reconciler.reconcile(recipeWithWebhook())
+
+ const logged = logSpy.mock.calls.flat().join('\n')
+ for (const name of sealed.keys()) {
+ // Negative half: the gate closed, no write went out.
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalledWith(
+ expect.objectContaining({ name })
+ )
+ // Positive half, and the reason this test is not vacuous: prove the second
+ // pass actually REACHED this policy. A negative assertion alone cannot tell
+ // "the gate closed" from "the code never ran" — patching the gateway loop to
+ // run only on the first pass leaves the negative half green. Both halves are
+ // required of any convergence test built on this shape (issue #581).
+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledWith(
+ expect.objectContaining({ name })
+ )
+ expect(logged).toContain(`NetworkPolicy "${name}" in`)
+ expect(logged).toContain('unchanged (spec-hash match); skipping update')
+ }
+ } finally {
+ logSpy.mockRestore()
}
})
diff --git a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts b/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts
index adcb5a37c..dbb541400 100644
--- a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts
+++ b/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts
@@ -3734,7 +3734,7 @@ export class WorkflowRecipeReconciler {
console.log(`[WR-Reconciler] Created ${label}`)
} catch (error: unknown) {
if (getErrorCode(error) === 409) {
- if (idempotency && (await this.applyIsNoop(idempotency))) {
+ if (idempotency && (await this.applyIsNoop(idempotency, label))) {
console.log(`[WR-Reconciler] ${label} unchanged (spec-hash match); skipping update`)
return
}
@@ -3761,14 +3761,27 @@ export class WorkflowRecipeReconciler {
* their own read in `replaceFn`: this catch is total by design (see
* `applyNetworkPolicy`).
*/
- private async applyIsNoop(idempotency: {
- manifest: { metadata?: { annotations?: { [key: string]: string } } }
- readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>
- }): Promise<boolean> {
+ private async applyIsNoop(
+ idempotency: {
+ manifest: { metadata?: { annotations?: { [key: string]: string } } }
+ readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>
+ },
+ label: string
+ ): Promise<boolean> {
try {
const existing = await idempotency.readExisting()
return specHashUnchanged(idempotency.manifest, existing ?? null)
- } catch {
+ } catch (error) {
+ // Falling through to the replace is correct — never skip an update we cannot
+ // prove unnecessary — but discarding the error left no trace of WHY: revoked
+ // RBAC or a persistently failing apiserver looked exactly like a healthy
+ // cache miss, and the gate silently stopped saving anything. The write still
+ // happens, so this is a diagnostic line, not a failure path; a genuine veto
+ // reaches the caller through `replaceFn`, loud.
+ console.warn(
+ `[WR-Reconciler] ${label}: idempotency pre-read failed, falling through to update:`,
+ error
+ )
return false
}
}
@@ -4892,14 +4905,17 @@ export class WorkflowRecipeReconciler {
*/
private async applyNetworkPolicy(policy: k8s.V1NetworkPolicy, namespace: string): Promise<void> {
const policyName = policy.metadata!.name!
- let existing: k8s.V1NetworkPolicy | undefined
+ // Set only by a read whose ownership assertion PASSED, which is what makes it
+ // safe to reuse in the replace path. Unset therefore means "no verified object
+ // in hand" — the read failed, or its assertion vetoed.
+ let assertedExisting: k8s.V1NetworkPolicy | undefined
const readExistingAndAssertOwnership = async (): Promise<k8s.V1NetworkPolicy> => {
const fetched = await this.networkingApi.readNamespacedNetworkPolicy({
name: policyName,
namespace,
})
this.assertInternalDependencyPolicyOwnership(policy, fetched, namespace)
- existing = fetched
+ assertedExisting = fetched
return fetched
}
await this.createOrReplace(
@@ -4909,9 +4925,9 @@ export class WorkflowRecipeReconciler {
body: policy,
}),
async () => {
- // `existing` is unset when the gate's read failed OR its assertion vetoed
- // (applyIsNoop swallows both); re-read so the veto surfaces from here, loud.
- const current = existing ?? (await readExistingAndAssertOwnership())
+ // Unset means the gate's read failed OR its assertion vetoed (applyIsNoop
+ // swallows both); re-read so the veto surfaces from here, loud.
+ const current = assertedExisting ?? (await readExistingAndAssertOwnership())
policy.metadata!.resourceVersion = current.metadata?.resourceVersion
return this.networkingApi.replaceNamespacedNetworkPolicy({
name: policyName,