Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(wrc): reconcile NetworkPolicies from live state

Alfredo Lopez8ddbe65ea350
diff --git a/docs/deploy/workflow-recipes-guide.md b/docs/deploy/workflow-recipes-guide.md
index dca6e5897..0ab802b46 100644
--- a/docs/deploy/workflow-recipes-guide.md
+++ b/docs/deploy/workflow-recipes-guide.md
@@ -301,53 +301,46 @@ kubectl delete workflowrecipe my-nginx -n sandbox-recipes
## Reconcile idempotency & drift behavior
-WRC reconciles every recipe on a periodic resync. Most managed objects (the recipe
-Deployment, StatefulSet, headless Service, CronJob, Job, DaemonSet, the McpServer CRD,
-and the NetworkPolicies `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*`,
-`wf-*-oauth-broker-egress` and the three webhook-gateway policies) are applied through a
-**spec-hash idempotency gate**: WRC stamps a `clerum.io/spec-hash` annotation (a hash of
-the _desired_ manifest) and, on the already-exists path, **skips the write entirely when
-the existing object's stamped hash matches**. This keeps `metadata.generation` from
-climbing on no-op reconciles (which previously produced a `degraded↔active` status flap
-and a downstream HCC NetworkPolicy no-op write storm).
-
-**What the gate does _not_ cover.** Do not read the list above as "everything":
-
-- `wl-egress-*` routes through the same method, but its manifest carries a
- `clerum.io/egress-fqdn-resolved-at` timestamp that is regenerated on every pass, so its
- desired hash never matches and **the gate never closes for it**. What actually keeps it
- quiet is the upstream issue #299 check (`egressWriteNeeded`), which compares the
- enforced egress rules. If that check ever regresses, the spec-hash gate will not catch it.
-- `ui-egress-*` and the `coordinator-to-gfs` policy do not use the spec-hash at all; they
- have their own ad-hoc comparisons (#299 and #579 respectively).
-- The **webhook gateway's own** ConfigMap, Deployment and Service are still ungated and
- rewritten on every pass — only its three NetworkPolicies were migrated.
+WRC reconciles every recipe on a periodic resync. Deployments, StatefulSets, headless
+Services, CronJobs, Jobs, DaemonSets and the McpServer CRD use the existing
+**spec-hash idempotency gate**. The recipe-derived NetworkPolicies (`ui-ingress-*`,
+`wl-ingress-*`, `wl-egress-*`, `wr-intdep-*`, `wf-*-oauth-broker-egress` and the three
+webhook-gateway policies) use a stronger **live convergence check** instead: WRC reads
+the current object, validates ownership/lifecycle, normalizes Kubernetes defaults and
+skips the write only when the live enforcement plus WRC-authored metadata equal the
+desired policy. A legacy `clerum.io/spec-hash` annotation is ignored for this decision;
+it cannot hide later drift.
+
+**What this path does _not_ cover.** Do not read the list above as "every policy writer":
+
+- `ui-egress-*` and the `coordinator-to-gfs` policy keep their existing dedicated
+ comparisons (#299 and #579 respectively); they do not use the recipe NetworkPolicy
+ helper.
+- `wl-egress-*` still uses the #299 content/renewal prefilter before the live convergence
+ helper. External/mixed egress carries temporal resolution state; a cluster-local-only
+ policy does not. In both cases, once the prefilter requires a write, the downstream
+ helper rechecks the live spec and cannot cancel a real repair because of a stale seal.
+- The **webhook gateway's own** ConfigMap, Deployment and Service remain ungated and are
+ rewritten on every pass — only its three NetworkPolicies use live convergence.
**Operational consequences:**
-- **Logs:** expect "unchanged (spec-hash match); skipping update" instead of a per-loop
- "Updated …". Per-loop write/`Updated` log volume drops sharply by design — re-baseline
- any alert keyed on it.
-- **Drift is no longer auto-reverted.** If an operator hand-edits a WRC-managed object
- (e.g. `kubectl edit`/`scale` on a recipe Deployment), WRC will **not** revert it until
- the recipe spec changes (which flips the hash). To force a re-apply, change the recipe
- spec (or delete the object and let WRC recreate it).
-- **For a Deployment the drift is still visible**: the recipe `status` reports the
- workload `degraded` when replica health drops.
-- **For a NetworkPolicy it is not visible by any path.** There is no replica health to
- degrade: the recipe stays `active` while the enforcement object diverges from the
- desired one, and nothing sweeps for it — `pruneStaleInternalDependencyPolicies`
- compares names, not content, and `deploy/scripts/verify-networkpolicies.sh` **does**
- detect content drift but only over policies an overlay renders. Recipe-derived
- policies (`ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*`, the gateway trio) are authored
- by the reconciler and live in no overlay, so the script never evaluates them at all —
- neither their existence nor their content. An edit that preserves the `clerum.io/spec-hash`
- annotation while changing the `spec` (widening `ingress`/`egress`, emptying
- `podSelector`, dropping a `policyTypes` entry) is **permanent and silent**. Treat any
- `kubectl edit` on a `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*` or gateway policy as
- permanent until a bounded resync exists (issue #581).
-- **First deploy after this change** does one stamping write per pre-existing managed
- object (they have no hash yet), then steady-state is quiet.
+- **Logs:** the live-convergence helper and the `wl-egress-*` prefilter use the structured
+ WRC logger. In steady state expect `network policy unchanged; skipping update` or
+ `network policy egress set unchanged; skipping live apply` with `policy`, `namespace`
+ and `family` fields. Re-baseline alerts keyed on the old `Updated …` console line.
+- **NetworkPolicy drift is auto-reverted.** A live change to `podSelector`,
+ `policyTypes`, `ingress`, `egress`, WRC-authored metadata or the gateway controller
+ owner forces one optimistic-concurrency replace using the `resourceVersion` from the
+ same validated read. The following reconcile is read-only.
+- **Other spec-hash-managed objects retain their existing drift contract.** For example,
+ a manually scaled recipe Deployment is not reverted until the desired manifest changes;
+ its replica-health drift remains visible in recipe status.
+- `deploy/scripts/verify-networkpolicies.sh` still covers only overlay-rendered policies.
+ Recipe-derived policy integrity is owned by the runtime reconciliation described here.
+- **First deploy after this change** does not require a NetworkPolicy stamping wave.
+ Equivalent policies are one GET and zero writes; real spec/metadata/owner drift is
+ repaired once.
## Automated Tests
diff --git a/workflow-recipes/package-lock.json b/workflow-recipes/package-lock.json
index 50a6bcac6..e63ee916f 100644
--- a/workflow-recipes/package-lock.json
+++ b/workflow-recipes/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "workload-recipes-controller",
- "version": "0.1.342",
+ "version": "0.1.343",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "workload-recipes-controller",
- "version": "0.1.342",
+ "version": "0.1.343",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@aws-sdk/client-s3": "^3.1009.0",
diff --git a/workflow-recipes/package.json b/workflow-recipes/package.json
index 03ba8569f..18fc8a122 100644
--- a/workflow-recipes/package.json
+++ b/workflow-recipes/package.json
@@ -1,6 +1,6 @@
{
"name": "workload-recipes-controller",
- "version": "0.1.342",
+ "version": "0.1.343",
"description": "Workload Recipes Controller (WRC) - WorkloadRecipe lifecycle management and MCP server interface",
"main": "dist/main.js",
"engines": {
diff --git a/workflow-recipes/src/reconciler/networkPolicyConvergence.test.ts b/workflow-recipes/src/reconciler/networkPolicyConvergence.test.ts
new file mode 100644
index 000000000..7997a107a
--- /dev/null
+++ b/workflow-recipes/src/reconciler/networkPolicyConvergence.test.ts
@@ -0,0 +1,234 @@
+import { describe, expect, it } from 'vitest'
+import * as k8s from '@kubernetes/client-node'
+import {
+ classifyNetworkPolicyOwnership,
+ decideNetworkPolicyConvergence,
+ networkPolicyMatchesDesired,
+ networkPolicyMetadataMatchesDesired,
+} from './networkPolicyConvergence'
+
+const SPEC_HASH = 'clerum.io/spec-hash'
+
+function policy(): k8s.V1NetworkPolicy {
+ return {
+ apiVersion: 'networking.k8s.io/v1',
+ kind: 'NetworkPolicy',
+ metadata: {
+ name: 'wl-ingress-r-w',
+ namespace: 'sandbox-recipes',
+ labels: {
+ 'clerum.io/managed-by': 'workflow-recipes',
+ 'clerum.io/recipe': 'r',
+ },
+ },
+ spec: {
+ podSelector: { matchLabels: { 'clerum.io/workload': 'w' } },
+ policyTypes: ['Ingress'],
+ ingress: [{ ports: [{ port: 8080, protocol: 'TCP' }] }],
+ },
+ }
+}
+
+function gatewayPolicy(ownerUid = 'uid-new'): k8s.V1NetworkPolicy {
+ return {
+ ...policy(),
+ metadata: {
+ name: 'allow-webhook-proxy-ingress-r',
+ namespace: 'sandbox-recipes',
+ labels: {
+ 'clerum.io/managed-by': 'workflow-recipes',
+ 'clerum.io/recipe-namespace': 'sandbox-recipes',
+ 'clerum.io/recipe-name': 'r',
+ 'clerum.io/webhook-gateway': 'true',
+ },
+ ownerReferences: [
+ {
+ apiVersion: 'clerum.io/v1alpha1',
+ kind: 'WorkflowRecipe',
+ name: 'r',
+ uid: ownerUid,
+ controller: true,
+ blockOwnerDeletion: true,
+ },
+ ],
+ },
+ }
+}
+
+describe('NetworkPolicy live convergence', () => {
+ it('treats API metadata/defaulting and the legacy spec-hash as equivalent', () => {
+ const desired = policy()
+ const live = structuredClone(desired)
+ live.metadata = {
+ ...live.metadata,
+ uid: 'policy-uid',
+ resourceVersion: '9',
+ generation: 3,
+ creationTimestamp: new Date('2026-09-04T00:00:00Z'),
+ annotations: { [SPEC_HASH]: 'legacy-seal' },
+ }
+ live.spec!.ingress![0].ports![0].protocol = undefined
+
+ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)
+ expect(decideNetworkPolicyConvergence('workload-ingress', desired, live)).toEqual({
+ action: 'unchanged',
+ })
+ })
+
+ it('normalizes empty rule arrays and API-defaulted policyTypes', () => {
+ const desired = policy()
+ desired.spec = { podSelector: {}, policyTypes: [], ingress: [], egress: [] }
+ const live = policy()
+ live.spec = { podSelector: {}, policyTypes: ['Ingress'] }
+
+ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)
+ })
+
+ it('normalizes the client model _from field and the Kubernetes wire from field', () => {
+ const desired = policy()
+ desired.spec!.ingress = [
+ {
+ _from: [{ podSelector: { matchLabels: { app: 'gateway' } } }],
+ ports: [{ port: 8090, protocol: 'TCP' }],
+ },
+ ]
+ const live = structuredClone(desired)
+ const liveRule = live.spec!.ingress![0] as k8s.V1NetworkPolicyIngressRule & {
+ from?: k8s.V1NetworkPolicyPeer[]
+ }
+ liveRule.from = liveRule._from
+ delete liveRule._from
+
+ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)
+ })
+
+ it('ignores admission metadata outside the keys authored by WRC', () => {
+ const desired = policy()
+ const live = structuredClone(desired)
+ live.metadata!.labels = { ...live.metadata!.labels, 'admission.example/tier': 'audited' }
+ live.metadata!.annotations = { 'admission.example/audit': 'true' }
+ live.metadata!.finalizers = ['admission.example/cleanup']
+
+ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)
+
+ live.metadata!.labels!['clerum.io/recipe'] = 'foreign'
+ expect(networkPolicyMatchesDesired(desired, live)).toBe(false)
+ })
+
+ it('repairs live enforcement drift even when the stored spec-hash is preserved', () => {
+ const desired = policy()
+ const live = structuredClone(desired)
+ live.metadata!.resourceVersion = '9'
+ live.metadata!.annotations = { [SPEC_HASH]: 'still-matches-the-old-desired' }
+ live.spec!.podSelector = {}
+ live.spec!.ingress = [{}]
+
+ expect(networkPolicyMatchesDesired(desired, live)).toBe(false)
+ expect(decideNetworkPolicyConvergence('workload-ingress', desired, live)).toEqual({
+ action: 'replace',
+ reason: 'live-drift',
+ })
+ })
+
+ it('makes workload-egress metadata drift bypass its temporal prefilter', () => {
+ const desired = policy()
+ const live = structuredClone(desired)
+ live.metadata!.annotations = { 'clerum.io/egress-fqdn-resolved-at': 'old-timestamp' }
+
+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(true)
+
+ live.metadata!.labels = { ...live.metadata!.labels, 'clerum.io/recipe': 'wrong' }
+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(false)
+
+ live.metadata!.labels = desired.metadata!.labels
+ live.metadata!.deletionTimestamp = new Date('2026-09-04T00:00:00Z')
+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(false)
+
+ delete live.metadata!.deletionTimestamp
+ live.metadata!.ownerReferences = [
+ {
+ apiVersion: 'apps/v1',
+ kind: 'Deployment',
+ name: 'foreign',
+ uid: 'foreign-uid',
+ controller: true,
+ },
+ ]
+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(false)
+ })
+
+ it('repairs a stale or missing gateway owner only for the same WRC recipe identity', () => {
+ const desired = gatewayPolicy('uid-new')
+ const stale = gatewayPolicy('uid-old')
+ const missing = gatewayPolicy('uid-new')
+ delete missing.metadata!.ownerReferences
+
+ expect(classifyNetworkPolicyOwnership('webhook-gateway', desired, stale)).toEqual({
+ kind: 'repairable-owner',
+ })
+ expect(classifyNetworkPolicyOwnership('webhook-gateway', desired, missing)).toEqual({
+ kind: 'repairable-owner',
+ })
+ expect(decideNetworkPolicyConvergence('webhook-gateway', desired, stale)).toEqual({
+ action: 'replace',
+ reason: 'owner-repair',
+ })
+ })
+
+ it('refuses a gateway policy controlled by a different owner', () => {
+ const desired = gatewayPolicy('uid-new')
+ const foreign = gatewayPolicy('uid-new')
+ foreign.metadata!.ownerReferences = [
+ {
+ apiVersion: 'apps/v1',
+ kind: 'Deployment',
+ name: 'foreign',
+ uid: 'foreign-uid',
+ controller: true,
+ },
+ ]
+
+ expect(classifyNetworkPolicyOwnership('webhook-gateway', desired, foreign)).toEqual({
+ kind: 'conflict',
+ reason: 'controller-owner-mismatch',
+ })
+ expect(decideNetworkPolicyConvergence('webhook-gateway', desired, foreign)).toEqual({
+ action: 'conflict',
+ reason: 'controller-owner-mismatch',
+ })
+ })
+
+ it('refuses a controller owner on a family that must be ownerless', () => {
+ const desired = policy()
+ const foreign = structuredClone(desired)
+ foreign.metadata!.ownerReferences = [
+ {
+ apiVersion: 'apps/v1',
+ kind: 'Deployment',
+ name: 'foreign',
+ uid: 'foreign-uid',
+ controller: true,
+ },
+ ]
+
+ expect(classifyNetworkPolicyOwnership('workload-ingress', desired, foreign)).toEqual({
+ kind: 'conflict',
+ reason: 'owner-reference-mismatch',
+ })
+ expect(decideNetworkPolicyConvergence('workload-ingress', desired, foreign)).toEqual({
+ action: 'conflict',
+ reason: 'owner-reference-mismatch',
+ })
+ })
+
+ it('never reports a terminating policy as unchanged or replaceable', () => {
+ const desired = gatewayPolicy()
+ const terminating = gatewayPolicy()
+ terminating.metadata!.deletionTimestamp = new Date('2026-09-04T00:00:00Z')
+
+ expect(decideNetworkPolicyConvergence('webhook-gateway', desired, terminating)).toEqual({
+ action: 'retry',
+ reason: 'terminating',
+ })
+ })
+})
diff --git a/workflow-recipes/src/reconciler/networkPolicyConvergence.ts b/workflow-recipes/src/reconciler/networkPolicyConvergence.ts
new file mode 100644
index 000000000..ee5ad05d7
--- /dev/null
+++ b/workflow-recipes/src/reconciler/networkPolicyConvergence.ts
@@ -0,0 +1,251 @@
+import * as k8s from '@kubernetes/client-node'
+import { SPEC_HASH_ANNOTATION } from './specHash'
+
+export type NetworkPolicyFamily =
+ | 'ui-ingress'
+ | 'workload-ingress'
+ | 'workload-egress'
+ | 'internal-dependency'
+ | 'oauth-broker-egress'
+ | 'webhook-gateway'
+
+export type NetworkPolicyOwnershipDecision =
+ | { kind: 'owned' }
+ | { kind: 'repairable-owner' }
+ | {
+ kind: 'conflict'
+ reason:
+ | 'desired-owner-uid-missing'
+ | 'identity-label-mismatch'
+ | 'controller-owner-mismatch'
+ | 'owner-reference-mismatch'
+ }
+
+export type NetworkPolicyConvergenceDecision =
+ | { action: 'unchanged' }
+ | { action: 'replace'; reason: 'live-drift' | 'owner-repair' }
+ | { action: 'retry'; reason: 'terminating' }
+ | {
+ action: 'conflict'
+ reason:
+ | 'desired-owner-uid-missing'
+ | 'identity-label-mismatch'
+ | 'controller-owner-mismatch'
+ | 'owner-reference-mismatch'
+ }
+
+const GATEWAY_IDENTITY_LABELS = [
+ 'clerum.io/managed-by',
+ 'clerum.io/recipe-namespace',
+ 'clerum.io/recipe-name',
+ 'clerum.io/webhook-gateway',
+] as const
+
+function canonicalizeValue(value: unknown): unknown {
+ if (Array.isArray(value)) return value.map(canonicalizeValue)
+ if (value && typeof value === 'object') {
+ const entries = Object.entries(value as Record<string, unknown>)
+ .filter(([, child]) => child !== undefined)
+ .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))
+ return Object.fromEntries(entries.map(([key, child]) => [key, canonicalizeValue(child)]))
+ }
+ return value
+}
+
+function projectAuthoredMap(
+ desired: Record<string, string> | undefined,
+ existing: Record<string, string> | undefined,
+ excludedKeys: ReadonlySet<string> = new Set()
+): Record<string, string> {
+ return Object.fromEntries(
+ Object.keys(desired ?? {})
+ .filter(key => !excludedKeys.has(key))
+ .flatMap(key => (existing?.[key] === undefined ? [] : [[key, existing[key]]]))
+ )
+}
+
+function projectNetworkPolicyForComparison(
+ desired: k8s.V1NetworkPolicy,
+ candidate: k8s.V1NetworkPolicy
+): k8s.V1NetworkPolicy {
+ const desiredMetadata = desired.metadata ?? {}
+ const candidateMetadata = candidate.metadata ?? {}
+ return {
+ apiVersion: candidate.apiVersion,
+ kind: candidate.kind,
+ metadata: {
+ name: candidateMetadata.name,
+ namespace: candidateMetadata.namespace,
+ labels: projectAuthoredMap(desiredMetadata.labels, candidateMetadata.labels),
+ annotations: projectAuthoredMap(
+ desiredMetadata.annotations,
+ candidateMetadata.annotations,
+ new Set([SPEC_HASH_ANNOTATION])
+ ),
+ ...(desiredMetadata.ownerReferences?.length
+ ? { ownerReferences: candidateMetadata.ownerReferences }
+ : {}),
+ },
+ spec: structuredClone(candidate.spec),
+ }
+}
+
+function normalizeIngressFrom(rule: k8s.V1NetworkPolicyIngressRule): void {
+ const record = rule as k8s.V1NetworkPolicyIngressRule & {
+ from?: k8s.V1NetworkPolicyPeer[]
+ }
+ if (record._from === undefined && record.from !== undefined) record._from = record.from
+ delete record.from
+}
+
+function normalizeProjectedNetworkPolicy(normalized: k8s.V1NetworkPolicy): unknown {
+ const spec = normalized.spec
+ if (spec) {
+ const hasEgressRules = (spec.egress?.length ?? 0) > 0
+ if (!spec.policyTypes?.length) {
+ spec.policyTypes = hasEgressRules ? ['Ingress', 'Egress'] : ['Ingress']
+ } else {
+ spec.policyTypes = [...spec.policyTypes].sort()
+ }
+ for (const rule of spec.ingress ?? []) normalizeIngressFrom(rule)
+ for (const rule of [...(spec.ingress ?? []), ...(spec.egress ?? [])]) {
+ for (const port of rule.ports ?? []) {
+ if (port.protocol === 'TCP') delete port.protocol
+ }
+ }
+ if (spec.ingress?.length === 0) delete spec.ingress
+ if (spec.egress?.length === 0) delete spec.egress
+ }
+
+ return canonicalizeValue(normalized)
+}
+
+export function normalizeNetworkPolicyForComparison(policy: k8s.V1NetworkPolicy): unknown {
+ return normalizeProjectedNetworkPolicy(projectNetworkPolicyForComparison(policy, policy))
+}
+
+export function networkPolicyMatchesDesired(
+ desired: k8s.V1NetworkPolicy | undefined,
+ existing: k8s.V1NetworkPolicy | undefined
+): boolean {
+ try {
+ if (!desired?.spec || !existing?.spec) return false
+ return (
+ JSON.stringify(
+ normalizeProjectedNetworkPolicy(projectNetworkPolicyForComparison(desired, desired))
+ ) ===
+ JSON.stringify(
+ normalizeProjectedNetworkPolicy(projectNetworkPolicyForComparison(desired, existing))
+ )
+ )
+ } catch {
+ return false
+ }
+}
+
+/**
+ * Compare the stable identity metadata that the workload-egress prefilter must
+ * never hide. Temporal DNS/state annotations are intentionally excluded: their
+ * persistence cadence is decided separately by the accumulator, while labels,
+ * lifecycle owner and a terminating object must still reach the live apply.
+ */
+export function networkPolicyMetadataMatchesDesired(
+ desired: k8s.V1NetworkPolicy,
+ existing: k8s.V1NetworkPolicy
+): boolean {
+ if (existing.metadata?.deletionTimestamp) return false
+ if (classifyNetworkPolicyOwnership('workload-egress', desired, existing).kind !== 'owned') {
+ return false
+ }
+ const project = (policy: k8s.V1NetworkPolicy) =>
+ canonicalizeValue({
+ apiVersion: policy.apiVersion,
+ kind: policy.kind,
+ name: policy.metadata?.name,
+ namespace: policy.metadata?.namespace,
+ labels: projectAuthoredMap(desired.metadata?.labels, policy.metadata?.labels),
+ ownerReferences: desired.metadata?.ownerReferences?.length
+ ? policy.metadata?.ownerReferences
+ : undefined,
+ })
+ return JSON.stringify(project(desired)) === JSON.stringify(project(existing))
+}
+
+function controllerOwners(policy: k8s.V1NetworkPolicy): k8s.V1OwnerReference[] {
+ return (policy.metadata?.ownerReferences ?? []).filter(owner => owner.controller === true)
+}
+
+function sameOwnerIdentity(desired: k8s.V1OwnerReference, existing: k8s.V1OwnerReference): boolean {
+ return (
+ desired.apiVersion === existing.apiVersion &&
+ desired.kind === existing.kind &&
+ desired.name === existing.name
+ )
+}
+
+function exactOwnerMatch(desired: k8s.V1OwnerReference, existing: k8s.V1OwnerReference): boolean {
+ return (
+ sameOwnerIdentity(desired, existing) &&
+ desired.uid === existing.uid &&
+ desired.controller === existing.controller &&
+ desired.blockOwnerDeletion === existing.blockOwnerDeletion
+ )
+}
+
+function gatewayIdentityLabelsMatch(
+ desired: k8s.V1NetworkPolicy,
+ existing: k8s.V1NetworkPolicy
+): boolean {
+ const desiredLabels = desired.metadata?.labels ?? {}
+ const existingLabels = existing.metadata?.labels ?? {}
+ return GATEWAY_IDENTITY_LABELS.every(
+ key => Boolean(desiredLabels[key]) && desiredLabels[key] === existingLabels[key]
+ )
+}
+
+export function classifyNetworkPolicyOwnership(
+ family: NetworkPolicyFamily,
+ desired: k8s.V1NetworkPolicy,
+ existing: k8s.V1NetworkPolicy
+): NetworkPolicyOwnershipDecision {
+ const desiredOwners = controllerOwners(desired)
+ const existingOwners = controllerOwners(existing)
+ if (desiredOwners.length === 0) {
+ return (existing.metadata?.ownerReferences?.length ?? 0) === 0
+ ? { kind: 'owned' }
+ : { kind: 'conflict', reason: 'owner-reference-mismatch' }
+ }
+
+ const desiredOwner = desiredOwners[0]
+ if (!desiredOwner.uid) return { kind: 'conflict', reason: 'desired-owner-uid-missing' }
+
+ if (existingOwners.length === 1 && exactOwnerMatch(desiredOwner, existingOwners[0])) {
+ return { kind: 'owned' }
+ }
+
+ if (family !== 'webhook-gateway' || !gatewayIdentityLabelsMatch(desired, existing)) {
+ return { kind: 'conflict', reason: 'identity-label-mismatch' }
+ }
+
+ if (existingOwners.length === 0) return { kind: 'repairable-owner' }
+ if (existingOwners.length === 1 && sameOwnerIdentity(desiredOwner, existingOwners[0])) {
+ return { kind: 'repairable-owner' }
+ }
+ return { kind: 'conflict', reason: 'controller-owner-mismatch' }
+}
+
+export function decideNetworkPolicyConvergence(
+ family: NetworkPolicyFamily,
+ desired: k8s.V1NetworkPolicy,
+ existing: k8s.V1NetworkPolicy
+): NetworkPolicyConvergenceDecision {
+ if (existing.metadata?.deletionTimestamp) return { action: 'retry', reason: 'terminating' }
+
+ const ownership = classifyNetworkPolicyOwnership(family, desired, existing)
+ if (ownership.kind === 'conflict') return { action: 'conflict', reason: ownership.reason }
+ if (ownership.kind === 'repairable-owner') {
+ return { action: 'replace', reason: 'owner-repair' }
+ }
+ if (networkPolicyMatchesDesired(desired, existing)) return { action: 'unchanged' }
+ return { action: 'replace', reason: 'live-drift' }
+}
diff --git a/workflow-recipes/src/reconciler/reconciler.test.ts b/workflow-recipes/src/reconciler/reconciler.test.ts
index cefa28266..bc891e54f 100644
--- a/workflow-recipes/src/reconciler/reconciler.test.ts
+++ b/workflow-recipes/src/reconciler/reconciler.test.ts
@@ -10,6 +10,7 @@ import { CronJobDef, WorkflowRecipeCRD, WorkflowRecipeGfsIntentSpec, WorkloadDef
import { INHERITED_PARENT_RESOURCES_ANNOTATION } from '../workflow/childRecipeFactory'
import { buildCoordinatorGfsNetworkPolicy } from '../workflow/networkPolicyFactory'
import { isRetryableInfraError } from './k8sErrors'
+import type { NetworkPolicyFamily } from './networkPolicyConvergence'
import * as brokerIssuer from './oauthBrokerTokenIssuerClient'
import { PLUGIN_WORKLOAD_SDK_PROVIDER_UNAVAILABLE_CONDITION_TYPE } from './pluginWorkloadSdkValidator'
import {
@@ -18,7 +19,6 @@ import {
resolveScopedWorkloadResourceName,
resolveWorkloadRuntimeResourceName,
} from './resourceBuilder'
-import { computeSpecHash, stampSpecHash } from './specHash'
import {
TRANSIENT_REQUEUE_BASE_MS,
WORKFLOW_PROGRESS_REQUEUE_BASE_MS,
@@ -224,18 +224,21 @@ describe('WorkflowRecipeReconciler', () => {
mockNetworkingApi.createNamespacedNetworkPolicy.mockReset()
mockNetworkingApi.createNamespacedNetworkPolicy.mockResolvedValue({})
mockNetworkingApi.readNamespacedNetworkPolicy.mockReset()
- mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(({ name }: { name: string }) =>
- Promise.resolve({
- metadata: {
- name,
- uid: `uid-${name}`,
- resourceVersion: '1',
- labels: {
- 'clerum.io/managed-by': 'wrc',
- 'clerum.io/recipe': name.replace(/-coordinator-to-gfs$/, ''),
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
+ ({ name }: { name: string }) => {
+ if (!name.endsWith('-coordinator-to-gfs')) return Promise.reject({ code: 404 })
+ return Promise.resolve({
+ metadata: {
+ name,
+ uid: `uid-${name}`,
+ resourceVersion: '1',
+ labels: {
+ 'clerum.io/managed-by': 'wrc',
+ 'clerum.io/recipe': name.replace(/-coordinator-to-gfs$/, ''),
+ },
},
- },
- })
+ })
+ }
)
mockNetworkingApi.replaceNamespacedNetworkPolicy.mockReset()
mockNetworkingApi.replaceNamespacedNetworkPolicy.mockResolvedValue({})
@@ -770,28 +773,27 @@ describe('WorkflowRecipeReconciler', () => {
])
})
- // ─── issue #575: the spec-hash gate applyNetworkPolicy never opted into ──────
- //
- // `createOrReplace` takes a fourth `idempotency` argument that stamps the
- // desired manifest with a SHA-256 of its content and compares it against the
- // annotation on the live object before replacing. Five callers pass it —
- // Deployment, headless Service, CronJob, Job, DaemonSet. `applyNetworkPolicy`
- // did not, so every WRC NetworkPolicy funnelling through it was rewritten on
- // every reconcile pass without comparing anything: ~12 200 PUTs/hour of objects
- // the apiserver then discarded as unchanged.
- //
- // The ownership assertion is why this is not a one-line change. It runs inside
- // the replace path today; a gate that returns first would skip it, so it moves
- // into `readExisting` — the very read the gate evaluates — and the result is
- // cached only AFTER it passes, so the replace path can never inherit a vetoed
- // object.
- describe('applyNetworkPolicy spec-hash gate (issue #575)', () => {
+ // ─── issue #575: live NetworkPolicy convergence ────────────────────────
+ describe('applyNetworkPolicy live convergence (issue #575)', () => {
const SPEC_HASH = 'clerum.io/spec-hash'
const NS = 'sandbox-recipes'
- const clone = <T>(v: T): T => JSON.parse(JSON.stringify(v)) as T
- type Priv = { applyNetworkPolicy: (p: k8s.V1NetworkPolicy, ns: string) => Promise<void> }
- const apply = (p: k8s.V1NetworkPolicy) =>
- (reconciler as unknown as Priv).applyNetworkPolicy(p, NS)
+ const clone = <T>(value: T): T => structuredClone(value)
+ type Priv = {
+ applyNetworkPolicy: (
+ policy: k8s.V1NetworkPolicy,
+ namespace: string,
+ options: { family: NetworkPolicyFamily; existing?: k8s.V1NetworkPolicy | null }
+ ) => Promise<void>
+ }
+ const apply = (
+ policy: k8s.V1NetworkPolicy,
+ family: NetworkPolicyFamily = 'workload-ingress',
+ existing?: k8s.V1NetworkPolicy | null
+ ) =>
+ (reconciler as unknown as Priv).applyNetworkPolicy(policy, NS, {
+ family,
+ ...(existing === undefined ? {} : { existing }),
+ })
const plainPolicy = (): k8s.V1NetworkPolicy => ({
apiVersion: 'networking.k8s.io/v1',
@@ -821,57 +823,101 @@ describe('WorkflowRecipeReconciler', () => {
},
})
- const ownedExisting = (extra: Record<string, unknown> = {}) => ({
+ const gatewayPolicy = (ownerUid = 'uid-current'): k8s.V1NetworkPolicy => ({
+ ...plainPolicy(),
metadata: {
- name: 'wl-ingress-r-w',
- resourceVersion: '9',
- labels: plainPolicy().metadata!.labels,
- ...extra,
+ name: 'allow-webhook-proxy-ingress-r',
+ namespace: NS,
+ labels: {
+ 'clerum.io/managed-by': 'workflow-recipes',
+ 'clerum.io/recipe-namespace': NS,
+ 'clerum.io/recipe-name': 'r',
+ 'clerum.io/webhook-gateway': 'true',
+ },
+ ownerReferences: [
+ {
+ apiVersion: 'clerum.io/v1alpha1',
+ kind: 'WorkflowRecipe',
+ name: 'r',
+ uid: ownerUid,
+ controller: true,
+ blockOwnerDeletion: true,
+ },
+ ],
},
})
- it('T1 does NOT replace a NetworkPolicy whose spec-hash is unchanged, and says so', async () => {
- await apply(plainPolicy())
- const sealed = clone(
- (
- mockNetworkingApi.createNamespacedNetworkPolicy.mock.calls.at(-1)![0] as {
- body: k8s.V1NetworkPolicy
- }
- ).body
- )
- // The exact hash, not merely "some annotation": a stamp that is present but
- // wrong would still skip the write, and would skip it for the wrong reason.
- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))
+ const livePolicy = (
+ desired: k8s.V1NetworkPolicy,
+ metadata: Partial<k8s.V1ObjectMeta> = {}
+ ): k8s.V1NetworkPolicy => ({
+ ...clone(desired),
+ metadata: {
+ ...clone(desired.metadata ?? {}),
+ resourceVersion: '9',
+ ...metadata,
+ },
+ })
+
+ function captureNetworkPolicyLogs(): {
+ entries: Array<Record<string, unknown>>
+ restore: () => void
+ } {
+ const previousLevel = process.env.LOG_LEVEL
+ process.env.LOG_LEVEL = 'info'
+ const entries: Array<Record<string, unknown>> = []
+ const spy = vi.spyOn(process.stdout, 'write').mockImplementation(((chunk: unknown) => {
+ const text = typeof chunk === 'string' ? chunk : Buffer.from(chunk as Uint8Array).toString()
+ for (const line of text.split('\n')) {
+ if (!line.startsWith('{')) continue
+ const parsed = JSON.parse(line) as Record<string, unknown>
+ if (String(parsed.msg).startsWith('network policy')) entries.push(parsed)
+ }
+ return true
+ }) as unknown as typeof process.stdout.write)
+ return {
+ entries,
+ restore: () => {
+ spy.mockRestore()
+ if (previousLevel === undefined) delete process.env.LOG_LEVEL
+ else process.env.LOG_LEVEL = previousLevel
+ },
+ }
+ }
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({
- ...sealed,
- metadata: { ...sealed.metadata, resourceVersion: '9' },
- })
- mockNetworkingApi.readNamespacedNetworkPolicy.mockClear()
- const logSpy = vi.spyOn(console, 'log').mockImplementation(() => undefined)
+ it('T1 skips a live-equivalent policy and emits a structured liveness witness', async () => {
+ const desired = plainPolicy()
+ const live = livePolicy(desired, { annotations: { [SPEC_HASH]: 'legacy-seal' } })
+ live.spec!.ingress![0].ports![0].protocol = undefined
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(live)
+ const captured = captureNetworkPolicyLogs()
try {
- await apply(plainPolicy())
- expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ await apply(desired)
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
- // The skip line is load-bearing, not decoration: it is what keeps the
- // ~16s reconcile cadence visible once the writes stop. A silent gate
- // would make the symptom of #460 invisible.
- expect(logSpy.mock.calls.flat().join(' ')).toContain(
- 'NetworkPolicy "wl-ingress-r-w" in sandbox-recipes unchanged (spec-hash match); skipping update'
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ expect(captured.entries).toContainEqual(
+ expect.objectContaining({
+ level: 'info',
+ msg: 'network policy unchanged; skipping update',
+ policy: 'wl-ingress-r-w',
+ namespace: NS,
+ family: 'workload-ingress',
+ })
)
} finally {
- logSpy.mockRestore()
+ captured.restore()
}
})
- it('T2 replaces when the spec-hash differs, reading once and using the live resourceVersion', async () => {
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(
- ownedExisting({ annotations: { [SPEC_HASH]: 'stale' } })
- )
+ it('T2 repairs live spec drift even when the legacy spec-hash is preserved', async () => {
+ const desired = plainPolicy()
+ const live = livePolicy(desired, { annotations: { [SPEC_HASH]: 'legacy-seal' } })
+ live.spec!.podSelector = {}
+ live.spec!.ingress = [{}]
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(live)
- await apply(plainPolicy())
+ await apply(desired)
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
const body = (
@@ -880,155 +926,265 @@ describe('WorkflowRecipeReconciler', () => {
}
).body
expect(body.metadata?.resourceVersion).toBe('9')
- expect(body.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))
- // One read serves both the gate and the replace, so the replace cannot act
- // on a different resourceVersion than the one the gate evaluated.
+ expect(body.metadata?.annotations?.[SPEC_HASH]).toBeUndefined()
+ expect(body.spec).toEqual(desired.spec)
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
})
- it('T3 writes once to seal a pre-existing policy with no spec-hash, then skips', async () => {
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(ownedExisting())
+ it('does not let a supplied cluster-local wl-egress snapshot veto detected drift', async () => {
+ const desired = plainPolicy()
+ desired.metadata!.name = 'wl-egress-r-w'
+ desired.spec = {
+ podSelector: { matchLabels: { 'clerum.io/workload': 'w' } },
+ policyTypes: ['Egress'],
+ egress: [
+ {
+ to: [{ podSelector: { matchLabels: { 'clerum.io/workload': 'db' } } }],
+ ports: [{ port: 5432, protocol: 'TCP' }],
+ },
+ ],
+ }
+ const live = livePolicy(desired, { annotations: { [SPEC_HASH]: 'legacy-seal' } })
+ live.spec!.podSelector = {}
+
+ await apply(desired, 'workload-egress', live)
- await apply(plainPolicy())
+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).not.toHaveBeenCalled()
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
- const sealed = clone(
+ expect(
(
mockNetworkingApi.replaceNamespacedNetworkPolicy.mock.calls[0][0] as {
body: k8s.V1NetworkPolicy
}
+ ).body.spec?.podSelector
+ ).toEqual(desired.spec.podSelector)
+ })
+
+ it('routes workload-egress label drift through the live apply prefilter', () => {
+ const desired = plainPolicy()
+ desired.metadata!.name = 'wl-egress-r-w'
+ desired.spec = { podSelector: {}, policyTypes: ['Egress'], egress: [] }
+ const live = livePolicy(desired)
+ live.metadata!.labels = { ...live.metadata!.labels, 'clerum.io/recipe': 'wrong' }
+
+ const writeNeeded = (
+ reconciler as unknown as {
+ egressWriteNeeded: (
+ existing: k8s.V1NetworkPolicy | null,
+ wanted: k8s.V1NetworkPolicy
+ ) => boolean
+ }
+ ).egressWriteNeeded(live, desired)
+
+ expect(writeNeeded).toBe(true)
+
+ live.metadata!.labels = desired.metadata!.labels
+ live.metadata!.ownerReferences = [
+ {
+ apiVersion: 'apps/v1',
+ kind: 'Deployment',
+ name: 'foreign',
+ uid: 'foreign-uid',
+ controller: true,
+ },
+ ]
+ expect(
+ (
+ reconciler as unknown as {
+ egressWriteNeeded: (
+ existing: k8s.V1NetworkPolicy | null,
+ wanted: k8s.V1NetworkPolicy
+ ) => boolean
+ }
+ ).egressWriteNeeded(live, desired)
+ ).toBe(true)
+ })
+
+ it('T3 creates on 404 and the next equivalent reconcile is read-only', async () => {
+ const desired = plainPolicy()
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 404 })
+
+ await apply(desired)
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ const created = clone(
+ (
+ mockNetworkingApi.createNamespacedNetworkPolicy.mock.calls[0][0] as {
+ body: k8s.V1NetworkPolicy
+ }
).body
)
- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))
+ expect(created.metadata?.annotations?.[SPEC_HASH]).toBeUndefined()
- mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(sealed)
+ mockNetworkingApi.createNamespacedNetworkPolicy.mockClear()
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(livePolicy(created))
- await apply(plainPolicy())
+ await apply(desired)
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
})
- it('T4 does not let a matching spec-hash shortcut the ownership assertion', async () => {
+ it('T4 propagates a foreign intdep ownership veto after exactly one read', async () => {
const desired = intdepPolicy()
- const sameHash = stampSpecHash(clone(desired))
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
- // Foreign owner, carrying a hash the gate would happily match on.
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({
- metadata: {
- name: 'wr-intdep-egress-r-w',
- resourceVersion: '9',
- annotations: { [SPEC_HASH]: sameHash },
- labels: { 'clerum.io/managed-by': 'hcc' },
- },
- })
+ mockNetworkingApi.readNamespacedNetworkPolicy
+ .mockResolvedValueOnce({
+ metadata: {
+ name: 'wr-intdep-egress-r-w',
+ resourceVersion: '9',
+ labels: { 'clerum.io/managed-by': 'hcc' },
+ },
+ })
+ .mockRejectedValueOnce({ code: 503 })
- const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined)
- const rejection = await apply(desired).then(
- () => {
- throw new Error('expected the ownership assertion to reject')
- },
- (e: unknown) => e as Error
+ const rejection = await apply(desired, 'internal-dependency').then(
+ () => new Error('expected the ownership assertion to reject'),
+ (error: unknown) => error as Error
)
- // Pin the CLASS, not just the message. `reconcileInternalDependencyPolicies`
- // keys the CRD condition `OwnershipConflict` off `instanceof`, so a bare
- // Error carrying the same text would set the wrong condition while passing a
- // message-only assertion. The class is module-local — exporting it just for
- // this test would be production code shaped by a test — so pin its `name`.
expect(rejection.name).toBe('NetworkPolicyOwnershipConflictError')
expect(rejection.message).toMatch(/Refusing to replace NetworkPolicy "wr-intdep-egress-r-w"/)
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
- // Two reads: the gate's read raises the veto, `applyIsNoop` treats any
- // throw as "cannot prove this is a no-op" and falls through, and the replace
- // path re-reads and re-asserts, where the veto propagates. The write never
- // happens either way. What this test rules out is the assertion living ONLY
- // in the replace path: there, a matching hash would make the gate skip and
- // this call would resolve instead of rejecting.
- expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ })
- // Review of #580, R1-L1. A veto is not a read failure, and `applyIsNoop`
- // must not announce it as one: the read SUCCEEDED and the caller's
- // assertion on the result is what threw. The accurate message arrives a
- // moment later from `replaceFn`, so a "pre-read failed" line here would only
- // precede the truth with a wrong diagnosis.
- //
- // Scoped to that one line rather than asserting console.warn was never
- // called: other warnings on this path are legitimate, and a blanket
- // assertion would break on any of them for reasons unrelated to the veto.
- const warned = warnSpy.mock.calls.flat().join(' ')
- expect(warned).not.toContain('idempotency pre-read failed')
- warnSpy.mockRestore()
- })
-
- it('T4b lets an internal-dependency policy WRC already owns through the gate', async () => {
- // T4 pins the veto; nothing pinned the other side. An assertion that rejected
- // EVERY internal-dependency policy would leave T4 green and silently stop the
- // whole intdep family from converging — the gate would never be reached and
- // every pass would set an OwnershipConflict condition. This is the test that
- // makes T4 mean "vetoes foreign owners" rather than "always vetoes".
+ it('T4b lets an intdep policy WRC owns converge read-only', async () => {
const desired = intdepPolicy()
- const ownHash = stampSpecHash(clone(desired))
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({
- metadata: {
- name: 'wr-intdep-egress-r-w',
- resourceVersion: '9',
- annotations: { [SPEC_HASH]: ownHash },
- labels: desired.metadata!.labels,
- },
- })
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(livePolicy(desired))
- await expect(apply(desired)).resolves.toBeUndefined()
+ await expect(apply(desired, 'internal-dependency')).resolves.toBeUndefined()
- expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
- // One read: the assertion passed inside the gate's own read, so the replace
- // path is never entered and never re-reads. Contrast T4's two.
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
})
- it('T5 falls through to the replace when the gate read fails with an HTTP status', async () => {
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
+ it('T5 retries an HTTP read failure, replaces with the recovered RV, and logs safely', async () => {
+ const desired = plainPolicy()
mockNetworkingApi.readNamespacedNetworkPolicy
- .mockRejectedValueOnce({ code: 500 })
- .mockResolvedValueOnce(ownedExisting())
+ .mockRejectedValueOnce({ code: 500, authorization: 'Bearer must-not-appear' })
+ .mockResolvedValueOnce(livePolicy(desired))
+ const captured = captureNetworkPolicyLogs()
+ try {
+ await apply(desired)
+
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
+ expect(
+ (
+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mock.calls[0][0] as {
+ body: k8s.V1NetworkPolicy
+ }
+ ).body.metadata?.resourceVersion
+ ).toBe('9')
+ const warning = captured.entries.find(
+ entry => entry.msg === 'network policy read unavailable; retrying once'
+ )
+ expect(warning).toEqual(
+ expect.objectContaining({
+ level: 'warn',
+ errorName: 'UnknownError',
+ errorCode: 500,
+ retryable: true,
+ })
+ )
+ expect(JSON.stringify(warning)).not.toContain('must-not-appear')
+ } finally {
+ captured.restore()
+ }
+ })
+
+ it('T5b retries a transport read failure once and replaces', async () => {
+ const desired = plainPolicy()
+ mockNetworkingApi.readNamespacedNetworkPolicy
+ .mockRejectedValueOnce({ code: 'ECONNRESET' })
+ .mockResolvedValueOnce(livePolicy(desired))
- await apply(plainPolicy())
+ await apply(desired)
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
})
- it('T5b falls through to the replace on a retryable transport error', async () => {
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
+ it('T5c retries a codeless read failure once and replaces', async () => {
+ const desired = plainPolicy()
mockNetworkingApi.readNamespacedNetworkPolicy
- .mockRejectedValueOnce({ code: 'ECONNRESET' })
- .mockResolvedValueOnce(ownedExisting())
+ .mockRejectedValueOnce(new Error('transient read timeout'))
+ .mockResolvedValueOnce(livePolicy(desired))
- await apply(plainPolicy())
+ await apply(desired)
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
})
- it('T5c falls through on a codeless read error, the shape `:1354` pins for Deployment', async () => {
- // The rule cannot key on the shape of the error. A bare Error with no code
- // is a legitimate read failure — the existing Deployment test injects
- // exactly `new Error('transient read timeout')` and asserts the write still
- // happens — so `applyIsNoop` swallows everything and falls open. A version
- // of this gate that classified errors instead broke that test; this pins the
- // behaviour for NetworkPolicy so the same idea cannot come back quietly.
- //
- // Not redundant with T5b despite the total catch: T5b's error carries a
- // `code`, and every earlier draft that regressed this did so by keying on
- // `code`. The codeless shape is the one that broke the Deployment test.
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
- mockNetworkingApi.readNamespacedNetworkPolicy
- .mockRejectedValueOnce(new Error('transient read timeout'))
- .mockResolvedValueOnce(ownedExisting())
+ it('repairs the gateway owner UID and refuses a foreign controller owner', async () => {
+ const desired = gatewayPolicy('uid-current')
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(
+ livePolicy(gatewayPolicy('uid-old'), { annotations: { [SPEC_HASH]: 'legacy-seal' } })
+ )
- await apply(plainPolicy())
+ await apply(desired, 'webhook-gateway')
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ const body = (
+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mock.calls[0][0] as {
+ body: k8s.V1NetworkPolicy
+ }
+ ).body
+ expect(body.metadata?.ownerReferences?.[0].uid).toBe('uid-current')
+
+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()
+ const foreign = livePolicy(desired)
+ foreign.metadata!.ownerReferences = [
+ {
+ apiVersion: 'apps/v1',
+ kind: 'Deployment',
+ name: 'foreign',
+ uid: 'foreign-uid',
+ controller: true,
+ },
+ ]
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(foreign)
+
+ await expect(apply(desired, 'webhook-gateway')).rejects.toMatchObject({
+ name: 'NetworkPolicyOwnershipConflictError',
+ })
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ })
+
+ it('handles a read/create race by re-reading instead of blind replacing', async () => {
+ const desired = plainPolicy()
+ mockNetworkingApi.readNamespacedNetworkPolicy
+ .mockRejectedValueOnce({ code: 404 })
+ .mockResolvedValueOnce(livePolicy(desired))
+ mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 409 })
+
+ await apply(desired)
+
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ })
+
+ it.each([404, 409])('classifies replace race status %s as retryable', async code => {
+ const desired = plainPolicy()
+ const live = livePolicy(desired)
+ live.spec!.podSelector = {}
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(live)
+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mockRejectedValueOnce({ code })
+
+ await expect(apply(desired)).rejects.toMatchObject({
+ name: 'RetryableReconcileError',
+ cause: { code },
+ })
+ })
+
+ it('refuses to create a gateway policy without a complete desired owner', async () => {
+ const desired = gatewayPolicy('')
+
+ await expect(apply(desired, 'webhook-gateway')).rejects.toMatchObject({
+ name: 'NetworkPolicyOwnershipConflictError',
+ })
+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()
})
})
@@ -1919,17 +2075,21 @@ describe('WorkflowRecipeReconciler', () => {
})
it('fails closed instead of replacing an existing non-WRC internal-dependency policy', async () => {
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 409 })
- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValueOnce({
- metadata: {
- resourceVersion: 'rv-hcc',
- labels: {
- 'clerum.io/managed-by': 'host-capability-controller',
- 'clerum.io/policy-type': 'binding-allow',
- 'clerum.io/recipe': 'test-recipe',
- },
- },
- })
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
+ ({ name }: { name: string }) => {
+ if (!name.startsWith('wr-intdep-')) return Promise.reject({ code: 404 })
+ return Promise.resolve({
+ metadata: {
+ resourceVersion: 'rv-hcc',
+ labels: {
+ 'clerum.io/managed-by': 'host-capability-controller',
+ 'clerum.io/policy-type': 'binding-allow',
+ 'clerum.io/recipe': 'test-recipe',
+ },
+ },
+ })
+ }
+ )
const result = await reconciler.reconcile(
makeRecipe({
@@ -1956,6 +2116,163 @@ describe('WorkflowRecipeReconciler', () => {
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
})
+ it('keeps a recipe retryable when an internal-dependency policy is terminating', async () => {
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
+ ({ name }: { name: string }) => {
+ if (!name.startsWith('wr-intdep-')) return Promise.reject({ code: 404 })
+ return Promise.resolve({
+ metadata: {
+ name,
+ resourceVersion: 'rv-terminating',
+ deletionTimestamp: new Date('2026-09-04T00:00:00Z'),
+ labels: {
+ 'clerum.io/managed-by': 'host-capability-controller',
+ 'clerum.io/policy-type': 'binding-allow',
+ 'clerum.io/recipe': 'test-recipe',
+ },
+ },
+ })
+ }
+ )
+
+ const result = await reconciler.reconcile(
+ makeRecipe({
+ spec: {
+ workloads: [
+ {
+ id: 'api',
+ type: 'deployment',
+ image: 'api:test',
+ port: 8080,
+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],
+ },
+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },
+ ],
+ },
+ })
+ )
+
+ expect(result.phase).toBe('degraded')
+ expect(result.message).toMatch(/is terminating; retrying after deletion/)
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ })
+
+ it('keeps a recipe retryable when an internal-dependency policy disappears before replace', async () => {
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
+ ({ name }: { name: string }) => {
+ if (!name.startsWith('wr-intdep-')) return Promise.reject({ code: 404 })
+ return Promise.resolve({
+ metadata: {
+ name,
+ resourceVersion: 'rv-before-delete',
+ labels: {
+ 'clerum.io/managed-by': 'workflow-recipes',
+ 'clerum.io/policy-type': 'internal-dependency',
+ 'clerum.io/recipe': 'test-recipe',
+ },
+ },
+ spec: { podSelector: {}, policyTypes: ['Ingress'] },
+ })
+ }
+ )
+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 404 })
+
+ const result = await reconciler.reconcile(
+ makeRecipe({
+ spec: {
+ workloads: [
+ {
+ id: 'api',
+ type: 'deployment',
+ image: 'api:test',
+ port: 8080,
+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],
+ },
+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },
+ ],
+ },
+ })
+ )
+
+ expect(result.phase).toBe('degraded')
+ expect(result.message).toMatch(/disappeared during replace/)
+ })
+
+ it.each([{ code: 500 }, new Error('network timeout')])(
+ 'preserves an active recipe and requeues when internal-dependency reads stay unavailable (%s)',
+ async readError => {
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
+ ({ name }: { name: string }) =>
+ name.startsWith('wr-intdep-') ? Promise.reject(readError) : Promise.reject({ code: 404 })
+ )
+
+ const result = await reconciler.reconcile(
+ makeRecipe({
+ status: { phase: 'active', message: 'healthy before API outage', workloads: [] },
+ spec: {
+ workloads: [
+ {
+ id: 'api',
+ type: 'deployment',
+ image: 'api:test',
+ port: 8080,
+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],
+ },
+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },
+ ],
+ },
+ })
+ )
+
+ expect(result.phase).toBe('active')
+ expect(result.message).toBe('healthy before API outage')
+ expect(result.skipStatusPatch).toBe(true)
+ expect(result.requeueAfterMs).toBe(TRANSIENT_REQUEUE_BASE_MS)
+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ }
+ )
+
+ it('keeps a workflow non-terminal when internal-dependency reads stay unavailable', async () => {
+ const workflowReconcile = vi.fn()
+ ;(
+ reconciler as unknown as {
+ workflowReconciler: {
+ reconcile: typeof workflowReconcile
+ validateWorkflowSpec: () => undefined
+ }
+ }
+ ).workflowReconciler = { reconcile: workflowReconcile, validateWorkflowSpec: () => undefined }
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
+ ({ name }: { name: string }) =>
+ name.startsWith('wr-intdep-')
+ ? Promise.reject({ code: 500 })
+ : Promise.reject({ code: 404 })
+ )
+
+ const result = await reconciler.reconcile(
+ makeRecipe({
+ spec: {
+ workloads: [
+ {
+ id: 'api',
+ type: 'deployment',
+ image: 'api:test',
+ port: 8080,
+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],
+ },
+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },
+ ],
+ steps: [{ id: 'run', run: snippetRun() }],
+ },
+ })
+ )
+
+ expect(result.phase).toBe('degraded')
+ expect(result.workflowPhase).toBeUndefined()
+ expect(result.message).toBe('Workflow workload infrastructure temporarily unavailable (500)')
+ expect(workflowReconcile).not.toHaveBeenCalled()
+ })
+
it('fails closed before workflow execution when a workflow StatefulSet hits a foreign PVC', async () => {
const workflowReconcile = vi.fn().mockResolvedValue({
phase: 'deploying',
@@ -9454,16 +9771,9 @@ describe('WorkflowRecipeReconciler', () => {
expect(allFalse).toBe(true)
})
- it('T6 (#575) applies the 3 gateway NetworkPolicies through the spec-hash gate', async () => {
- // Two jobs. First: the gateway policies used to be written by an inline
- // create-or-replace with no gate, so they were three no-op PUTs per pass.
- // Second, and the reason this test is here rather than asserted by
- // inspection: it proves the gateway builder is DETERMINISTIC. A gate only
- // closes if the manifest hashes the same on the next pass, and a builder
- // that iterated a Set or stamped a timestamp would leave the gate open
- // forever — the failure mode of #413 and #579. If this test goes red, the
- // builder acquired non-determinism; do not relax the assertion.
+ it('T6 (#575) converges the 3 gateway NetworkPolicies through live reads', async () => {
mockNetworkingApi.createNamespacedNetworkPolicy.mockClear()
+ mockNetworkingApi.readNamespacedNetworkPolicy.mockRejectedValue({ code: 404 })
await reconciler.reconcile(recipeWithWebhook())
const sealed = new Map<string, unknown>()
@@ -9479,7 +9789,6 @@ describe('WorkflowRecipeReconciler', () => {
'allow-webhook-proxy-ingress-wf-wh-recipe',
])
- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })
mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(
({ name }: { name: string }) => {
const previous = sealed.get(name) as k8s.V1NetworkPolicy | undefined
@@ -9491,31 +9800,44 @@ describe('WorkflowRecipeReconciler', () => {
}
)
mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()
+ mockNetworkingApi.createNamespacedNetworkPolicy.mockClear()
mockNetworkingApi.readNamespacedNetworkPolicy.mockClear()
- const logSpy = vi.spyOn(console, 'log').mockImplementation(() => undefined)
+ const previousLevel = process.env.LOG_LEVEL
+ process.env.LOG_LEVEL = 'info'
+ const logEntries: Array<Record<string, unknown>> = []
+ const logSpy = vi.spyOn(process.stdout, 'write').mockImplementation(((chunk: unknown) => {
+ const text = typeof chunk === 'string' ? chunk : Buffer.from(chunk as Uint8Array).toString()
+ for (const line of text.split('\n')) {
+ if (!line.startsWith('{')) continue
+ const parsed = JSON.parse(line) as Record<string, unknown>
+ if (parsed.msg === 'network policy unchanged; skipping update') logEntries.push(parsed)
+ }
+ return true
+ }) as unknown as typeof process.stdout.write)
try {
await reconciler.reconcile(recipeWithWebhook())
- const logged = logSpy.mock.calls.flat().join('\n')
for (const name of sealed.keys()) {
- // Negative half: the gate closed, no write went out.
expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalledWith(
expect.objectContaining({ name })
)
- // Positive half, and the reason this test is not vacuous: prove the second
- // pass actually REACHED this policy. A negative assertion alone cannot tell
- // "the gate closed" from "the code never ran" — patching the gateway loop to
- // run only on the first pass leaves the negative half green. Both halves are
- // required of any convergence test built on this shape (issue #581).
expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledWith(
expect.objectContaining({ name })
)
- expect(logged).toContain(`NetworkPolicy "${name}" in`)
- expect(logged).toContain('unchanged (spec-hash match); skipping update')
+ expect(logEntries).toContainEqual(
+ expect.objectContaining({
+ policy: name,
+ family: 'webhook-gateway',
+ msg: 'network policy unchanged; skipping update',
+ })
+ )
}
+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()
} finally {
logSpy.mockRestore()
+ if (previousLevel === undefined) delete process.env.LOG_LEVEL
+ else process.env.LOG_LEVEL = previousLevel
}
})
diff --git a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts b/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts
index 0bc37f1cf..20df497a0 100644
--- a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts
+++ b/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts
@@ -89,6 +89,11 @@ import {
waitForExternalEgressReady,
waitForNetworkReady,
} from './mcpDelegation'
+import {
+ type NetworkPolicyFamily,
+ decideNetworkPolicyConvergence,
+ networkPolicyMetadataMatchesDesired,
+} from './networkPolicyConvergence'
import { issueOAuthBrokerToken } from './oauthBrokerTokenIssuerClient'
import {
PLUGIN_WORKLOAD_SDK_CONDITION_TYPE,
@@ -472,6 +477,26 @@ class NetworkPolicyOwnershipConflictError extends Error {
}
}
+class NetworkPolicyInfraError extends Error {
+ readonly code?: string | number
+
+ constructor(message: string, cause: unknown, code?: string | number) {
+ super(message, { cause })
+ this.name = 'NetworkPolicyInfraError'
+ this.code = code
+ }
+}
+
+interface NetworkPolicyApplyOptions {
+ family: NetworkPolicyFamily
+ existing?: k8s.V1NetworkPolicy | null
+}
+
+interface NetworkPolicyReadResult {
+ existing: k8s.V1NetworkPolicy | null
+ recoveredFromReadFailure: boolean
+}
+
class InternalDependencyReconcileError extends Error {
constructor(
message: string,
@@ -1904,13 +1929,21 @@ export class WorkflowRecipeReconciler {
error instanceof InternalDependencyReconcileError ? error.conditions : undefined
const workloadConditions =
error instanceof ImmutableStatefulSetDriftError ? [error.condition] : undefined
- if (error instanceof RetryableReconcileError) {
+ if (
+ error instanceof RetryableReconcileError ||
+ error instanceof NetworkPolicyInfraError
+ ) {
+ const safe = this.safeInfrastructureErrorFields(error)
+ const retryableMessage =
+ error instanceof RetryableReconcileError
+ ? message
+ : `Workflow workload infrastructure temporarily unavailable (${safe.errorCode ?? safe.errorName})`
// Transient infra failure (e.g. DNS SERVFAIL on egress). Degrade
// instead of failing so the next reconcile retries before the
// workflow runs — don't mark workflowPhase failed.
return {
phase: 'degraded' as RecipePhase,
- message,
+ message: retryableMessage,
workloadStatuses: [],
internalDependencyConditions,
workloadConditions,
@@ -2700,7 +2733,10 @@ export class WorkflowRecipeReconciler {
// decision and hiding the permanent misconfiguration.
const isEgressResolutionError =
error instanceof Error && error.message.includes('egress resolution failed')
- if (!isEgressResolutionError && isRetryableInfraError(error)) {
+ if (
+ !isEgressResolutionError &&
+ (error instanceof NetworkPolicyInfraError || isRetryableInfraError(error))
+ ) {
console.warn(
`[WR-Reconciler] Transient infra error reconciling "${name}" — keeping phase "${currentPhase}", will retry: ${String(error)}`
)
@@ -3031,19 +3067,16 @@ export class WorkflowRecipeReconciler {
`WebhookGateway Service "${gatewayServiceName(recipeName)}"`
)
- // Same spec-hash gate as every other WRC NetworkPolicy (issue #575): these
- // three are rebuilt verbatim from the recipe on every pass, so the inline
- // replace was three no-op PUTs per reconcile. The wr-intdep-* ownership
- // assertion inside applyNetworkPolicy returns immediately for these — their
- // policy-type label differs — so routing through it changes the write gate and
- // nothing else. The log label changes from `WebhookGateway NetworkPolicy "x"`
- // to `NetworkPolicy "x" in <ns>`, which no test, alert or doc consumes.
+ // The gateway policies use the same live convergence path as other
+ // WRC-authored NetworkPolicies (issue #575). Their controller ownerReference
+ // is part of the comparison, so recreating a recipe under the same name repairs
+ // a stale UID instead of accepting the old garbage-collection lineage.
for (const policy of [
built.proxyIngressPolicy,
built.handlerEgressPolicy,
built.handlerIngressPolicy,
]) {
- await this.applyNetworkPolicy(policy, ns)
+ await this.applyNetworkPolicy(policy, ns, { family: 'webhook-gateway' })
}
}
@@ -3734,7 +3767,7 @@ export class WorkflowRecipeReconciler {
console.log(`[WR-Reconciler] Created ${label}`)
} catch (error: unknown) {
if (getErrorCode(error) === 409) {
- if (idempotency && (await this.applyIsNoop(idempotency, label))) {
+ if (idempotency && (await this.applyIsNoop(idempotency))) {
console.log(`[WR-Reconciler] ${label} unchanged (spec-hash match); skipping update`)
return
}
@@ -3754,43 +3787,17 @@ export class WorkflowRecipeReconciler {
/**
* True when the existing object already carries the desired spec-hash, so the
- * replace would be a no-op. Anything thrown by `readExisting` — a read failure,
- * or an invariant the caller asserts on what it just read — returns false and
- * falls through to the replace: never skip an update we cannot prove is
- * unnecessary. Callers whose `readExisting` asserts invariants must re-assert on
- * their own read in `replaceFn`: this catch is total by design (see
- * `applyNetworkPolicy`).
+ * replace would be a no-op. A read failure returns false (fall through to
+ * replace) — never skip an update we cannot prove is unnecessary.
*/
- private async applyIsNoop(
- idempotency: {
- manifest: { metadata?: { annotations?: { [key: string]: string } } }
- readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>
- },
- label: string
- ): Promise<boolean> {
+ private async applyIsNoop(idempotency: {
+ manifest: { metadata?: { annotations?: { [key: string]: string } } }
+ readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>
+ }): Promise<boolean> {
try {
const existing = await idempotency.readExisting()
return specHashUnchanged(idempotency.manifest, existing ?? null)
- } catch (error) {
- // An ownership veto is NOT a read failure, and saying so here would be
- // actively misleading: the read succeeded, and what threw was the caller's
- // assertion on what it read. Stay quiet — `replaceFn` re-reads, re-asserts
- // and raises the conflict with its own message a moment later, so the
- // accurate line follows immediately and this one would only precede it with
- // a wrong diagnosis. Nothing is swallowed; the veto still reaches the
- // caller, loud (T4).
- if (!(error instanceof NetworkPolicyOwnershipConflictError)) {
- // Everything else IS a read failure. Falling through to the replace is
- // correct — never skip an update we cannot prove unnecessary — but
- // discarding the error left no trace of WHY: revoked RBAC or a
- // persistently failing apiserver looked exactly like a healthy cache
- // miss, and the gate silently stopped saving anything. The write still
- // happens, so this is a diagnostic line, not a failure path.
- console.warn(
- `[WR-Reconciler] ${label}: idempotency pre-read failed, falling through to update:`,
- error
- )
- }
+ } catch {
return false
}
}
@@ -4297,7 +4304,7 @@ export class WorkflowRecipeReconciler {
await this.deleteUiIngressIfExists(recipe.metadata.name, w.id, ns)
continue
}
- await this.applyNetworkPolicy(policy, ns)
+ await this.applyNetworkPolicy(policy, ns, { family: 'ui-ingress' })
}
}
@@ -4447,7 +4454,7 @@ export class WorkflowRecipeReconciler {
)
if (!policy) continue
rememberDesired(policy)
- await this.applyNetworkPolicy(policy, sourceNamespace)
+ await this.applyNetworkPolicy(policy, sourceNamespace, { family: 'internal-dependency' })
}
for (const [targetWorkloadId, sources] of sourcesByTarget.entries()) {
@@ -4462,11 +4469,14 @@ export class WorkflowRecipeReconciler {
)
if (!policy) continue
rememberDesired(policy)
- await this.applyNetworkPolicy(policy, targetNamespace)
+ await this.applyNetworkPolicy(policy, targetNamespace, { family: 'internal-dependency' })
}
await this.pruneStaleInternalDependencyPolicies(recipe, desiredByNamespace)
} catch (error) {
+ if (error instanceof RetryableReconcileError || error instanceof NetworkPolicyInfraError) {
+ throw error
+ }
const reason =
error instanceof NetworkPolicyOwnershipConflictError
? 'OwnershipConflict'
@@ -4669,12 +4679,16 @@ export class WorkflowRecipeReconciler {
!wlEgressRenewalDue &&
!wlEgressStateChanged
) {
- console.log(
- `[WR-Reconciler] NetworkPolicy "${wlPolicyName}" in ${wlNs} egress set unchanged — no-op`
+ createLogger('wrc', recipe.metadata.name).info(
+ 'network policy egress set unchanged; skipping live apply',
+ { policy: wlPolicyName, namespace: wlNs, family: 'workload-egress' }
)
continue
}
- await this.applyNetworkPolicy(policy, wlNs)
+ await this.applyNetworkPolicy(policy, wlNs, {
+ family: 'workload-egress',
+ existing: existingWlPolicy,
+ })
// #299: the policy has landed — record the set it actually enforces.
this.logResolvedEgressSet(
recipe.metadata.name,
@@ -4701,7 +4715,7 @@ export class WorkflowRecipeReconciler {
await this.deleteWorkloadIngressIfExists(recipe.metadata.name, w.id, wlNs)
continue
}
- await this.applyNetworkPolicy(policy, wlNs)
+ await this.applyNetworkPolicy(policy, wlNs, { family: 'workload-ingress' })
}
}
@@ -4782,6 +4796,7 @@ export class WorkflowRecipeReconciler {
desired: k8s.V1NetworkPolicy
): boolean {
if (!existing) return true
+ if (!networkPolicyMetadataMatchesDesired(desired, existing)) return true
// Decide the write off the ENFORCED rules (the ipBlock set + ports), NOT the
// raw state annotation: serializeState embeds expiresAt/lastObservedAt, which
// renew on every OK tick, so comparing the annotation string would rewrite
@@ -4894,59 +4909,213 @@ export class WorkflowRecipeReconciler {
}
/**
- * Apply a WRC-authored NetworkPolicy through the spec-hash idempotency gate
- * (issue #575). Every family funnelling through here — ui-ingress, wl-ingress,
- * wr-intdep-*, oauth-broker egress, webhook-gateway, and wl-egress once #299 has
- * decided a write is due — is rebuilt deterministically from the recipe on every
- * pass, so the unconditional replace was one PUT per policy per ~16s that etcd
- * never recorded: pure apiserver CPU and priority-and-fairness budget.
- * Deployment, Service, CronJob, Job and DaemonSet already opted in; this was the
- * kind whose skip counter stayed at zero.
- *
- * The wr-intdep-* ownership assertion runs INSIDE `readExisting`, i.e. on the
- * very read the gate evaluates, so a spec-hash match can never shortcut it.
- * `applyIsNoop` swallows EVERY error `readExisting` throws by design — a read
- * failure must fall open to the replace — so the veto reaches the caller through
- * `replaceFn`, which re-reads and re-asserts. The read is cached only AFTER the
- * assertion passed: the replace path can never inherit a vetoed object, and the
- * PUT carries the resourceVersion of the object that passed, so the apiserver
- * rejects any change in between.
+ * Reconcile one WRC-authored NetworkPolicy from the live object, not from the
+ * legacy spec-hash seal. NetworkPolicy is an enforcement resource: a stored
+ * annotation can describe what WRC wrote previously while the live spec or
+ * lifecycle ownership has since drifted. One validated live snapshot therefore
+ * owns both the no-op decision and the resourceVersion used by a replace.
*/
- private async applyNetworkPolicy(policy: k8s.V1NetworkPolicy, namespace: string): Promise<void> {
- const policyName = policy.metadata!.name!
- // Set only by a read whose ownership assertion PASSED, which is what makes it
- // safe to reuse in the replace path. Unset therefore means "no verified object
- // in hand" — the read failed, or its assertion vetoed.
- let assertedExisting: k8s.V1NetworkPolicy | undefined
- const readExistingAndAssertOwnership = async (): Promise<k8s.V1NetworkPolicy> => {
- const fetched = await this.networkingApi.readNamespacedNetworkPolicy({
+ private async applyNetworkPolicy(
+ policy: k8s.V1NetworkPolicy,
+ namespace: string,
+ options: NetworkPolicyApplyOptions
+ ): Promise<void> {
+ const desired = structuredClone(policy)
+ const policyName = desired.metadata?.name
+ if (!policyName) throw new Error('Cannot apply a NetworkPolicy without metadata.name')
+ desired.metadata = desired.metadata ?? {}
+ desired.metadata.namespace = desired.metadata.namespace ?? namespace
+ this.assertDesiredNetworkPolicyOwner(desired, options.family, namespace)
+
+ const recipeName =
+ desired.metadata?.labels?.['clerum.io/recipe'] ??
+ desired.metadata?.labels?.['clerum.io/recipe-name'] ??
+ policyName
+ const log = createLogger('wrc', recipeName)
+ const fields = { policy: policyName, namespace, family: options.family }
+ const hasSuppliedExisting = Object.prototype.hasOwnProperty.call(options, 'existing')
+ let readResult: NetworkPolicyReadResult = hasSuppliedExisting
+ ? { existing: options.existing ?? null, recoveredFromReadFailure: false }
+ : await this.readNetworkPolicyForApply(policyName, namespace, log, fields)
+ let existing = readResult.existing
+
+ if (!existing) {
+ try {
+ await this.networkingApi.createNamespacedNetworkPolicy({ namespace, body: desired })
+ log.info('network policy created', fields)
+ return
+ } catch (error: unknown) {
+ if (getErrorCode(error) !== 409) {
+ log.error('network policy create failed', {
+ ...fields,
+ ...this.safeInfrastructureErrorFields(error),
+ })
+ if (isRetryableInfraError(error)) {
+ const safe = this.safeInfrastructureErrorFields(error)
+ throw new NetworkPolicyInfraError(
+ `NetworkPolicy "${policyName}" in ${namespace} create temporarily unavailable (${safe.errorCode ?? safe.errorName})`,
+ error,
+ safe.errorCode
+ )
+ }
+ throw error
+ }
+ // A real create race is exceptional in the read-first path. Re-read once
+ // and evaluate the winner from scratch; never blind-replace after 409.
+ readResult = await this.readNetworkPolicyForApply(policyName, namespace, log, fields)
+ existing = readResult.existing
+ if (!existing) {
+ throw new RetryableReconcileError(
+ `NetworkPolicy "${policyName}" in ${namespace} still absent after create conflict`,
+ { cause: error }
+ )
+ }
+ }
+ }
+
+ // Deletion wins over ownership: a terminating object cannot be adopted or
+ // safely classified from partial labels. Wait for 404, then recreate from the
+ // current desired state.
+ if (existing.metadata?.deletionTimestamp) {
+ throw new RetryableReconcileError(
+ `NetworkPolicy "${policyName}" in ${namespace} is terminating; retrying after deletion`
+ )
+ }
+
+ // Domain invariants run outside the read catch. A confirmed veto propagates
+ // immediately and can never be replaced by a second transport error.
+ this.assertInternalDependencyPolicyOwnership(desired, existing, namespace)
+ const decision = decideNetworkPolicyConvergence(options.family, desired, existing)
+ if (decision.action === 'conflict') {
+ throw new NetworkPolicyOwnershipConflictError(
+ `Refusing to replace NetworkPolicy "${policyName}" in ${namespace}: ${decision.reason}`
+ )
+ }
+ if (decision.action === 'retry') {
+ throw new RetryableReconcileError(
+ `NetworkPolicy "${policyName}" in ${namespace} is terminating; retrying after deletion`
+ )
+ }
+ if (decision.action === 'unchanged' && !readResult.recoveredFromReadFailure) {
+ log.info('network policy unchanged; skipping update', fields)
+ return
+ }
+
+ const resourceVersion = existing.metadata?.resourceVersion
+ if (!resourceVersion) {
+ throw new RetryableReconcileError(
+ `NetworkPolicy "${policyName}" in ${namespace} has no resourceVersion`
+ )
+ }
+ desired.metadata.resourceVersion = resourceVersion
+ try {
+ await this.networkingApi.replaceNamespacedNetworkPolicy({
name: policyName,
namespace,
+ body: desired,
+ })
+ log.info('network policy replaced', {
+ ...fields,
+ reason: decision.action === 'unchanged' ? 'read-recovery' : decision.reason,
})
- this.assertInternalDependencyPolicyOwnership(policy, fetched, namespace)
- assertedExisting = fetched
- return fetched
+ } catch (error: unknown) {
+ log.error('network policy replace failed', {
+ ...fields,
+ ...this.safeInfrastructureErrorFields(error),
+ })
+ const errorCode = getErrorCode(error)
+ if (errorCode === 404 || errorCode === 409) {
+ throw new RetryableReconcileError(
+ `NetworkPolicy "${policyName}" in ${namespace} ${errorCode === 404 ? 'disappeared' : 'changed'} during replace`,
+ { cause: error }
+ )
+ }
+ if (isRetryableInfraError(error)) {
+ const safe = this.safeInfrastructureErrorFields(error)
+ throw new NetworkPolicyInfraError(
+ `NetworkPolicy "${policyName}" in ${namespace} replace temporarily unavailable (${safe.errorCode ?? safe.errorName})`,
+ error,
+ safe.errorCode
+ )
+ }
+ throw error
}
- await this.createOrReplace(
- () =>
- this.networkingApi.createNamespacedNetworkPolicy({
- namespace,
- body: policy,
- }),
- async () => {
- // Unset means the gate's read failed OR its assertion vetoed (applyIsNoop
- // swallows both); re-read so the veto surfaces from here, loud.
- const current = assertedExisting ?? (await readExistingAndAssertOwnership())
- policy.metadata!.resourceVersion = current.metadata?.resourceVersion
- return this.networkingApi.replaceNamespacedNetworkPolicy({
- name: policyName,
- namespace,
- body: policy,
- })
- },
- `NetworkPolicy "${policyName}" in ${namespace}`,
- { manifest: policy, readExisting: readExistingAndAssertOwnership }
+ }
+
+ private assertDesiredNetworkPolicyOwner(
+ desired: k8s.V1NetworkPolicy,
+ family: NetworkPolicyFamily,
+ namespace: string
+ ): void {
+ const controllerOwners = (desired.metadata?.ownerReferences ?? []).filter(
+ owner => owner.controller === true
)
+ const invalidControllerOwner =
+ controllerOwners.length > 1 || controllerOwners.some(owner => !owner.uid)
+ const gatewayOwnerMissing = family === 'webhook-gateway' && controllerOwners.length !== 1
+ if (invalidControllerOwner || gatewayOwnerMissing) {
+ throw new NetworkPolicyOwnershipConflictError(
+ `Refusing to apply NetworkPolicy "${desired.metadata?.name ?? '<unknown>'}" in ${namespace}: desired controller owner is incomplete`
+ )
+ }
+ }
+
+ private async readNetworkPolicyForApply(
+ name: string,
+ namespace: string,
+ log: ReturnType<typeof createLogger>,
+ fields: { policy: string; namespace: string; family: NetworkPolicyFamily }
+ ): Promise<NetworkPolicyReadResult> {
+ for (let attempt = 1; attempt <= 2; attempt += 1) {
+ try {
+ return {
+ existing: await this.networkingApi.readNamespacedNetworkPolicy({ name, namespace }),
+ recoveredFromReadFailure: attempt > 1,
+ }
+ } catch (error: unknown) {
+ if (getErrorCode(error) === 404) {
+ return { existing: null, recoveredFromReadFailure: attempt > 1 }
+ }
+ if (attempt === 2) {
+ if (isRetryableInfraError(error)) {
+ const safe = this.safeInfrastructureErrorFields(error)
+ throw new NetworkPolicyInfraError(
+ `NetworkPolicy "${name}" in ${namespace} read temporarily unavailable (${safe.errorCode ?? safe.errorName})`,
+ error,
+ safe.errorCode
+ )
+ }
+ throw error
+ }
+ log.warn('network policy read unavailable; retrying once', {
+ ...fields,
+ ...this.safeInfrastructureErrorFields(error),
+ })
+ }
+ }
+ return { existing: null, recoveredFromReadFailure: false }
+ }
+
+ private safeInfrastructureErrorFields(error: unknown): {
+ errorName: string
+ errorCode?: string | number
+ retryable: boolean
+ } {
+ const candidateCode = (error as { code?: unknown } | null)?.code
+ const candidateName = error instanceof Error ? error.name : 'UnknownError'
+ const errorCode =
+ typeof candidateCode === 'number'
+ ? candidateCode
+ : typeof candidateCode === 'string' && /^[A-Z0-9_]{1,64}$/.test(candidateCode)
+ ? candidateCode
+ : getErrorCode(error)
+ return {
+ errorName: /^[A-Za-z][A-Za-z0-9_.-]{0,63}$/.test(candidateName)
+ ? candidateName
+ : 'UnknownError',
+ ...(errorCode === undefined ? {} : { errorCode }),
+ retryable: isRetryableInfraError(error),
+ }
}
private coordinatorGfsNetworkPolicy(recipe: WorkflowRecipeCRD): k8s.V1NetworkPolicy {
@@ -6266,7 +6435,7 @@ export class WorkflowRecipeReconciler {
)
return
}
- await this.applyNetworkPolicy(policy, ns)
+ await this.applyNetworkPolicy(policy, ns, { family: 'oauth-broker-egress' })
}
private async ensureConfigMap(