Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(wrc): reconcile NetworkPolicies from live state
Alfredo Lopez8ddbe65ea350
diff --git a/docs/deploy/workflow-recipes-guide.md b/docs/deploy/workflow-recipes-guide.mdindex dca6e5897..0ab802b46 100644--- a/docs/deploy/workflow-recipes-guide.md+++ b/docs/deploy/workflow-recipes-guide.md@@ -301,53 +301,46 @@ kubectl delete workflowrecipe my-nginx -n sandbox-recipes## Reconcile idempotency & drift behavior-WRC reconciles every recipe on a periodic resync. Most managed objects (the recipe-Deployment, StatefulSet, headless Service, CronJob, Job, DaemonSet, the McpServer CRD,-and the NetworkPolicies `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*`,-`wf-*-oauth-broker-egress` and the three webhook-gateway policies) are applied through a-**spec-hash idempotency gate**: WRC stamps a `clerum.io/spec-hash` annotation (a hash of-the _desired_ manifest) and, on the already-exists path, **skips the write entirely when-the existing object's stamped hash matches**. This keeps `metadata.generation` from-climbing on no-op reconciles (which previously produced a `degraded↔active` status flap-and a downstream HCC NetworkPolicy no-op write storm).--**What the gate does _not_ cover.** Do not read the list above as "everything":--- `wl-egress-*` routes through the same method, but its manifest carries a- `clerum.io/egress-fqdn-resolved-at` timestamp that is regenerated on every pass, so its- desired hash never matches and **the gate never closes for it**. What actually keeps it- quiet is the upstream issue #299 check (`egressWriteNeeded`), which compares the- enforced egress rules. If that check ever regresses, the spec-hash gate will not catch it.-- `ui-egress-*` and the `coordinator-to-gfs` policy do not use the spec-hash at all; they- have their own ad-hoc comparisons (#299 and #579 respectively).-- The **webhook gateway's own** ConfigMap, Deployment and Service are still ungated and- rewritten on every pass — only its three NetworkPolicies were migrated.+WRC reconciles every recipe on a periodic resync. Deployments, StatefulSets, headless+Services, CronJobs, Jobs, DaemonSets and the McpServer CRD use the existing+**spec-hash idempotency gate**. The recipe-derived NetworkPolicies (`ui-ingress-*`,+`wl-ingress-*`, `wl-egress-*`, `wr-intdep-*`, `wf-*-oauth-broker-egress` and the three+webhook-gateway policies) use a stronger **live convergence check** instead: WRC reads+the current object, validates ownership/lifecycle, normalizes Kubernetes defaults and+skips the write only when the live enforcement plus WRC-authored metadata equal the+desired policy. A legacy `clerum.io/spec-hash` annotation is ignored for this decision;+it cannot hide later drift.++**What this path does _not_ cover.** Do not read the list above as "every policy writer":++- `ui-egress-*` and the `coordinator-to-gfs` policy keep their existing dedicated+ comparisons (#299 and #579 respectively); they do not use the recipe NetworkPolicy+ helper.+- `wl-egress-*` still uses the #299 content/renewal prefilter before the live convergence+ helper. External/mixed egress carries temporal resolution state; a cluster-local-only+ policy does not. In both cases, once the prefilter requires a write, the downstream+ helper rechecks the live spec and cannot cancel a real repair because of a stale seal.+- The **webhook gateway's own** ConfigMap, Deployment and Service remain ungated and are+ rewritten on every pass — only its three NetworkPolicies use live convergence.**Operational consequences:**-- **Logs:** expect "unchanged (spec-hash match); skipping update" instead of a per-loop- "Updated …". Per-loop write/`Updated` log volume drops sharply by design — re-baseline- any alert keyed on it.-- **Drift is no longer auto-reverted.** If an operator hand-edits a WRC-managed object- (e.g. `kubectl edit`/`scale` on a recipe Deployment), WRC will **not** revert it until- the recipe spec changes (which flips the hash). To force a re-apply, change the recipe- spec (or delete the object and let WRC recreate it).-- **For a Deployment the drift is still visible**: the recipe `status` reports the- workload `degraded` when replica health drops.-- **For a NetworkPolicy it is not visible by any path.** There is no replica health to- degrade: the recipe stays `active` while the enforcement object diverges from the- desired one, and nothing sweeps for it — `pruneStaleInternalDependencyPolicies`- compares names, not content, and `deploy/scripts/verify-networkpolicies.sh` **does**- detect content drift but only over policies an overlay renders. Recipe-derived- policies (`ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*`, the gateway trio) are authored- by the reconciler and live in no overlay, so the script never evaluates them at all —- neither their existence nor their content. An edit that preserves the `clerum.io/spec-hash`- annotation while changing the `spec` (widening `ingress`/`egress`, emptying- `podSelector`, dropping a `policyTypes` entry) is **permanent and silent**. Treat any- `kubectl edit` on a `ui-ingress-*`, `wl-ingress-*`, `wr-intdep-*` or gateway policy as- permanent until a bounded resync exists (issue #581).-- **First deploy after this change** does one stamping write per pre-existing managed- object (they have no hash yet), then steady-state is quiet.+- **Logs:** the live-convergence helper and the `wl-egress-*` prefilter use the structured+ WRC logger. In steady state expect `network policy unchanged; skipping update` or+ `network policy egress set unchanged; skipping live apply` with `policy`, `namespace`+ and `family` fields. Re-baseline alerts keyed on the old `Updated …` console line.+- **NetworkPolicy drift is auto-reverted.** A live change to `podSelector`,+ `policyTypes`, `ingress`, `egress`, WRC-authored metadata or the gateway controller+ owner forces one optimistic-concurrency replace using the `resourceVersion` from the+ same validated read. The following reconcile is read-only.+- **Other spec-hash-managed objects retain their existing drift contract.** For example,+ a manually scaled recipe Deployment is not reverted until the desired manifest changes;+ its replica-health drift remains visible in recipe status.+- `deploy/scripts/verify-networkpolicies.sh` still covers only overlay-rendered policies.+ Recipe-derived policy integrity is owned by the runtime reconciliation described here.+- **First deploy after this change** does not require a NetworkPolicy stamping wave.+ Equivalent policies are one GET and zero writes; real spec/metadata/owner drift is+ repaired once.## Automated Testsdiff --git a/workflow-recipes/package-lock.json b/workflow-recipes/package-lock.jsonindex 50a6bcac6..e63ee916f 100644--- a/workflow-recipes/package-lock.json+++ b/workflow-recipes/package-lock.json@@ -1,12 +1,12 @@{"name": "workload-recipes-controller",- "version": "0.1.342",+ "version": "0.1.343","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "workload-recipes-controller",- "version": "0.1.342",+ "version": "0.1.343","license": "SEE LICENSE IN LICENSE","dependencies": {"@aws-sdk/client-s3": "^3.1009.0",diff --git a/workflow-recipes/package.json b/workflow-recipes/package.jsonindex 03ba8569f..18fc8a122 100644--- a/workflow-recipes/package.json+++ b/workflow-recipes/package.json@@ -1,6 +1,6 @@{"name": "workload-recipes-controller",- "version": "0.1.342",+ "version": "0.1.343","description": "Workload Recipes Controller (WRC) - WorkloadRecipe lifecycle management and MCP server interface","main": "dist/main.js","engines": {diff --git a/workflow-recipes/src/reconciler/networkPolicyConvergence.test.ts b/workflow-recipes/src/reconciler/networkPolicyConvergence.test.tsnew file mode 100644index 000000000..7997a107a--- /dev/null+++ b/workflow-recipes/src/reconciler/networkPolicyConvergence.test.ts@@ -0,0 +1,234 @@+import { describe, expect, it } from 'vitest'+import * as k8s from '@kubernetes/client-node'+import {+ classifyNetworkPolicyOwnership,+ decideNetworkPolicyConvergence,+ networkPolicyMatchesDesired,+ networkPolicyMetadataMatchesDesired,+} from './networkPolicyConvergence'++const SPEC_HASH = 'clerum.io/spec-hash'++function policy(): k8s.V1NetworkPolicy {+ return {+ apiVersion: 'networking.k8s.io/v1',+ kind: 'NetworkPolicy',+ metadata: {+ name: 'wl-ingress-r-w',+ namespace: 'sandbox-recipes',+ labels: {+ 'clerum.io/managed-by': 'workflow-recipes',+ 'clerum.io/recipe': 'r',+ },+ },+ spec: {+ podSelector: { matchLabels: { 'clerum.io/workload': 'w' } },+ policyTypes: ['Ingress'],+ ingress: [{ ports: [{ port: 8080, protocol: 'TCP' }] }],+ },+ }+}++function gatewayPolicy(ownerUid = 'uid-new'): k8s.V1NetworkPolicy {+ return {+ ...policy(),+ metadata: {+ name: 'allow-webhook-proxy-ingress-r',+ namespace: 'sandbox-recipes',+ labels: {+ 'clerum.io/managed-by': 'workflow-recipes',+ 'clerum.io/recipe-namespace': 'sandbox-recipes',+ 'clerum.io/recipe-name': 'r',+ 'clerum.io/webhook-gateway': 'true',+ },+ ownerReferences: [+ {+ apiVersion: 'clerum.io/v1alpha1',+ kind: 'WorkflowRecipe',+ name: 'r',+ uid: ownerUid,+ controller: true,+ blockOwnerDeletion: true,+ },+ ],+ },+ }+}++describe('NetworkPolicy live convergence', () => {+ it('treats API metadata/defaulting and the legacy spec-hash as equivalent', () => {+ const desired = policy()+ const live = structuredClone(desired)+ live.metadata = {+ ...live.metadata,+ uid: 'policy-uid',+ resourceVersion: '9',+ generation: 3,+ creationTimestamp: new Date('2026-09-04T00:00:00Z'),+ annotations: { [SPEC_HASH]: 'legacy-seal' },+ }+ live.spec!.ingress![0].ports![0].protocol = undefined++ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)+ expect(decideNetworkPolicyConvergence('workload-ingress', desired, live)).toEqual({+ action: 'unchanged',+ })+ })++ it('normalizes empty rule arrays and API-defaulted policyTypes', () => {+ const desired = policy()+ desired.spec = { podSelector: {}, policyTypes: [], ingress: [], egress: [] }+ const live = policy()+ live.spec = { podSelector: {}, policyTypes: ['Ingress'] }++ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)+ })++ it('normalizes the client model _from field and the Kubernetes wire from field', () => {+ const desired = policy()+ desired.spec!.ingress = [+ {+ _from: [{ podSelector: { matchLabels: { app: 'gateway' } } }],+ ports: [{ port: 8090, protocol: 'TCP' }],+ },+ ]+ const live = structuredClone(desired)+ const liveRule = live.spec!.ingress![0] as k8s.V1NetworkPolicyIngressRule & {+ from?: k8s.V1NetworkPolicyPeer[]+ }+ liveRule.from = liveRule._from+ delete liveRule._from++ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)+ })++ it('ignores admission metadata outside the keys authored by WRC', () => {+ const desired = policy()+ const live = structuredClone(desired)+ live.metadata!.labels = { ...live.metadata!.labels, 'admission.example/tier': 'audited' }+ live.metadata!.annotations = { 'admission.example/audit': 'true' }+ live.metadata!.finalizers = ['admission.example/cleanup']++ expect(networkPolicyMatchesDesired(desired, live)).toBe(true)++ live.metadata!.labels!['clerum.io/recipe'] = 'foreign'+ expect(networkPolicyMatchesDesired(desired, live)).toBe(false)+ })++ it('repairs live enforcement drift even when the stored spec-hash is preserved', () => {+ const desired = policy()+ const live = structuredClone(desired)+ live.metadata!.resourceVersion = '9'+ live.metadata!.annotations = { [SPEC_HASH]: 'still-matches-the-old-desired' }+ live.spec!.podSelector = {}+ live.spec!.ingress = [{}]++ expect(networkPolicyMatchesDesired(desired, live)).toBe(false)+ expect(decideNetworkPolicyConvergence('workload-ingress', desired, live)).toEqual({+ action: 'replace',+ reason: 'live-drift',+ })+ })++ it('makes workload-egress metadata drift bypass its temporal prefilter', () => {+ const desired = policy()+ const live = structuredClone(desired)+ live.metadata!.annotations = { 'clerum.io/egress-fqdn-resolved-at': 'old-timestamp' }++ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(true)++ live.metadata!.labels = { ...live.metadata!.labels, 'clerum.io/recipe': 'wrong' }+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(false)++ live.metadata!.labels = desired.metadata!.labels+ live.metadata!.deletionTimestamp = new Date('2026-09-04T00:00:00Z')+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(false)++ delete live.metadata!.deletionTimestamp+ live.metadata!.ownerReferences = [+ {+ apiVersion: 'apps/v1',+ kind: 'Deployment',+ name: 'foreign',+ uid: 'foreign-uid',+ controller: true,+ },+ ]+ expect(networkPolicyMetadataMatchesDesired(desired, live)).toBe(false)+ })++ it('repairs a stale or missing gateway owner only for the same WRC recipe identity', () => {+ const desired = gatewayPolicy('uid-new')+ const stale = gatewayPolicy('uid-old')+ const missing = gatewayPolicy('uid-new')+ delete missing.metadata!.ownerReferences++ expect(classifyNetworkPolicyOwnership('webhook-gateway', desired, stale)).toEqual({+ kind: 'repairable-owner',+ })+ expect(classifyNetworkPolicyOwnership('webhook-gateway', desired, missing)).toEqual({+ kind: 'repairable-owner',+ })+ expect(decideNetworkPolicyConvergence('webhook-gateway', desired, stale)).toEqual({+ action: 'replace',+ reason: 'owner-repair',+ })+ })++ it('refuses a gateway policy controlled by a different owner', () => {+ const desired = gatewayPolicy('uid-new')+ const foreign = gatewayPolicy('uid-new')+ foreign.metadata!.ownerReferences = [+ {+ apiVersion: 'apps/v1',+ kind: 'Deployment',+ name: 'foreign',+ uid: 'foreign-uid',+ controller: true,+ },+ ]++ expect(classifyNetworkPolicyOwnership('webhook-gateway', desired, foreign)).toEqual({+ kind: 'conflict',+ reason: 'controller-owner-mismatch',+ })+ expect(decideNetworkPolicyConvergence('webhook-gateway', desired, foreign)).toEqual({+ action: 'conflict',+ reason: 'controller-owner-mismatch',+ })+ })++ it('refuses a controller owner on a family that must be ownerless', () => {+ const desired = policy()+ const foreign = structuredClone(desired)+ foreign.metadata!.ownerReferences = [+ {+ apiVersion: 'apps/v1',+ kind: 'Deployment',+ name: 'foreign',+ uid: 'foreign-uid',+ controller: true,+ },+ ]++ expect(classifyNetworkPolicyOwnership('workload-ingress', desired, foreign)).toEqual({+ kind: 'conflict',+ reason: 'owner-reference-mismatch',+ })+ expect(decideNetworkPolicyConvergence('workload-ingress', desired, foreign)).toEqual({+ action: 'conflict',+ reason: 'owner-reference-mismatch',+ })+ })++ it('never reports a terminating policy as unchanged or replaceable', () => {+ const desired = gatewayPolicy()+ const terminating = gatewayPolicy()+ terminating.metadata!.deletionTimestamp = new Date('2026-09-04T00:00:00Z')++ expect(decideNetworkPolicyConvergence('webhook-gateway', desired, terminating)).toEqual({+ action: 'retry',+ reason: 'terminating',+ })+ })+})diff --git a/workflow-recipes/src/reconciler/networkPolicyConvergence.ts b/workflow-recipes/src/reconciler/networkPolicyConvergence.tsnew file mode 100644index 000000000..ee5ad05d7--- /dev/null+++ b/workflow-recipes/src/reconciler/networkPolicyConvergence.ts@@ -0,0 +1,251 @@+import * as k8s from '@kubernetes/client-node'+import { SPEC_HASH_ANNOTATION } from './specHash'++export type NetworkPolicyFamily =+ | 'ui-ingress'+ | 'workload-ingress'+ | 'workload-egress'+ | 'internal-dependency'+ | 'oauth-broker-egress'+ | 'webhook-gateway'++export type NetworkPolicyOwnershipDecision =+ | { kind: 'owned' }+ | { kind: 'repairable-owner' }+ | {+ kind: 'conflict'+ reason:+ | 'desired-owner-uid-missing'+ | 'identity-label-mismatch'+ | 'controller-owner-mismatch'+ | 'owner-reference-mismatch'+ }++export type NetworkPolicyConvergenceDecision =+ | { action: 'unchanged' }+ | { action: 'replace'; reason: 'live-drift' | 'owner-repair' }+ | { action: 'retry'; reason: 'terminating' }+ | {+ action: 'conflict'+ reason:+ | 'desired-owner-uid-missing'+ | 'identity-label-mismatch'+ | 'controller-owner-mismatch'+ | 'owner-reference-mismatch'+ }++const GATEWAY_IDENTITY_LABELS = [+ 'clerum.io/managed-by',+ 'clerum.io/recipe-namespace',+ 'clerum.io/recipe-name',+ 'clerum.io/webhook-gateway',+] as const++function canonicalizeValue(value: unknown): unknown {+ if (Array.isArray(value)) return value.map(canonicalizeValue)+ if (value && typeof value === 'object') {+ const entries = Object.entries(value as Record<string, unknown>)+ .filter(([, child]) => child !== undefined)+ .sort(([left], [right]) => (left < right ? -1 : left > right ? 1 : 0))+ return Object.fromEntries(entries.map(([key, child]) => [key, canonicalizeValue(child)]))+ }+ return value+}++function projectAuthoredMap(+ desired: Record<string, string> | undefined,+ existing: Record<string, string> | undefined,+ excludedKeys: ReadonlySet<string> = new Set()+): Record<string, string> {+ return Object.fromEntries(+ Object.keys(desired ?? {})+ .filter(key => !excludedKeys.has(key))+ .flatMap(key => (existing?.[key] === undefined ? [] : [[key, existing[key]]]))+ )+}++function projectNetworkPolicyForComparison(+ desired: k8s.V1NetworkPolicy,+ candidate: k8s.V1NetworkPolicy+): k8s.V1NetworkPolicy {+ const desiredMetadata = desired.metadata ?? {}+ const candidateMetadata = candidate.metadata ?? {}+ return {+ apiVersion: candidate.apiVersion,+ kind: candidate.kind,+ metadata: {+ name: candidateMetadata.name,+ namespace: candidateMetadata.namespace,+ labels: projectAuthoredMap(desiredMetadata.labels, candidateMetadata.labels),+ annotations: projectAuthoredMap(+ desiredMetadata.annotations,+ candidateMetadata.annotations,+ new Set([SPEC_HASH_ANNOTATION])+ ),+ ...(desiredMetadata.ownerReferences?.length+ ? { ownerReferences: candidateMetadata.ownerReferences }+ : {}),+ },+ spec: structuredClone(candidate.spec),+ }+}++function normalizeIngressFrom(rule: k8s.V1NetworkPolicyIngressRule): void {+ const record = rule as k8s.V1NetworkPolicyIngressRule & {+ from?: k8s.V1NetworkPolicyPeer[]+ }+ if (record._from === undefined && record.from !== undefined) record._from = record.from+ delete record.from+}++function normalizeProjectedNetworkPolicy(normalized: k8s.V1NetworkPolicy): unknown {+ const spec = normalized.spec+ if (spec) {+ const hasEgressRules = (spec.egress?.length ?? 0) > 0+ if (!spec.policyTypes?.length) {+ spec.policyTypes = hasEgressRules ? ['Ingress', 'Egress'] : ['Ingress']+ } else {+ spec.policyTypes = [...spec.policyTypes].sort()+ }+ for (const rule of spec.ingress ?? []) normalizeIngressFrom(rule)+ for (const rule of [...(spec.ingress ?? []), ...(spec.egress ?? [])]) {+ for (const port of rule.ports ?? []) {+ if (port.protocol === 'TCP') delete port.protocol+ }+ }+ if (spec.ingress?.length === 0) delete spec.ingress+ if (spec.egress?.length === 0) delete spec.egress+ }++ return canonicalizeValue(normalized)+}++export function normalizeNetworkPolicyForComparison(policy: k8s.V1NetworkPolicy): unknown {+ return normalizeProjectedNetworkPolicy(projectNetworkPolicyForComparison(policy, policy))+}++export function networkPolicyMatchesDesired(+ desired: k8s.V1NetworkPolicy | undefined,+ existing: k8s.V1NetworkPolicy | undefined+): boolean {+ try {+ if (!desired?.spec || !existing?.spec) return false+ return (+ JSON.stringify(+ normalizeProjectedNetworkPolicy(projectNetworkPolicyForComparison(desired, desired))+ ) ===+ JSON.stringify(+ normalizeProjectedNetworkPolicy(projectNetworkPolicyForComparison(desired, existing))+ )+ )+ } catch {+ return false+ }+}++/**+ * Compare the stable identity metadata that the workload-egress prefilter must+ * never hide. Temporal DNS/state annotations are intentionally excluded: their+ * persistence cadence is decided separately by the accumulator, while labels,+ * lifecycle owner and a terminating object must still reach the live apply.+ */+export function networkPolicyMetadataMatchesDesired(+ desired: k8s.V1NetworkPolicy,+ existing: k8s.V1NetworkPolicy+): boolean {+ if (existing.metadata?.deletionTimestamp) return false+ if (classifyNetworkPolicyOwnership('workload-egress', desired, existing).kind !== 'owned') {+ return false+ }+ const project = (policy: k8s.V1NetworkPolicy) =>+ canonicalizeValue({+ apiVersion: policy.apiVersion,+ kind: policy.kind,+ name: policy.metadata?.name,+ namespace: policy.metadata?.namespace,+ labels: projectAuthoredMap(desired.metadata?.labels, policy.metadata?.labels),+ ownerReferences: desired.metadata?.ownerReferences?.length+ ? policy.metadata?.ownerReferences+ : undefined,+ })+ return JSON.stringify(project(desired)) === JSON.stringify(project(existing))+}++function controllerOwners(policy: k8s.V1NetworkPolicy): k8s.V1OwnerReference[] {+ return (policy.metadata?.ownerReferences ?? []).filter(owner => owner.controller === true)+}++function sameOwnerIdentity(desired: k8s.V1OwnerReference, existing: k8s.V1OwnerReference): boolean {+ return (+ desired.apiVersion === existing.apiVersion &&+ desired.kind === existing.kind &&+ desired.name === existing.name+ )+}++function exactOwnerMatch(desired: k8s.V1OwnerReference, existing: k8s.V1OwnerReference): boolean {+ return (+ sameOwnerIdentity(desired, existing) &&+ desired.uid === existing.uid &&+ desired.controller === existing.controller &&+ desired.blockOwnerDeletion === existing.blockOwnerDeletion+ )+}++function gatewayIdentityLabelsMatch(+ desired: k8s.V1NetworkPolicy,+ existing: k8s.V1NetworkPolicy+): boolean {+ const desiredLabels = desired.metadata?.labels ?? {}+ const existingLabels = existing.metadata?.labels ?? {}+ return GATEWAY_IDENTITY_LABELS.every(+ key => Boolean(desiredLabels[key]) && desiredLabels[key] === existingLabels[key]+ )+}++export function classifyNetworkPolicyOwnership(+ family: NetworkPolicyFamily,+ desired: k8s.V1NetworkPolicy,+ existing: k8s.V1NetworkPolicy+): NetworkPolicyOwnershipDecision {+ const desiredOwners = controllerOwners(desired)+ const existingOwners = controllerOwners(existing)+ if (desiredOwners.length === 0) {+ return (existing.metadata?.ownerReferences?.length ?? 0) === 0+ ? { kind: 'owned' }+ : { kind: 'conflict', reason: 'owner-reference-mismatch' }+ }++ const desiredOwner = desiredOwners[0]+ if (!desiredOwner.uid) return { kind: 'conflict', reason: 'desired-owner-uid-missing' }++ if (existingOwners.length === 1 && exactOwnerMatch(desiredOwner, existingOwners[0])) {+ return { kind: 'owned' }+ }++ if (family !== 'webhook-gateway' || !gatewayIdentityLabelsMatch(desired, existing)) {+ return { kind: 'conflict', reason: 'identity-label-mismatch' }+ }++ if (existingOwners.length === 0) return { kind: 'repairable-owner' }+ if (existingOwners.length === 1 && sameOwnerIdentity(desiredOwner, existingOwners[0])) {+ return { kind: 'repairable-owner' }+ }+ return { kind: 'conflict', reason: 'controller-owner-mismatch' }+}++export function decideNetworkPolicyConvergence(+ family: NetworkPolicyFamily,+ desired: k8s.V1NetworkPolicy,+ existing: k8s.V1NetworkPolicy+): NetworkPolicyConvergenceDecision {+ if (existing.metadata?.deletionTimestamp) return { action: 'retry', reason: 'terminating' }++ const ownership = classifyNetworkPolicyOwnership(family, desired, existing)+ if (ownership.kind === 'conflict') return { action: 'conflict', reason: ownership.reason }+ if (ownership.kind === 'repairable-owner') {+ return { action: 'replace', reason: 'owner-repair' }+ }+ if (networkPolicyMatchesDesired(desired, existing)) return { action: 'unchanged' }+ return { action: 'replace', reason: 'live-drift' }+}diff --git a/workflow-recipes/src/reconciler/reconciler.test.ts b/workflow-recipes/src/reconciler/reconciler.test.tsindex cefa28266..bc891e54f 100644--- a/workflow-recipes/src/reconciler/reconciler.test.ts+++ b/workflow-recipes/src/reconciler/reconciler.test.ts@@ -10,6 +10,7 @@ import { CronJobDef, WorkflowRecipeCRD, WorkflowRecipeGfsIntentSpec, WorkloadDefimport { INHERITED_PARENT_RESOURCES_ANNOTATION } from '../workflow/childRecipeFactory'import { buildCoordinatorGfsNetworkPolicy } from '../workflow/networkPolicyFactory'import { isRetryableInfraError } from './k8sErrors'+import type { NetworkPolicyFamily } from './networkPolicyConvergence'import * as brokerIssuer from './oauthBrokerTokenIssuerClient'import { PLUGIN_WORKLOAD_SDK_PROVIDER_UNAVAILABLE_CONDITION_TYPE } from './pluginWorkloadSdkValidator'import {@@ -18,7 +19,6 @@ import {resolveScopedWorkloadResourceName,resolveWorkloadRuntimeResourceName,} from './resourceBuilder'-import { computeSpecHash, stampSpecHash } from './specHash'import {TRANSIENT_REQUEUE_BASE_MS,WORKFLOW_PROGRESS_REQUEUE_BASE_MS,@@ -224,18 +224,21 @@ describe('WorkflowRecipeReconciler', () => {mockNetworkingApi.createNamespacedNetworkPolicy.mockReset()mockNetworkingApi.createNamespacedNetworkPolicy.mockResolvedValue({})mockNetworkingApi.readNamespacedNetworkPolicy.mockReset()- mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(({ name }: { name: string }) =>- Promise.resolve({- metadata: {- name,- uid: `uid-${name}`,- resourceVersion: '1',- labels: {- 'clerum.io/managed-by': 'wrc',- 'clerum.io/recipe': name.replace(/-coordinator-to-gfs$/, ''),+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(+ ({ name }: { name: string }) => {+ if (!name.endsWith('-coordinator-to-gfs')) return Promise.reject({ code: 404 })+ return Promise.resolve({+ metadata: {+ name,+ uid: `uid-${name}`,+ resourceVersion: '1',+ labels: {+ 'clerum.io/managed-by': 'wrc',+ 'clerum.io/recipe': name.replace(/-coordinator-to-gfs$/, ''),+ },},- },- })+ })+ })mockNetworkingApi.replaceNamespacedNetworkPolicy.mockReset()mockNetworkingApi.replaceNamespacedNetworkPolicy.mockResolvedValue({})@@ -770,28 +773,27 @@ describe('WorkflowRecipeReconciler', () => {])})- // ─── issue #575: the spec-hash gate applyNetworkPolicy never opted into ──────- //- // `createOrReplace` takes a fourth `idempotency` argument that stamps the- // desired manifest with a SHA-256 of its content and compares it against the- // annotation on the live object before replacing. Five callers pass it —- // Deployment, headless Service, CronJob, Job, DaemonSet. `applyNetworkPolicy`- // did not, so every WRC NetworkPolicy funnelling through it was rewritten on- // every reconcile pass without comparing anything: ~12 200 PUTs/hour of objects- // the apiserver then discarded as unchanged.- //- // The ownership assertion is why this is not a one-line change. It runs inside- // the replace path today; a gate that returns first would skip it, so it moves- // into `readExisting` — the very read the gate evaluates — and the result is- // cached only AFTER it passes, so the replace path can never inherit a vetoed- // object.- describe('applyNetworkPolicy spec-hash gate (issue #575)', () => {+ // ─── issue #575: live NetworkPolicy convergence ────────────────────────+ describe('applyNetworkPolicy live convergence (issue #575)', () => {const SPEC_HASH = 'clerum.io/spec-hash'const NS = 'sandbox-recipes'- const clone = <T>(v: T): T => JSON.parse(JSON.stringify(v)) as T- type Priv = { applyNetworkPolicy: (p: k8s.V1NetworkPolicy, ns: string) => Promise<void> }- const apply = (p: k8s.V1NetworkPolicy) =>- (reconciler as unknown as Priv).applyNetworkPolicy(p, NS)+ const clone = <T>(value: T): T => structuredClone(value)+ type Priv = {+ applyNetworkPolicy: (+ policy: k8s.V1NetworkPolicy,+ namespace: string,+ options: { family: NetworkPolicyFamily; existing?: k8s.V1NetworkPolicy | null }+ ) => Promise<void>+ }+ const apply = (+ policy: k8s.V1NetworkPolicy,+ family: NetworkPolicyFamily = 'workload-ingress',+ existing?: k8s.V1NetworkPolicy | null+ ) =>+ (reconciler as unknown as Priv).applyNetworkPolicy(policy, NS, {+ family,+ ...(existing === undefined ? {} : { existing }),+ })const plainPolicy = (): k8s.V1NetworkPolicy => ({apiVersion: 'networking.k8s.io/v1',@@ -821,57 +823,101 @@ describe('WorkflowRecipeReconciler', () => {},})- const ownedExisting = (extra: Record<string, unknown> = {}) => ({+ const gatewayPolicy = (ownerUid = 'uid-current'): k8s.V1NetworkPolicy => ({+ ...plainPolicy(),metadata: {- name: 'wl-ingress-r-w',- resourceVersion: '9',- labels: plainPolicy().metadata!.labels,- ...extra,+ name: 'allow-webhook-proxy-ingress-r',+ namespace: NS,+ labels: {+ 'clerum.io/managed-by': 'workflow-recipes',+ 'clerum.io/recipe-namespace': NS,+ 'clerum.io/recipe-name': 'r',+ 'clerum.io/webhook-gateway': 'true',+ },+ ownerReferences: [+ {+ apiVersion: 'clerum.io/v1alpha1',+ kind: 'WorkflowRecipe',+ name: 'r',+ uid: ownerUid,+ controller: true,+ blockOwnerDeletion: true,+ },+ ],},})- it('T1 does NOT replace a NetworkPolicy whose spec-hash is unchanged, and says so', async () => {- await apply(plainPolicy())- const sealed = clone(- (- mockNetworkingApi.createNamespacedNetworkPolicy.mock.calls.at(-1)![0] as {- body: k8s.V1NetworkPolicy- }- ).body- )- // The exact hash, not merely "some annotation": a stamp that is present but- // wrong would still skip the write, and would skip it for the wrong reason.- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))+ const livePolicy = (+ desired: k8s.V1NetworkPolicy,+ metadata: Partial<k8s.V1ObjectMeta> = {}+ ): k8s.V1NetworkPolicy => ({+ ...clone(desired),+ metadata: {+ ...clone(desired.metadata ?? {}),+ resourceVersion: '9',+ ...metadata,+ },+ })++ function captureNetworkPolicyLogs(): {+ entries: Array<Record<string, unknown>>+ restore: () => void+ } {+ const previousLevel = process.env.LOG_LEVEL+ process.env.LOG_LEVEL = 'info'+ const entries: Array<Record<string, unknown>> = []+ const spy = vi.spyOn(process.stdout, 'write').mockImplementation(((chunk: unknown) => {+ const text = typeof chunk === 'string' ? chunk : Buffer.from(chunk as Uint8Array).toString()+ for (const line of text.split('\n')) {+ if (!line.startsWith('{')) continue+ const parsed = JSON.parse(line) as Record<string, unknown>+ if (String(parsed.msg).startsWith('network policy')) entries.push(parsed)+ }+ return true+ }) as unknown as typeof process.stdout.write)+ return {+ entries,+ restore: () => {+ spy.mockRestore()+ if (previousLevel === undefined) delete process.env.LOG_LEVEL+ else process.env.LOG_LEVEL = previousLevel+ },+ }+ }- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({- ...sealed,- metadata: { ...sealed.metadata, resourceVersion: '9' },- })- mockNetworkingApi.readNamespacedNetworkPolicy.mockClear()- const logSpy = vi.spyOn(console, 'log').mockImplementation(() => undefined)+ it('T1 skips a live-equivalent policy and emits a structured liveness witness', async () => {+ const desired = plainPolicy()+ const live = livePolicy(desired, { annotations: { [SPEC_HASH]: 'legacy-seal' } })+ live.spec!.ingress![0].ports![0].protocol = undefined+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(live)+ const captured = captureNetworkPolicyLogs()try {- await apply(plainPolicy())- expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ await apply(desired)expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)- // The skip line is load-bearing, not decoration: it is what keeps the- // ~16s reconcile cadence visible once the writes stop. A silent gate- // would make the symptom of #460 invisible.- expect(logSpy.mock.calls.flat().join(' ')).toContain(- 'NetworkPolicy "wl-ingress-r-w" in sandbox-recipes unchanged (spec-hash match); skipping update'+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ expect(captured.entries).toContainEqual(+ expect.objectContaining({+ level: 'info',+ msg: 'network policy unchanged; skipping update',+ policy: 'wl-ingress-r-w',+ namespace: NS,+ family: 'workload-ingress',+ }))} finally {- logSpy.mockRestore()+ captured.restore()}})- it('T2 replaces when the spec-hash differs, reading once and using the live resourceVersion', async () => {- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(- ownedExisting({ annotations: { [SPEC_HASH]: 'stale' } })- )+ it('T2 repairs live spec drift even when the legacy spec-hash is preserved', async () => {+ const desired = plainPolicy()+ const live = livePolicy(desired, { annotations: { [SPEC_HASH]: 'legacy-seal' } })+ live.spec!.podSelector = {}+ live.spec!.ingress = [{}]+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(live)- await apply(plainPolicy())+ await apply(desired)expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)const body = (@@ -880,155 +926,265 @@ describe('WorkflowRecipeReconciler', () => {}).bodyexpect(body.metadata?.resourceVersion).toBe('9')- expect(body.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))- // One read serves both the gate and the replace, so the replace cannot act- // on a different resourceVersion than the one the gate evaluated.+ expect(body.metadata?.annotations?.[SPEC_HASH]).toBeUndefined()+ expect(body.spec).toEqual(desired.spec)expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)})- it('T3 writes once to seal a pre-existing policy with no spec-hash, then skips', async () => {- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(ownedExisting())+ it('does not let a supplied cluster-local wl-egress snapshot veto detected drift', async () => {+ const desired = plainPolicy()+ desired.metadata!.name = 'wl-egress-r-w'+ desired.spec = {+ podSelector: { matchLabels: { 'clerum.io/workload': 'w' } },+ policyTypes: ['Egress'],+ egress: [+ {+ to: [{ podSelector: { matchLabels: { 'clerum.io/workload': 'db' } } }],+ ports: [{ port: 5432, protocol: 'TCP' }],+ },+ ],+ }+ const live = livePolicy(desired, { annotations: { [SPEC_HASH]: 'legacy-seal' } })+ live.spec!.podSelector = {}++ await apply(desired, 'workload-egress', live)- await apply(plainPolicy())+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).not.toHaveBeenCalled()expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)- const sealed = clone(+ expect((mockNetworkingApi.replaceNamespacedNetworkPolicy.mock.calls[0][0] as {body: k8s.V1NetworkPolicy}+ ).body.spec?.podSelector+ ).toEqual(desired.spec.podSelector)+ })++ it('routes workload-egress label drift through the live apply prefilter', () => {+ const desired = plainPolicy()+ desired.metadata!.name = 'wl-egress-r-w'+ desired.spec = { podSelector: {}, policyTypes: ['Egress'], egress: [] }+ const live = livePolicy(desired)+ live.metadata!.labels = { ...live.metadata!.labels, 'clerum.io/recipe': 'wrong' }++ const writeNeeded = (+ reconciler as unknown as {+ egressWriteNeeded: (+ existing: k8s.V1NetworkPolicy | null,+ wanted: k8s.V1NetworkPolicy+ ) => boolean+ }+ ).egressWriteNeeded(live, desired)++ expect(writeNeeded).toBe(true)++ live.metadata!.labels = desired.metadata!.labels+ live.metadata!.ownerReferences = [+ {+ apiVersion: 'apps/v1',+ kind: 'Deployment',+ name: 'foreign',+ uid: 'foreign-uid',+ controller: true,+ },+ ]+ expect(+ (+ reconciler as unknown as {+ egressWriteNeeded: (+ existing: k8s.V1NetworkPolicy | null,+ wanted: k8s.V1NetworkPolicy+ ) => boolean+ }+ ).egressWriteNeeded(live, desired)+ ).toBe(true)+ })++ it('T3 creates on 404 and the next equivalent reconcile is read-only', async () => {+ const desired = plainPolicy()+ mockNetworkingApi.readNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 404 })++ await apply(desired)+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ const created = clone(+ (+ mockNetworkingApi.createNamespacedNetworkPolicy.mock.calls[0][0] as {+ body: k8s.V1NetworkPolicy+ }).body)- expect(sealed.metadata?.annotations?.[SPEC_HASH]).toBe(computeSpecHash(plainPolicy()))+ expect(created.metadata?.annotations?.[SPEC_HASH]).toBeUndefined()- mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(sealed)+ mockNetworkingApi.createNamespacedNetworkPolicy.mockClear()+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(livePolicy(created))- await apply(plainPolicy())+ await apply(desired)+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()})- it('T4 does not let a matching spec-hash shortcut the ownership assertion', async () => {+ it('T4 propagates a foreign intdep ownership veto after exactly one read', async () => {const desired = intdepPolicy()- const sameHash = stampSpecHash(clone(desired))- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })- // Foreign owner, carrying a hash the gate would happily match on.- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({- metadata: {- name: 'wr-intdep-egress-r-w',- resourceVersion: '9',- annotations: { [SPEC_HASH]: sameHash },- labels: { 'clerum.io/managed-by': 'hcc' },- },- })+ mockNetworkingApi.readNamespacedNetworkPolicy+ .mockResolvedValueOnce({+ metadata: {+ name: 'wr-intdep-egress-r-w',+ resourceVersion: '9',+ labels: { 'clerum.io/managed-by': 'hcc' },+ },+ })+ .mockRejectedValueOnce({ code: 503 })- const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined)- const rejection = await apply(desired).then(- () => {- throw new Error('expected the ownership assertion to reject')- },- (e: unknown) => e as Error+ const rejection = await apply(desired, 'internal-dependency').then(+ () => new Error('expected the ownership assertion to reject'),+ (error: unknown) => error as Error)- // Pin the CLASS, not just the message. `reconcileInternalDependencyPolicies`- // keys the CRD condition `OwnershipConflict` off `instanceof`, so a bare- // Error carrying the same text would set the wrong condition while passing a- // message-only assertion. The class is module-local — exporting it just for- // this test would be production code shaped by a test — so pin its `name`.expect(rejection.name).toBe('NetworkPolicyOwnershipConflictError')expect(rejection.message).toMatch(/Refusing to replace NetworkPolicy "wr-intdep-egress-r-w"/)+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()- // Two reads: the gate's read raises the veto, `applyIsNoop` treats any- // throw as "cannot prove this is a no-op" and falls through, and the replace- // path re-reads and re-asserts, where the veto propagates. The write never- // happens either way. What this test rules out is the assertion living ONLY- // in the replace path: there, a matching hash would make the gate skip and- // this call would resolve instead of rejecting.- expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ })- // Review of #580, R1-L1. A veto is not a read failure, and `applyIsNoop`- // must not announce it as one: the read SUCCEEDED and the caller's- // assertion on the result is what threw. The accurate message arrives a- // moment later from `replaceFn`, so a "pre-read failed" line here would only- // precede the truth with a wrong diagnosis.- //- // Scoped to that one line rather than asserting console.warn was never- // called: other warnings on this path are legitimate, and a blanket- // assertion would break on any of them for reasons unrelated to the veto.- const warned = warnSpy.mock.calls.flat().join(' ')- expect(warned).not.toContain('idempotency pre-read failed')- warnSpy.mockRestore()- })-- it('T4b lets an internal-dependency policy WRC already owns through the gate', async () => {- // T4 pins the veto; nothing pinned the other side. An assertion that rejected- // EVERY internal-dependency policy would leave T4 green and silently stop the- // whole intdep family from converging — the gate would never be reached and- // every pass would set an OwnershipConflict condition. This is the test that- // makes T4 mean "vetoes foreign owners" rather than "always vetoes".+ it('T4b lets an intdep policy WRC owns converge read-only', async () => {const desired = intdepPolicy()- const ownHash = stampSpecHash(clone(desired))- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue({- metadata: {- name: 'wr-intdep-egress-r-w',- resourceVersion: '9',- annotations: { [SPEC_HASH]: ownHash },- labels: desired.metadata!.labels,- },- })+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(livePolicy(desired))- await expect(apply(desired)).resolves.toBeUndefined()+ await expect(apply(desired, 'internal-dependency')).resolves.toBeUndefined()- expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()- // One read: the assertion passed inside the gate's own read, so the replace- // path is never entered and never re-reads. Contrast T4's two.expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()})- it('T5 falls through to the replace when the gate read fails with an HTTP status', async () => {- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })+ it('T5 retries an HTTP read failure, replaces with the recovered RV, and logs safely', async () => {+ const desired = plainPolicy()mockNetworkingApi.readNamespacedNetworkPolicy- .mockRejectedValueOnce({ code: 500 })- .mockResolvedValueOnce(ownedExisting())+ .mockRejectedValueOnce({ code: 500, authorization: 'Bearer must-not-appear' })+ .mockResolvedValueOnce(livePolicy(desired))+ const captured = captureNetworkPolicyLogs()+ try {+ await apply(desired)++ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)+ expect(+ (+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mock.calls[0][0] as {+ body: k8s.V1NetworkPolicy+ }+ ).body.metadata?.resourceVersion+ ).toBe('9')+ const warning = captured.entries.find(+ entry => entry.msg === 'network policy read unavailable; retrying once'+ )+ expect(warning).toEqual(+ expect.objectContaining({+ level: 'warn',+ errorName: 'UnknownError',+ errorCode: 500,+ retryable: true,+ })+ )+ expect(JSON.stringify(warning)).not.toContain('must-not-appear')+ } finally {+ captured.restore()+ }+ })++ it('T5b retries a transport read failure once and replaces', async () => {+ const desired = plainPolicy()+ mockNetworkingApi.readNamespacedNetworkPolicy+ .mockRejectedValueOnce({ code: 'ECONNRESET' })+ .mockResolvedValueOnce(livePolicy(desired))- await apply(plainPolicy())+ await apply(desired)expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)})- it('T5b falls through to the replace on a retryable transport error', async () => {- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })+ it('T5c retries a codeless read failure once and replaces', async () => {+ const desired = plainPolicy()mockNetworkingApi.readNamespacedNetworkPolicy- .mockRejectedValueOnce({ code: 'ECONNRESET' })- .mockResolvedValueOnce(ownedExisting())+ .mockRejectedValueOnce(new Error('transient read timeout'))+ .mockResolvedValueOnce(livePolicy(desired))- await apply(plainPolicy())+ await apply(desired)expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)})- it('T5c falls through on a codeless read error, the shape `:1354` pins for Deployment', async () => {- // The rule cannot key on the shape of the error. A bare Error with no code- // is a legitimate read failure — the existing Deployment test injects- // exactly `new Error('transient read timeout')` and asserts the write still- // happens — so `applyIsNoop` swallows everything and falls open. A version- // of this gate that classified errors instead broke that test; this pins the- // behaviour for NetworkPolicy so the same idea cannot come back quietly.- //- // Not redundant with T5b despite the total catch: T5b's error carries a- // `code`, and every earlier draft that regressed this did so by keying on- // `code`. The codeless shape is the one that broke the Deployment test.- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })- mockNetworkingApi.readNamespacedNetworkPolicy- .mockRejectedValueOnce(new Error('transient read timeout'))- .mockResolvedValueOnce(ownedExisting())+ it('repairs the gateway owner UID and refuses a foreign controller owner', async () => {+ const desired = gatewayPolicy('uid-current')+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(+ livePolicy(gatewayPolicy('uid-old'), { annotations: { [SPEC_HASH]: 'legacy-seal' } })+ )- await apply(plainPolicy())+ await apply(desired, 'webhook-gateway')expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ const body = (+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mock.calls[0][0] as {+ body: k8s.V1NetworkPolicy+ }+ ).body+ expect(body.metadata?.ownerReferences?.[0].uid).toBe('uid-current')++ mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()+ const foreign = livePolicy(desired)+ foreign.metadata!.ownerReferences = [+ {+ apiVersion: 'apps/v1',+ kind: 'Deployment',+ name: 'foreign',+ uid: 'foreign-uid',+ controller: true,+ },+ ]+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(foreign)++ await expect(apply(desired, 'webhook-gateway')).rejects.toMatchObject({+ name: 'NetworkPolicyOwnershipConflictError',+ })+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ })++ it('handles a read/create race by re-reading instead of blind replacing', async () => {+ const desired = plainPolicy()+ mockNetworkingApi.readNamespacedNetworkPolicy+ .mockRejectedValueOnce({ code: 404 })+ .mockResolvedValueOnce(livePolicy(desired))+ mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 409 })++ await apply(desired)+expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ })++ it.each([404, 409])('classifies replace race status %s as retryable', async code => {+ const desired = plainPolicy()+ const live = livePolicy(desired)+ live.spec!.podSelector = {}+ mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValue(live)+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mockRejectedValueOnce({ code })++ await expect(apply(desired)).rejects.toMatchObject({+ name: 'RetryableReconcileError',+ cause: { code },+ })+ })++ it('refuses to create a gateway policy without a complete desired owner', async () => {+ const desired = gatewayPolicy('')++ await expect(apply(desired, 'webhook-gateway')).rejects.toMatchObject({+ name: 'NetworkPolicyOwnershipConflictError',+ })+ expect(mockNetworkingApi.readNamespacedNetworkPolicy).not.toHaveBeenCalled()+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()})})@@ -1919,17 +2075,21 @@ describe('WorkflowRecipeReconciler', () => {})it('fails closed instead of replacing an existing non-WRC internal-dependency policy', async () => {- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 409 })- mockNetworkingApi.readNamespacedNetworkPolicy.mockResolvedValueOnce({- metadata: {- resourceVersion: 'rv-hcc',- labels: {- 'clerum.io/managed-by': 'host-capability-controller',- 'clerum.io/policy-type': 'binding-allow',- 'clerum.io/recipe': 'test-recipe',- },- },- })+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(+ ({ name }: { name: string }) => {+ if (!name.startsWith('wr-intdep-')) return Promise.reject({ code: 404 })+ return Promise.resolve({+ metadata: {+ resourceVersion: 'rv-hcc',+ labels: {+ 'clerum.io/managed-by': 'host-capability-controller',+ 'clerum.io/policy-type': 'binding-allow',+ 'clerum.io/recipe': 'test-recipe',+ },+ },+ })+ }+ )const result = await reconciler.reconcile(makeRecipe({@@ -1956,6 +2116,163 @@ describe('WorkflowRecipeReconciler', () => {expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()})+ it('keeps a recipe retryable when an internal-dependency policy is terminating', async () => {+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(+ ({ name }: { name: string }) => {+ if (!name.startsWith('wr-intdep-')) return Promise.reject({ code: 404 })+ return Promise.resolve({+ metadata: {+ name,+ resourceVersion: 'rv-terminating',+ deletionTimestamp: new Date('2026-09-04T00:00:00Z'),+ labels: {+ 'clerum.io/managed-by': 'host-capability-controller',+ 'clerum.io/policy-type': 'binding-allow',+ 'clerum.io/recipe': 'test-recipe',+ },+ },+ })+ }+ )++ const result = await reconciler.reconcile(+ makeRecipe({+ spec: {+ workloads: [+ {+ id: 'api',+ type: 'deployment',+ image: 'api:test',+ port: 8080,+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],+ },+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },+ ],+ },+ })+ )++ expect(result.phase).toBe('degraded')+ expect(result.message).toMatch(/is terminating; retrying after deletion/)+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ })++ it('keeps a recipe retryable when an internal-dependency policy disappears before replace', async () => {+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(+ ({ name }: { name: string }) => {+ if (!name.startsWith('wr-intdep-')) return Promise.reject({ code: 404 })+ return Promise.resolve({+ metadata: {+ name,+ resourceVersion: 'rv-before-delete',+ labels: {+ 'clerum.io/managed-by': 'workflow-recipes',+ 'clerum.io/policy-type': 'internal-dependency',+ 'clerum.io/recipe': 'test-recipe',+ },+ },+ spec: { podSelector: {}, policyTypes: ['Ingress'] },+ })+ }+ )+ mockNetworkingApi.replaceNamespacedNetworkPolicy.mockRejectedValueOnce({ code: 404 })++ const result = await reconciler.reconcile(+ makeRecipe({+ spec: {+ workloads: [+ {+ id: 'api',+ type: 'deployment',+ image: 'api:test',+ port: 8080,+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],+ },+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },+ ],+ },+ })+ )++ expect(result.phase).toBe('degraded')+ expect(result.message).toMatch(/disappeared during replace/)+ })++ it.each([{ code: 500 }, new Error('network timeout')])(+ 'preserves an active recipe and requeues when internal-dependency reads stay unavailable (%s)',+ async readError => {+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(+ ({ name }: { name: string }) =>+ name.startsWith('wr-intdep-') ? Promise.reject(readError) : Promise.reject({ code: 404 })+ )++ const result = await reconciler.reconcile(+ makeRecipe({+ status: { phase: 'active', message: 'healthy before API outage', workloads: [] },+ spec: {+ workloads: [+ {+ id: 'api',+ type: 'deployment',+ image: 'api:test',+ port: 8080,+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],+ },+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },+ ],+ },+ })+ )++ expect(result.phase).toBe('active')+ expect(result.message).toBe('healthy before API outage')+ expect(result.skipStatusPatch).toBe(true)+ expect(result.requeueAfterMs).toBe(TRANSIENT_REQUEUE_BASE_MS)+ expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()+ }+ )++ it('keeps a workflow non-terminal when internal-dependency reads stay unavailable', async () => {+ const workflowReconcile = vi.fn()+ ;(+ reconciler as unknown as {+ workflowReconciler: {+ reconcile: typeof workflowReconcile+ validateWorkflowSpec: () => undefined+ }+ }+ ).workflowReconciler = { reconcile: workflowReconcile, validateWorkflowSpec: () => undefined }+ mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(+ ({ name }: { name: string }) =>+ name.startsWith('wr-intdep-')+ ? Promise.reject({ code: 500 })+ : Promise.reject({ code: 404 })+ )++ const result = await reconciler.reconcile(+ makeRecipe({+ spec: {+ workloads: [+ {+ id: 'api',+ type: 'deployment',+ image: 'api:test',+ port: 8080,+ env: [{ name: 'DB_HOST', value: '{{db:host}}' }],+ },+ { id: 'db', type: 'deployment', image: 'postgres:16', port: 5432 },+ ],+ steps: [{ id: 'run', run: snippetRun() }],+ },+ })+ )++ expect(result.phase).toBe('degraded')+ expect(result.workflowPhase).toBeUndefined()+ expect(result.message).toBe('Workflow workload infrastructure temporarily unavailable (500)')+ expect(workflowReconcile).not.toHaveBeenCalled()+ })+it('fails closed before workflow execution when a workflow StatefulSet hits a foreign PVC', async () => {const workflowReconcile = vi.fn().mockResolvedValue({phase: 'deploying',@@ -9454,16 +9771,9 @@ describe('WorkflowRecipeReconciler', () => {expect(allFalse).toBe(true)})- it('T6 (#575) applies the 3 gateway NetworkPolicies through the spec-hash gate', async () => {- // Two jobs. First: the gateway policies used to be written by an inline- // create-or-replace with no gate, so they were three no-op PUTs per pass.- // Second, and the reason this test is here rather than asserted by- // inspection: it proves the gateway builder is DETERMINISTIC. A gate only- // closes if the manifest hashes the same on the next pass, and a builder- // that iterated a Set or stamped a timestamp would leave the gate open- // forever — the failure mode of #413 and #579. If this test goes red, the- // builder acquired non-determinism; do not relax the assertion.+ it('T6 (#575) converges the 3 gateway NetworkPolicies through live reads', async () => {mockNetworkingApi.createNamespacedNetworkPolicy.mockClear()+ mockNetworkingApi.readNamespacedNetworkPolicy.mockRejectedValue({ code: 404 })await reconciler.reconcile(recipeWithWebhook())const sealed = new Map<string, unknown>()@@ -9479,7 +9789,6 @@ describe('WorkflowRecipeReconciler', () => {'allow-webhook-proxy-ingress-wf-wh-recipe',])- mockNetworkingApi.createNamespacedNetworkPolicy.mockRejectedValue({ code: 409 })mockNetworkingApi.readNamespacedNetworkPolicy.mockImplementation(({ name }: { name: string }) => {const previous = sealed.get(name) as k8s.V1NetworkPolicy | undefined@@ -9491,31 +9800,44 @@ describe('WorkflowRecipeReconciler', () => {})mockNetworkingApi.replaceNamespacedNetworkPolicy.mockClear()+ mockNetworkingApi.createNamespacedNetworkPolicy.mockClear()mockNetworkingApi.readNamespacedNetworkPolicy.mockClear()- const logSpy = vi.spyOn(console, 'log').mockImplementation(() => undefined)+ const previousLevel = process.env.LOG_LEVEL+ process.env.LOG_LEVEL = 'info'+ const logEntries: Array<Record<string, unknown>> = []+ const logSpy = vi.spyOn(process.stdout, 'write').mockImplementation(((chunk: unknown) => {+ const text = typeof chunk === 'string' ? chunk : Buffer.from(chunk as Uint8Array).toString()+ for (const line of text.split('\n')) {+ if (!line.startsWith('{')) continue+ const parsed = JSON.parse(line) as Record<string, unknown>+ if (parsed.msg === 'network policy unchanged; skipping update') logEntries.push(parsed)+ }+ return true+ }) as unknown as typeof process.stdout.write)try {await reconciler.reconcile(recipeWithWebhook())- const logged = logSpy.mock.calls.flat().join('\n')for (const name of sealed.keys()) {- // Negative half: the gate closed, no write went out.expect(mockNetworkingApi.replaceNamespacedNetworkPolicy).not.toHaveBeenCalledWith(expect.objectContaining({ name }))- // Positive half, and the reason this test is not vacuous: prove the second- // pass actually REACHED this policy. A negative assertion alone cannot tell- // "the gate closed" from "the code never ran" — patching the gateway loop to- // run only on the first pass leaves the negative half green. Both halves are- // required of any convergence test built on this shape (issue #581).expect(mockNetworkingApi.readNamespacedNetworkPolicy).toHaveBeenCalledWith(expect.objectContaining({ name }))- expect(logged).toContain(`NetworkPolicy "${name}" in`)- expect(logged).toContain('unchanged (spec-hash match); skipping update')+ expect(logEntries).toContainEqual(+ expect.objectContaining({+ policy: name,+ family: 'webhook-gateway',+ msg: 'network policy unchanged; skipping update',+ })+ )}+ expect(mockNetworkingApi.createNamespacedNetworkPolicy).not.toHaveBeenCalled()} finally {logSpy.mockRestore()+ if (previousLevel === undefined) delete process.env.LOG_LEVEL+ else process.env.LOG_LEVEL = previousLevel}})diff --git a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts b/workflow-recipes/src/reconciler/workflowRecipeReconciler.tsindex 0bc37f1cf..20df497a0 100644--- a/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts+++ b/workflow-recipes/src/reconciler/workflowRecipeReconciler.ts@@ -89,6 +89,11 @@ import {waitForExternalEgressReady,waitForNetworkReady,} from './mcpDelegation'+import {+ type NetworkPolicyFamily,+ decideNetworkPolicyConvergence,+ networkPolicyMetadataMatchesDesired,+} from './networkPolicyConvergence'import { issueOAuthBrokerToken } from './oauthBrokerTokenIssuerClient'import {PLUGIN_WORKLOAD_SDK_CONDITION_TYPE,@@ -472,6 +477,26 @@ class NetworkPolicyOwnershipConflictError extends Error {}}+class NetworkPolicyInfraError extends Error {+ readonly code?: string | number++ constructor(message: string, cause: unknown, code?: string | number) {+ super(message, { cause })+ this.name = 'NetworkPolicyInfraError'+ this.code = code+ }+}++interface NetworkPolicyApplyOptions {+ family: NetworkPolicyFamily+ existing?: k8s.V1NetworkPolicy | null+}++interface NetworkPolicyReadResult {+ existing: k8s.V1NetworkPolicy | null+ recoveredFromReadFailure: boolean+}+class InternalDependencyReconcileError extends Error {constructor(message: string,@@ -1904,13 +1929,21 @@ export class WorkflowRecipeReconciler {error instanceof InternalDependencyReconcileError ? error.conditions : undefinedconst workloadConditions =error instanceof ImmutableStatefulSetDriftError ? [error.condition] : undefined- if (error instanceof RetryableReconcileError) {+ if (+ error instanceof RetryableReconcileError ||+ error instanceof NetworkPolicyInfraError+ ) {+ const safe = this.safeInfrastructureErrorFields(error)+ const retryableMessage =+ error instanceof RetryableReconcileError+ ? message+ : `Workflow workload infrastructure temporarily unavailable (${safe.errorCode ?? safe.errorName})`// Transient infra failure (e.g. DNS SERVFAIL on egress). Degrade// instead of failing so the next reconcile retries before the// workflow runs — don't mark workflowPhase failed.return {phase: 'degraded' as RecipePhase,- message,+ message: retryableMessage,workloadStatuses: [],internalDependencyConditions,workloadConditions,@@ -2700,7 +2733,10 @@ export class WorkflowRecipeReconciler {// decision and hiding the permanent misconfiguration.const isEgressResolutionError =error instanceof Error && error.message.includes('egress resolution failed')- if (!isEgressResolutionError && isRetryableInfraError(error)) {+ if (+ !isEgressResolutionError &&+ (error instanceof NetworkPolicyInfraError || isRetryableInfraError(error))+ ) {console.warn(`[WR-Reconciler] Transient infra error reconciling "${name}" — keeping phase "${currentPhase}", will retry: ${String(error)}`)@@ -3031,19 +3067,16 @@ export class WorkflowRecipeReconciler {`WebhookGateway Service "${gatewayServiceName(recipeName)}"`)- // Same spec-hash gate as every other WRC NetworkPolicy (issue #575): these- // three are rebuilt verbatim from the recipe on every pass, so the inline- // replace was three no-op PUTs per reconcile. The wr-intdep-* ownership- // assertion inside applyNetworkPolicy returns immediately for these — their- // policy-type label differs — so routing through it changes the write gate and- // nothing else. The log label changes from `WebhookGateway NetworkPolicy "x"`- // to `NetworkPolicy "x" in <ns>`, which no test, alert or doc consumes.+ // The gateway policies use the same live convergence path as other+ // WRC-authored NetworkPolicies (issue #575). Their controller ownerReference+ // is part of the comparison, so recreating a recipe under the same name repairs+ // a stale UID instead of accepting the old garbage-collection lineage.for (const policy of [built.proxyIngressPolicy,built.handlerEgressPolicy,built.handlerIngressPolicy,]) {- await this.applyNetworkPolicy(policy, ns)+ await this.applyNetworkPolicy(policy, ns, { family: 'webhook-gateway' })}}@@ -3734,7 +3767,7 @@ export class WorkflowRecipeReconciler {console.log(`[WR-Reconciler] Created ${label}`)} catch (error: unknown) {if (getErrorCode(error) === 409) {- if (idempotency && (await this.applyIsNoop(idempotency, label))) {+ if (idempotency && (await this.applyIsNoop(idempotency))) {console.log(`[WR-Reconciler] ${label} unchanged (spec-hash match); skipping update`)return}@@ -3754,43 +3787,17 @@ export class WorkflowRecipeReconciler {/*** True when the existing object already carries the desired spec-hash, so the- * replace would be a no-op. Anything thrown by `readExisting` — a read failure,- * or an invariant the caller asserts on what it just read — returns false and- * falls through to the replace: never skip an update we cannot prove is- * unnecessary. Callers whose `readExisting` asserts invariants must re-assert on- * their own read in `replaceFn`: this catch is total by design (see- * `applyNetworkPolicy`).+ * replace would be a no-op. A read failure returns false (fall through to+ * replace) — never skip an update we cannot prove is unnecessary.*/- private async applyIsNoop(- idempotency: {- manifest: { metadata?: { annotations?: { [key: string]: string } } }- readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>- },- label: string- ): Promise<boolean> {+ private async applyIsNoop(idempotency: {+ manifest: { metadata?: { annotations?: { [key: string]: string } } }+ readExisting: () => Promise<{ metadata?: { annotations?: { [key: string]: string } } } | null>+ }): Promise<boolean> {try {const existing = await idempotency.readExisting()return specHashUnchanged(idempotency.manifest, existing ?? null)- } catch (error) {- // An ownership veto is NOT a read failure, and saying so here would be- // actively misleading: the read succeeded, and what threw was the caller's- // assertion on what it read. Stay quiet — `replaceFn` re-reads, re-asserts- // and raises the conflict with its own message a moment later, so the- // accurate line follows immediately and this one would only precede it with- // a wrong diagnosis. Nothing is swallowed; the veto still reaches the- // caller, loud (T4).- if (!(error instanceof NetworkPolicyOwnershipConflictError)) {- // Everything else IS a read failure. Falling through to the replace is- // correct — never skip an update we cannot prove unnecessary — but- // discarding the error left no trace of WHY: revoked RBAC or a- // persistently failing apiserver looked exactly like a healthy cache- // miss, and the gate silently stopped saving anything. The write still- // happens, so this is a diagnostic line, not a failure path.- console.warn(- `[WR-Reconciler] ${label}: idempotency pre-read failed, falling through to update:`,- error- )- }+ } catch {return false}}@@ -4297,7 +4304,7 @@ export class WorkflowRecipeReconciler {await this.deleteUiIngressIfExists(recipe.metadata.name, w.id, ns)continue}- await this.applyNetworkPolicy(policy, ns)+ await this.applyNetworkPolicy(policy, ns, { family: 'ui-ingress' })}}@@ -4447,7 +4454,7 @@ export class WorkflowRecipeReconciler {)if (!policy) continuerememberDesired(policy)- await this.applyNetworkPolicy(policy, sourceNamespace)+ await this.applyNetworkPolicy(policy, sourceNamespace, { family: 'internal-dependency' })}for (const [targetWorkloadId, sources] of sourcesByTarget.entries()) {@@ -4462,11 +4469,14 @@ export class WorkflowRecipeReconciler {)if (!policy) continuerememberDesired(policy)- await this.applyNetworkPolicy(policy, targetNamespace)+ await this.applyNetworkPolicy(policy, targetNamespace, { family: 'internal-dependency' })}await this.pruneStaleInternalDependencyPolicies(recipe, desiredByNamespace)} catch (error) {+ if (error instanceof RetryableReconcileError || error instanceof NetworkPolicyInfraError) {+ throw error+ }const reason =error instanceof NetworkPolicyOwnershipConflictError? 'OwnershipConflict'@@ -4669,12 +4679,16 @@ export class WorkflowRecipeReconciler {!wlEgressRenewalDue &&!wlEgressStateChanged) {- console.log(- `[WR-Reconciler] NetworkPolicy "${wlPolicyName}" in ${wlNs} egress set unchanged — no-op`+ createLogger('wrc', recipe.metadata.name).info(+ 'network policy egress set unchanged; skipping live apply',+ { policy: wlPolicyName, namespace: wlNs, family: 'workload-egress' })continue}- await this.applyNetworkPolicy(policy, wlNs)+ await this.applyNetworkPolicy(policy, wlNs, {+ family: 'workload-egress',+ existing: existingWlPolicy,+ })// #299: the policy has landed — record the set it actually enforces.this.logResolvedEgressSet(recipe.metadata.name,@@ -4701,7 +4715,7 @@ export class WorkflowRecipeReconciler {await this.deleteWorkloadIngressIfExists(recipe.metadata.name, w.id, wlNs)continue}- await this.applyNetworkPolicy(policy, wlNs)+ await this.applyNetworkPolicy(policy, wlNs, { family: 'workload-ingress' })}}@@ -4782,6 +4796,7 @@ export class WorkflowRecipeReconciler {desired: k8s.V1NetworkPolicy): boolean {if (!existing) return true+ if (!networkPolicyMetadataMatchesDesired(desired, existing)) return true// Decide the write off the ENFORCED rules (the ipBlock set + ports), NOT the// raw state annotation: serializeState embeds expiresAt/lastObservedAt, which// renew on every OK tick, so comparing the annotation string would rewrite@@ -4894,59 +4909,213 @@ export class WorkflowRecipeReconciler {}/**- * Apply a WRC-authored NetworkPolicy through the spec-hash idempotency gate- * (issue #575). Every family funnelling through here — ui-ingress, wl-ingress,- * wr-intdep-*, oauth-broker egress, webhook-gateway, and wl-egress once #299 has- * decided a write is due — is rebuilt deterministically from the recipe on every- * pass, so the unconditional replace was one PUT per policy per ~16s that etcd- * never recorded: pure apiserver CPU and priority-and-fairness budget.- * Deployment, Service, CronJob, Job and DaemonSet already opted in; this was the- * kind whose skip counter stayed at zero.- *- * The wr-intdep-* ownership assertion runs INSIDE `readExisting`, i.e. on the- * very read the gate evaluates, so a spec-hash match can never shortcut it.- * `applyIsNoop` swallows EVERY error `readExisting` throws by design — a read- * failure must fall open to the replace — so the veto reaches the caller through- * `replaceFn`, which re-reads and re-asserts. The read is cached only AFTER the- * assertion passed: the replace path can never inherit a vetoed object, and the- * PUT carries the resourceVersion of the object that passed, so the apiserver- * rejects any change in between.+ * Reconcile one WRC-authored NetworkPolicy from the live object, not from the+ * legacy spec-hash seal. NetworkPolicy is an enforcement resource: a stored+ * annotation can describe what WRC wrote previously while the live spec or+ * lifecycle ownership has since drifted. One validated live snapshot therefore+ * owns both the no-op decision and the resourceVersion used by a replace.*/- private async applyNetworkPolicy(policy: k8s.V1NetworkPolicy, namespace: string): Promise<void> {- const policyName = policy.metadata!.name!- // Set only by a read whose ownership assertion PASSED, which is what makes it- // safe to reuse in the replace path. Unset therefore means "no verified object- // in hand" — the read failed, or its assertion vetoed.- let assertedExisting: k8s.V1NetworkPolicy | undefined- const readExistingAndAssertOwnership = async (): Promise<k8s.V1NetworkPolicy> => {- const fetched = await this.networkingApi.readNamespacedNetworkPolicy({+ private async applyNetworkPolicy(+ policy: k8s.V1NetworkPolicy,+ namespace: string,+ options: NetworkPolicyApplyOptions+ ): Promise<void> {+ const desired = structuredClone(policy)+ const policyName = desired.metadata?.name+ if (!policyName) throw new Error('Cannot apply a NetworkPolicy without metadata.name')+ desired.metadata = desired.metadata ?? {}+ desired.metadata.namespace = desired.metadata.namespace ?? namespace+ this.assertDesiredNetworkPolicyOwner(desired, options.family, namespace)++ const recipeName =+ desired.metadata?.labels?.['clerum.io/recipe'] ??+ desired.metadata?.labels?.['clerum.io/recipe-name'] ??+ policyName+ const log = createLogger('wrc', recipeName)+ const fields = { policy: policyName, namespace, family: options.family }+ const hasSuppliedExisting = Object.prototype.hasOwnProperty.call(options, 'existing')+ let readResult: NetworkPolicyReadResult = hasSuppliedExisting+ ? { existing: options.existing ?? null, recoveredFromReadFailure: false }+ : await this.readNetworkPolicyForApply(policyName, namespace, log, fields)+ let existing = readResult.existing++ if (!existing) {+ try {+ await this.networkingApi.createNamespacedNetworkPolicy({ namespace, body: desired })+ log.info('network policy created', fields)+ return+ } catch (error: unknown) {+ if (getErrorCode(error) !== 409) {+ log.error('network policy create failed', {+ ...fields,+ ...this.safeInfrastructureErrorFields(error),+ })+ if (isRetryableInfraError(error)) {+ const safe = this.safeInfrastructureErrorFields(error)+ throw new NetworkPolicyInfraError(+ `NetworkPolicy "${policyName}" in ${namespace} create temporarily unavailable (${safe.errorCode ?? safe.errorName})`,+ error,+ safe.errorCode+ )+ }+ throw error+ }+ // A real create race is exceptional in the read-first path. Re-read once+ // and evaluate the winner from scratch; never blind-replace after 409.+ readResult = await this.readNetworkPolicyForApply(policyName, namespace, log, fields)+ existing = readResult.existing+ if (!existing) {+ throw new RetryableReconcileError(+ `NetworkPolicy "${policyName}" in ${namespace} still absent after create conflict`,+ { cause: error }+ )+ }+ }+ }++ // Deletion wins over ownership: a terminating object cannot be adopted or+ // safely classified from partial labels. Wait for 404, then recreate from the+ // current desired state.+ if (existing.metadata?.deletionTimestamp) {+ throw new RetryableReconcileError(+ `NetworkPolicy "${policyName}" in ${namespace} is terminating; retrying after deletion`+ )+ }++ // Domain invariants run outside the read catch. A confirmed veto propagates+ // immediately and can never be replaced by a second transport error.+ this.assertInternalDependencyPolicyOwnership(desired, existing, namespace)+ const decision = decideNetworkPolicyConvergence(options.family, desired, existing)+ if (decision.action === 'conflict') {+ throw new NetworkPolicyOwnershipConflictError(+ `Refusing to replace NetworkPolicy "${policyName}" in ${namespace}: ${decision.reason}`+ )+ }+ if (decision.action === 'retry') {+ throw new RetryableReconcileError(+ `NetworkPolicy "${policyName}" in ${namespace} is terminating; retrying after deletion`+ )+ }+ if (decision.action === 'unchanged' && !readResult.recoveredFromReadFailure) {+ log.info('network policy unchanged; skipping update', fields)+ return+ }++ const resourceVersion = existing.metadata?.resourceVersion+ if (!resourceVersion) {+ throw new RetryableReconcileError(+ `NetworkPolicy "${policyName}" in ${namespace} has no resourceVersion`+ )+ }+ desired.metadata.resourceVersion = resourceVersion+ try {+ await this.networkingApi.replaceNamespacedNetworkPolicy({name: policyName,namespace,+ body: desired,+ })+ log.info('network policy replaced', {+ ...fields,+ reason: decision.action === 'unchanged' ? 'read-recovery' : decision.reason,})- this.assertInternalDependencyPolicyOwnership(policy, fetched, namespace)- assertedExisting = fetched- return fetched+ } catch (error: unknown) {+ log.error('network policy replace failed', {+ ...fields,+ ...this.safeInfrastructureErrorFields(error),+ })+ const errorCode = getErrorCode(error)+ if (errorCode === 404 || errorCode === 409) {+ throw new RetryableReconcileError(+ `NetworkPolicy "${policyName}" in ${namespace} ${errorCode === 404 ? 'disappeared' : 'changed'} during replace`,+ { cause: error }+ )+ }+ if (isRetryableInfraError(error)) {+ const safe = this.safeInfrastructureErrorFields(error)+ throw new NetworkPolicyInfraError(+ `NetworkPolicy "${policyName}" in ${namespace} replace temporarily unavailable (${safe.errorCode ?? safe.errorName})`,+ error,+ safe.errorCode+ )+ }+ throw error}- await this.createOrReplace(- () =>- this.networkingApi.createNamespacedNetworkPolicy({- namespace,- body: policy,- }),- async () => {- // Unset means the gate's read failed OR its assertion vetoed (applyIsNoop- // swallows both); re-read so the veto surfaces from here, loud.- const current = assertedExisting ?? (await readExistingAndAssertOwnership())- policy.metadata!.resourceVersion = current.metadata?.resourceVersion- return this.networkingApi.replaceNamespacedNetworkPolicy({- name: policyName,- namespace,- body: policy,- })- },- `NetworkPolicy "${policyName}" in ${namespace}`,- { manifest: policy, readExisting: readExistingAndAssertOwnership }+ }++ private assertDesiredNetworkPolicyOwner(+ desired: k8s.V1NetworkPolicy,+ family: NetworkPolicyFamily,+ namespace: string+ ): void {+ const controllerOwners = (desired.metadata?.ownerReferences ?? []).filter(+ owner => owner.controller === true)+ const invalidControllerOwner =+ controllerOwners.length > 1 || controllerOwners.some(owner => !owner.uid)+ const gatewayOwnerMissing = family === 'webhook-gateway' && controllerOwners.length !== 1+ if (invalidControllerOwner || gatewayOwnerMissing) {+ throw new NetworkPolicyOwnershipConflictError(+ `Refusing to apply NetworkPolicy "${desired.metadata?.name ?? '<unknown>'}" in ${namespace}: desired controller owner is incomplete`+ )+ }+ }++ private async readNetworkPolicyForApply(+ name: string,+ namespace: string,+ log: ReturnType<typeof createLogger>,+ fields: { policy: string; namespace: string; family: NetworkPolicyFamily }+ ): Promise<NetworkPolicyReadResult> {+ for (let attempt = 1; attempt <= 2; attempt += 1) {+ try {+ return {+ existing: await this.networkingApi.readNamespacedNetworkPolicy({ name, namespace }),+ recoveredFromReadFailure: attempt > 1,+ }+ } catch (error: unknown) {+ if (getErrorCode(error) === 404) {+ return { existing: null, recoveredFromReadFailure: attempt > 1 }+ }+ if (attempt === 2) {+ if (isRetryableInfraError(error)) {+ const safe = this.safeInfrastructureErrorFields(error)+ throw new NetworkPolicyInfraError(+ `NetworkPolicy "${name}" in ${namespace} read temporarily unavailable (${safe.errorCode ?? safe.errorName})`,+ error,+ safe.errorCode+ )+ }+ throw error+ }+ log.warn('network policy read unavailable; retrying once', {+ ...fields,+ ...this.safeInfrastructureErrorFields(error),+ })+ }+ }+ return { existing: null, recoveredFromReadFailure: false }+ }++ private safeInfrastructureErrorFields(error: unknown): {+ errorName: string+ errorCode?: string | number+ retryable: boolean+ } {+ const candidateCode = (error as { code?: unknown } | null)?.code+ const candidateName = error instanceof Error ? error.name : 'UnknownError'+ const errorCode =+ typeof candidateCode === 'number'+ ? candidateCode+ : typeof candidateCode === 'string' && /^[A-Z0-9_]{1,64}$/.test(candidateCode)+ ? candidateCode+ : getErrorCode(error)+ return {+ errorName: /^[A-Za-z][A-Za-z0-9_.-]{0,63}$/.test(candidateName)+ ? candidateName+ : 'UnknownError',+ ...(errorCode === undefined ? {} : { errorCode }),+ retryable: isRetryableInfraError(error),+ }}private coordinatorGfsNetworkPolicy(recipe: WorkflowRecipeCRD): k8s.V1NetworkPolicy {@@ -6266,7 +6435,7 @@ export class WorkflowRecipeReconciler {)return}- await this.applyNetworkPolicy(policy, ns)+ await this.applyNetworkPolicy(policy, ns, { family: 'oauth-broker-egress' })}private async ensureConfigMap(