Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(hcc): certify after an orphan-sweep cap trip (#478)
Alfredo Lopez08ef8c043e49
diff --git a/host-context-controller/README.md b/host-context-controller/README.mdindex f10fdcf55..e9e958c6a 100644--- a/host-context-controller/README.md+++ b/host-context-controller/README.md@@ -439,16 +439,16 @@ Read by `src/gfsConfig.ts`. Invalid values **throw at startup** rather than sile### NetworkPolicy-| Variable | Default | Description |-| ------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |-| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |-| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |-| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |-| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |-| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval (merge default). Startup still converges once. Default `0` logs a `console.warn` on every boot (same pattern as Host/SFS/GFS/LlmHook) — expected; it names the self-heal gap. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). Governs **every** `fullReconcile` (startup, event-driven, and the resync timer), not only a periodic tick. A legitimate bulk offboarding above the cap withholds certification until an operator raises it. Read-only census: `scripts/ops/hcc-netpol-orphan-census.sh` — `live_cap_would_trip` is Deployment env only; `controller_cap_would_trip` applies these compiled defaults when the env is unset. |-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. Same every-`fullReconcile` scope as the absolute cap. |-| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |+| Variable | Default | Description |+| ------------------------------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |+| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |+| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |+| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |+| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval (merge default). Startup still converges once. Default `0` logs a `console.warn` on every boot (same pattern as Host/SFS/GFS/LlmHook) — expected; it names the self-heal gap. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse **deletes** and increment `clerum_hcc_netpol_orphan_sweep_capped_total`; the pass **still certifies** (#478: alert and do not delete). Governs every `fullReconcile` (startup, event-driven, timer). Absolute is checked first, so at a fleet ≥ 50 with these defaults the percent rule never binds. Census: `scripts/ops/hcc-netpol-orphan-census.sh` — `live_cap_would_trip` is Deployment env only; `controller_cap_would_trip` applies compiled defaults when unset. |+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. Same every-`fullReconcile` scope as the absolute cap; only binds when the fleet is small enough that 20% is below the absolute cap of 10. |+| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |### Plugin image allowlistdiff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex dbbd87fd5..f02d8db39 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.249",+ "version": "0.1.250","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.249",+ "version": "0.1.250","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/image-policy": "file:../packages/image-policy",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex d3d738da8..189a692dd 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.249",+ "version": "0.1.250","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/config.ts b/host-context-controller/src/config.tsindex 271733220..ff96962df 100644--- a/host-context-controller/src/config.ts+++ b/host-context-controller/src/config.ts@@ -178,8 +178,9 @@ export interface Config {netPolResyncIntervalSec: number// Absolute orphan-delete cap for a NetworkPolicy fullReconcile sweep.- // Candidates strictly above this count are refused (no deletes, no- // certification). Set via CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP.+ // Candidates strictly above this count refuse deletes and increment the+ // cap metric; the pass still certifies. Set via+ // CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP.netPolOrphanDeleteCap: number// Percent orphan-delete cap (0-100) against the listed managed fleet.@@ -761,7 +762,8 @@ export const config: Config = {// sweep. Unset or 0 → no interval.netPolResyncIntervalSec: getEnvInt('CONTEXT_MAPPER_NETPOL_RESYNC_SEC', 0),// Mass-delete guard for the namespace-wide orphan sweep. Thresholds are- // strict `>`: exactly N candidates still delete; N+1 refuses the pass.+ // strict `>`: exactly N candidates still delete; N+1 refuses deletes and+ // still certifies.netPolOrphanDeleteCap: getEnvInt('CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP', 10),netPolOrphanDeleteCapPercent: getEnvInt('CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT', 20),diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.tsindex 72a6b62ba..19d630f30 100644--- a/host-context-controller/src/metrics.ts+++ b/host-context-controller/src/metrics.ts@@ -145,7 +145,7 @@ export const netPolOrphansDeletedTotal = counter({export const netPolOrphanSweepCappedTotal = counter({name: 'clerum_hcc_netpol_orphan_sweep_capped_total',- help: 'NetworkPolicy orphan sweeps that refused deletes because the candidate count exceeded the absolute or percent cap.',+ help: 'NetworkPolicy orphan sweeps that refused deletes because the candidate count exceeded the absolute or percent cap. The pass still certifies.',labelNames: ['reason'] as const,})diff --git a/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts b/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.tsindex 140792ce0..61f1a9e43 100644--- a/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts+++ b/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts@@ -248,7 +248,7 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {expect(netPolOrphanSweepCappedTotal.inc).not.toHaveBeenCalled()})- it('M4: N+1 orphans refuse every delete, increment absolute cap, skip certification', async () => {+ it('M4: N+1 orphans refuse every delete, increment absolute cap, and still certify', async () => {const orphans = Array.from({ length: 11 }, (_, i) =>orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`))@@ -256,24 +256,39 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {const complete = vi.fn()const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})- await expect(- reconciler.fullReconcile([], [], {- ensureDefaults: false,- contextInventoryAuthoritative: () => true,- serverInventoryAuthoritative: () => true,- onAuthoritativeRevocationComplete: complete,- })- ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)+ await reconciler.fullReconcile([], [], {+ ensureDefaults: false,+ contextInventoryAuthoritative: () => true,+ serverInventoryAuthoritative: () => true,+ onAuthoritativeRevocationComplete: complete,+ })+ await reconciler.fullReconcile([], [], {+ ensureDefaults: false,+ contextInventoryAuthoritative: () => true,+ serverInventoryAuthoritative: () => true,+ onAuthoritativeRevocationComplete: complete,+ })+ await reconciler.fullReconcile([], [], {+ ensureDefaults: false,+ contextInventoryAuthoritative: () => true,+ serverInventoryAuthoritative: () => true,+ onAuthoritativeRevocationComplete: complete,+ })expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'absolute' })- expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledTimes(1)+ expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledTimes(3)expect(netPolOrphansDeletedTotal.inc).not.toHaveBeenCalled()- expect(complete).not.toHaveBeenCalled()+ expect(complete).toHaveBeenCalledTimes(3)+ expect(+ warnSpy.mock.calls.some(call =>+ String(call[0]).includes(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)+ )+ ).toBe(true)warnSpy.mockRestore()})- it('M4 sibling: orphans over the percent share refuse with reason=percent', async () => {+ it('M4 sibling: orphans over the percent share refuse deletes with reason=percent and still certify', async () => {const orphans = Array.from({ length: 5 }, (_, i) =>orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`))@@ -281,18 +296,16 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {const complete = vi.fn()const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})- await expect(- reconciler.fullReconcile([], [], {- ensureDefaults: false,- contextInventoryAuthoritative: () => true,- serverInventoryAuthoritative: () => true,- onAuthoritativeRevocationComplete: complete,- })- ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)+ await reconciler.fullReconcile([], [], {+ ensureDefaults: false,+ contextInventoryAuthoritative: () => true,+ serverInventoryAuthoritative: () => true,+ onAuthoritativeRevocationComplete: complete,+ })expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'percent' })- expect(complete).not.toHaveBeenCalled()+ expect(complete).toHaveBeenCalledOnce()warnSpy.mockRestore()})it('S2: absolute cap is strict >; exactly N deletes and N+1 refuses', async () => {@@ -319,17 +332,15 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {const completeEleven = vi.fn()const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})- await expect(- reconciler.fullReconcile([], [], {- ensureDefaults: false,- contextInventoryAuthoritative: () => true,- serverInventoryAuthoritative: () => true,- onAuthoritativeRevocationComplete: completeEleven,- })- ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)+ await reconciler.fullReconcile([], [], {+ ensureDefaults: false,+ contextInventoryAuthoritative: () => true,+ serverInventoryAuthoritative: () => true,+ onAuthoritativeRevocationComplete: completeEleven,+ })expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'absolute' })- expect(completeEleven).not.toHaveBeenCalled()+ expect(completeEleven).toHaveBeenCalledOnce()warnSpy.mockRestore()})diff --git a/host-context-controller/src/networkPolicyReconciler.ts b/host-context-controller/src/networkPolicyReconciler.tsindex b7a3078bb..456ef52ab 100644--- a/host-context-controller/src/networkPolicyReconciler.ts+++ b/host-context-controller/src/networkPolicyReconciler.ts@@ -90,8 +90,9 @@ export type NetPolOrphanSweepCapReason = 'absolute' | 'percent'* real orphan cannot look like a cache-wipe.** Runs on every `fullReconcile` (startup, event-driven, and the resync- * timer), not only a periodic tick. A legitimate bulk offboarding above- * the cap withholds certification until an operator raises the env.+ * timer), not only a periodic tick. A trip refuses the orphan deletes and+ * increments the cap metric; the pass still certifies the rest of the+ * inventory (#478: alert and do not delete — not "and do not certify").*/export function evaluateNetPolOrphanSweepCap(orphanCount: number,@@ -1476,40 +1477,47 @@ export class NetworkPolicyReconciler {if (capReason) {netPolOrphanSweepCappedTotal.inc({ reason: capReason })console.warn(- `[NetPol] Orphan sweep capped (${capReason}): ${orphanCandidates.length} candidates of ${listedManagedFleet} listed managed policies; refusing deletes and withholding certification`+ `[NetPol] ${NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE} (${capReason}): ${orphanCandidates.length} candidates of ${listedManagedFleet} listed managed policies; refusing deletes and continuing certification`)- throw new Error(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)}// Delete orphaned Context policies across every L2 lane. Each lane uses// the canonical contextId effect key and rechecks authority plus live CRD- // presence immediately before its delete.- await cleanupOrphanedContextPolicies(- allContextPolicies,- policy =>- this.deleteSafetyPolicySnapshot(config.namespace, policy, contextCleanupAuthoritative),- (name, contextId) =>- `[NetPol] Deleting orphaned policy "${name}" (context "${contextId}" no longer exists)`,- 'context-ingress'- )- await cleanupOrphanedContextPolicies(- allContextEgressPolicies,- policy =>- this.deleteSafetyPolicySnapshot(config.hostNamespace, policy, contextCleanupAuthoritative),- name => `[NetPol] Deleting orphaned L2 egress policy "${name}"`,- 'context-host-egress'- )- await cleanupOrphanedContextPolicies(- allRpcProxyEgressPolicies,- policy =>- this.deleteSafetyPolicySnapshot(- config.rpcProxyNamespace,- policy,- contextCleanupAuthoritative- ),- name => `[NetPol] Deleting orphaned rpc-proxy egress policy "${name}"`,- 'context-rpc-egress'- )+ // presence immediately before its delete. A cap trip skips this sweep+ // (and the external-orphan loop below) so a blown cache cannot mass-delete,+ // then the pass still certifies live inventory.+ if (!capReason) {+ await cleanupOrphanedContextPolicies(+ allContextPolicies,+ policy =>+ this.deleteSafetyPolicySnapshot(config.namespace, policy, contextCleanupAuthoritative),+ (name, contextId) =>+ `[NetPol] Deleting orphaned policy "${name}" (context "${contextId}" no longer exists)`,+ 'context-ingress'+ )+ await cleanupOrphanedContextPolicies(+ allContextEgressPolicies,+ policy =>+ this.deleteSafetyPolicySnapshot(+ config.hostNamespace,+ policy,+ contextCleanupAuthoritative+ ),+ name => `[NetPol] Deleting orphaned L2 egress policy "${name}"`,+ 'context-host-egress'+ )+ await cleanupOrphanedContextPolicies(+ allRpcProxyEgressPolicies,+ policy =>+ this.deleteSafetyPolicySnapshot(+ config.rpcProxyNamespace,+ policy,+ contextCleanupAuthoritative+ ),+ name => `[NetPol] Deleting orphaned rpc-proxy egress policy "${name}"`,+ 'context-rpc-egress'+ )+ }let liveDesiredRevisionChanged = falsefor (const context of contexts) {@@ -1547,7 +1555,7 @@ export class NetworkPolicyReconciler {}// Clean up external egress policies for servers that no longer exist.- if (options.serverInventoryComplete !== false && serverCleanupAuthoritative()) {+ if (!capReason && options.serverInventoryComplete !== false && serverCleanupAuthoritative()) {for (const policy of allExternalPolicies) {const serverName = policy.metadata?.labels?.[MCPSERVER_LABEL]if (!serverName) {diff --git a/scripts/ops/hcc-netpol-orphan-census.sh b/scripts/ops/hcc-netpol-orphan-census.shindex 8b70dd040..088cc1eb1 100755--- a/scripts/ops/hcc-netpol-orphan-census.sh+++ b/scripts/ops/hcc-netpol-orphan-census.sh@@ -13,7 +13,8 @@# The running controller still applies compiled defaults in that case# (absolute 10, percent 20 — host-context-controller/src/config.ts).# `controller_cap_would_trip` answers that question so a reader cannot-# treat live_cap_would_trip=none as "the sweep will proceed".+# treat live_cap_would_trip=none as "the controller will delete".+# A trip refuses deletes only; the pass still certifies.## Double-samples 90s apart. Adjudicates only when the desired Context +# McpServer identity set is identical across samples; otherwise