Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(hcc): skip in-flight NetworkPolicy resync ticks (#478)
Alfredo Lopez36b832c020a0
diff --git a/host-context-controller/README.md b/host-context-controller/README.mdindex 7bf380e66..edc083460 100644--- a/host-context-controller/README.md+++ b/host-context-controller/README.md@@ -439,16 +439,16 @@ Read by `src/gfsConfig.ts`. Invalid values **throw at startup** rather than sile### NetworkPolicy-| Variable | Default | Description |-| ------------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |-| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |-| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |-| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |-| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass with `ensureDefaults: true` so a tick covers orphan sweep (class A) and L0/L1 defaults (class D). `0` disables the interval (merge default). Startup still converges once; `start()` already applied defaults as a bootstrap barrier. Default `0` logs a `console.warn` on every boot (same pattern as Host/SFS/GFS/LlmHook) — expected; it names the self-heal gap. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse **deletes** and increment `clerum_hcc_netpol_orphan_sweep_capped_total`; the pass **still certifies** (#478: alert and do not delete). Governs every `fullReconcile` (startup, event-driven, timer). Absolute is checked first, so at a fleet ≥ 50 with these defaults the percent rule never binds. `0` refuses every orphan delete; a negative value fails loud at load. Census: `scripts/ops/hcc-netpol-orphan-census.sh` — `live_cap_would_trip` is Deployment env only; `controller_cap_would_trip` applies compiled defaults when unset. When the Deployment omits `CONTEXT_MAPPER_RPC_PROXY_NAMESPACE`, the census assumes the compiled `rpc-proxy` default. |-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. Same every-`fullReconcile` scope as the absolute cap; only binds when the fleet is small enough that 20% is below the absolute cap of 10. |-| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |+| Variable | Default | Description |+| ------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |+| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |+| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |+| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |+| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass with `ensureDefaults: true` so a tick covers orphan sweep (class A) and L0/L1 defaults (class D). A tick while a pass is in flight is **skipped** (`clerum_hcc_netpol_resync_ticks_skipped_total{reason="pass-in-flight"}`) and does not coalesce; event-driven trailing is unchanged. `0` disables the interval (merge default). Startup still converges once; `start()` already applied defaults as a bootstrap barrier. Default `0` logs a `console.warn` on every boot (same pattern as Host/SFS/GFS/LlmHook) — expected; it names the self-heal gap. **Enable is a separate ops decision** (issue #478, not this merge): with the skip guard the chosen value is **`1500`**; without the guard the floor is **`3600`**. `300` is not a valid enable while passes last minutes. Treadmill ⇔ pass duration ≥ period. Enable preconditions live on #478: image deployed, census read, histogram p99/max ≪ T, then infra **dev first**. |+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse **deletes** and increment `clerum_hcc_netpol_orphan_sweep_capped_total`; the pass **still certifies** (#478: alert and do not delete). Governs every `fullReconcile` (startup, event-driven, timer). Absolute is checked first, so at a fleet ≥ 50 with these defaults the percent rule never binds. `0` refuses every orphan delete; a negative value fails loud at load. Census: `scripts/ops/hcc-netpol-orphan-census.sh` — `live_cap_would_trip` is Deployment env only; `controller_cap_would_trip` applies compiled defaults when unset. When the Deployment omits `CONTEXT_MAPPER_RPC_PROXY_NAMESPACE`, the census assumes the compiled `rpc-proxy` default. |+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. Same every-`fullReconcile` scope as the absolute cap; only binds when the fleet is small enough that 20% is below the absolute cap of 10. |+| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |### Plugin image allowlistdiff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex ac6fc1214..97d84521c 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.252",+ "version": "0.1.253","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.252",+ "version": "0.1.253","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/image-policy": "file:../packages/image-policy",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex 4729d0eb1..80a149d81 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.252",+ "version": "0.1.253","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/k8sClient.test.ts b/host-context-controller/src/k8sClient.test.tsindex 8d654cebe..d3c6cab7f 100644--- a/host-context-controller/src/k8sClient.test.ts+++ b/host-context-controller/src/k8sClient.test.ts@@ -54,7 +54,8 @@ async function readLabeledConvergenceMetric(name:| 'clerum_hcc_initial_convergence_swallowed_total'| 'clerum_hcc_initial_convergence_effects_dropped_total'- | 'clerum_hcc_initial_convergence_pass_results_total',+ | 'clerum_hcc_initial_convergence_pass_results_total'+ | 'clerum_hcc_netpol_resync_ticks_skipped_total',labels: Record<string, string>): Promise<number> {const metric = registry.getSingleMetric(name)@@ -11350,7 +11351,7 @@ describe('McpServerWatcher NetworkPolicy periodic resync (#478)', () => {)await watcher.stop()})- it('M2: a tick while a pass is in flight keeps one concurrent fullReconcile and trails after settle', async () => {+ it('M2: a tick while a pass is in flight is skipped and does not trail', async () => {vi.useFakeTimers()mockConfig.netPolResyncIntervalSec = 300const activePass = deferred()@@ -11367,15 +11368,26 @@ describe('McpServerWatcher NetworkPolicy periodic resync (#478)', () => {})const watcher = await startWatcherForNetPolResync()await vi.waitFor(() => expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1))+ const skippedBefore = await readLabeledConvergenceMetric(+ 'clerum_hcc_netpol_resync_ticks_skipped_total',+ { reason: 'pass-in-flight' }+ )await vi.advanceTimersByTimeAsync(300_000)+ expect((watcher as any).initialConvergenceRuns.get('NetworkPolicy')?.trailingRequested).toBe(+ false+ )expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1)+ expect(+ await readLabeledConvergenceMetric('clerum_hcc_netpol_resync_ticks_skipped_total', {+ reason: 'pass-in-flight',+ })+ ).toBe(skippedBefore + 1)activePass.resolve()- await vi.waitFor(() => expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(2))- expect(mocks.netPolFullReconcile.mock.calls[1]?.[2]).toEqual(- expect.objectContaining({ ensureDefaults: true })- )+ await flushMicrotasks()+ await vi.advanceTimersByTimeAsync(0)+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1)await watcher.stop()})diff --git a/host-context-controller/src/k8sClient.ts b/host-context-controller/src/k8sClient.tsindex 6e0eabd7c..44c395d42 100644--- a/host-context-controller/src/k8sClient.ts+++ b/host-context-controller/src/k8sClient.ts@@ -64,6 +64,7 @@ import {initialConvergencePassResultsTotal,initialConvergenceRetriesTotal,initialConvergenceSwallowedTotal,+ netPolResyncTicksSkippedTotal,} from './metrics'import {DESIRED_NETWORKPOLICY_INVENTORY_CHANGED_MESSAGE,@@ -2853,12 +2854,18 @@ export class McpServerWatcher implements McpServerProvider {)}- // Interval > 0 is an ops enable, not a merge default. Issue #478: before- // any non-zero value, add a freshness gate or use a controller-runtime- // 10h-scale period. 300s is too aggressive while passes last minutes.+ // Interval > 0 is an ops enable, not a merge default. Issue #478: with+ // this skip-if-in-flight guard the chosen enable is 1500s. Without it the+ // floor would be 3600s. 300s is not a valid enable while passes last+ // minutes (treadmill ⇔ pass duration ≥ period). Event-driven coalescing+ // is unchanged; only this timer trigger is filtered.const netPolResyncSec = config.netPolResyncIntervalSecif (netPolResyncSec > 0) {this.netPolResyncTimer = setInterval(() => {+ if (this.initialConvergenceRuns.has('NetworkPolicy')) {+ netPolResyncTicksSkippedTotal.inc({ reason: 'pass-in-flight' })+ return+ }void this.runInitialNetworkPolicyConvergence({ ensureDefaults: true })}, netPolResyncSec * 1000)console.log(`[K8s] NetworkPolicy periodic resync enabled (every ${netPolResyncSec}s)`)diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.tsindex 19d630f30..5f2accf79 100644--- a/host-context-controller/src/metrics.ts+++ b/host-context-controller/src/metrics.ts@@ -149,6 +149,12 @@ export const netPolOrphanSweepCappedTotal = counter({labelNames: ['reason'] as const,})+export const netPolResyncTicksSkippedTotal = counter({+ name: 'clerum_hcc_netpol_resync_ticks_skipped_total',+ help: 'Periodic NetworkPolicy resync ticks skipped because a convergence pass was already in flight.',+ labelNames: ['reason'] as const,+})+export const contextReconciliationsTotal = counter({name: 'clerum_hcc_context_reconciliations_total',help: 'Total Context CRD reconciliations',