Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(hcc): add high-tail pass-duration buckets and document resync enable (#478)
Alfredo Lopez31f06b95a00d
diff --git a/host-context-controller/README.md b/host-context-controller/README.mdindex 24f5c257b..35a182d9d 100644--- a/host-context-controller/README.md+++ b/host-context-controller/README.md@@ -439,16 +439,16 @@ Read by `src/gfsConfig.ts`. Invalid values **throw at startup** rather than sile### NetworkPolicy-| Variable | Default | Description |-| ------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |-| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |-| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |-| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |-| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval. Startup still converges once. |-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). |-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. |-| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |+| Variable | Default | Description |+| ------------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |+| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |+| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |+| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |+| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval (merge default). Startup still converges once. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). |+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. |+| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |### Plugin image allowlistdiff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex 1d7ff668d..6055cad7b 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.247",+ "version": "0.1.248","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.247",+ "version": "0.1.248","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/image-policy": "file:../packages/image-policy",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex 44516712a..b4d204505 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.247",+ "version": "0.1.248","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/k8sClient.ts b/host-context-controller/src/k8sClient.tsindex 751e40b33..0858cc84f 100644--- a/host-context-controller/src/k8sClient.ts+++ b/host-context-controller/src/k8sClient.ts@@ -2849,6 +2849,9 @@ export class McpServerWatcher implements McpServerProvider {)}+ // Interval > 0 is an ops enable, not a merge default. Issue #478: before+ // any non-zero value, add a freshness gate or use a controller-runtime+ // 10h-scale period. 300s is too aggressive while passes last minutes.const netPolResyncSec = config.netPolResyncIntervalSecif (netPolResyncSec > 0) {this.netPolResyncTimer = setInterval(() => {diff --git a/host-context-controller/src/metrics.test.ts b/host-context-controller/src/metrics.test.tsnew file mode 100644index 000000000..f5d86e304--- /dev/null+++ b/host-context-controller/src/metrics.test.ts@@ -0,0 +1,54 @@+import { describe, expect, it } from 'vitest'+import {+ NETWORKPOLICY_PASS_DURATION_BUCKETS,+ initialConvergencePassDurationSeconds,+ networkPolicySafetyPassDurationSeconds,+ registry,+} from './metrics'++async function readHistogramBucket(name: string, labels: Record<string, string>): Promise<number> {+ const metric = registry.getSingleMetric(name)+ if (!metric) throw new Error(`${name} is not registered`)+ const snapshot = await metric.get()+ return (+ snapshot.values.find(entry =>+ Object.entries(labels).every(([key, value]) => String(entry.labels[key]) === value)+ )?.value ?? 0+ )+}++describe('NetworkPolicy pass-duration histogram buckets', () => {+ it('includes the 300s GKE recycle cut and longer diagnostic tails', () => {+ expect([...NETWORKPOLICY_PASS_DURATION_BUCKETS]).toEqual([+ 1, 5, 15, 60, 120, 300, 600, 1200, 1800, 3600,+ ])+ })++ it('records a 250s initial-convergence pass in the 300s bucket, not +Inf-only', async () => {+ const name = 'clerum_hcc_initial_convergence_pass_duration_seconds'+ const labels = { lane: 'NetworkPolicy', result: 'certified' }+ const before120 = await readHistogramBucket(name, { ...labels, le: '120' })+ const before300 = await readHistogramBucket(name, { ...labels, le: '300' })+ const beforeInf = await readHistogramBucket(name, { ...labels, le: '+Inf' })++ initialConvergencePassDurationSeconds.observe(labels, 250)++ expect(await readHistogramBucket(name, { ...labels, le: '120' })).toBe(before120)+ expect(await readHistogramBucket(name, { ...labels, le: '300' })).toBe(before300 + 1)+ expect(await readHistogramBucket(name, { ...labels, le: '+Inf' })).toBe(beforeInf + 1)+ })++ it('records a 250s safety pass in the 300s bucket, not +Inf-only', async () => {+ const name = 'clerum_hcc_networkpolicy_safety_pass_duration_seconds'+ const labels = { outcome: 'completed' }+ const before120 = await readHistogramBucket(name, { ...labels, le: '120' })+ const before300 = await readHistogramBucket(name, { ...labels, le: '300' })+ const beforeInf = await readHistogramBucket(name, { ...labels, le: '+Inf' })++ networkPolicySafetyPassDurationSeconds.observe(labels, 250)++ expect(await readHistogramBucket(name, { ...labels, le: '120' })).toBe(before120)+ expect(await readHistogramBucket(name, { ...labels, le: '300' })).toBe(before300 + 1)+ expect(await readHistogramBucket(name, { ...labels, le: '+Inf' })).toBe(beforeInf + 1)+ })+})diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.tsindex 10c51b19b..72a6b62ba 100644--- a/host-context-controller/src/metrics.ts+++ b/host-context-controller/src/metrics.ts@@ -49,6 +49,14 @@ function gauge(options: {*/const RECONCILE_LATENCY_BUCKETS = [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 15, 45, 120] as const+/**+ * Pass-duration buckets. Shared admission histograms top at 120s; GKE watch+ * recycle and the #462 duration criterion need an explicit 300s cut plus tails.+ */+export const NETWORKPOLICY_PASS_DURATION_BUCKETS = [+ 1, 5, 15, 60, 120, 300, 600, 1200, 1800, 3600,+] as const+/** Idempotent Histogram helper mirroring counter()/gauge(). */function histogram(options: {name: string@@ -113,12 +121,14 @@ export const initialConvergencePassDurationSeconds = histogram({name: 'clerum_hcc_initial_convergence_pass_duration_seconds',help: 'Seconds spent in an initial background convergence pass, labeled by named result.',labelNames: ['lane', 'result'] as const,+ buckets: NETWORKPOLICY_PASS_DURATION_BUCKETS,})export const networkPolicySafetyPassDurationSeconds = histogram({name: 'clerum_hcc_networkpolicy_safety_pass_duration_seconds',help: 'Seconds until an authoritative NetworkPolicy safety pass has revoked stale allows.',labelNames: ['outcome'] as const,+ buckets: NETWORKPOLICY_PASS_DURATION_BUCKETS,})export const networkPolicySafetyPassPoliciesTotal = counter({