Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(hcc): add high-tail pass-duration buckets and document resync enable (#478)

Alfredo Lopez31f06b95a00d
diff --git a/host-context-controller/README.md b/host-context-controller/README.md
index 24f5c257b..35a182d9d 100644
--- a/host-context-controller/README.md
+++ b/host-context-controller/README.md
@@ -439,16 +439,16 @@ Read by `src/gfsConfig.ts`. Invalid values **throw at startup** rather than sile
### NetworkPolicy
-| Variable | Default | Description |
-| ------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |
-| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |
-| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |
-| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |
-| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval. Startup still converges once. |
-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). |
-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. |
-| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |
+| Variable | Default | Description |
+| ------------------------------------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |
+| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |
+| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |
+| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |
+| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval (merge default). Startup still converges once. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |
+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). |
+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. |
+| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |
### Plugin image allowlist
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index 1d7ff668d..6055cad7b 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.247",
+ "version": "0.1.248",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.247",
+ "version": "0.1.248",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index 44516712a..b4d204505 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.247",
+ "version": "0.1.248",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/k8sClient.ts b/host-context-controller/src/k8sClient.ts
index 751e40b33..0858cc84f 100644
--- a/host-context-controller/src/k8sClient.ts
+++ b/host-context-controller/src/k8sClient.ts
@@ -2849,6 +2849,9 @@ export class McpServerWatcher implements McpServerProvider {
)
}
+ // Interval > 0 is an ops enable, not a merge default. Issue #478: before
+ // any non-zero value, add a freshness gate or use a controller-runtime
+ // 10h-scale period. 300s is too aggressive while passes last minutes.
const netPolResyncSec = config.netPolResyncIntervalSec
if (netPolResyncSec > 0) {
this.netPolResyncTimer = setInterval(() => {
diff --git a/host-context-controller/src/metrics.test.ts b/host-context-controller/src/metrics.test.ts
new file mode 100644
index 000000000..f5d86e304
--- /dev/null
+++ b/host-context-controller/src/metrics.test.ts
@@ -0,0 +1,54 @@
+import { describe, expect, it } from 'vitest'
+import {
+ NETWORKPOLICY_PASS_DURATION_BUCKETS,
+ initialConvergencePassDurationSeconds,
+ networkPolicySafetyPassDurationSeconds,
+ registry,
+} from './metrics'
+
+async function readHistogramBucket(name: string, labels: Record<string, string>): Promise<number> {
+ const metric = registry.getSingleMetric(name)
+ if (!metric) throw new Error(`${name} is not registered`)
+ const snapshot = await metric.get()
+ return (
+ snapshot.values.find(entry =>
+ Object.entries(labels).every(([key, value]) => String(entry.labels[key]) === value)
+ )?.value ?? 0
+ )
+}
+
+describe('NetworkPolicy pass-duration histogram buckets', () => {
+ it('includes the 300s GKE recycle cut and longer diagnostic tails', () => {
+ expect([...NETWORKPOLICY_PASS_DURATION_BUCKETS]).toEqual([
+ 1, 5, 15, 60, 120, 300, 600, 1200, 1800, 3600,
+ ])
+ })
+
+ it('records a 250s initial-convergence pass in the 300s bucket, not +Inf-only', async () => {
+ const name = 'clerum_hcc_initial_convergence_pass_duration_seconds'
+ const labels = { lane: 'NetworkPolicy', result: 'certified' }
+ const before120 = await readHistogramBucket(name, { ...labels, le: '120' })
+ const before300 = await readHistogramBucket(name, { ...labels, le: '300' })
+ const beforeInf = await readHistogramBucket(name, { ...labels, le: '+Inf' })
+
+ initialConvergencePassDurationSeconds.observe(labels, 250)
+
+ expect(await readHistogramBucket(name, { ...labels, le: '120' })).toBe(before120)
+ expect(await readHistogramBucket(name, { ...labels, le: '300' })).toBe(before300 + 1)
+ expect(await readHistogramBucket(name, { ...labels, le: '+Inf' })).toBe(beforeInf + 1)
+ })
+
+ it('records a 250s safety pass in the 300s bucket, not +Inf-only', async () => {
+ const name = 'clerum_hcc_networkpolicy_safety_pass_duration_seconds'
+ const labels = { outcome: 'completed' }
+ const before120 = await readHistogramBucket(name, { ...labels, le: '120' })
+ const before300 = await readHistogramBucket(name, { ...labels, le: '300' })
+ const beforeInf = await readHistogramBucket(name, { ...labels, le: '+Inf' })
+
+ networkPolicySafetyPassDurationSeconds.observe(labels, 250)
+
+ expect(await readHistogramBucket(name, { ...labels, le: '120' })).toBe(before120)
+ expect(await readHistogramBucket(name, { ...labels, le: '300' })).toBe(before300 + 1)
+ expect(await readHistogramBucket(name, { ...labels, le: '+Inf' })).toBe(beforeInf + 1)
+ })
+})
diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.ts
index 10c51b19b..72a6b62ba 100644
--- a/host-context-controller/src/metrics.ts
+++ b/host-context-controller/src/metrics.ts
@@ -49,6 +49,14 @@ function gauge(options: {
*/
const RECONCILE_LATENCY_BUCKETS = [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 15, 45, 120] as const
+/**
+ * Pass-duration buckets. Shared admission histograms top at 120s; GKE watch
+ * recycle and the #462 duration criterion need an explicit 300s cut plus tails.
+ */
+export const NETWORKPOLICY_PASS_DURATION_BUCKETS = [
+ 1, 5, 15, 60, 120, 300, 600, 1200, 1800, 3600,
+] as const
+
/** Idempotent Histogram helper mirroring counter()/gauge(). */
function histogram(options: {
name: string
@@ -113,12 +121,14 @@ export const initialConvergencePassDurationSeconds = histogram({
name: 'clerum_hcc_initial_convergence_pass_duration_seconds',
help: 'Seconds spent in an initial background convergence pass, labeled by named result.',
labelNames: ['lane', 'result'] as const,
+ buckets: NETWORKPOLICY_PASS_DURATION_BUCKETS,
})
export const networkPolicySafetyPassDurationSeconds = histogram({
name: 'clerum_hcc_networkpolicy_safety_pass_duration_seconds',
help: 'Seconds until an authoritative NetworkPolicy safety pass has revoked stale allows.',
labelNames: ['outcome'] as const,
+ buckets: NETWORKPOLICY_PASS_DURATION_BUCKETS,
})
export const networkPolicySafetyPassPoliciesTotal = counter({