Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(hcc): certify after an orphan-sweep cap trip (#478)

Alfredo Lopez08ef8c043e49
diff --git a/host-context-controller/README.md b/host-context-controller/README.md
index f10fdcf55..e9e958c6a 100644
--- a/host-context-controller/README.md
+++ b/host-context-controller/README.md
@@ -439,16 +439,16 @@ Read by `src/gfsConfig.ts`. Invalid values **throw at startup** rather than sile
### NetworkPolicy
-| Variable | Default | Description |
-| ------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |
-| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |
-| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |
-| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |
-| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval (merge default). Startup still converges once. Default `0` logs a `console.warn` on every boot (same pattern as Host/SFS/GFS/LlmHook) — expected; it names the self-heal gap. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |
-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). Governs **every** `fullReconcile` (startup, event-driven, and the resync timer), not only a periodic tick. A legitimate bulk offboarding above the cap withholds certification until an operator raises it. Read-only census: `scripts/ops/hcc-netpol-orphan-census.sh` — `live_cap_would_trip` is Deployment env only; `controller_cap_would_trip` applies these compiled defaults when the env is unset. |
-| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. Same every-`fullReconcile` scope as the absolute cap. |
-| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |
+| Variable | Default | Description |
+| ------------------------------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |
+| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |
+| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |
+| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |
+| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval (merge default). Startup still converges once. Default `0` logs a `console.warn` on every boot (same pattern as Host/SFS/GFS/LlmHook) — expected; it names the self-heal gap. **Enable is a separate ops decision** (issue #478): before any non-zero value, add a freshness gate **or** use a controller-runtime 10h-scale period. Do not treat `300` as the enable value while passes last minutes. |
+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse **deletes** and increment `clerum_hcc_netpol_orphan_sweep_capped_total`; the pass **still certifies** (#478: alert and do not delete). Governs every `fullReconcile` (startup, event-driven, timer). Absolute is checked first, so at a fleet ≥ 50 with these defaults the percent rule never binds. Census: `scripts/ops/hcc-netpol-orphan-census.sh` — `live_cap_would_trip` is Deployment env only; `controller_cap_would_trip` applies compiled defaults when unset. |
+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. Same every-`fullReconcile` scope as the absolute cap; only binds when the fleet is small enough that 20% is below the absolute cap of 10. |
+| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |
### Plugin image allowlist
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index dbbd87fd5..f02d8db39 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.249",
+ "version": "0.1.250",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.249",
+ "version": "0.1.250",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index d3d738da8..189a692dd 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.249",
+ "version": "0.1.250",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/config.ts b/host-context-controller/src/config.ts
index 271733220..ff96962df 100644
--- a/host-context-controller/src/config.ts
+++ b/host-context-controller/src/config.ts
@@ -178,8 +178,9 @@ export interface Config {
netPolResyncIntervalSec: number
// Absolute orphan-delete cap for a NetworkPolicy fullReconcile sweep.
- // Candidates strictly above this count are refused (no deletes, no
- // certification). Set via CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP.
+ // Candidates strictly above this count refuse deletes and increment the
+ // cap metric; the pass still certifies. Set via
+ // CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP.
netPolOrphanDeleteCap: number
// Percent orphan-delete cap (0-100) against the listed managed fleet.
@@ -761,7 +762,8 @@ export const config: Config = {
// sweep. Unset or 0 → no interval.
netPolResyncIntervalSec: getEnvInt('CONTEXT_MAPPER_NETPOL_RESYNC_SEC', 0),
// Mass-delete guard for the namespace-wide orphan sweep. Thresholds are
- // strict `>`: exactly N candidates still delete; N+1 refuses the pass.
+ // strict `>`: exactly N candidates still delete; N+1 refuses deletes and
+ // still certifies.
netPolOrphanDeleteCap: getEnvInt('CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP', 10),
netPolOrphanDeleteCapPercent: getEnvInt('CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT', 20),
diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.ts
index 72a6b62ba..19d630f30 100644
--- a/host-context-controller/src/metrics.ts
+++ b/host-context-controller/src/metrics.ts
@@ -145,7 +145,7 @@ export const netPolOrphansDeletedTotal = counter({
export const netPolOrphanSweepCappedTotal = counter({
name: 'clerum_hcc_netpol_orphan_sweep_capped_total',
- help: 'NetworkPolicy orphan sweeps that refused deletes because the candidate count exceeded the absolute or percent cap.',
+ help: 'NetworkPolicy orphan sweeps that refused deletes because the candidate count exceeded the absolute or percent cap. The pass still certifies.',
labelNames: ['reason'] as const,
})
diff --git a/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts b/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts
index 140792ce0..61f1a9e43 100644
--- a/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts
+++ b/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts
@@ -248,7 +248,7 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {
expect(netPolOrphanSweepCappedTotal.inc).not.toHaveBeenCalled()
})
- it('M4: N+1 orphans refuse every delete, increment absolute cap, skip certification', async () => {
+ it('M4: N+1 orphans refuse every delete, increment absolute cap, and still certify', async () => {
const orphans = Array.from({ length: 11 }, (_, i) =>
orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`)
)
@@ -256,24 +256,39 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {
const complete = vi.fn()
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
- await expect(
- reconciler.fullReconcile([], [], {
- ensureDefaults: false,
- contextInventoryAuthoritative: () => true,
- serverInventoryAuthoritative: () => true,
- onAuthoritativeRevocationComplete: complete,
- })
- ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()
expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'absolute' })
- expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledTimes(1)
+ expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledTimes(3)
expect(netPolOrphansDeletedTotal.inc).not.toHaveBeenCalled()
- expect(complete).not.toHaveBeenCalled()
+ expect(complete).toHaveBeenCalledTimes(3)
+ expect(
+ warnSpy.mock.calls.some(call =>
+ String(call[0]).includes(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+ )
+ ).toBe(true)
warnSpy.mockRestore()
})
- it('M4 sibling: orphans over the percent share refuse with reason=percent', async () => {
+ it('M4 sibling: orphans over the percent share refuse deletes with reason=percent and still certify', async () => {
const orphans = Array.from({ length: 5 }, (_, i) =>
orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`)
)
@@ -281,18 +296,16 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {
const complete = vi.fn()
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
- await expect(
- reconciler.fullReconcile([], [], {
- ensureDefaults: false,
- contextInventoryAuthoritative: () => true,
- serverInventoryAuthoritative: () => true,
- onAuthoritativeRevocationComplete: complete,
- })
- ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()
expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'percent' })
- expect(complete).not.toHaveBeenCalled()
+ expect(complete).toHaveBeenCalledOnce()
warnSpy.mockRestore()
})
it('S2: absolute cap is strict >; exactly N deletes and N+1 refuses', async () => {
@@ -319,17 +332,15 @@ describe('NetworkPolicy orphan sweep cap (#478)', () => {
const completeEleven = vi.fn()
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
- await expect(
- reconciler.fullReconcile([], [], {
- ensureDefaults: false,
- contextInventoryAuthoritative: () => true,
- serverInventoryAuthoritative: () => true,
- onAuthoritativeRevocationComplete: completeEleven,
- })
- ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: completeEleven,
+ })
expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()
expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'absolute' })
- expect(completeEleven).not.toHaveBeenCalled()
+ expect(completeEleven).toHaveBeenCalledOnce()
warnSpy.mockRestore()
})
diff --git a/host-context-controller/src/networkPolicyReconciler.ts b/host-context-controller/src/networkPolicyReconciler.ts
index b7a3078bb..456ef52ab 100644
--- a/host-context-controller/src/networkPolicyReconciler.ts
+++ b/host-context-controller/src/networkPolicyReconciler.ts
@@ -90,8 +90,9 @@ export type NetPolOrphanSweepCapReason = 'absolute' | 'percent'
* real orphan cannot look like a cache-wipe.
*
* Runs on every `fullReconcile` (startup, event-driven, and the resync
- * timer), not only a periodic tick. A legitimate bulk offboarding above
- * the cap withholds certification until an operator raises the env.
+ * timer), not only a periodic tick. A trip refuses the orphan deletes and
+ * increments the cap metric; the pass still certifies the rest of the
+ * inventory (#478: alert and do not delete — not "and do not certify").
*/
export function evaluateNetPolOrphanSweepCap(
orphanCount: number,
@@ -1476,40 +1477,47 @@ export class NetworkPolicyReconciler {
if (capReason) {
netPolOrphanSweepCappedTotal.inc({ reason: capReason })
console.warn(
- `[NetPol] Orphan sweep capped (${capReason}): ${orphanCandidates.length} candidates of ${listedManagedFleet} listed managed policies; refusing deletes and withholding certification`
+ `[NetPol] ${NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE} (${capReason}): ${orphanCandidates.length} candidates of ${listedManagedFleet} listed managed policies; refusing deletes and continuing certification`
)
- throw new Error(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
}
// Delete orphaned Context policies across every L2 lane. Each lane uses
// the canonical contextId effect key and rechecks authority plus live CRD
- // presence immediately before its delete.
- await cleanupOrphanedContextPolicies(
- allContextPolicies,
- policy =>
- this.deleteSafetyPolicySnapshot(config.namespace, policy, contextCleanupAuthoritative),
- (name, contextId) =>
- `[NetPol] Deleting orphaned policy "${name}" (context "${contextId}" no longer exists)`,
- 'context-ingress'
- )
- await cleanupOrphanedContextPolicies(
- allContextEgressPolicies,
- policy =>
- this.deleteSafetyPolicySnapshot(config.hostNamespace, policy, contextCleanupAuthoritative),
- name => `[NetPol] Deleting orphaned L2 egress policy "${name}"`,
- 'context-host-egress'
- )
- await cleanupOrphanedContextPolicies(
- allRpcProxyEgressPolicies,
- policy =>
- this.deleteSafetyPolicySnapshot(
- config.rpcProxyNamespace,
- policy,
- contextCleanupAuthoritative
- ),
- name => `[NetPol] Deleting orphaned rpc-proxy egress policy "${name}"`,
- 'context-rpc-egress'
- )
+ // presence immediately before its delete. A cap trip skips this sweep
+ // (and the external-orphan loop below) so a blown cache cannot mass-delete,
+ // then the pass still certifies live inventory.
+ if (!capReason) {
+ await cleanupOrphanedContextPolicies(
+ allContextPolicies,
+ policy =>
+ this.deleteSafetyPolicySnapshot(config.namespace, policy, contextCleanupAuthoritative),
+ (name, contextId) =>
+ `[NetPol] Deleting orphaned policy "${name}" (context "${contextId}" no longer exists)`,
+ 'context-ingress'
+ )
+ await cleanupOrphanedContextPolicies(
+ allContextEgressPolicies,
+ policy =>
+ this.deleteSafetyPolicySnapshot(
+ config.hostNamespace,
+ policy,
+ contextCleanupAuthoritative
+ ),
+ name => `[NetPol] Deleting orphaned L2 egress policy "${name}"`,
+ 'context-host-egress'
+ )
+ await cleanupOrphanedContextPolicies(
+ allRpcProxyEgressPolicies,
+ policy =>
+ this.deleteSafetyPolicySnapshot(
+ config.rpcProxyNamespace,
+ policy,
+ contextCleanupAuthoritative
+ ),
+ name => `[NetPol] Deleting orphaned rpc-proxy egress policy "${name}"`,
+ 'context-rpc-egress'
+ )
+ }
let liveDesiredRevisionChanged = false
for (const context of contexts) {
@@ -1547,7 +1555,7 @@ export class NetworkPolicyReconciler {
}
// Clean up external egress policies for servers that no longer exist.
- if (options.serverInventoryComplete !== false && serverCleanupAuthoritative()) {
+ if (!capReason && options.serverInventoryComplete !== false && serverCleanupAuthoritative()) {
for (const policy of allExternalPolicies) {
const serverName = policy.metadata?.labels?.[MCPSERVER_LABEL]
if (!serverName) {
diff --git a/scripts/ops/hcc-netpol-orphan-census.sh b/scripts/ops/hcc-netpol-orphan-census.sh
index 8b70dd040..088cc1eb1 100755
--- a/scripts/ops/hcc-netpol-orphan-census.sh
+++ b/scripts/ops/hcc-netpol-orphan-census.sh
@@ -13,7 +13,8 @@
# The running controller still applies compiled defaults in that case
# (absolute 10, percent 20 — host-context-controller/src/config.ts).
# `controller_cap_would_trip` answers that question so a reader cannot
-# treat live_cap_would_trip=none as "the sweep will proceed".
+# treat live_cap_would_trip=none as "the controller will delete".
+# A trip refuses deletes only; the pass still certifies.
#
# Double-samples 90s apart. Adjudicates only when the desired Context +
# McpServer identity set is identical across samples; otherwise