Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(hcc): resync NetworkPolicies through coordinated convergence (#478)

Alfredo Lopez65d514722ed1
diff --git a/host-context-controller/README.md b/host-context-controller/README.md
index d494b15bf..24f5c257b 100644
--- a/host-context-controller/README.md
+++ b/host-context-controller/README.md
@@ -439,13 +439,16 @@ Read by `src/gfsConfig.ts`. Invalid values **throw at startup** rather than sile
### NetworkPolicy
-| Variable | Default | Description |
-| ----------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |
-| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |
-| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |
-| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |
-| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |
+| Variable | Default | Description |
+| ------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `CONTEXT_MAPPER_RUNTIME_NAMESPACES` | `mcp-server,mcp-host,sandbox-recipes,rpc-proxy` | Namespaces that get L0 deny-all + L1 infrastructure policies. Manifest overrides to `mcp-server,sandbox-recipes,rpc-proxy,sandbox-ui` — note it **drops `mcp-host`**, whose policies are static (`deploy/base/mcp-host/networkpolicies.yaml`), because an HCC deny-all would block mcp-host from reaching LLM APIs. |
+| `CONTEXT_MAPPER_MINIMAL_INFRA_NAMESPACES` | `` (empty) | Subset of runtime namespaces that get **only** deny-all + DNS egress — no HCC-API or K8s-API egress. Manifest sets `sandbox-ui`. |
+| `CONTEXT_MAPPER_K8S_API_CIDRS` | _(unset → `[]`)_ | K8s API-server CIDRs for `allow-k8s-api-egress-*`. Empty falls back to `KUBERNETES_SERVICE_HOST`. **Validated fail-closed at module load**: a malformed entry, or one broader than `/24` (IPv4) or `/120` (IPv6), **crashes startup** rather than programming a permissive policy. |
+| `CONTEXT_MAPPER_NODELOCAL_DNS_CIDR` | `` (empty) | DNS infrastructure CIDR for NodeLocal DNSCache / kube-dns. Must be exactly one IPv4 `/32`; anything else throws at startup. |
+| `CONTEXT_MAPPER_NETPOL_RESYNC_SEC` | `0` | Periodic NetworkPolicy coordinated convergence (seconds). Re-enters the single-flight pass; `0` disables the interval. Startup still converges once. |
+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP` | `10` | Absolute orphan-delete cap for a namespace-wide NetworkPolicy sweep. Candidates **strictly above** this count refuse the pass (no deletes, no certification). |
+| `CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT` | `20` | Percent orphan-delete cap against the listed managed fleet. Strict `>`. Inert on a tiny inventory where `percent * listed < 1`. |
+| `KUBERNETES_SERVICE_HOST` | _(injected by Kubernetes)_ | Not set by the operator — the `/32` fallback source when `CONTEXT_MAPPER_K8S_API_CIDRS` is empty. |
### Plugin image allowlist
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index 1cd5126c3..1d7ff668d 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.246",
+ "version": "0.1.247",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.246",
+ "version": "0.1.247",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index b813d7c9f..44516712a 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.246",
+ "version": "0.1.247",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/config.netPolResync.test.ts b/host-context-controller/src/config.netPolResync.test.ts
new file mode 100644
index 000000000..1c5f041e6
--- /dev/null
+++ b/host-context-controller/src/config.netPolResync.test.ts
@@ -0,0 +1,47 @@
+import { afterEach, describe, expect, it, vi } from 'vitest'
+
+const originalEnv = process.env
+
+const KNOBS = [
+ 'CONTEXT_MAPPER_NETPOL_RESYNC_SEC',
+ 'CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP',
+ 'CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT',
+] as const
+
+async function loadConfig(overrides: Partial<Record<(typeof KNOBS)[number], string>>) {
+ vi.resetModules()
+ process.env = { ...originalEnv }
+ for (const k of KNOBS) delete process.env[k]
+ for (const [k, v] of Object.entries(overrides)) process.env[k] = v
+ return import('./config')
+}
+
+afterEach(() => {
+ process.env = originalEnv
+ vi.resetModules()
+})
+
+describe('NetworkPolicy resync and orphan-sweep config (#478)', () => {
+ it('defaults resync to 0 and caps to 10/20 when unset', async () => {
+ const { config } = await loadConfig({})
+ expect(config.netPolResyncIntervalSec).toBe(0)
+ expect(config.netPolOrphanDeleteCap).toBe(10)
+ expect(config.netPolOrphanDeleteCapPercent).toBe(20)
+ })
+
+ it('keeps periodic resync disabled when the interval is 0', async () => {
+ const { config } = await loadConfig({ CONTEXT_MAPPER_NETPOL_RESYNC_SEC: '0' })
+ expect(config.netPolResyncIntervalSec).toBe(0)
+ })
+
+ it('accepts an explicit resync interval and cap overrides', async () => {
+ const { config } = await loadConfig({
+ CONTEXT_MAPPER_NETPOL_RESYNC_SEC: '300',
+ CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP: '8',
+ CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT: '25',
+ })
+ expect(config.netPolResyncIntervalSec).toBe(300)
+ expect(config.netPolOrphanDeleteCap).toBe(8)
+ expect(config.netPolOrphanDeleteCapPercent).toBe(25)
+ })
+})
diff --git a/host-context-controller/src/config.ts b/host-context-controller/src/config.ts
index 2ac77c749..271733220 100644
--- a/host-context-controller/src/config.ts
+++ b/host-context-controller/src/config.ts
@@ -171,6 +171,23 @@ export interface Config {
// event drops on long K8s watch disconnects. 0 disables.
hostResyncIntervalSec: number
+ // Periodic NetworkPolicy convergence interval (seconds). Re-enters the
+ // coordinated single-flight pass (not a naked fullReconcile) so a dropped
+ // watch cannot leave stale allows. Default 0 disables the interval; startup
+ // still runs one convergence pass. Set via CONTEXT_MAPPER_NETPOL_RESYNC_SEC.
+ netPolResyncIntervalSec: number
+
+ // Absolute orphan-delete cap for a NetworkPolicy fullReconcile sweep.
+ // Candidates strictly above this count are refused (no deletes, no
+ // certification). Set via CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP.
+ netPolOrphanDeleteCap: number
+
+ // Percent orphan-delete cap (0-100) against the listed managed fleet.
+ // Candidates strictly above this fraction are refused. Inert on a tiny
+ // inventory where percent*fleet < 1. Set via
+ // CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT.
+ netPolOrphanDeleteCapPercent: number
+
// Per-request deadline for finite Kubernetes calls on the serialized Host
// convergence path. Watches keep their independent long-lived lifecycle.
hostK8sRequestTimeoutMs: number
@@ -739,6 +756,15 @@ export const config: Config = {
// for runtime-auth degraded mcp-host pods; use only for diagnostic runs.
hostResyncIntervalSec: getEnvInt('CONTEXT_MAPPER_HOST_RESYNC_SEC', 300),
+ // Periodic NetworkPolicy resync. Default 0 (disabled): startup still
+ // converges once; the interval is opt-in so a bad cache cannot periodically
+ // sweep. Unset or 0 → no interval.
+ netPolResyncIntervalSec: getEnvInt('CONTEXT_MAPPER_NETPOL_RESYNC_SEC', 0),
+ // Mass-delete guard for the namespace-wide orphan sweep. Thresholds are
+ // strict `>`: exactly N candidates still delete; N+1 refuses the pass.
+ netPolOrphanDeleteCap: getEnvInt('CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP', 10),
+ netPolOrphanDeleteCapPercent: getEnvInt('CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT', 20),
+
// Every finite Kubernetes request that can hold Host convergence gets its
// own deadline. Watch streams are intentionally excluded.
hostK8sRequestTimeoutMs: parseHostK8sRequestTimeoutMs(getEnv('HCC_HOST_K8S_REQUEST_TIMEOUT_MS')),
diff --git a/host-context-controller/src/k8sClient.test.ts b/host-context-controller/src/k8sClient.test.ts
index a065444c1..a048805b1 100644
--- a/host-context-controller/src/k8sClient.test.ts
+++ b/host-context-controller/src/k8sClient.test.ts
@@ -248,6 +248,7 @@ const mockConfig = vi.hoisted(() => ({
runtimeNamespaces: ['mcp-server', 'mcp-host', 'sandbox-recipes', 'rpc-proxy'],
hostK8sRequestTimeoutMs: 30_000,
hostResyncIntervalSec: 0,
+ netPolResyncIntervalSec: 0,
externalEgressResyncIntervalSec: 0,
externalEgressRefreshFloorSec: 5,
controlApiBaseUrl: 'http://control-api.test:8090',
@@ -607,6 +608,7 @@ describe('McpServerWatcher startup', () => {
// per-test discipline.
mocks.hasCertifiedSafetyInventory.mockReset().mockReturnValue(true)
mockConfig.externalEgressResyncIntervalSec = 0 // keep periodic resync off by default
+ mockConfig.netPolResyncIntervalSec = 0
mocks.ensureDefaultPolicies.mockReset().mockResolvedValue(undefined)
mocks.netPolFullReconcile.mockReset().mockImplementation(async (...args: unknown[]) => {
const options = args[2] as { onAuthoritativeRevocationComplete?: () => void } | undefined
@@ -10867,3 +10869,130 @@ describe('McpServerWatcher watch-recovery retry backoff (exponential, jittered,
}
)
})
+
+describe('McpServerWatcher NetworkPolicy periodic resync (#478)', () => {
+ async function startWatcherForNetPolResync(): Promise<McpServerWatcher> {
+ mocks.netPolFullReconcile.mockClear()
+ mocks.watch.mockReset().mockResolvedValue({ abort: vi.fn() })
+ mocks.listNamespacedCustomObject.mockImplementation(async ({ plural }: { plural: string }) => {
+ if (plural === 'communicationchannels') {
+ return { metadata: { resourceVersion: 'netpol-resync-cc-rv' }, items: [] }
+ }
+ return { items: [] }
+ })
+ const watcher = new McpServerWatcher()
+ stubAuthoritativeInventoryWatch(watcher, 'McpServer')
+ stubAuthoritativeInventoryWatch(watcher, 'Context')
+ vi.spyOn(watcher as any, 'startSharedFileSystemWatch').mockResolvedValue(undefined)
+ vi.spyOn(watcher as any, 'startGlobalFileSystemWatch').mockResolvedValue(undefined)
+ vi.spyOn(watcher as any, 'startCommunicationChannelWatch').mockResolvedValue(undefined)
+ vi.spyOn(watcher as any, 'startLlmHookWatch').mockResolvedValue(undefined)
+ await watcher.start()
+ await vi.advanceTimersByTimeAsync(0)
+ return watcher
+ }
+
+ afterEach(() => {
+ vi.useRealTimers()
+ mockConfig.netPolResyncIntervalSec = 0
+ mocks.netPolFullReconcile.mockReset().mockImplementation(async (...args: unknown[]) => {
+ const options = args[2] as { onAuthoritativeRevocationComplete?: () => void } | undefined
+ options?.onAuthoritativeRevocationComplete?.()
+ })
+ })
+
+ it('M1: timer tick enters coordinated fullReconcile with the single-writer options bag', async () => {
+ vi.useFakeTimers()
+ mockConfig.netPolResyncIntervalSec = 300
+ const watcher = await startWatcherForNetPolResync()
+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1)
+
+ await vi.advanceTimersByTimeAsync(300_000)
+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(2)
+ expect(mocks.netPolFullReconcile).toHaveBeenLastCalledWith(
+ expect.any(Array),
+ expect.any(Array),
+ expect.objectContaining({
+ onAuthoritativeRevocationComplete: expect.any(Function),
+ runContextEffect: expect.any(Function),
+ contextInventoryAuthoritative: expect.any(Function),
+ })
+ )
+ await watcher.stop()
+ })
+ it('M2: a tick while a pass is in flight keeps one concurrent fullReconcile and trails after settle', async () => {
+ vi.useFakeTimers()
+ mockConfig.netPolResyncIntervalSec = 300
+ const activePass = deferred()
+ mocks.netPolFullReconcile
+ .mockReset()
+ .mockImplementationOnce(async (...args: unknown[]) => {
+ const options = args[2] as { onAuthoritativeRevocationComplete?: () => void } | undefined
+ options?.onAuthoritativeRevocationComplete?.()
+ await activePass.promise
+ })
+ .mockImplementation(async (...args: unknown[]) => {
+ const options = args[2] as { onAuthoritativeRevocationComplete?: () => void } | undefined
+ options?.onAuthoritativeRevocationComplete?.()
+ })
+ const watcher = await startWatcherForNetPolResync()
+ await vi.waitFor(() => expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1))
+
+ await vi.advanceTimersByTimeAsync(300_000)
+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1)
+
+ activePass.resolve()
+ await vi.waitFor(() => expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(2))
+ await watcher.stop()
+ })
+
+ it('M5: unset/0 interval warns, registers no timer, and still runs startup once', async () => {
+ vi.useFakeTimers()
+ mockConfig.netPolResyncIntervalSec = 0
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+ const watcher = await startWatcherForNetPolResync()
+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1)
+ expect(
+ warnSpy.mock.calls.some(call =>
+ String(call[0]).includes('NetworkPolicy periodic resync disabled')
+ )
+ ).toBe(true)
+ expect((watcher as any).netPolResyncTimer).toBeNull()
+
+ await vi.advanceTimersByTimeAsync(300_000)
+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1)
+ warnSpy.mockRestore()
+ await watcher.stop()
+ })
+
+ it('M6: stop() clears the resync interval so later ticks do not reconcile', async () => {
+ vi.useFakeTimers()
+ mockConfig.netPolResyncIntervalSec = 300
+ const watcher = await startWatcherForNetPolResync()
+ const callsAfterStart = mocks.netPolFullReconcile.mock.calls.length
+ expect((watcher as any).netPolResyncTimer).not.toBeNull()
+
+ await watcher.stop()
+ expect((watcher as any).netPolResyncTimer).toBeNull()
+ await vi.advanceTimersByTimeAsync(300_000)
+ expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(callsAfterStart)
+ })
+
+ it('S1: a timer tick while caches are unsynced defers and does not call fullReconcile', async () => {
+ vi.useFakeTimers()
+ mockConfig.netPolResyncIntervalSec = 300
+ const watcher = await startWatcherForNetPolResync()
+ await vi.waitFor(() => expect(mocks.netPolFullReconcile).toHaveBeenCalledTimes(1))
+ mocks.netPolFullReconcile.mockClear()
+ ;(watcher as any).contextCacheSynced = false
+ ;(watcher as any).mcpServerCacheSynced = false
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+
+ await vi.advanceTimersByTimeAsync(300_000)
+ expect(mocks.netPolFullReconcile).not.toHaveBeenCalled()
+ expect(warnSpy.mock.calls.some(call => String(call[0]).includes('caches unsynced'))).toBe(true)
+ expect((watcher as any).initialConvergenceRetryTimers.has('NetworkPolicy')).toBe(true)
+ warnSpy.mockRestore()
+ await watcher.stop()
+ })
+})
diff --git a/host-context-controller/src/k8sClient.ts b/host-context-controller/src/k8sClient.ts
index 54baa54af..751e40b33 100644
--- a/host-context-controller/src/k8sClient.ts
+++ b/host-context-controller/src/k8sClient.ts
@@ -889,6 +889,10 @@ export class McpServerWatcher implements McpServerProvider {
// GlobalFileSystem stuck at Initializing converges to Ready (and seeds its
// root directories) once the writer is up. Disabled when interval <= 0 (tests).
private gfsResyncTimer: ReturnType<typeof setInterval> | null = null
+ // Periodic NetworkPolicy resync (#478): re-enters the coordinated
+ // single-flight pass. Disabled by default (interval <= 0); startup still
+ // runs one convergence. Never call netPolReconciler.fullReconcile from here.
+ private netPolResyncTimer: ReturnType<typeof setInterval> | null = null
constructor() {
if (!kc) {
@@ -2845,6 +2849,18 @@ export class McpServerWatcher implements McpServerProvider {
)
}
+ const netPolResyncSec = config.netPolResyncIntervalSec
+ if (netPolResyncSec > 0) {
+ this.netPolResyncTimer = setInterval(() => {
+ void this.runInitialNetworkPolicyConvergence()
+ }, netPolResyncSec * 1000)
+ console.log(`[K8s] NetworkPolicy periodic resync enabled (every ${netPolResyncSec}s)`)
+ } else {
+ console.warn(
+ '[K8s] NetworkPolicy periodic resync disabled; dropped watch events will not self-heal stale NetworkPolicy allows through controller-driven convergence until another Context or McpServer event triggers reconciliation.'
+ )
+ }
+
const externalEgressResyncSec = config.externalEgressResyncIntervalSec
// #205 delegates external-egress periodic resync to the convergence
// coordinator. Its runResyncCore drives reconcileExternalEgress, so the
@@ -4516,6 +4532,10 @@ export class McpServerWatcher implements McpServerProvider {
clearInterval(this.gfsResyncTimer)
this.gfsResyncTimer = null
}
+ if (this.netPolResyncTimer) {
+ clearInterval(this.netPolResyncTimer)
+ this.netPolResyncTimer = null
+ }
for (const timer of this.initialConvergenceRetryTimers.values()) {
clearTimeout(timer)
}
diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.ts
index eb61fee25..10c51b19b 100644
--- a/host-context-controller/src/metrics.ts
+++ b/host-context-controller/src/metrics.ts
@@ -127,6 +127,18 @@ export const networkPolicySafetyPassPoliciesTotal = counter({
labelNames: ['operation'] as const,
})
+export const netPolOrphansDeletedTotal = counter({
+ name: 'clerum_hcc_netpol_orphans_deleted_total',
+ help: 'Orphan NetworkPolicies deleted by an HCC fullReconcile sweep.',
+ labelNames: ['lane'] as const,
+})
+
+export const netPolOrphanSweepCappedTotal = counter({
+ name: 'clerum_hcc_netpol_orphan_sweep_capped_total',
+ help: 'NetworkPolicy orphan sweeps that refused deletes because the candidate count exceeded the absolute or percent cap.',
+ labelNames: ['reason'] as const,
+})
+
export const contextReconciliationsTotal = counter({
name: 'clerum_hcc_context_reconciliations_total',
help: 'Total Context CRD reconciliations',
diff --git a/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts b/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts
new file mode 100644
index 000000000..140792ce0
--- /dev/null
+++ b/host-context-controller/src/networkPolicyReconciler.orphanSweep.test.ts
@@ -0,0 +1,380 @@
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+import * as k8s from '@kubernetes/client-node'
+import { config } from './config'
+import { netPolOrphanSweepCappedTotal, netPolOrphansDeletedTotal } from './metrics'
+import {
+ NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE,
+ NetworkPolicyReconciler,
+} from './networkPolicyReconciler'
+
+vi.mock('./config', () => ({
+ config: {
+ namespace: 'mcp-server',
+ controlPlaneNamespace: 'control-plane',
+ hostNamespace: 'mcp-host',
+ rpcProxyNamespace: 'rpc-proxy',
+ port: 8081,
+ runtimeNamespaces: ['mcp-server', 'mcp-host', 'sandbox-recipes', 'rpc-proxy'],
+ k8sApiCidrs: [],
+ externalEgressDnsResolveTimeoutMs: 5_000,
+ externalEgressOverlapSec: 300,
+ externalEgressMaxEntries: 128,
+ netPolOrphanDeleteCap: 10,
+ netPolOrphanDeleteCapPercent: 20,
+ },
+}))
+
+vi.mock('node:dns/promises', () => ({
+ resolve4: vi.fn().mockResolvedValue([
+ { address: '1.2.3.4', ttl: 300 },
+ { address: '5.6.7.8', ttl: 300 },
+ ]),
+}))
+
+vi.mock('./metrics', () => ({
+ networkPolicySafetyPassDurationSeconds: { observe: vi.fn() },
+ networkPolicySafetyPassPoliciesTotal: { inc: vi.fn() },
+ netPolOrphansDeletedTotal: { inc: vi.fn() },
+ netPolOrphanSweepCappedTotal: { inc: vi.fn() },
+}))
+function makeMockNetworkingApi() {
+ return {
+ createNamespacedNetworkPolicy: vi.fn().mockResolvedValue({}),
+ readNamespacedNetworkPolicy: vi.fn().mockResolvedValue({ metadata: { resourceVersion: '1' } }),
+ replaceNamespacedNetworkPolicy: vi.fn().mockResolvedValue({}),
+ listNamespacedNetworkPolicy: vi.fn().mockResolvedValue({ items: [] }),
+ deleteNamespacedNetworkPolicy: vi.fn().mockResolvedValue({}),
+ }
+}
+
+function makeMockCustomApi() {
+ return {
+ getNamespacedCustomObject: vi
+ .fn()
+ .mockRejectedValue(Object.assign(new Error('not found'), { code: 404 })),
+ listNamespacedCustomObject: vi.fn().mockResolvedValue({ items: [] }),
+ getNamespacedCustomObjectStatus: vi.fn().mockResolvedValue({
+ metadata: { generation: 7, resourceVersion: 'status-rv-default' },
+ status: {},
+ }),
+ patchNamespacedCustomObjectStatus: vi.fn().mockResolvedValue({}),
+ }
+}
+
+function makeReconciler(
+ mockApi: ReturnType<typeof makeMockNetworkingApi>,
+ mockCustomApi: ReturnType<typeof makeMockCustomApi> = makeMockCustomApi()
+): NetworkPolicyReconciler {
+ const kc = new k8s.KubeConfig()
+ kc.loadFromOptions({
+ clusters: [{ name: 'test', server: 'https://test' }],
+ users: [{ name: 'test' }],
+ contexts: [{ name: 'test', cluster: 'test', user: 'test' }],
+ currentContext: 'test',
+ })
+ const reconciler = new NetworkPolicyReconciler(kc, new Map())
+ const listedPolicies = new Map<string, k8s.V1NetworkPolicy>()
+ const networkingApi = {
+ ...mockApi,
+ listNamespacedNetworkPolicy: async (args: { namespace: string }) => {
+ const response = await mockApi.listNamespacedNetworkPolicy(args)
+ const items = (response.items ?? []).map((policy: k8s.V1NetworkPolicy, index: number) => {
+ const name = policy.metadata?.name ?? `unnamed-${index}`
+ const materialized = {
+ ...policy,
+ metadata: {
+ ...policy.metadata,
+ name,
+ namespace: policy.metadata?.namespace ?? args.namespace,
+ uid: policy.metadata?.uid ?? `${args.namespace}:${name}:uid`,
+ resourceVersion: policy.metadata?.resourceVersion ?? '1',
+ },
+ }
+ listedPolicies.set(`${args.namespace}/${name}`, materialized)
+ return materialized
+ })
+ return { ...response, items }
+ },
+ readNamespacedNetworkPolicy: async (args: { name: string; namespace: string }) => {
+ const listed = listedPolicies.get(`${args.namespace}/${args.name}`)
+ return (
+ listed ?? {
+ metadata: { name: args.name, namespace: args.namespace, uid: 'u', resourceVersion: '1' },
+ }
+ )
+ },
+ }
+ ;(reconciler as unknown as { networkingApi: unknown }).networkingApi = networkingApi
+ ;(reconciler as unknown as { customApi: unknown }).customApi = mockCustomApi
+ return reconciler
+}
+function orphanContextAllow(name: string, contextId: string): k8s.V1NetworkPolicy {
+ return {
+ metadata: {
+ name,
+ labels: {
+ 'clerum.io/managed-by': 'host-context-controller',
+ 'clerum.io/policy-type': 'context-allow',
+ 'clerum.io/context': contextId,
+ 'clerum.io/mcpserver': 'old-server',
+ },
+ },
+ // #471 fail-open: a missing spec is treated as unknown/write. Sweep
+ // fixtures must include spec so the candidate is a real listed policy.
+ spec: {
+ podSelector: {},
+ policyTypes: ['Ingress'],
+ ingress: [{ _from: [{ podSelector: { matchLabels: { app: 'mcp-host' } } }] }],
+ },
+ }
+}
+
+function orphanHostEgress(name: string, contextId: string): k8s.V1NetworkPolicy {
+ return {
+ metadata: {
+ name,
+ labels: {
+ 'clerum.io/managed-by': 'host-context-controller',
+ 'clerum.io/policy-type': 'context-allow',
+ 'clerum.io/context': contextId,
+ 'clerum.io/mcpserver': 'old-server',
+ },
+ },
+ spec: {
+ podSelector: {},
+ policyTypes: ['Egress'],
+ egress: [{ to: [{ podSelector: { matchLabels: { app: 'mcp-server' } } }] }],
+ },
+ }
+}
+
+function orphanRpcEgress(name: string, contextId: string): k8s.V1NetworkPolicy {
+ return {
+ metadata: {
+ name,
+ labels: {
+ 'clerum.io/managed-by': 'host-context-controller',
+ 'clerum.io/policy-type': 'rpc-proxy-egress',
+ 'clerum.io/context': contextId,
+ 'clerum.io/mcpserver': 'old-server',
+ },
+ },
+ spec: {
+ podSelector: {},
+ policyTypes: ['Egress'],
+ egress: [{ to: [{ podSelector: { matchLabels: { app: 'mcp-server' } } }] }],
+ },
+ }
+}
+
+function orphanExternalEgress(name: string, serverName: string): k8s.V1NetworkPolicy {
+ return {
+ metadata: {
+ name,
+ labels: {
+ 'clerum.io/managed-by': 'host-context-controller',
+ 'clerum.io/policy-type': 'external-egress',
+ 'clerum.io/mcpserver': serverName,
+ },
+ },
+ spec: {
+ podSelector: {},
+ policyTypes: ['Egress'],
+ egress: [{ to: [{ ipBlock: { cidr: '1.2.3.4/32' } }] }],
+ },
+ }
+}
+function stubOrphanLists(
+ mockApi: ReturnType<typeof makeMockNetworkingApi>,
+ items: {
+ context?: k8s.V1NetworkPolicy[]
+ hostEgress?: k8s.V1NetworkPolicy[]
+ rpc?: k8s.V1NetworkPolicy[]
+ external?: k8s.V1NetworkPolicy[]
+ }
+): void {
+ mockApi.listNamespacedNetworkPolicy.mockImplementation(
+ async ({ namespace, labelSelector }: { namespace?: string; labelSelector?: string }) => {
+ if (namespace === 'mcp-server' && labelSelector?.includes('context-allow')) {
+ return { items: items.context ?? [] }
+ }
+ if (namespace === 'mcp-host' && labelSelector?.includes('context-allow')) {
+ return { items: items.hostEgress ?? [] }
+ }
+ if (namespace === 'rpc-proxy' && labelSelector?.includes('rpc-proxy-egress')) {
+ return { items: items.rpc ?? [] }
+ }
+ if (namespace === 'mcp-server' && labelSelector?.includes('external-egress')) {
+ return { items: items.external ?? [] }
+ }
+ return { items: [] }
+ }
+ )
+}
+
+describe('NetworkPolicy orphan sweep cap (#478)', () => {
+ let mockApi: ReturnType<typeof makeMockNetworkingApi>
+ let reconciler: NetworkPolicyReconciler
+
+ beforeEach(() => {
+ config.netPolOrphanDeleteCap = 10
+ config.netPolOrphanDeleteCapPercent = 20
+ mockApi = makeMockNetworkingApi()
+ reconciler = makeReconciler(mockApi)
+ vi.clearAllMocks()
+ })
+ it('M3: deletes one under-cap context-allow orphan and increments orphans_deleted_total{lane=context-ingress}', async () => {
+ stubOrphanLists(mockApi, {
+ context: [orphanContextAllow('ctx-orphan-context-old-server', 'orphan-context')],
+ })
+ const complete = vi.fn()
+
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
+
+ expect(mockApi.deleteNamespacedNetworkPolicy).toHaveBeenCalledWith(
+ expect.objectContaining({
+ name: 'ctx-orphan-context-old-server',
+ namespace: 'mcp-server',
+ })
+ )
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledWith({ lane: 'context-ingress' })
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledTimes(1)
+ expect(complete).toHaveBeenCalledOnce()
+ expect(netPolOrphanSweepCappedTotal.inc).not.toHaveBeenCalled()
+ })
+
+ it('M4: N+1 orphans refuse every delete, increment absolute cap, skip certification', async () => {
+ const orphans = Array.from({ length: 11 }, (_, i) =>
+ orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`)
+ )
+ stubOrphanLists(mockApi, { context: orphans })
+ const complete = vi.fn()
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+
+ await expect(
+ reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
+ ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+
+ expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'absolute' })
+ expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledTimes(1)
+ expect(netPolOrphansDeletedTotal.inc).not.toHaveBeenCalled()
+ expect(complete).not.toHaveBeenCalled()
+ warnSpy.mockRestore()
+ })
+
+ it('M4 sibling: orphans over the percent share refuse with reason=percent', async () => {
+ const orphans = Array.from({ length: 5 }, (_, i) =>
+ orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`)
+ )
+ stubOrphanLists(mockApi, { context: orphans })
+ const complete = vi.fn()
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+
+ await expect(
+ reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
+ ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+
+ expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'percent' })
+ expect(complete).not.toHaveBeenCalled()
+ warnSpy.mockRestore()
+ })
+ it('S2: absolute cap is strict >; exactly N deletes and N+1 refuses', async () => {
+ config.netPolOrphanDeleteCapPercent = 100
+ const ten = Array.from({ length: 10 }, (_, i) =>
+ orphanContextAllow(`ctx-orphan-${i}-old-server`, `orphan-context-${i}`)
+ )
+ stubOrphanLists(mockApi, { context: ten })
+ const completeTen = vi.fn()
+
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: completeTen,
+ })
+ expect(mockApi.deleteNamespacedNetworkPolicy).toHaveBeenCalledTimes(10)
+ expect(completeTen).toHaveBeenCalledOnce()
+
+ mockApi.deleteNamespacedNetworkPolicy.mockClear()
+ vi.mocked(netPolOrphanSweepCappedTotal.inc).mockClear()
+ const eleven = [...ten, orphanContextAllow('ctx-orphan-10-old-server', 'orphan-context-10')]
+ stubOrphanLists(mockApi, { context: eleven })
+ const completeEleven = vi.fn()
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+
+ await expect(
+ reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: completeEleven,
+ })
+ ).rejects.toThrow(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+ expect(mockApi.deleteNamespacedNetworkPolicy).not.toHaveBeenCalled()
+ expect(netPolOrphanSweepCappedTotal.inc).toHaveBeenCalledWith({ reason: 'absolute' })
+ expect(completeEleven).not.toHaveBeenCalled()
+ warnSpy.mockRestore()
+ })
+
+ it('S3: deleted orphans increment the matching lane label', async () => {
+ stubOrphanLists(mockApi, {
+ context: [orphanContextAllow('ctx-orphan-context-old-server', 'orphan-context')],
+ hostEgress: [orphanHostEgress('ctx-orphan-context-old-server-egress', 'orphan-context')],
+ rpc: [orphanRpcEgress('rpc-egress-orphan-context-old-server', 'orphan-context')],
+ external: [
+ orphanExternalEgress('ext-egress-orphan-server-old-example-com-443', 'orphan-server'),
+ ],
+ })
+
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: vi.fn(),
+ })
+
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledWith({ lane: 'context-ingress' })
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledWith({ lane: 'context-host-egress' })
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledWith({ lane: 'context-rpc-egress' })
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledWith({ lane: 'external-egress' })
+ expect(netPolOrphansDeletedTotal.inc).toHaveBeenCalledTimes(4)
+ })
+
+ it('S4: percent cap is inert on a tiny listed fleet', async () => {
+ stubOrphanLists(mockApi, {
+ context: [
+ orphanContextAllow('ctx-orphan-a-old-server', 'orphan-a'),
+ orphanContextAllow('ctx-orphan-b-old-server', 'orphan-b'),
+ ],
+ })
+ const complete = vi.fn()
+
+ await reconciler.fullReconcile([], [], {
+ ensureDefaults: false,
+ contextInventoryAuthoritative: () => true,
+ serverInventoryAuthoritative: () => true,
+ onAuthoritativeRevocationComplete: complete,
+ })
+
+ expect(mockApi.deleteNamespacedNetworkPolicy).toHaveBeenCalledTimes(2)
+ expect(netPolOrphanSweepCappedTotal.inc).not.toHaveBeenCalled()
+ expect(complete).toHaveBeenCalledOnce()
+ })
+})
diff --git a/host-context-controller/src/networkPolicyReconciler.test.ts b/host-context-controller/src/networkPolicyReconciler.test.ts
index bb393858f..5aa8fc4ae 100644
--- a/host-context-controller/src/networkPolicyReconciler.test.ts
+++ b/host-context-controller/src/networkPolicyReconciler.test.ts
@@ -5,6 +5,7 @@ import * as dns from 'node:dns/promises'
import { readFileSync } from 'node:fs'
import { join } from 'node:path'
import { asApiserverNetworkPolicy } from './__tests__/asApiserverNetworkPolicy'
+import { config } from './config'
import { confirmAuthoritativeMcpServerAbsence } from './mcpServerSafety'
import {
networkPolicySafetyPassDurationSeconds,
@@ -56,6 +57,8 @@ vi.mock('./config', () => ({
// #299 sliding-window knobs read by reconcileExternalEgress.
externalEgressOverlapSec: 300,
externalEgressMaxEntries: 128,
+ netPolOrphanDeleteCap: 10,
+ netPolOrphanDeleteCapPercent: 20,
},
}))
@@ -73,6 +76,8 @@ vi.mock('node:dns/promises', () => ({
vi.mock('./metrics', () => ({
networkPolicySafetyPassDurationSeconds: { observe: vi.fn() },
networkPolicySafetyPassPoliciesTotal: { inc: vi.fn() },
+ netPolOrphansDeletedTotal: { inc: vi.fn() },
+ netPolOrphanSweepCappedTotal: { inc: vi.fn() },
}))
function makeMockNetworkingApi() {
@@ -177,6 +182,8 @@ describe('NetworkPolicyReconciler', () => {
beforeEach(() => {
vi.useRealTimers()
+ config.netPolOrphanDeleteCap = 10
+ config.netPolOrphanDeleteCapPercent = 20
mockApi = makeMockNetworkingApi()
mockCustomApi = makeMockCustomApi()
reconciler = makeReconciler(mockApi, undefined, mockCustomApi)
diff --git a/host-context-controller/src/networkPolicyReconciler.ts b/host-context-controller/src/networkPolicyReconciler.ts
index b23a76c91..fd7a605e8 100644
--- a/host-context-controller/src/networkPolicyReconciler.ts
+++ b/host-context-controller/src/networkPolicyReconciler.ts
@@ -32,6 +32,8 @@ import {
sameMcpServerDesiredRevision,
} from './mcpServerSafety'
import {
+ netPolOrphanSweepCappedTotal,
+ netPolOrphansDeletedTotal,
networkPolicySafetyPassDurationSeconds,
networkPolicySafetyPassPoliciesTotal,
} from './metrics'
@@ -76,6 +78,29 @@ type NetworkPolicyMutationOptions = {
export const DESIRED_NETWORKPOLICY_INVENTORY_CHANGED_MESSAGE =
'Desired NetworkPolicy inventory changed during authoritative revocation'
+export const NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE =
+ 'NetworkPolicy orphan sweep refused: candidate count exceeds the delete cap'
+
+export type NetPolOrphanSweepCapReason = 'absolute' | 'percent'
+
+/**
+ * Mass-delete guard for the namespace-wide orphan sweep. Thresholds are
+ * strict `>`: exactly `absoluteCap` candidates still delete. The percent
+ * rule is inert on a tiny listed fleet (`percent*listed < 1`) so a single
+ * real orphan cannot look like a cache-wipe.
+ */
+export function evaluateNetPolOrphanSweepCap(
+ orphanCount: number,
+ listedManaged: number,
+ absoluteCap = config.netPolOrphanDeleteCap,
+ percentCap = config.netPolOrphanDeleteCapPercent
+): NetPolOrphanSweepCapReason | null {
+ if (orphanCount > absoluteCap) return 'absolute'
+ const percentThreshold = (listedManaged * percentCap) / 100
+ if (percentThreshold >= 1 && orphanCount > percentThreshold) return 'percent'
+ return null
+}
+
export function sameContextDesiredRevision(expected: ContextCRD, current: ContextCRD): boolean {
return (
expected.name === current.name &&
@@ -1313,10 +1338,17 @@ export class NetworkPolicyReconciler {
}
const desiredContextIds = new Set(contexts.map(c => c.spec.contextId))
+ const desiredServerNames = new Set(servers.map(s => s.name))
+ const recordOrphanDelete = (lane: SafetyInventoryLane, deleted: boolean): void => {
+ if (!deleted) return
+ recordSafetyPassRevocation()
+ netPolOrphansDeletedTotal.inc({ lane })
+ }
const cleanupOrphanedContextPolicies = async (
policies: k8s.V1NetworkPolicy[],
deletePolicy: (policy: k8s.V1NetworkPolicy) => Promise<boolean>,
- describePolicy: (name: string, contextId: string) => string
+ describePolicy: (name: string, contextId: string) => string,
+ lane: SafetyInventoryLane
): Promise<void> => {
if (!contextCleanupAuthoritative()) return
for (const policy of policies) {
@@ -1330,7 +1362,7 @@ export class NetworkPolicyReconciler {
console.warn(
`[NetPol] Deleting malformed HCC-managed policy "${name}" without ${CONTEXT_LABEL}`
)
- if (await deletePolicy(policy)) recordSafetyPassRevocation()
+ recordOrphanDelete(lane, await deletePolicy(policy))
continue
}
if (!desiredContextIds.has(contextId)) {
@@ -1346,7 +1378,7 @@ export class NetworkPolicyReconciler {
return
}
console.log(describePolicy(name, contextId))
- if (await deletePolicy(policy)) recordSafetyPassRevocation()
+ recordOrphanDelete(lane, await deletePolicy(policy))
})
}
}
@@ -1413,6 +1445,38 @@ export class NetworkPolicyReconciler {
allRpcProxyEgressPolicies.length +
allExternalPolicies.length
+ // Count namespace-wide orphan candidates BEFORE any delete. A blown cache
+ // that lists a mass-delete must refuse the pass rather than sweep.
+ const isContextLaneOrphan = (policy: k8s.V1NetworkPolicy): boolean => {
+ const contextId = policy.metadata?.labels?.[CONTEXT_LABEL]
+ return !contextId || !desiredContextIds.has(contextId)
+ }
+ const isExternalLaneOrphan = (policy: k8s.V1NetworkPolicy): boolean => {
+ const serverName = policy.metadata?.labels?.[MCPSERVER_LABEL]
+ return !serverName || !desiredServerNames.has(serverName)
+ }
+ const orphanCandidates = [
+ ...allContextPolicies.filter(isContextLaneOrphan),
+ ...allContextEgressPolicies.filter(isContextLaneOrphan),
+ ...allRpcProxyEgressPolicies.filter(isContextLaneOrphan),
+ ...(options.serverInventoryComplete !== false && serverCleanupAuthoritative()
+ ? allExternalPolicies.filter(isExternalLaneOrphan)
+ : []),
+ ]
+ const listedManagedFleet =
+ allContextPolicies.length +
+ allContextEgressPolicies.length +
+ allRpcProxyEgressPolicies.length +
+ allExternalPolicies.length
+ const capReason = evaluateNetPolOrphanSweepCap(orphanCandidates.length, listedManagedFleet)
+ if (capReason) {
+ netPolOrphanSweepCappedTotal.inc({ reason: capReason })
+ console.warn(
+ `[NetPol] Orphan sweep capped (${capReason}): ${orphanCandidates.length} candidates of ${listedManagedFleet} listed managed policies; refusing deletes and withholding certification`
+ )
+ throw new Error(NETWORKPOLICY_ORPHAN_SWEEP_CAPPED_MESSAGE)
+ }
+
// Delete orphaned Context policies across every L2 lane. Each lane uses
// the canonical contextId effect key and rechecks authority plus live CRD
// presence immediately before its delete.
@@ -1421,13 +1485,15 @@ export class NetworkPolicyReconciler {
policy =>
this.deleteSafetyPolicySnapshot(config.namespace, policy, contextCleanupAuthoritative),
(name, contextId) =>
- `[NetPol] Deleting orphaned policy "${name}" (context "${contextId}" no longer exists)`
+ `[NetPol] Deleting orphaned policy "${name}" (context "${contextId}" no longer exists)`,
+ 'context-ingress'
)
await cleanupOrphanedContextPolicies(
allContextEgressPolicies,
policy =>
this.deleteSafetyPolicySnapshot(config.hostNamespace, policy, contextCleanupAuthoritative),
- name => `[NetPol] Deleting orphaned L2 egress policy "${name}"`
+ name => `[NetPol] Deleting orphaned L2 egress policy "${name}"`,
+ 'context-host-egress'
)
await cleanupOrphanedContextPolicies(
allRpcProxyEgressPolicies,
@@ -1437,7 +1503,8 @@ export class NetworkPolicyReconciler {
policy,
contextCleanupAuthoritative
),
- name => `[NetPol] Deleting orphaned rpc-proxy egress policy "${name}"`
+ name => `[NetPol] Deleting orphaned rpc-proxy egress policy "${name}"`,
+ 'context-rpc-egress'
)
let liveDesiredRevisionChanged = false
@@ -1477,7 +1544,6 @@ export class NetworkPolicyReconciler {
// Clean up external egress policies for servers that no longer exist.
if (options.serverInventoryComplete !== false && serverCleanupAuthoritative()) {
- const desiredServerNames = new Set(servers.map(s => s.name))
for (const policy of allExternalPolicies) {
const serverName = policy.metadata?.labels?.[MCPSERVER_LABEL]
if (!serverName) {
@@ -1491,15 +1557,14 @@ export class NetworkPolicyReconciler {
console.warn(
`[NetPol] Deleting malformed HCC-managed policy "${name}" without ${MCPSERVER_LABEL}`
)
- if (
+ recordOrphanDelete(
+ 'external-egress',
await this.deleteSafetyPolicySnapshot(
config.namespace,
policy,
serverCleanupAuthoritative
)
- ) {
- recordSafetyPassRevocation()
- }
+ )
continue
}
if (!desiredServerNames.has(serverName)) {
@@ -1522,15 +1587,14 @@ export class NetworkPolicyReconciler {
return
}
console.log(`[NetPol] Deleting orphaned external egress policy "${name}"`)
- if (
+ recordOrphanDelete(
+ 'external-egress',
await this.deleteSafetyPolicySnapshot(
config.namespace,
policy,
serverCleanupAuthoritative
)
- ) {
- recordSafetyPassRevocation()
- }
+ )
})
}
}
diff --git a/scripts/ops/hcc-netpol-orphan-census.sh b/scripts/ops/hcc-netpol-orphan-census.sh
new file mode 100755
index 000000000..958b57734
--- /dev/null
+++ b/scripts/ops/hcc-netpol-orphan-census.sh
@@ -0,0 +1,173 @@
+#!/bin/bash
+# Job 4.1-D: read-only census of HCC-managed NetworkPolicy orphans.
+#
+# get/list only — never create, patch, replace, or delete.
+# Classifies by clerum.io/policy-type in {context-allow, rpc-proxy-egress,
+# external-egress}. Do not treat managed-by alone as membership.
+#
+# CONTEXT_MAPPER_* knobs are read from the live host-context-controller
+# Deployment. Code defaults are never printed as cluster facts: an absent
+# env is UNSET.
+#
+# Double-samples 90s apart. Adjudicates only when the desired Context +
+# McpServer identity set is identical across samples; otherwise
+# INCONCLUSIVE_RERUN.
+#
+# Usage:
+# CONTEXT=<kube-context> ./scripts/ops/hcc-netpol-orphan-census.sh
+
+set -euo pipefail
+umask 077
+
+: "${CONTEXT:?must set CONTEXT (kubectl context)}"
+
+HCC_NS="${HCC_NS:-control-plane}"
+HCC_DEPLOYMENT="${HCC_DEPLOYMENT:-host-context-controller}"
+SAMPLE_GAP_SEC="${SAMPLE_GAP_SEC:-90}"
+
+echo "[census] context=${CONTEXT} deployment=${HCC_NS}/${HCC_DEPLOYMENT}"
+
+kc() {
+ command kubectl --context "${CONTEXT}" "$@"
+}
+
+# Live Deployment env only. Missing keys are UNSET — never substitute
+# controller source defaults as cluster facts.
+deployment_env() {
+ local key="$1"
+ local value
+ value="$(kc -n "${HCC_NS}" get deploy "${HCC_DEPLOYMENT}" -o "jsonpath={.spec.template.spec.containers[0].env[?(@.name==\"${key}\")].value}" 2>/dev/null || true)"
+ if [[ -z "${value}" ]]; then
+ printf "UNSET"
+ else
+ printf "%s" "${value}"
+ fi
+}
+
+MAPPER_NS="$(deployment_env CONTEXT_MAPPER_NAMESPACE)"
+HOST_NS="$(deployment_env CONTEXT_MAPPER_HOST_NAMESPACE)"
+RPC_NS="$(deployment_env CONTEXT_MAPPER_RPC_PROXY_NAMESPACE)"
+RESYNC_SEC="$(deployment_env CONTEXT_MAPPER_NETPOL_RESYNC_SEC)"
+ORPHAN_CAP="$(deployment_env CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP)"
+ORPHAN_CAP_PCT="$(deployment_env CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT)"
+
+echo "[census] live Deployment env:"
+echo " CONTEXT_MAPPER_NAMESPACE=${MAPPER_NS}"
+echo " CONTEXT_MAPPER_HOST_NAMESPACE=${HOST_NS}"
+echo " CONTEXT_MAPPER_RPC_PROXY_NAMESPACE=${RPC_NS}"
+echo " CONTEXT_MAPPER_NETPOL_RESYNC_SEC=${RESYNC_SEC}"
+echo " CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP=${ORPHAN_CAP}"
+echo " CONTEXT_MAPPER_NETPOL_ORPHAN_DELETE_CAP_PERCENT=${ORPHAN_CAP_PCT}"
+
+if [[ "${MAPPER_NS}" == "UNSET" || "${HOST_NS}" == "UNSET" || "${RPC_NS}" == "UNSET" ]]; then
+ echo "[census] FATAL: Deployment did not declare mapper/host/rpc-proxy namespaces; refusing to invent defaults" >&2
+ exit 2
+fi
+
+# One read-only snapshot of desired identities + orphan candidates.
+# Filters on policy-type, not managed-by.
+sample_state() {
+ local contexts_json servers_json np_json
+ contexts_json="$(kc get contexts.clerum.io -n "${MAPPER_NS}" -o json)"
+ servers_json="$(kc get mcpservers.clerum.io -n "${MAPPER_NS}" -o json)"
+ np_json="$(
+ {
+ kc get networkpolicy -n "${MAPPER_NS}" -o json
+ kc get networkpolicy -n "${HOST_NS}" -o json
+ kc get networkpolicy -n "${RPC_NS}" -o json
+ } | jq -s "{items: [.[].items[]?]}"
+ )"
+
+ local desired_contexts desired_servers listed orphans orphan_count
+ desired_contexts="$(printf "%s" "${contexts_json}" | jq -r '[.items[]?.spec.contextId // empty] | unique | sort | join(",")')"
+ desired_servers="$(printf "%s" "${servers_json}" | jq -r '[.items[]?.metadata.name // empty] | unique | sort | join(",")')"
+ listed="$(printf "%s" "${np_json}" | jq -r '[.items[] | select((.metadata.labels["clerum.io/policy-type"] == "context-allow") or (.metadata.labels["clerum.io/policy-type"] == "rpc-proxy-egress") or (.metadata.labels["clerum.io/policy-type"] == "external-egress"))] | length')"
+
+ orphans="$(printf "%s" "${np_json}" | jq -r --arg ctxs "${desired_contexts}" --arg srvs "${desired_servers}" '
+ ($ctxs | split(",") | map(select(length>0))) as $desiredCtx
+ | ($srvs | split(",") | map(select(length>0))) as $desiredSrv
+ | .items[]
+ | .metadata.labels["clerum.io/policy-type"] as $t
+ | select($t == "context-allow" or $t == "rpc-proxy-egress" or $t == "external-egress")
+ | . as $p
+ | (
+ if $t == "external-egress" then
+ ($p.metadata.labels["clerum.io/mcpserver"] as $s | ($s == null) or (($desiredSrv | index($s)) == null))
+ else
+ ($p.metadata.labels["clerum.io/context"] as $c | ($c == null) or (($desiredCtx | index($c)) == null))
+ end
+ )
+ | select(.)
+ | [$p.metadata.namespace, $p.metadata.name, $t]
+ | @tsv
+ ')"
+
+ if [[ -z "${orphans}" ]]; then
+ orphan_count=0
+ else
+ orphan_count="$(printf "%s\n" "${orphans}" | grep -c . || true)"
+ fi
+
+ printf "DESIRED_CONTEXTS=%s\n" "${desired_contexts}"
+ printf "DESIRED_SERVERS=%s\n" "${desired_servers}"
+ printf "LISTED_MANAGED=%s\n" "${listed}"
+ printf "ORPHAN_COUNT=%s\n" "${orphan_count}"
+ printf "ORPHANS<<EOF\n"
+ printf "%s\n" "${orphans}"
+ printf "EOF\n"
+}
+
+parse_field() {
+ local blob="$1" key="$2"
+ printf "%s" "${blob}" | sed -n "s/^${key}=//p" | head -n1
+}
+
+echo "[census] sample 1"
+SAMPLE1="$(sample_state)"
+echo "${SAMPLE1}" | sed "/^ORPHANS<<EOF/,/^EOF/d"
+
+echo "[census] waiting ${SAMPLE_GAP_SEC}s for second sample"
+sleep "${SAMPLE_GAP_SEC}"
+
+echo "[census] sample 2"
+SAMPLE2="$(sample_state)"
+echo "${SAMPLE2}" | sed "/^ORPHANS<<EOF/,/^EOF/d"
+
+CTX1="$(parse_field "${SAMPLE1}" DESIRED_CONTEXTS)"
+SRV1="$(parse_field "${SAMPLE1}" DESIRED_SERVERS)"
+CTX2="$(parse_field "${SAMPLE2}" DESIRED_CONTEXTS)"
+SRV2="$(parse_field "${SAMPLE2}" DESIRED_SERVERS)"
+
+if [[ "${CTX1}" != "${CTX2}" || "${SRV1}" != "${SRV2}" ]]; then
+ echo "[census] VERDICT=INCONCLUSIVE_RERUN"
+ echo "[census] desired Context+McpServer set changed between samples; do not adjudicate"
+ echo " sample1 contexts=${CTX1}"
+ echo " sample2 contexts=${CTX2}"
+ echo " sample1 servers=${SRV1}"
+ echo " sample2 servers=${SRV2}"
+ exit 3
+fi
+
+ORPHAN_COUNT="$(parse_field "${SAMPLE2}" ORPHAN_COUNT)"
+LISTED="$(parse_field "${SAMPLE2}" LISTED_MANAGED)"
+echo "[census] desired set identical across samples"
+echo "[census] listed_managed=${LISTED} orphan_count=${ORPHAN_COUNT}"
+echo "[census] orphans (namespace name policy-type):"
+printf "%s" "${SAMPLE2}" | sed -n "/^ORPHANS<<EOF/,/^EOF/{ /^ORPHANS<<EOF/d; /^EOF/d; p; }"
+
+CAP_REASON="none"
+if [[ "${ORPHAN_CAP}" != "UNSET" && "${ORPHAN_COUNT}" -gt "${ORPHAN_CAP}" ]]; then
+ CAP_REASON="absolute"
+elif [[ "${ORPHAN_CAP_PCT}" != "UNSET" && "${LISTED}" -gt 0 ]]; then
+ # Integer compare of orphan*100 > listed*percent, inert when percent*listed < 100.
+ if [[ $((LISTED * ORPHAN_CAP_PCT)) -ge 100 && $((ORPHAN_COUNT * 100)) -gt $((LISTED * ORPHAN_CAP_PCT)) ]]; then
+ CAP_REASON="percent"
+ fi
+fi
+
+echo "[census] live_cap_would_trip=${CAP_REASON}"
+if [[ "${ORPHAN_COUNT}" -eq 0 ]]; then
+ echo "[census] VERDICT=CLEAN"
+else
+ echo "[census] VERDICT=ORPHANS_PRESENT"
+fi