Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(hcc): close A1 review findings on the probe vs data-path split (#477)
Alfredo Lopez46c350fdd721
diff --git a/docs/architecture/hcc-readiness-flows-diagram.svg b/docs/architecture/hcc-readiness-flows-diagram.svgindex db0985b89..280620ddb 100644--- a/docs/architecture/hcc-readiness-flows-diagram.svg+++ b/docs/architecture/hcc-readiness-flows-diagram.svg@@ -29,9 +29,9 @@<text x="640.0" y="178" text-anchor="middle" font-size="11" fill="#475569">· anchored to desired revision</text><rect x="779.5" y="104" width="214" height="92" rx="11" fill="#dbeafe" stroke="#1d4ed8" stroke-width="1.8" filter="url(#sh)"/><text x="886.5" y="127" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">/ready = 200</text>-<text x="886.5" y="146" text-anchor="middle" font-size="11" fill="#475569">only if inventory certified</text>-<text x="886.5" y="162" text-anchor="middle" font-size="11" fill="#475569">and no stale allow</text>-<text x="886.5" y="178" text-anchor="middle" font-size="11" fill="#475569">survives</text>+<text x="886.5" y="146" text-anchor="middle" font-size="11" fill="#475569">watch freshness only</text>+<text x="886.5" y="162" text-anchor="middle" font-size="11" fill="#475569">phase-2 stays on the</text>+<text x="886.5" y="178" text-anchor="middle" font-size="11" fill="#475569">per-request data path</text><rect x="1026.0" y="104" width="214" height="92" rx="11" fill="#eff6ff" stroke="#2563eb" stroke-width="1.8" filter="url(#sh)"/><text x="1133.0" y="127" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">Additive convergence</text><text x="1133.0" y="146" text-anchor="middle" font-size="11" fill="#475569">fleet converges AFTER</text>@@ -52,7 +52,7 @@<text x="40" y="275" font-size="13" font-weight="700" fill="#b91c1c">2 · Fail-closed fences (R3 · R4 · J3)</text><rect x="40.0" y="284" width="380.0" height="76" rx="11" fill="#fef2f2" stroke="#dc2626" stroke-width="1.8" filter="url(#sh)"/><text x="230.0" y="307" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">Readiness gate — default () => false</text>-<text x="230.0" y="326" text-anchor="middle" font-size="11" fill="#475569">/ready = ready && providerAuthoritativeFn()</text>+<text x="230.0" y="326" text-anchor="middle" font-size="11" fill="#475569">/ready = ready && probeAuthoritativeFn()</text><text x="230.0" y="342" text-anchor="middle" font-size="11" fill="#475569">throw → 503 · never a wrong 200</text><rect x="450.0" y="284" width="380.0" height="76" rx="11" fill="#fef2f2" stroke="#dc2626" stroke-width="1.8" filter="url(#sh)"/><text x="640.0" y="307" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">G1 — deny without inventory-authority</text>@@ -76,7 +76,7 @@<text x="693.0" y="456" text-anchor="middle" font-size="11" fill="#475569">same protocol/port/ownership/cidrs</text><text x="693.0" y="472" text-anchor="middle" font-size="11" fill="#475569">· HCC stays Ready (modulo cidr)</text><rect x="540" y="496" width="306" height="60" rx="11" fill="#fef2f2" stroke="#dc2626" stroke-width="1.8" filter="url(#sh)"/>-<text x="693.0" y="519" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">REVOKE before readiness</text>+<text x="693.0" y="519" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">REVOKE before data-path certify</text><text x="693.0" y="538" text-anchor="middle" font-size="11" fill="#475569">drift TCP→UDP · port · ownership · zero-cidr</text><text x="693.0" y="554" text-anchor="middle" font-size="11" fill="#475569">→ unprovable → cleanup fail-closed</text><path d="M 472,470 H 506.0 V 444 H 540" fill="none" stroke="#059669" stroke-width="2.6" marker-end="url(#aGreen)" stroke-linejoin="round"/>diff --git a/docs/architecture/hcc-readiness-flows.md b/docs/architecture/hcc-readiness-flows.mdindex 5d158b3e4..049cdbb59 100644--- a/docs/architecture/hcc-readiness-flows.md+++ b/docs/architecture/hcc-readiness-flows.md@@ -4,7 +4,7 @@> `host-context-controller` (HCC). Addresses the "no design/decision doc for the> 500+-line external-egress coordinator and the certification barrier" review gap.-+## 1. Readiness lifecycle — decoupled from initial fleet convergence@@ -12,8 +12,9 @@ Before this work, `McpServerWatcher.start()` awaited full initial fleetreconciliation, so a large MCP/Context/Host fleet could keep `provider.start()`and `/ready` unavailable until every individual reconcile finished.-Now readiness is gated on an **authoritative, fail-closed safety boundary**, not on-full convergence:+Readiness is now two lanes. Watch freshness (clauses 1–4) stays fail-closed on+both. Phase-2 certification stays fail-closed on the **data path** and is off+the kubelet probe (#477 A1):1. **HCC start** — four watch lanes (Context ingress, mcp-host egress, rpc-proxyegress, external egress) start. A transient LIST/WATCH failure on any lane enters@@ -21,16 +22,20 @@ full convergence:2. **4-LIST inventory** — bounded inventory across the four lanes.3. **Certify safety inventory** — the authoritative pass revokes any stale /unprovable allow **before** recording a certificate (fail-closed).-4. **`/ready = 200`** — served **only** when the inventory is certified and no stale- allow survives.-5. **Additive convergence** — the remaining fleet converges **after** ready,- asynchronously, without blocking it.+4. **`/ready = 200`** — served when the process is past warm-up and the three+ watches are synced. Phase-2 certification does **not** hold the probe.+5. **Per-request data path** — inventory, credential, and other API routes stay+ on the 6-clause gate. An uncertified safety pass returns 503 without making+ kubelet evict the Pod.+6. **Additive convergence** — the remaining fleet converges **after** the probe+ is ready, asynchronously, without blocking it.## 2. Fail-closed fences| Fence | Rule | Source ||-------|------|--------|-| Readiness gate | `/ready = ready && providerAuthoritativeFn()`; constructor default `() => false`; a throwing gate → 503 | `server.ts`, `readinessGate.ts` |+| Probe gate | `/ready = ready && probeAuthoritativeFn()` (watch freshness). Omit the 10th constructor arg → fall back to the 6-clause gate so omission cannot weaken kubelet readiness. A throwing gate → 503 | `server.ts`, `readinessGate.ts` |+| Per-request gate | data-path 503 unless `ready && providerAuthoritativeFn()` (6 clauses). Constructor default `() => false` | `server.ts`, `readinessGate.ts` || G1 — unconfigured authority | `currentWhenUnconfigured()` returns `false` + warns once; prod always wires the source, dev never constructs the reconciler | `reconciler.ts` || F1 — readiness-poll rebind | rebinds `isCurrent`/`server` on a superseding reconcile so the poll window does not self-destruct on a stale fence | `reconciler.ts` || Authority fence — `isCurrent` | **required** (not an optional/defaulted arg) on all three allow-granting reconcilers — `bindingPolicyReconciler`, `reconcileContext` (L2 context-allow), `reconcileExternalEgress` (L3 external-egress ALLOW). No `?? (() => true)` fail-open default: a caller cannot compile without supplying the fence (TS2554/TS2345) | `networkPolicyReconciler.ts`, `bindingPolicyReconciler.ts` |@@ -48,16 +53,20 @@ reconstructed from the **live** resolved CIDRs ("modulo cidr"). The decision:| Observed change | Classification | Action | Rationale ||-----------------|----------------|--------|-----------|| identity unchanged (same protocol, port, selector, ownership; cidrs only) | identity-stable | **RETAIN** while Ready; refresh cidrs in the additive lane | revoking a healthy binding every pass created a guaranteed deny window |-| protocol drift (TCP→UDP) | identity-changed | **REVOKE before readiness** | old allow is unprovable/stale |-| spec-level port drift | identity-changed | **REVOKE before readiness** | " |-| ownership drift | identity-changed | **REVOKE before readiness** | " |+| protocol drift (TCP→UDP) | identity-changed | **REVOKE before certifying the data path** | old allow is unprovable/stale |+| spec-level port drift | identity-changed | **REVOKE before certifying the data path** | " |+| ownership drift | identity-changed | **REVOKE before certifying the data path** | " || zero-cidr (deny-in-disguise) | never retained | **REVOKE** | an empty cidr set is never a valid retained allow |Revocation of an unprovable policy runs immediately (`cleanupExternalEgress`) under-the `isCurrent()` fence, **before** the safety inventory is certified — so `/ready`-never reports 200 with a stale divergent policy present. The unit suite pins both-directions (retain identity-stable; revoke on each drift), and the branch-owned-Minikube gate `e2e-hcc-mcp-context-readiness.sh` exercises both phases live.+the `isCurrent()` fence, **before** the safety inventory is certified — so the+**data path** never serves 200 with a stale divergent policy present. After A1+the kubelet `/ready` probe may return 200 while that revocation is still in+flight; the cluster gate waits for `/ready` 200 **and** the stale policy to be+absent, and treats `/ready` 200 with the stale allow still present as retryable.+The unit suite pins both directions (retain identity-stable; revoke on each+drift), and the branch-owned Minikube gate `e2e-hcc-mcp-context-readiness.sh`+exercises both phases live.## 4. ExternalEgressConvergenceCoordinatordiff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex 1d7ff668d..6055cad7b 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.247",+ "version": "0.1.248","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.247",+ "version": "0.1.248","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/image-policy": "file:../packages/image-policy",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex 44516712a..b4d204505 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.247",+ "version": "0.1.248","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/readinessGate.test.ts b/host-context-controller/src/readinessGate.test.tsindex b124689ff..0f98bf4c8 100644--- a/host-context-controller/src/readinessGate.test.ts+++ b/host-context-controller/src/readinessGate.test.ts@@ -8,6 +8,7 @@ import {resolveProbeAuthoritativeFn,resolveProviderAuthoritativeFn,resolveReadinessDetailFn,+ watchFreshnessReasonsFromDetail,} from './readinessGate'function authoritativeDetail(@@ -173,6 +174,18 @@ describe('resolveProbeAuthoritativeFn', () => {})})+describe('watchFreshnessReasonsFromDetail', () => {+ it('keeps the probe predicate identical to an empty watch-reason list', () => {+ const uncertified = authoritativeDetail({+ safetyInventoryCertified: false,+ contextRevisionAligned: false,+ })+ expect(watchFreshnessReasonsFromDetail(uncertified)).toEqual([])+ expect(isProbeReadinessAuthoritative(uncertified)).toBe(true)+ expect(probeReadinessReasonsFromDetail(uncertified)).toEqual([])+ })+})+describe('probeReadinessReasonsFromDetail (A1-T04)', () => {it('emits only the host watch reason when phase-2 clauses are also down', () => {expect(diff --git a/host-context-controller/src/readinessGate.ts b/host-context-controller/src/readinessGate.tsindex 3bf8d516c..dae6fa9c0 100644--- a/host-context-controller/src/readinessGate.ts+++ b/host-context-controller/src/readinessGate.ts@@ -17,12 +17,20 @@ export type ReadinessReason =| 'safety_pass_uncertified'| 'revocation_revision_mismatch'-export function readinessReasonsFromDetail(detail: ReadinessInventoryDetail): ReadinessReason[] {+/** Watch-freshness reasons shared by the kubelet probe and the 6-clause gate. */+export function watchFreshnessReasonsFromDetail(+ detail: ReadinessInventoryDetail+): ReadinessReason[] {const reasons: ReadinessReason[] = []if (detail.stopped) reasons.push('controller_stopped')if (!detail.mcpServerCacheSynced) reasons.push('mcp_watch_unsynced')if (!detail.contextCacheSynced) reasons.push('context_watch_unsynced')if (!detail.hostCacheSynced) reasons.push('host_watch_unsynced')+ return reasons+}++export function readinessReasonsFromDetail(detail: ReadinessInventoryDetail): ReadinessReason[] {+ const reasons = watchFreshnessReasonsFromDetail(detail)if (!detail.safetyInventoryCertified) reasons.push('safety_pass_uncertified')if (!detail.contextRevisionAligned || !detail.serverRevisionAligned) {reasons.push('revocation_revision_mismatch')@@ -46,14 +54,15 @@ export function resolveReadinessDetailFn(/*** Resolve the readiness-authority gate for the server from the provider.*- * Production (McpServerWatcher present): readiness follows the live inventory- * certification — /ready is 200 only once stale allows are revoked.+ * Production (McpServerWatcher present): the per-request data-path gate+ * follows live inventory certification (6 clauses). Kubelet /ready uses+ * resolveProbeAuthoritativeFn instead — watch freshness only.** Dev (DevMcpServerProvider — no watcher): there is NO NetworkPolicy inventory* to certify, so authority is unconditional. The ContextMapperServer default* for a missing gate is fail-closed (`() => false`), which is correct for a- * caller that forgets to wire one — but in dev it would pin /ready (and every- * data endpoint) at 503 forever. Dev must therefore pass an explicit+ * caller that forgets to wire one — but in dev it would pin every data+ * endpoint at 503 forever. Dev must therefore pass an explicit* always-authoritative gate; this function is that single, named decision so* the affirmative literal never gets copy-pasted into a production path.*/@@ -93,12 +102,7 @@ export function resolveHostAuthoritativeFn(* endpoints without making kubelet evict the Pod.*/export function isProbeReadinessAuthoritative(detail: ReadinessInventoryDetail): boolean {- return (- !detail.stopped &&- detail.mcpServerCacheSynced &&- detail.contextCacheSynced &&- detail.hostCacheSynced- )+ return watchFreshnessReasonsFromDetail(detail).length === 0}/**@@ -110,12 +114,7 @@ export function isProbeReadinessAuthoritative(detail: ReadinessInventoryDetail):export function probeReadinessReasonsFromDetail(detail: ReadinessInventoryDetail): ReadinessReason[] {- const reasons: ReadinessReason[] = []- if (detail.stopped) reasons.push('controller_stopped')- if (!detail.mcpServerCacheSynced) reasons.push('mcp_watch_unsynced')- if (!detail.contextCacheSynced) reasons.push('context_watch_unsynced')- if (!detail.hostCacheSynced) reasons.push('host_watch_unsynced')- return reasons+ return watchFreshnessReasonsFromDetail(detail)}/**diff --git a/host-context-controller/src/readinessSafetyFence.integration.test.ts b/host-context-controller/src/readinessSafetyFence.integration.test.tsindex d8b853023..9533b8c6b 100644--- a/host-context-controller/src/readinessSafetyFence.integration.test.ts+++ b/host-context-controller/src/readinessSafetyFence.integration.test.ts@@ -176,7 +176,7 @@ function alignRevocationCounters(watcher: InstanceType<typeof McpServerWatcher>)w.networkPolicyRevocationServerRevision = w.mcpServerDesiredRevision}-describe('readiness withdrawal on a real lost delete fence', () => {+describe('lost delete fence: per-request 503 while /ready stays 200', () => {let watcher: InstanceType<typeof McpServerWatcher>let server: InstanceType<typeof ContextMapperServer>@@ -211,7 +211,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {await watcher.stop()})- it('turns /ready from 200 to 503 when the additive phase loses a delete fence, and back on a clean pass', async () => {+ it('503s the data path when the additive phase loses a delete fence, keeps /ready 200, and reopens the API on a clean pass', async () => {const reconciler = (watcher as unknown as { netPolReconciler: any }).netPolReconcileralignRevocationCounters(watcher)@@ -225,6 +225,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {onAuthoritativeRevocationComplete: () => {},})alignRevocationCounters(watcher)+ expect(await apiStatus(server)).toBe(200)expect(await readyStatus(server)).toBe(200)// The stale allow only becomes visible after the authoritative phase has@@ -270,10 +271,11 @@ describe('readiness withdrawal on a real lost delete fence', () => {})alignRevocationCounters(watcher)expect(reconciler.hasCertifiedSafetyInventory()).toBe(true)+ expect(await apiStatus(server)).toBe(200)expect(await readyStatus(server)).toBe(200)})- it('withholds readiness when the authoritative pass loses a delete fence under a provided safety snapshot', async () => {+ it('503s the data path when the authoritative pass loses a delete fence under a provided safety snapshot', async () => {const reconciler = (watcher as unknown as { netPolReconciler: any }).netPolReconcileralignRevocationCounters(watcher)@@ -287,6 +289,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {onAuthoritativeRevocationComplete: () => {},})alignRevocationCounters(watcher)+ expect(await apiStatus(server)).toBe(200)expect(await readyStatus(server)).toBe(200)// Unlike the additive-phase test above, the stale allow is visible to the@@ -349,6 +352,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {})alignRevocationCounters(watcher)expect(reconciler.hasCertifiedSafetyInventory()).toBe(true)+ expect(await apiStatus(server)).toBe(200)expect(await readyStatus(server)).toBe(200)})@@ -387,6 +391,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {onAuthoritativeRevocationComplete: () => {},})alignRevocationCounters(watcher)+ expect(await apiStatus(server)).toBe(200)expect(await readyStatus(server)).toBe(200)// The authoritative pass loses its delete fence under the provided@@ -487,6 +492,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {onAuthoritativeRevocationComplete: () => {},})alignRevocationCounters(watcher)+ expect(await apiStatus(server)).toBe(200)expect(await readyStatus(server)).toBe(200)// The delta certificate is captured now, while the fence is still certified.diff --git a/host-context-controller/src/server.test.ts b/host-context-controller/src/server.test.tsindex 32c62e5a7..88a4f173d 100644--- a/host-context-controller/src/server.test.ts+++ b/host-context-controller/src/server.test.ts@@ -1090,4 +1090,56 @@ describe('ContextMapperServer /ready probe cut (A1)', () => {inventory = await invoke(server, '/api/v1/mcpservers')expect(inventory.statusCode).toBe(200)})++ it('A1-T15 logs distinct lanes when the probe is ready and the data path is not', async () => {+ const infoSpy = vi.spyOn(console, 'log').mockImplementation(() => {})+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})+ try {+ const detailRef = {+ current: authoritativeDetail({+ safetyInventoryCertified: false,+ contextRevisionAligned: false,+ serverRevisionAligned: false,+ }),+ }+ server = mainLikeServer(detailRef)+ server.setReady(true)++ expect((await invoke(server, '/ready')).statusCode).toBe(200)+ expect((await invoke(server, '/health')).statusCode).toBe(200)+ expect(JSON.parse((await invoke(server, '/health')).body).ready).toBe(false)++ const infoLines = infoSpy.mock.calls.map(call => String(call[0]))+ const warnLines = warnSpy.mock.calls.map(call => String(call[0]))+ expect(infoLines.some(line => line.includes('"lane":"probe"'))).toBe(true)+ expect(warnLines.some(line => line.includes('"lane":"request"'))).toBe(true)+ } finally {+ infoSpy.mockRestore()+ warnSpy.mockRestore()+ }+ })++ it('A1-T16 describes /ready as watch-freshness on GET /', async () => {+ server = mainLikeServer({ current: authoritativeDetail() })+ const response = await invoke(server, '/')+ expect(response.statusCode).toBe(200)+ const body = JSON.parse(response.body) as { endpoints: Record<string, string> }+ expect(body.endpoints['GET /ready']).toContain('watch inventory is fresh')+ expect(body.endpoints['GET /ready']).not.toContain('provider inventory is authoritative')+ })++ it('A1-T17 keeps /health.ready on the 6-clause gate while /ready is 200', async () => {+ const detailRef = {+ current: authoritativeDetail({ safetyInventoryCertified: false }),+ }+ server = mainLikeServer(detailRef)+ server.setReady(true)++ const ready = await invoke(server, '/ready')+ const health = await invoke(server, '/health')+ expect(ready.statusCode).toBe(200)+ expect(JSON.parse(ready.body)).toEqual({ status: 'ready', ready: true })+ expect(health.statusCode).toBe(200)+ expect(JSON.parse(health.body)).toEqual({ status: 'ok', ready: false })+ })})diff --git a/host-context-controller/src/server.ts b/host-context-controller/src/server.tsindex 71e3bcb3e..b75a14a14 100644--- a/host-context-controller/src/server.ts+++ b/host-context-controller/src/server.ts@@ -81,6 +81,8 @@ export class McpHostApiRateLimiter {}}+type ReadinessLane = 'probe' | 'request'+type ReadinessState =| { ready: true; status: 'ready' }| {@@ -162,74 +164,82 @@ export class ContextMapperServer {this.ready = ready}- private collectReadinessReasons(): ReadinessReason[] | undefined {+ private collectLaneReasons(lane: ReadinessLane): ReadinessReason[] | undefined {if (!this.readinessDetailFn) return undefinedtry {- return readinessReasonsFromDetail(this.readinessDetailFn())+ const detail = this.readinessDetailFn()+ return lane === 'probe'+ ? probeReadinessReasonsFromDetail(detail)+ : readinessReasonsFromDetail(detail)} catch (err) {- readinessLog.error('readiness detail check failed', { err })- return undefined- }- }-- private collectProbeReadinessReasons(): ReadinessReason[] | undefined {- if (!this.readinessDetailFn) return undefined- try {- return probeReadinessReasonsFromDetail(this.readinessDetailFn())- } catch (err) {- readinessLog.error('readiness detail check failed', { err })+ readinessLog.error(+ lane === 'probe' ? 'probe readiness detail check failed' : 'readiness detail check failed',+ { err, lane }+ )return undefined}}- private emitReadinessTransition(- state: ReadinessState,- keyHolder:- | 'lastReadinessTransitionKey'- | 'lastProbeReadinessTransitionKey' = 'lastReadinessTransitionKey'- ): void {+ private emitReadinessTransition(state: ReadinessState, lane: ReadinessLane): void {+ const keyHolder =+ lane === 'probe' ? 'lastProbeReadinessTransitionKey' : 'lastReadinessTransitionKey'const reasons = !state.ready ? (state.reasons ?? []) : []const key = `${state.ready}:${state.status}:${reasons.join(',')}`if (key === this[keyHolder]) returnthis[keyHolder] = keyif (state.ready) {- readinessLog.info('readiness became ready')+ readinessLog.info('readiness became ready', { lane })return}- readinessLog.warn('readiness not ready', { status: state.status, reasons })+ readinessLog.warn('readiness not ready', { lane, status: state.status, reasons })}- private getReadinessState(): ReadinessState {+ private evaluateAuthoritativeLane(+ lane: ReadinessLane,+ authoritativeFn: () => boolean+ ): ReadinessState {if (!this.ready) {const starting: ReadinessState = {ready: false,status: 'starting',message: 'Context mapper is still starting',}- this.emitReadinessTransition(starting)+ this.emitReadinessTransition(starting, lane)return starting}try {- if (this.providerAuthoritativeFn()) {+ if (authoritativeFn()) {const ready: ReadinessState = { ready: true, status: 'ready' }- this.emitReadinessTransition(ready)+ this.emitReadinessTransition(ready, lane)return ready}} catch (err) {- readinessLog.error('provider authority readiness check failed', { err })+ readinessLog.error(+ lane === 'probe'+ ? 'probe authority readiness check failed'+ : 'provider authority readiness check failed',+ { err, lane }+ )}const degraded: ReadinessState = {ready: false,status: 'degraded',- message: 'Context mapper provider inventory is not authoritative',- reasons: this.collectReadinessReasons(),+ message:+ lane === 'probe'+ ? 'Context mapper watch inventory is not fresh'+ : 'Context mapper provider inventory is not authoritative',+ reasons: this.collectLaneReasons(lane),}- this.emitReadinessTransition(degraded)+ this.emitReadinessTransition(degraded, lane)return degraded}+ private getReadinessState(): ReadinessState {+ return this.evaluateAuthoritativeLane('request', this.providerAuthoritativeFn)+ }+/*** Kubelet /ready state. When probeAuthoritativeFn is omitted, fall back to* the 6-clause provider gate so omitting the 10th constructor argument@@ -239,34 +249,7 @@ export class ContextMapperServer {if (this.probeAuthoritativeFn === undefined) {return this.getReadinessState()}- if (!this.ready) {- const starting: ReadinessState = {- ready: false,- status: 'starting',- message: 'Context mapper is still starting',- }- this.emitReadinessTransition(starting, 'lastProbeReadinessTransitionKey')- return starting- }-- try {- if (this.probeAuthoritativeFn()) {- const ready: ReadinessState = { ready: true, status: 'ready' }- this.emitReadinessTransition(ready, 'lastProbeReadinessTransitionKey')- return ready- }- } catch (err) {- readinessLog.error('probe authority readiness check failed', { err })- }-- const degraded: ReadinessState = {- ready: false,- status: 'degraded',- message: 'Context mapper provider inventory is not authoritative',- reasons: this.collectProbeReadinessReasons(),- }- this.emitReadinessTransition(degraded, 'lastProbeReadinessTransitionKey')- return degraded+ return this.evaluateAuthoritativeLane('probe', this.probeAuthoritativeFn)}private async handleRequest(req: http.IncomingMessage, res: http.ServerResponse): Promise<void> {@@ -662,7 +645,7 @@ export class ContextMapperServer {'GET /': 'This information page','GET /health': "Health check endpoint - returns { status: 'ok', ready: boolean }",'GET /ready':- 'Readiness endpoint - returns 200 after warm-up while provider inventory is authoritative',+ 'Readiness endpoint - returns 200 after warm-up while watch inventory is fresh','GET /api/v1/mcpservers': 'List all McpServer resources','GET /api/v2/hosts/self/mcpservers': 'List MCP servers for the authenticated Host','POST /api/v2/hosts/self/mcpservers/credential':diff --git a/scripts/e2e/e2e-hcc-mcp-context-readiness.sh b/scripts/e2e/e2e-hcc-mcp-context-readiness.shindex d4b0e6664..2527cd075 100755--- a/scripts/e2e/e2e-hcc-mcp-context-readiness.sh+++ b/scripts/e2e/e2e-hcc-mcp-context-readiness.sh@@ -165,9 +165,10 @@ wait_until_fast() {while :; do"$@" && return 0rc=$?- # rc=2 is a fatal, non-retryable ordering violation (e.g. 200 observed with- # a stale policy still present): stop polling immediately so a later clean- # poll cannot mask it. rc=1 is benign "not ready yet".+ # rc=2 is a fatal, non-retryable ordering violation (e.g. identity-stable+ # allow already gone when /ready first becomes 200): stop polling immediately+ # so a later clean poll cannot mask it. rc=1 is benign "not ready yet".+ # After A1, /ready 200 with a divergent stale allow is rc=1, not rc=2.if [ "$rc" -eq 2 ]; thenecho "FATAL ordering violation while waiting for ${description}: ${HCC_READY_PROBE_DIAGNOSTIC:-unset}" >&2return 2@@ -877,13 +878,12 @@ safety_pass_metrics_certify() {}hcc_ready_after_stale_policy_revoked() {- # A single poll that sees 200 WHILE the stale policy is still present is a- # safety-contract violation and must be FATAL, never retried: retrying lets a- # later poll (policy already revoked) turn the violation into a false PASS.- # rc=2 signals that fatal ordering violation to wait_until_fast; rc=1 is a- # benign "not ready yet, keep polling". The sub-0.5s window where revocation- # races the probe inside one poll stays unobservable on THIS signal — the- # third signal (M6, /metrics) covers it.+ # A1: /ready is watch freshness only. 200 while a divergent policy is still+ # present is the intended kubelet window, not a fatal ordering violation.+ # This wait succeeds only when BOTH the probe is ready AND the stale allow+ # is gone. rc=1 = retry. Do not return rc=2 from /ready+stale — that would+ # abort the first poll after watches sync. The third signal (M6, /metrics)+ # still proves an authoritative pass measured and revoked.local policy_absent_before=0external_egress_policy_absent && policy_absent_before=1probe_new_hcc_ready_endpoint || return 1@@ -893,8 +893,7 @@ hcc_ready_after_stale_policy_revoked() {# this poll cannot adjudicate the order. Retry to adjudicate cleanly.return 1fi- HCC_READY_PROBE_DIAGNOSTIC="HCC_READY_PROBE category=ready-200-with-stale-policy-present"- return 2+ return 1}# Canonical, order-stable projection of the single external-egress policy for thediff --git a/scripts/tests/test-hcc-watch-recovery-gate.sh b/scripts/tests/test-hcc-watch-recovery-gate.shindex 5a2620513..4f711a573 100755--- a/scripts/tests/test-hcc-watch-recovery-gate.sh+++ b/scripts/tests/test-hcc-watch-recovery-gate.sh@@ -983,17 +983,17 @@ if (eval "$hcc_ready_after_revoke_function"rc=0hcc_ready_after_stale_policy_revoked || rc=$?- # A 200 observed WITH the stale policy present is a fatal ordering violation- # (rc=2), not a benign retryable "not ready" (rc=1).- [ "$rc" -eq 2 ] &&- [ "$HCC_READY_PROBE_DIAGNOSTIC" = 'HCC_READY_PROBE category=ready-200-with-stale-policy-present' ]+ # A1: /ready 200 WITH the stale policy present is the kubelet window.+ # Retry (rc=1). rc=2 here would abort the first poll after watches sync.+ [ "$rc" -eq 1 ]); then- pass "global readiness evidence flags a surviving divergent policy as a fatal rc=2 ordering violation"+ pass "global readiness retries while a divergent policy still survives after /ready 200"else- fail "global readiness can self-confirm (or only soft-fail) while a divergent policy survives"+ fail "global readiness treats /ready 200 with a surviving divergent policy as terminal"fi-# The loop must PROPAGATE the fatal rc=2 immediately, not retry until a later-# clean poll turns the violation into a false PASS (the round-2 bug).+# The loop must KEEP polling through that window and PASS only once the+# stale policy is gone. wait_until_fast still treats rc=2 as fatal for the+# identity-stable retain path; this predicate must not emit rc=2.wait_until_fast_function="$(sed -n '/^wait_until_fast() {$/,/^}$/p' "$MCP_READINESS_GATE")"@@ -1001,21 +1001,19 @@ if (HCC_READY_PROBE_DIAGNOSTIC=""POLL=0probe_new_hcc_ready_endpoint() { return 0; }- # Stale policy present for the first polls, revoked only later. The buggy- # retry loop would poll past the fatal first observation and PASS here.+ # Stale for the first polls, revoked later. A1 must retry, not die on poll 1.external_egress_policy_absent() { POLL=$((POLL + 1)); [ "$POLL" -gt 4 ]; }eval "$hcc_ready_after_revoke_function"eval "$wait_until_fast_function"rc=0- wait_until_fast 10 "rc=2 propagation" hcc_ready_after_stale_policy_revoked || rc=$?- # rc must be the fatal 2, and it must have stopped inside the first predicate- # poll (which calls external_egress_policy_absent twice) — POLL<3 proves no- # retry occurred.- [ "$rc" -eq 2 ] && [ "$POLL" -lt 3 ]+ wait_until_fast 10 "A1 probe window retry" hcc_ready_after_stale_policy_revoked || rc=$?+ # Each predicate call samples absence twice. Success needs POLL>4 so the+ # before/after pair is both absent.+ [ "$rc" -eq 0 ] && [ "$POLL" -gt 4 ]); then- pass "wait_until_fast propagates a fatal rc=2 on the first poll instead of retrying to a false PASS"+ pass "wait_until_fast retries /ready 200 with a surviving policy until revocation"else- fail "wait_until_fast retried past a fatal ordering violation to a later clean poll"+ fail "wait_until_fast no longer waits through the A1 probe window"fi# M6: the third contract signal adjudicates a real /metrics scrape. Extract the