Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(hcc): close A1 review findings on the probe vs data-path split (#477)

Alfredo Lopez46c350fdd721
diff --git a/docs/architecture/hcc-readiness-flows-diagram.svg b/docs/architecture/hcc-readiness-flows-diagram.svg
index db0985b89..280620ddb 100644
--- a/docs/architecture/hcc-readiness-flows-diagram.svg
+++ b/docs/architecture/hcc-readiness-flows-diagram.svg
@@ -29,9 +29,9 @@
<text x="640.0" y="178" text-anchor="middle" font-size="11" fill="#475569">· anchored to desired revision</text>
<rect x="779.5" y="104" width="214" height="92" rx="11" fill="#dbeafe" stroke="#1d4ed8" stroke-width="1.8" filter="url(#sh)"/>
<text x="886.5" y="127" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">/ready = 200</text>
-<text x="886.5" y="146" text-anchor="middle" font-size="11" fill="#475569">only if inventory certified</text>
-<text x="886.5" y="162" text-anchor="middle" font-size="11" fill="#475569">and no stale allow</text>
-<text x="886.5" y="178" text-anchor="middle" font-size="11" fill="#475569">survives</text>
+<text x="886.5" y="146" text-anchor="middle" font-size="11" fill="#475569">watch freshness only</text>
+<text x="886.5" y="162" text-anchor="middle" font-size="11" fill="#475569">phase-2 stays on the</text>
+<text x="886.5" y="178" text-anchor="middle" font-size="11" fill="#475569">per-request data path</text>
<rect x="1026.0" y="104" width="214" height="92" rx="11" fill="#eff6ff" stroke="#2563eb" stroke-width="1.8" filter="url(#sh)"/>
<text x="1133.0" y="127" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">Additive convergence</text>
<text x="1133.0" y="146" text-anchor="middle" font-size="11" fill="#475569">fleet converges AFTER</text>
@@ -52,7 +52,7 @@
<text x="40" y="275" font-size="13" font-weight="700" fill="#b91c1c">2 · Fail-closed fences (R3 · R4 · J3)</text>
<rect x="40.0" y="284" width="380.0" height="76" rx="11" fill="#fef2f2" stroke="#dc2626" stroke-width="1.8" filter="url(#sh)"/>
<text x="230.0" y="307" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">Readiness gate — default () =&gt; false</text>
-<text x="230.0" y="326" text-anchor="middle" font-size="11" fill="#475569">/ready = ready &amp;&amp; providerAuthoritativeFn()</text>
+<text x="230.0" y="326" text-anchor="middle" font-size="11" fill="#475569">/ready = ready &amp;&amp; probeAuthoritativeFn()</text>
<text x="230.0" y="342" text-anchor="middle" font-size="11" fill="#475569">throw → 503 · never a wrong 200</text>
<rect x="450.0" y="284" width="380.0" height="76" rx="11" fill="#fef2f2" stroke="#dc2626" stroke-width="1.8" filter="url(#sh)"/>
<text x="640.0" y="307" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">G1 — deny without inventory-authority</text>
@@ -76,7 +76,7 @@
<text x="693.0" y="456" text-anchor="middle" font-size="11" fill="#475569">same protocol/port/ownership/cidrs</text>
<text x="693.0" y="472" text-anchor="middle" font-size="11" fill="#475569">· HCC stays Ready (modulo cidr)</text>
<rect x="540" y="496" width="306" height="60" rx="11" fill="#fef2f2" stroke="#dc2626" stroke-width="1.8" filter="url(#sh)"/>
-<text x="693.0" y="519" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">REVOKE before readiness</text>
+<text x="693.0" y="519" text-anchor="middle" font-size="14" font-weight="700" fill="#0f172a">REVOKE before data-path certify</text>
<text x="693.0" y="538" text-anchor="middle" font-size="11" fill="#475569">drift TCP→UDP · port · ownership · zero-cidr</text>
<text x="693.0" y="554" text-anchor="middle" font-size="11" fill="#475569">→ unprovable → cleanup fail-closed</text>
<path d="M 472,470 H 506.0 V 444 H 540" fill="none" stroke="#059669" stroke-width="2.6" marker-end="url(#aGreen)" stroke-linejoin="round"/>
diff --git a/docs/architecture/hcc-readiness-flows.md b/docs/architecture/hcc-readiness-flows.md
index 5d158b3e4..049cdbb59 100644
--- a/docs/architecture/hcc-readiness-flows.md
+++ b/docs/architecture/hcc-readiness-flows.md
@@ -4,7 +4,7 @@
> `host-context-controller` (HCC). Addresses the "no design/decision doc for the
> 500+-line external-egress coordinator and the certification barrier" review gap.
-![HCC readiness flows resolved by #382](./hcc-readiness-flows-diagram.png)
+![HCC readiness flows — probe vs data-path lanes after #477 A1](./hcc-readiness-flows-diagram.svg)
## 1. Readiness lifecycle — decoupled from initial fleet convergence
@@ -12,8 +12,9 @@ Before this work, `McpServerWatcher.start()` awaited full initial fleet
reconciliation, so a large MCP/Context/Host fleet could keep `provider.start()`
and `/ready` unavailable until every individual reconcile finished.
-Now readiness is gated on an **authoritative, fail-closed safety boundary**, not on
-full convergence:
+Readiness is now two lanes. Watch freshness (clauses 1–4) stays fail-closed on
+both. Phase-2 certification stays fail-closed on the **data path** and is off
+the kubelet probe (#477 A1):
1. **HCC start** — four watch lanes (Context ingress, mcp-host egress, rpc-proxy
egress, external egress) start. A transient LIST/WATCH failure on any lane enters
@@ -21,16 +22,20 @@ full convergence:
2. **4-LIST inventory** — bounded inventory across the four lanes.
3. **Certify safety inventory** — the authoritative pass revokes any stale /
unprovable allow **before** recording a certificate (fail-closed).
-4. **`/ready = 200`** — served **only** when the inventory is certified and no stale
- allow survives.
-5. **Additive convergence** — the remaining fleet converges **after** ready,
- asynchronously, without blocking it.
+4. **`/ready = 200`** — served when the process is past warm-up and the three
+ watches are synced. Phase-2 certification does **not** hold the probe.
+5. **Per-request data path** — inventory, credential, and other API routes stay
+ on the 6-clause gate. An uncertified safety pass returns 503 without making
+ kubelet evict the Pod.
+6. **Additive convergence** — the remaining fleet converges **after** the probe
+ is ready, asynchronously, without blocking it.
## 2. Fail-closed fences
| Fence | Rule | Source |
|-------|------|--------|
-| Readiness gate | `/ready = ready && providerAuthoritativeFn()`; constructor default `() => false`; a throwing gate → 503 | `server.ts`, `readinessGate.ts` |
+| Probe gate | `/ready = ready && probeAuthoritativeFn()` (watch freshness). Omit the 10th constructor arg → fall back to the 6-clause gate so omission cannot weaken kubelet readiness. A throwing gate → 503 | `server.ts`, `readinessGate.ts` |
+| Per-request gate | data-path 503 unless `ready && providerAuthoritativeFn()` (6 clauses). Constructor default `() => false` | `server.ts`, `readinessGate.ts` |
| G1 — unconfigured authority | `currentWhenUnconfigured()` returns `false` + warns once; prod always wires the source, dev never constructs the reconciler | `reconciler.ts` |
| F1 — readiness-poll rebind | rebinds `isCurrent`/`server` on a superseding reconcile so the poll window does not self-destruct on a stale fence | `reconciler.ts` |
| Authority fence — `isCurrent` | **required** (not an optional/defaulted arg) on all three allow-granting reconcilers — `bindingPolicyReconciler`, `reconcileContext` (L2 context-allow), `reconcileExternalEgress` (L3 external-egress ALLOW). No `?? (() => true)` fail-open default: a caller cannot compile without supplying the fence (TS2554/TS2345) | `networkPolicyReconciler.ts`, `bindingPolicyReconciler.ts` |
@@ -48,16 +53,20 @@ reconstructed from the **live** resolved CIDRs ("modulo cidr"). The decision:
| Observed change | Classification | Action | Rationale |
|-----------------|----------------|--------|-----------|
| identity unchanged (same protocol, port, selector, ownership; cidrs only) | identity-stable | **RETAIN** while Ready; refresh cidrs in the additive lane | revoking a healthy binding every pass created a guaranteed deny window |
-| protocol drift (TCP→UDP) | identity-changed | **REVOKE before readiness** | old allow is unprovable/stale |
-| spec-level port drift | identity-changed | **REVOKE before readiness** | " |
-| ownership drift | identity-changed | **REVOKE before readiness** | " |
+| protocol drift (TCP→UDP) | identity-changed | **REVOKE before certifying the data path** | old allow is unprovable/stale |
+| spec-level port drift | identity-changed | **REVOKE before certifying the data path** | " |
+| ownership drift | identity-changed | **REVOKE before certifying the data path** | " |
| zero-cidr (deny-in-disguise) | never retained | **REVOKE** | an empty cidr set is never a valid retained allow |
Revocation of an unprovable policy runs immediately (`cleanupExternalEgress`) under
-the `isCurrent()` fence, **before** the safety inventory is certified — so `/ready`
-never reports 200 with a stale divergent policy present. The unit suite pins both
-directions (retain identity-stable; revoke on each drift), and the branch-owned
-Minikube gate `e2e-hcc-mcp-context-readiness.sh` exercises both phases live.
+the `isCurrent()` fence, **before** the safety inventory is certified — so the
+**data path** never serves 200 with a stale divergent policy present. After A1
+the kubelet `/ready` probe may return 200 while that revocation is still in
+flight; the cluster gate waits for `/ready` 200 **and** the stale policy to be
+absent, and treats `/ready` 200 with the stale allow still present as retryable.
+The unit suite pins both directions (retain identity-stable; revoke on each
+drift), and the branch-owned Minikube gate `e2e-hcc-mcp-context-readiness.sh`
+exercises both phases live.
## 4. ExternalEgressConvergenceCoordinator
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index 1d7ff668d..6055cad7b 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.247",
+ "version": "0.1.248",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.247",
+ "version": "0.1.248",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/image-policy": "file:../packages/image-policy",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index 44516712a..b4d204505 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.247",
+ "version": "0.1.248",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/readinessGate.test.ts b/host-context-controller/src/readinessGate.test.ts
index b124689ff..0f98bf4c8 100644
--- a/host-context-controller/src/readinessGate.test.ts
+++ b/host-context-controller/src/readinessGate.test.ts
@@ -8,6 +8,7 @@ import {
resolveProbeAuthoritativeFn,
resolveProviderAuthoritativeFn,
resolveReadinessDetailFn,
+ watchFreshnessReasonsFromDetail,
} from './readinessGate'
function authoritativeDetail(
@@ -173,6 +174,18 @@ describe('resolveProbeAuthoritativeFn', () => {
})
})
+describe('watchFreshnessReasonsFromDetail', () => {
+ it('keeps the probe predicate identical to an empty watch-reason list', () => {
+ const uncertified = authoritativeDetail({
+ safetyInventoryCertified: false,
+ contextRevisionAligned: false,
+ })
+ expect(watchFreshnessReasonsFromDetail(uncertified)).toEqual([])
+ expect(isProbeReadinessAuthoritative(uncertified)).toBe(true)
+ expect(probeReadinessReasonsFromDetail(uncertified)).toEqual([])
+ })
+})
+
describe('probeReadinessReasonsFromDetail (A1-T04)', () => {
it('emits only the host watch reason when phase-2 clauses are also down', () => {
expect(
diff --git a/host-context-controller/src/readinessGate.ts b/host-context-controller/src/readinessGate.ts
index 3bf8d516c..dae6fa9c0 100644
--- a/host-context-controller/src/readinessGate.ts
+++ b/host-context-controller/src/readinessGate.ts
@@ -17,12 +17,20 @@ export type ReadinessReason =
| 'safety_pass_uncertified'
| 'revocation_revision_mismatch'
-export function readinessReasonsFromDetail(detail: ReadinessInventoryDetail): ReadinessReason[] {
+/** Watch-freshness reasons shared by the kubelet probe and the 6-clause gate. */
+export function watchFreshnessReasonsFromDetail(
+ detail: ReadinessInventoryDetail
+): ReadinessReason[] {
const reasons: ReadinessReason[] = []
if (detail.stopped) reasons.push('controller_stopped')
if (!detail.mcpServerCacheSynced) reasons.push('mcp_watch_unsynced')
if (!detail.contextCacheSynced) reasons.push('context_watch_unsynced')
if (!detail.hostCacheSynced) reasons.push('host_watch_unsynced')
+ return reasons
+}
+
+export function readinessReasonsFromDetail(detail: ReadinessInventoryDetail): ReadinessReason[] {
+ const reasons = watchFreshnessReasonsFromDetail(detail)
if (!detail.safetyInventoryCertified) reasons.push('safety_pass_uncertified')
if (!detail.contextRevisionAligned || !detail.serverRevisionAligned) {
reasons.push('revocation_revision_mismatch')
@@ -46,14 +54,15 @@ export function resolveReadinessDetailFn(
/**
* Resolve the readiness-authority gate for the server from the provider.
*
- * Production (McpServerWatcher present): readiness follows the live inventory
- * certification — /ready is 200 only once stale allows are revoked.
+ * Production (McpServerWatcher present): the per-request data-path gate
+ * follows live inventory certification (6 clauses). Kubelet /ready uses
+ * resolveProbeAuthoritativeFn instead — watch freshness only.
*
* Dev (DevMcpServerProvider — no watcher): there is NO NetworkPolicy inventory
* to certify, so authority is unconditional. The ContextMapperServer default
* for a missing gate is fail-closed (`() => false`), which is correct for a
- * caller that forgets to wire one — but in dev it would pin /ready (and every
- * data endpoint) at 503 forever. Dev must therefore pass an explicit
+ * caller that forgets to wire one — but in dev it would pin every data
+ * endpoint at 503 forever. Dev must therefore pass an explicit
* always-authoritative gate; this function is that single, named decision so
* the affirmative literal never gets copy-pasted into a production path.
*/
@@ -93,12 +102,7 @@ export function resolveHostAuthoritativeFn(
* endpoints without making kubelet evict the Pod.
*/
export function isProbeReadinessAuthoritative(detail: ReadinessInventoryDetail): boolean {
- return (
- !detail.stopped &&
- detail.mcpServerCacheSynced &&
- detail.contextCacheSynced &&
- detail.hostCacheSynced
- )
+ return watchFreshnessReasonsFromDetail(detail).length === 0
}
/**
@@ -110,12 +114,7 @@ export function isProbeReadinessAuthoritative(detail: ReadinessInventoryDetail):
export function probeReadinessReasonsFromDetail(
detail: ReadinessInventoryDetail
): ReadinessReason[] {
- const reasons: ReadinessReason[] = []
- if (detail.stopped) reasons.push('controller_stopped')
- if (!detail.mcpServerCacheSynced) reasons.push('mcp_watch_unsynced')
- if (!detail.contextCacheSynced) reasons.push('context_watch_unsynced')
- if (!detail.hostCacheSynced) reasons.push('host_watch_unsynced')
- return reasons
+ return watchFreshnessReasonsFromDetail(detail)
}
/**
diff --git a/host-context-controller/src/readinessSafetyFence.integration.test.ts b/host-context-controller/src/readinessSafetyFence.integration.test.ts
index d8b853023..9533b8c6b 100644
--- a/host-context-controller/src/readinessSafetyFence.integration.test.ts
+++ b/host-context-controller/src/readinessSafetyFence.integration.test.ts
@@ -176,7 +176,7 @@ function alignRevocationCounters(watcher: InstanceType<typeof McpServerWatcher>)
w.networkPolicyRevocationServerRevision = w.mcpServerDesiredRevision
}
-describe('readiness withdrawal on a real lost delete fence', () => {
+describe('lost delete fence: per-request 503 while /ready stays 200', () => {
let watcher: InstanceType<typeof McpServerWatcher>
let server: InstanceType<typeof ContextMapperServer>
@@ -211,7 +211,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {
await watcher.stop()
})
- it('turns /ready from 200 to 503 when the additive phase loses a delete fence, and back on a clean pass', async () => {
+ it('503s the data path when the additive phase loses a delete fence, keeps /ready 200, and reopens the API on a clean pass', async () => {
const reconciler = (watcher as unknown as { netPolReconciler: any }).netPolReconciler
alignRevocationCounters(watcher)
@@ -225,6 +225,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {
onAuthoritativeRevocationComplete: () => {},
})
alignRevocationCounters(watcher)
+ expect(await apiStatus(server)).toBe(200)
expect(await readyStatus(server)).toBe(200)
// The stale allow only becomes visible after the authoritative phase has
@@ -270,10 +271,11 @@ describe('readiness withdrawal on a real lost delete fence', () => {
})
alignRevocationCounters(watcher)
expect(reconciler.hasCertifiedSafetyInventory()).toBe(true)
+ expect(await apiStatus(server)).toBe(200)
expect(await readyStatus(server)).toBe(200)
})
- it('withholds readiness when the authoritative pass loses a delete fence under a provided safety snapshot', async () => {
+ it('503s the data path when the authoritative pass loses a delete fence under a provided safety snapshot', async () => {
const reconciler = (watcher as unknown as { netPolReconciler: any }).netPolReconciler
alignRevocationCounters(watcher)
@@ -287,6 +289,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {
onAuthoritativeRevocationComplete: () => {},
})
alignRevocationCounters(watcher)
+ expect(await apiStatus(server)).toBe(200)
expect(await readyStatus(server)).toBe(200)
// Unlike the additive-phase test above, the stale allow is visible to the
@@ -349,6 +352,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {
})
alignRevocationCounters(watcher)
expect(reconciler.hasCertifiedSafetyInventory()).toBe(true)
+ expect(await apiStatus(server)).toBe(200)
expect(await readyStatus(server)).toBe(200)
})
@@ -387,6 +391,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {
onAuthoritativeRevocationComplete: () => {},
})
alignRevocationCounters(watcher)
+ expect(await apiStatus(server)).toBe(200)
expect(await readyStatus(server)).toBe(200)
// The authoritative pass loses its delete fence under the provided
@@ -487,6 +492,7 @@ describe('readiness withdrawal on a real lost delete fence', () => {
onAuthoritativeRevocationComplete: () => {},
})
alignRevocationCounters(watcher)
+ expect(await apiStatus(server)).toBe(200)
expect(await readyStatus(server)).toBe(200)
// The delta certificate is captured now, while the fence is still certified.
diff --git a/host-context-controller/src/server.test.ts b/host-context-controller/src/server.test.ts
index 32c62e5a7..88a4f173d 100644
--- a/host-context-controller/src/server.test.ts
+++ b/host-context-controller/src/server.test.ts
@@ -1090,4 +1090,56 @@ describe('ContextMapperServer /ready probe cut (A1)', () => {
inventory = await invoke(server, '/api/v1/mcpservers')
expect(inventory.statusCode).toBe(200)
})
+
+ it('A1-T15 logs distinct lanes when the probe is ready and the data path is not', async () => {
+ const infoSpy = vi.spyOn(console, 'log').mockImplementation(() => {})
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+ try {
+ const detailRef = {
+ current: authoritativeDetail({
+ safetyInventoryCertified: false,
+ contextRevisionAligned: false,
+ serverRevisionAligned: false,
+ }),
+ }
+ server = mainLikeServer(detailRef)
+ server.setReady(true)
+
+ expect((await invoke(server, '/ready')).statusCode).toBe(200)
+ expect((await invoke(server, '/health')).statusCode).toBe(200)
+ expect(JSON.parse((await invoke(server, '/health')).body).ready).toBe(false)
+
+ const infoLines = infoSpy.mock.calls.map(call => String(call[0]))
+ const warnLines = warnSpy.mock.calls.map(call => String(call[0]))
+ expect(infoLines.some(line => line.includes('"lane":"probe"'))).toBe(true)
+ expect(warnLines.some(line => line.includes('"lane":"request"'))).toBe(true)
+ } finally {
+ infoSpy.mockRestore()
+ warnSpy.mockRestore()
+ }
+ })
+
+ it('A1-T16 describes /ready as watch-freshness on GET /', async () => {
+ server = mainLikeServer({ current: authoritativeDetail() })
+ const response = await invoke(server, '/')
+ expect(response.statusCode).toBe(200)
+ const body = JSON.parse(response.body) as { endpoints: Record<string, string> }
+ expect(body.endpoints['GET /ready']).toContain('watch inventory is fresh')
+ expect(body.endpoints['GET /ready']).not.toContain('provider inventory is authoritative')
+ })
+
+ it('A1-T17 keeps /health.ready on the 6-clause gate while /ready is 200', async () => {
+ const detailRef = {
+ current: authoritativeDetail({ safetyInventoryCertified: false }),
+ }
+ server = mainLikeServer(detailRef)
+ server.setReady(true)
+
+ const ready = await invoke(server, '/ready')
+ const health = await invoke(server, '/health')
+ expect(ready.statusCode).toBe(200)
+ expect(JSON.parse(ready.body)).toEqual({ status: 'ready', ready: true })
+ expect(health.statusCode).toBe(200)
+ expect(JSON.parse(health.body)).toEqual({ status: 'ok', ready: false })
+ })
})
diff --git a/host-context-controller/src/server.ts b/host-context-controller/src/server.ts
index 71e3bcb3e..b75a14a14 100644
--- a/host-context-controller/src/server.ts
+++ b/host-context-controller/src/server.ts
@@ -81,6 +81,8 @@ export class McpHostApiRateLimiter {
}
}
+type ReadinessLane = 'probe' | 'request'
+
type ReadinessState =
| { ready: true; status: 'ready' }
| {
@@ -162,74 +164,82 @@ export class ContextMapperServer {
this.ready = ready
}
- private collectReadinessReasons(): ReadinessReason[] | undefined {
+ private collectLaneReasons(lane: ReadinessLane): ReadinessReason[] | undefined {
if (!this.readinessDetailFn) return undefined
try {
- return readinessReasonsFromDetail(this.readinessDetailFn())
+ const detail = this.readinessDetailFn()
+ return lane === 'probe'
+ ? probeReadinessReasonsFromDetail(detail)
+ : readinessReasonsFromDetail(detail)
} catch (err) {
- readinessLog.error('readiness detail check failed', { err })
- return undefined
- }
- }
-
- private collectProbeReadinessReasons(): ReadinessReason[] | undefined {
- if (!this.readinessDetailFn) return undefined
- try {
- return probeReadinessReasonsFromDetail(this.readinessDetailFn())
- } catch (err) {
- readinessLog.error('readiness detail check failed', { err })
+ readinessLog.error(
+ lane === 'probe' ? 'probe readiness detail check failed' : 'readiness detail check failed',
+ { err, lane }
+ )
return undefined
}
}
- private emitReadinessTransition(
- state: ReadinessState,
- keyHolder:
- | 'lastReadinessTransitionKey'
- | 'lastProbeReadinessTransitionKey' = 'lastReadinessTransitionKey'
- ): void {
+ private emitReadinessTransition(state: ReadinessState, lane: ReadinessLane): void {
+ const keyHolder =
+ lane === 'probe' ? 'lastProbeReadinessTransitionKey' : 'lastReadinessTransitionKey'
const reasons = !state.ready ? (state.reasons ?? []) : []
const key = `${state.ready}:${state.status}:${reasons.join(',')}`
if (key === this[keyHolder]) return
this[keyHolder] = key
if (state.ready) {
- readinessLog.info('readiness became ready')
+ readinessLog.info('readiness became ready', { lane })
return
}
- readinessLog.warn('readiness not ready', { status: state.status, reasons })
+ readinessLog.warn('readiness not ready', { lane, status: state.status, reasons })
}
- private getReadinessState(): ReadinessState {
+ private evaluateAuthoritativeLane(
+ lane: ReadinessLane,
+ authoritativeFn: () => boolean
+ ): ReadinessState {
if (!this.ready) {
const starting: ReadinessState = {
ready: false,
status: 'starting',
message: 'Context mapper is still starting',
}
- this.emitReadinessTransition(starting)
+ this.emitReadinessTransition(starting, lane)
return starting
}
try {
- if (this.providerAuthoritativeFn()) {
+ if (authoritativeFn()) {
const ready: ReadinessState = { ready: true, status: 'ready' }
- this.emitReadinessTransition(ready)
+ this.emitReadinessTransition(ready, lane)
return ready
}
} catch (err) {
- readinessLog.error('provider authority readiness check failed', { err })
+ readinessLog.error(
+ lane === 'probe'
+ ? 'probe authority readiness check failed'
+ : 'provider authority readiness check failed',
+ { err, lane }
+ )
}
const degraded: ReadinessState = {
ready: false,
status: 'degraded',
- message: 'Context mapper provider inventory is not authoritative',
- reasons: this.collectReadinessReasons(),
+ message:
+ lane === 'probe'
+ ? 'Context mapper watch inventory is not fresh'
+ : 'Context mapper provider inventory is not authoritative',
+ reasons: this.collectLaneReasons(lane),
}
- this.emitReadinessTransition(degraded)
+ this.emitReadinessTransition(degraded, lane)
return degraded
}
+ private getReadinessState(): ReadinessState {
+ return this.evaluateAuthoritativeLane('request', this.providerAuthoritativeFn)
+ }
+
/**
* Kubelet /ready state. When probeAuthoritativeFn is omitted, fall back to
* the 6-clause provider gate so omitting the 10th constructor argument
@@ -239,34 +249,7 @@ export class ContextMapperServer {
if (this.probeAuthoritativeFn === undefined) {
return this.getReadinessState()
}
- if (!this.ready) {
- const starting: ReadinessState = {
- ready: false,
- status: 'starting',
- message: 'Context mapper is still starting',
- }
- this.emitReadinessTransition(starting, 'lastProbeReadinessTransitionKey')
- return starting
- }
-
- try {
- if (this.probeAuthoritativeFn()) {
- const ready: ReadinessState = { ready: true, status: 'ready' }
- this.emitReadinessTransition(ready, 'lastProbeReadinessTransitionKey')
- return ready
- }
- } catch (err) {
- readinessLog.error('probe authority readiness check failed', { err })
- }
-
- const degraded: ReadinessState = {
- ready: false,
- status: 'degraded',
- message: 'Context mapper provider inventory is not authoritative',
- reasons: this.collectProbeReadinessReasons(),
- }
- this.emitReadinessTransition(degraded, 'lastProbeReadinessTransitionKey')
- return degraded
+ return this.evaluateAuthoritativeLane('probe', this.probeAuthoritativeFn)
}
private async handleRequest(req: http.IncomingMessage, res: http.ServerResponse): Promise<void> {
@@ -662,7 +645,7 @@ export class ContextMapperServer {
'GET /': 'This information page',
'GET /health': "Health check endpoint - returns { status: 'ok', ready: boolean }",
'GET /ready':
- 'Readiness endpoint - returns 200 after warm-up while provider inventory is authoritative',
+ 'Readiness endpoint - returns 200 after warm-up while watch inventory is fresh',
'GET /api/v1/mcpservers': 'List all McpServer resources',
'GET /api/v2/hosts/self/mcpservers': 'List MCP servers for the authenticated Host',
'POST /api/v2/hosts/self/mcpservers/credential':
diff --git a/scripts/e2e/e2e-hcc-mcp-context-readiness.sh b/scripts/e2e/e2e-hcc-mcp-context-readiness.sh
index d4b0e6664..2527cd075 100755
--- a/scripts/e2e/e2e-hcc-mcp-context-readiness.sh
+++ b/scripts/e2e/e2e-hcc-mcp-context-readiness.sh
@@ -165,9 +165,10 @@ wait_until_fast() {
while :; do
"$@" && return 0
rc=$?
- # rc=2 is a fatal, non-retryable ordering violation (e.g. 200 observed with
- # a stale policy still present): stop polling immediately so a later clean
- # poll cannot mask it. rc=1 is benign "not ready yet".
+ # rc=2 is a fatal, non-retryable ordering violation (e.g. identity-stable
+ # allow already gone when /ready first becomes 200): stop polling immediately
+ # so a later clean poll cannot mask it. rc=1 is benign "not ready yet".
+ # After A1, /ready 200 with a divergent stale allow is rc=1, not rc=2.
if [ "$rc" -eq 2 ]; then
echo "FATAL ordering violation while waiting for ${description}: ${HCC_READY_PROBE_DIAGNOSTIC:-unset}" >&2
return 2
@@ -877,13 +878,12 @@ safety_pass_metrics_certify() {
}
hcc_ready_after_stale_policy_revoked() {
- # A single poll that sees 200 WHILE the stale policy is still present is a
- # safety-contract violation and must be FATAL, never retried: retrying lets a
- # later poll (policy already revoked) turn the violation into a false PASS.
- # rc=2 signals that fatal ordering violation to wait_until_fast; rc=1 is a
- # benign "not ready yet, keep polling". The sub-0.5s window where revocation
- # races the probe inside one poll stays unobservable on THIS signal — the
- # third signal (M6, /metrics) covers it.
+ # A1: /ready is watch freshness only. 200 while a divergent policy is still
+ # present is the intended kubelet window, not a fatal ordering violation.
+ # This wait succeeds only when BOTH the probe is ready AND the stale allow
+ # is gone. rc=1 = retry. Do not return rc=2 from /ready+stale — that would
+ # abort the first poll after watches sync. The third signal (M6, /metrics)
+ # still proves an authoritative pass measured and revoked.
local policy_absent_before=0
external_egress_policy_absent && policy_absent_before=1
probe_new_hcc_ready_endpoint || return 1
@@ -893,8 +893,7 @@ hcc_ready_after_stale_policy_revoked() {
# this poll cannot adjudicate the order. Retry to adjudicate cleanly.
return 1
fi
- HCC_READY_PROBE_DIAGNOSTIC="HCC_READY_PROBE category=ready-200-with-stale-policy-present"
- return 2
+ return 1
}
# Canonical, order-stable projection of the single external-egress policy for the
diff --git a/scripts/tests/test-hcc-watch-recovery-gate.sh b/scripts/tests/test-hcc-watch-recovery-gate.sh
index 5a2620513..4f711a573 100755
--- a/scripts/tests/test-hcc-watch-recovery-gate.sh
+++ b/scripts/tests/test-hcc-watch-recovery-gate.sh
@@ -983,17 +983,17 @@ if (
eval "$hcc_ready_after_revoke_function"
rc=0
hcc_ready_after_stale_policy_revoked || rc=$?
- # A 200 observed WITH the stale policy present is a fatal ordering violation
- # (rc=2), not a benign retryable "not ready" (rc=1).
- [ "$rc" -eq 2 ] &&
- [ "$HCC_READY_PROBE_DIAGNOSTIC" = 'HCC_READY_PROBE category=ready-200-with-stale-policy-present' ]
+ # A1: /ready 200 WITH the stale policy present is the kubelet window.
+ # Retry (rc=1). rc=2 here would abort the first poll after watches sync.
+ [ "$rc" -eq 1 ]
); then
- pass "global readiness evidence flags a surviving divergent policy as a fatal rc=2 ordering violation"
+ pass "global readiness retries while a divergent policy still survives after /ready 200"
else
- fail "global readiness can self-confirm (or only soft-fail) while a divergent policy survives"
+ fail "global readiness treats /ready 200 with a surviving divergent policy as terminal"
fi
-# The loop must PROPAGATE the fatal rc=2 immediately, not retry until a later
-# clean poll turns the violation into a false PASS (the round-2 bug).
+# The loop must KEEP polling through that window and PASS only once the
+# stale policy is gone. wait_until_fast still treats rc=2 as fatal for the
+# identity-stable retain path; this predicate must not emit rc=2.
wait_until_fast_function="$(
sed -n '/^wait_until_fast() {$/,/^}$/p' "$MCP_READINESS_GATE"
)"
@@ -1001,21 +1001,19 @@ if (
HCC_READY_PROBE_DIAGNOSTIC=""
POLL=0
probe_new_hcc_ready_endpoint() { return 0; }
- # Stale policy present for the first polls, revoked only later. The buggy
- # retry loop would poll past the fatal first observation and PASS here.
+ # Stale for the first polls, revoked later. A1 must retry, not die on poll 1.
external_egress_policy_absent() { POLL=$((POLL + 1)); [ "$POLL" -gt 4 ]; }
eval "$hcc_ready_after_revoke_function"
eval "$wait_until_fast_function"
rc=0
- wait_until_fast 10 "rc=2 propagation" hcc_ready_after_stale_policy_revoked || rc=$?
- # rc must be the fatal 2, and it must have stopped inside the first predicate
- # poll (which calls external_egress_policy_absent twice) — POLL<3 proves no
- # retry occurred.
- [ "$rc" -eq 2 ] && [ "$POLL" -lt 3 ]
+ wait_until_fast 10 "A1 probe window retry" hcc_ready_after_stale_policy_revoked || rc=$?
+ # Each predicate call samples absence twice. Success needs POLL>4 so the
+ # before/after pair is both absent.
+ [ "$rc" -eq 0 ] && [ "$POLL" -gt 4 ]
); then
- pass "wait_until_fast propagates a fatal rc=2 on the first poll instead of retrying to a false PASS"
+ pass "wait_until_fast retries /ready 200 with a surviving policy until revocation"
else
- fail "wait_until_fast retried past a fatal ordering violation to a later clean poll"
+ fail "wait_until_fast no longer waits through the A1 probe window"
fi
# M6: the third contract signal adjudicates a real /metrics scrape. Extract the