Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(np08): raisable HCC staleness ceiling so HCC rollout doesn't drop MCP fleet (#425)

Alfredo Lopez9497625d8c62
diff --git a/deploy/overlays/minikube/configmaps/mcp-host-config.yaml b/deploy/overlays/minikube/configmaps/mcp-host-config.yaml
index e9f55075b..ebb02bb04 100644
--- a/deploy/overlays/minikube/configmaps/mcp-host-config.yaml
+++ b/deploy/overlays/minikube/configmaps/mcp-host-config.yaml
@@ -11,6 +11,10 @@ data:
CLERUM_CONTEXT_MAPPER_URL: 'http://host-context-controller-api-gateway.control-plane.svc.cluster.local:8081'
CLERUM_CONTEXT_MAPPER_POLL_INTERVAL: '30000'
MCP_PROXY_ENABLED: 'false'
+ # e2e lane pin: 60s keeps the revoke-on-staleness scenario fast and
+ # deterministic. Production defaults to 180s (bounded 600s) in
+ # mcp-host/src/config.ts so a normal HCC rollout (~77s in clerum-dev) does not
+ # tear down the MCP fleet. See issue #425.
HCC_AUTHORITY_MAX_STALENESS_MS: '60000'
MCP_PROXY_URL: 'http://mcp-proxy.mcp-server.svc.cluster.local:8083'
CLERUM_AGENT_TASK_DELAY: '3'
diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md
index 00cb1f8ab..7ea628254 100644
--- a/docs/architecture/overview.md
+++ b/docs/architecture/overview.md
@@ -766,10 +766,13 @@ HTTP 410; the remaining global v1 inventory is metadata-only transitional
surface for the separately reviewed mcp-proxy PR2 and is not used by mcp-host.
Polling interval: 30 seconds (configurable via `CLERUM_CONTEXT_MAPPER_POLL_INTERVAL`).
-In cluster mode it must remain strictly below the 60-second
-`HCC_AUTHORITY_MAX_STALENESS_MS` ceiling; mcp-host rejects an invalid relation at
-startup rather than allowing the fleet to flap between successful polls and
-authority revocation.
+In cluster mode it must remain strictly below `HCC_AUTHORITY_MAX_STALENESS_MS`
+(default 180 s, operator-tunable up to a 600 s hard ceiling); mcp-host rejects an
+invalid relation at startup rather than allowing the fleet to flap between
+successful polls and authority revocation. The staleness window governs only the
+`unavailable` failure class (HCC unreachable / 5xx): it is sized to absorb a
+normal HCC `Recreate` rollout without tearing down the MCP fleet. Identity/401
+revocations are immediate and are not gated by this window (issue #425).
On each poll, mcp-host compares new server list with previous state and:
@@ -822,7 +825,7 @@ The HTTP response is held open until the agent finishes processing. The response
| `CLERUM_SERVER_PORT` | `8080` | HTTP server port |
| `CLERUM_CONTEXT_MAPPER_URL` | auto | Context-mapper service URL |
| `CLERUM_CONTEXT_MAPPER_POLL_INTERVAL` | `30000` | Poll interval (ms) |
-| `HCC_AUTHORITY_MAX_STALENESS_MS` | `60000` | Maximum retained HCC authority (ms); must be greater than the poll interval in cluster mode |
+| `HCC_AUTHORITY_MAX_STALENESS_MS` | `180000` | Max retained HCC authority while HCC is unreachable (ms); operator-tunable, hard-capped at 600000; must be greater than the poll interval in cluster mode (issue #425) |
| `CLERUM_MODEL_PROVIDER` | auto-detected | `"openai"`, `"claude"`, `"zai"`, or `"bailian"` |
| `CLERUM_MODEL_NAME` | provider default | Specific model name |
| `OPENAI_API_KEY` | - | OpenAI key (dev mode) |
diff --git a/mcp-host/package-lock.json b/mcp-host/package-lock.json
index 908000646..96f47cfe4 100644
--- a/mcp-host/package-lock.json
+++ b/mcp-host/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-mcp-host",
- "version": "0.1.275",
+ "version": "0.1.276",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-mcp-host",
- "version": "0.1.275",
+ "version": "0.1.276",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@anthropic-ai/sdk": "^0.32.0",
diff --git a/mcp-host/package.json b/mcp-host/package.json
index 9e8a37131..feb6bd6ea 100644
--- a/mcp-host/package.json
+++ b/mcp-host/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-mcp-host",
- "version": "0.1.275",
+ "version": "0.1.276",
"description": "MCP Host - reads Host CRD and provides LLM access via OpenAI/Claude",
"main": "dist/main.js",
"engines": {
diff --git a/mcp-host/src/config.hccAuthority.test.ts b/mcp-host/src/config.hccAuthority.test.ts
index 0296814d9..eac421207 100644
--- a/mcp-host/src/config.hccAuthority.test.ts
+++ b/mcp-host/src/config.hccAuthority.test.ts
@@ -23,20 +23,24 @@ afterEach(() => {
})
describe('HCC_AUTHORITY_MAX_STALENESS_MS', () => {
- it('defaults to the 60-second authority ceiling', async () => {
- await expect(loadAuthorityMaxStaleness(undefined)).resolves.toBe(60_000)
+ it('defaults to the 180-second rollout-tolerant window', async () => {
+ await expect(loadAuthorityMaxStaleness(undefined)).resolves.toBe(180_000)
})
it('allows an operator to fail closed sooner', async () => {
await expect(loadAuthorityMaxStaleness('15000', '5000')).resolves.toBe(15_000)
})
- it('cannot be configured beyond the 60-second authority ceiling', async () => {
- await expect(loadAuthorityMaxStaleness('120000')).resolves.toBe(60_000)
+ it('allows an operator to raise the window for a slower-starting cluster (issue #425)', async () => {
+ await expect(loadAuthorityMaxStaleness('300000')).resolves.toBe(300_000)
})
- it('uses the safe ceiling for an invalid value', async () => {
- await expect(loadAuthorityMaxStaleness('not-a-duration')).resolves.toBe(60_000)
+ it('cannot be configured beyond the 600-second hard ceiling', async () => {
+ await expect(loadAuthorityMaxStaleness('700000')).resolves.toBe(600_000)
+ })
+
+ it('uses the default window for an invalid value', async () => {
+ await expect(loadAuthorityMaxStaleness('not-a-duration')).resolves.toBe(180_000)
})
it('rejects a poll interval that reaches the authority ceiling', async () => {
@@ -53,8 +57,8 @@ describe('HCC_AUTHORITY_MAX_STALENESS_MS', () => {
)
})
- it('accepts the default 30-second poll inside the 60-second ceiling', async () => {
+ it('accepts the default 30-second poll inside the default staleness window', async () => {
const { validateHccAuthorityTiming } = await import('./config')
- expect(() => validateHccAuthorityTiming(30_000, 60_000)).not.toThrow()
+ expect(() => validateHccAuthorityTiming(30_000, 180_000)).not.toThrow()
})
})
diff --git a/mcp-host/src/config.ts b/mcp-host/src/config.ts
index 71b4db83c..27fe2480d 100644
--- a/mcp-host/src/config.ts
+++ b/mcp-host/src/config.ts
@@ -579,9 +579,27 @@ const contextMapperPollInterval = parseInt(
getEnv('CLERUM_CONTEXT_MAPPER_POLL_INTERVAL', '30000')!,
10
)
+// Fail-closed window for an UNREACHABLE HCC (5xx / transport failures only).
+// It is bounded so a revoked-but-unconfirmable authority cannot linger forever,
+// yet generous enough to survive a normal HCC `Recreate` rollout (measured ~77s
+// in clerum-dev, with a startupProbe budget of up to 120s) without tearing down
+// the whole MCP fleet and aborting in-flight tool calls.
+//
+// Crucially, this window ONLY governs the `unavailable` failure class. Identity /
+// authorization failures (401/403 — Host UID change, revoked grant) revoke
+// immediately on the next reachable call and are NOT gated by this value, and a
+// grant change while HCC is healthy is applied by the next successful poll. So
+// raising this ceiling does not delay any real authority revocation and does not
+// widen the NP-08 credential-disclosure surface (which lives entirely in HCC's
+// per-request authorization). See issue #425.
+//
+// Operators may lower it, or raise it up to HCC_AUTHORITY_MAX_STALENESS_CEILING_MS
+// for slower-starting clusters. The minikube e2e lane pins 60000 explicitly so the
+// revoke-on-staleness scenario stays fast and deterministic.
+const HCC_AUTHORITY_MAX_STALENESS_CEILING_MS = 600_000 // 10 min hard upper bound
const hccAuthorityMaxStalenessMs = Math.min(
- getEnvNumber('HCC_AUTHORITY_MAX_STALENESS_MS', 60_000),
- 60_000
+ getEnvNumber('HCC_AUTHORITY_MAX_STALENESS_MS', 180_000),
+ HCC_AUTHORITY_MAX_STALENESS_CEILING_MS
)
// Dev mode does not retain a Kubernetes-authoritative MCP fleet, so the
diff --git a/scripts/tests/test-np08-render-contract.sh b/scripts/tests/test-np08-render-contract.sh
index 433da1cc0..a14487509 100755
--- a/scripts/tests/test-np08-render-contract.sh
+++ b/scripts/tests/test-np08-render-contract.sh
@@ -30,7 +30,11 @@ for rendered in "$@"; do
host_config = require_resource.call("ConfigMap", "mcp-host-config", "mcp-host")
abort("mcp-host must keep direct MCP transport") unless host_config.dig("data", "MCP_PROXY_ENABLED") == "false"
- abort("HCC authority staleness must stay bounded at 60 seconds") unless host_config.dig("data", "HCC_AUTHORITY_MAX_STALENESS_MS") == "60000"
+ # The minikube e2e lane pins staleness to 60s so the revoke-on-staleness
+ # scenario is fast and deterministic. Production defaults to 180s (bounded at
+ # 600s) in mcp-host/src/config.ts so a normal HCC rollout does not tear down
+ # the MCP fleet — see issue #425. This assertion guards the e2e-lane pin.
+ abort("HCC authority staleness must stay pinned to 60s on the minikube e2e lane") unless host_config.dig("data", "HCC_AUTHORITY_MAX_STALENESS_MS") == "60000"
hcc = require_resource.call("Deployment", "host-context-controller", "control-plane")
hcc_container = Array(hcc.dig("spec", "template", "spec", "containers")).find do |container|