Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(control-api): refuse a remote install whose asProfile the apiserver pruned (BR1-M2)

Diego Zacariasb001d28925c9
diff --git a/control-api/package-lock.json b/control-api/package-lock.json
index 6b5e83041..ba7fd77ac 100644
--- a/control-api/package-lock.json
+++ b/control-api/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-control-api",
- "version": "0.1.811",
+ "version": "0.1.812",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-control-api",
- "version": "0.1.811",
+ "version": "0.1.812",
"dependencies": {
"@clerum/display-field": "file:../packages/display-field",
"@clerum/grok-provider-attempt-contract": "file:../packages/grok-provider-attempt-contract",
diff --git a/control-api/package.json b/control-api/package.json
index 6f4277d55..45819e1dc 100644
--- a/control-api/package.json
+++ b/control-api/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-control-api",
- "version": "0.1.811",
+ "version": "0.1.812",
"description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets",
"main": "dist/main.js",
"engines": {
diff --git a/control-api/src/oauth/mcpServerOAuthSpec.ts b/control-api/src/oauth/mcpServerOAuthSpec.ts
index 0de071225..8efd190a2 100644
--- a/control-api/src/oauth/mcpServerOAuthSpec.ts
+++ b/control-api/src/oauth/mcpServerOAuthSpec.ts
@@ -303,6 +303,17 @@ function remoteOAuthSpecIncoherence(
) {
return 'as_profile_issuer_mismatch'
}
+ // The converse: the install always pins the profile its issuer is recognised from, so
+ // a tenant issuer without one was written outside it, typically by an apiserver whose
+ // CRD predates the field and pruned it. On the generic wire that issuer either rejects
+ // the exchange or picks the audience from the browser's scopes.
+ if (
+ oauth.asProfile === undefined &&
+ typeof oauth.issuer === 'string' &&
+ detectAsProfile(oauth.issuer) !== null
+ ) {
+ return 'as_profile_issuer_mismatch'
+ }
// Without `resource` the scopes alone pick the token's audience, and the scopes or the
// base URL can be edited after the install (API or kubectl). The hard audience rules
// are therefore re-checked against the base URL in force now: a known API's GUID scope
@@ -359,7 +370,9 @@ function asEndpointsOnIssuerSite(oauth: McpServerOAuthDecl): boolean {
/**
* Runtime mirror of the remote-carril coherence rules of the McpServer CRD
* (REMOTE-SECRET-PAIRING for `public`, REMOTE-BEARER-HEADER, AS-PROFILE-ISSUER), of the
- * install's scope-audience rules for an authorization-server profile, and of the install's
+ * install's profile pinning (a recognised issuer always carries its profile; the CRD does
+ * not require it, and an older CRD prunes the field), of the install's
+ * scope-audience rules for an authorization-server profile, and of the install's
* per-server callback rules (no CIMD client and same-site AS endpoints when the AS
* does not return `iss`, which the CRD cannot express). Throws
* {@link RemoteOAuthSpecIncoherentError} for a remote server whose shape no
diff --git a/control-api/src/routes/admin/remoteMcp.ts b/control-api/src/routes/admin/remoteMcp.ts
index 02cd0d951..2105e7b2b 100644
--- a/control-api/src/routes/admin/remoteMcp.ts
+++ b/control-api/src/routes/admin/remoteMcp.ts
@@ -258,6 +258,12 @@ export function effectiveRemoteScopes(result: DiscoveryResult, override?: string
return [...new Set(scopes)]
}
+/** `spec.oauth.asProfile` of the object the apiserver returned for a write. */
+function persistedAsProfile(persisted: unknown): unknown {
+ const spec = (persisted as { spec?: { oauth?: { asProfile?: unknown } } } | null)?.spec
+ return spec?.oauth?.asProfile
+}
+
export function buildRemoteOAuthSpec(
result: DiscoveryResult,
opts: {
@@ -1412,6 +1418,7 @@ export function createAdminRemoteMcpRouter(
let createdClientSecretSnapshot: SecretSnapshot | null = null
let createdServerPreconditions: ResourcePreconditions | undefined
let createdServerUid: string | undefined
+ let createdServer: unknown
if (effectiveClientMode === 'confidential' && clientSecretName) {
try {
createdClientSecretSnapshot = await gateway.createSecret({
@@ -1445,7 +1452,7 @@ export function createAdminRemoteMcpRouter(
// ── Saga step 2: create the McpServer CR (rollback Secret on failure) ──
try {
- const createdServer = await gateway.createResource(
+ createdServer = await gateway.createResource(
'mcpservers',
{ metadata: { name: serverName, labels: managedLabels }, spec: mcpServerSpec },
targetNs
@@ -1488,28 +1495,86 @@ export function createAdminRemoteMcpRouter(
)
}
+ /** Fenced delete of the CR this install created; false when it could not be removed. */
+ const rollbackCreatedServer = async (): Promise<boolean> => {
+ try {
+ await gateway.deleteResource(
+ 'mcpservers',
+ serverName,
+ targetNs,
+ createdServerPreconditions
+ )
+ await waitForDeletion(
+ () => gateway.getResource('mcpservers', serverName, targetNs),
+ `McpServer/${serverName}`
+ )
+ return true
+ } catch {
+ // Best-effort rollback; preserve the reported failure.
+ return false
+ }
+ }
+
+ // ── Saga step 2a: the profile must survive the write ──────────────────
+ // An apiserver whose McpServer CRD predates `asProfile` prunes it and still answers
+ // 201. The CR would keep the Entra issuer on the generic wire (RFC 8707 `resource`,
+ // no scope on exchange and refresh), and since the profile's presence is immutable
+ // under the newer CRD, applying it later cannot repair the object: it must not stay.
+ const requestedAsProfile = oauthSpec.asProfile
+ if (
+ requestedAsProfile !== undefined &&
+ persistedAsProfile(createdServer) !== requestedAsProfile
+ ) {
+ const serverRemoved = await rollbackCreatedServer()
+ let secretRemoved = true
+ if (createdClientSecretSnapshot && clientSecretName) {
+ try {
+ await gateway.deleteSecret(clientSecretName, targetNs, {
+ uid: createdClientSecretSnapshot.uid,
+ resourceVersion: createdClientSecretSnapshot.resourceVersion,
+ })
+ } catch {
+ secretRemoved = false
+ }
+ }
+ await rollbackDynamicClient()
+ reqLog.warn(
+ {
+ event: 'remote_install_crd_outdated',
+ serverName,
+ namespace: targetNs,
+ field: 'spec.oauth.asProfile',
+ },
+ 'remote install refused: the apiserver pruned spec.oauth.asProfile'
+ )
+ if (!serverRemoved || !secretRemoved) {
+ reqLog.error(
+ {
+ event: 'remote_install_crd_outdated_rollback_failed',
+ serverName,
+ namespace: targetNs,
+ serverRemoved,
+ secretRemoved,
+ },
+ 'remote install rollback after a pruned asProfile left objects behind'
+ )
+ }
+ res.status(409).json({
+ error: 'mcpserver_crd_outdated',
+ code: 'mcpserver_crd_outdated',
+ message:
+ 'McpServer spec.oauth.asProfile was not persisted by Kubernetes. ' +
+ 'Apply the latest clerum-crds before installing this server.',
+ })
+ return
+ }
+
// ── Saga step 2b (DCR only): bind the pending row to the created CR's uid ──
// Only DCR wrote a dynamic_clients row; other modes skip this. Binding seals the
// row to the McpServer's metadata.uid so a later teardown/reclaim can fence by the
// exact installation. A missing uid or a lost bind means the row is no longer ours
// — compensate the CR (fenced) and this install rather than leave a dangling row.
if (body.mode === 'dcr' && dcrRegistered) {
- const rollbackCreatedServer = async (): Promise<void> => {
- try {
- await gateway.deleteResource(
- 'mcpservers',
- serverName,
- targetNs,
- createdServerPreconditions
- )
- await waitForDeletion(
- () => gateway.getResource('mcpservers', serverName, targetNs),
- `McpServer/${serverName}`
- )
- } catch {
- // Best-effort rollback; preserve the reported failure.
- }
- }
if (createdServerUid === undefined) {
// Without a uid the row can never be bound (it would strand as pending and be
// reclaimed by the TTL). Compensate now instead of leaving that residue.
diff --git a/control-api/test/fixtures/perServerInstall.ts b/control-api/test/fixtures/perServerInstall.ts
index e1fde81f0..2bce25d78 100644
--- a/control-api/test/fixtures/perServerInstall.ts
+++ b/control-api/test/fixtures/perServerInstall.ts
@@ -93,6 +93,8 @@ export async function installRemoteServer(opts: {
discovery: DiscoveryResult
body: Record<string, unknown>
registration?: Record<string, unknown>
+ /** Stands in for the correlation middleware's `req.log`, for suites that assert logs. */
+ reqLog?: unknown
}): Promise<InstallOutcome> {
const { transport, calls } = makeDcrTransport({
registrationEndpoint: opts.discovery.endpoints.registration,
@@ -101,6 +103,12 @@ export async function installRemoteServer(opts: {
})
const app = express()
app.use(express.json())
+ if (opts.reqLog) {
+ app.use((req, _res, next) => {
+ ;(req as { log?: unknown }).log = opts.reqLog
+ next()
+ })
+ }
app.use(
createAdminRemoteMcpRouter(opts.gateway as unknown as K8sGateway, {
db: opts.db,
diff --git a/control-api/test/oauth.remoteSpecCoherence.test.ts b/control-api/test/oauth.remoteSpecCoherence.test.ts
index e3494567a..113dcb858 100644
--- a/control-api/test/oauth.remoteSpecCoherence.test.ts
+++ b/control-api/test/oauth.remoteSpecCoherence.test.ts
@@ -127,6 +127,12 @@ function workIqWithScopes(scopes: string[]): Record<string, unknown> {
return { ...workIqGuid.spec.oauth, scopes }
}
+/** The installed Work IQ `spec.oauth` minus the profile, as an outdated CRD persists it. */
+function workIqWithoutProfile(): Record<string, unknown> {
+ const { asProfile: _pruned, ...oauth } = workIqGuid.spec.oauth
+ return oauth
+}
+
/** A Notion pre-registered CR with the profile stamped on it: no producer, a direct write. */
function profileOnNotionOAuth(): Record<string, unknown> {
const oauth = buildRemoteOAuthSpec(notionDiscovery, {
@@ -271,6 +277,14 @@ const INCOHERENT: ReadonlyArray<{
reason: 'as_profile_issuer_mismatch',
consentOnly: false,
},
+ {
+ // What an apiserver with a CRD older than the field leaves of a real Work IQ install.
+ label: 'Entra tenant issuer without asProfile (pruned by an outdated CRD)',
+ oauth: workIqWithoutProfile,
+ remote: workIqRemote,
+ reason: 'as_profile_issuer_mismatch',
+ consentOnly: false,
+ },
// Hard audience rules of the profile, against the base URL in force.
{
label: 'entra-v2 with the Microsoft Graph GUID scope',
diff --git a/control-api/test/routes.adminRemoteMcp.crdOutdated.test.ts b/control-api/test/routes.adminRemoteMcp.crdOutdated.test.ts
new file mode 100644
index 000000000..3bef962cb
--- /dev/null
+++ b/control-api/test/routes.adminRemoteMcp.crdOutdated.test.ts
@@ -0,0 +1,224 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
+import { config } from '../src/config.js'
+import { remoteWireForProfile } from '../src/oauth/asProfiles.js'
+import type { DiscoveryResult } from '../src/oauth/discovery.js'
+import {
+ WORKIQ_GUID_SCOPE,
+ discoverPilot,
+ installRemoteServer,
+ seedContext,
+ withRfc9207,
+} from './fixtures/perServerInstall.js'
+import {
+ ENTRA_TENANT_AUTHORITY,
+ ENTRA_UNKNOWN_GUID_PILOT,
+ ENTRA_WORKIQ_PILOT,
+ PILOTS,
+ type PilotFixture,
+ makeInMemoryDynamicClientsDb,
+} from './fixtures/remoteOAuthDiscovery.js'
+import { MockGateway } from './mockGateway.js'
+
+/**
+ * A remote install against an apiserver whose McpServer CRD predates `spec.oauth.asProfile`.
+ * Such an apiserver prunes the unknown field and still answers 201, so the gateway below
+ * stores and returns exactly what the route sent minus that one field (T1: the object is
+ * derived from the route's own write, never hand-authored). Discovery runs the real client
+ * over the recorded Entra probe fixtures.
+ */
+
+vi.mock('node:dns/promises', () => ({
+ lookup: vi.fn(async () => [{ address: '93.184.216.34', family: 4 }]),
+}))
+
+const NS = config.mcpServersNamespace
+const CONTEXT = 'ctx-a'
+const UNKNOWN_GUID_SCOPE = '11111111-2222-3333-4444-555555555555/Tools.Read'
+
+/** The apiserver of an outdated CRD: `spec.oauth.asProfile` does not survive a create. */
+type CreateArgs = Parameters<MockGateway['createResource']>
+
+class PruningGateway extends MockGateway {
+ override async createResource(
+ plural: CreateArgs[0],
+ body: CreateArgs[1],
+ namespace?: CreateArgs[2]
+ ): Promise<unknown> {
+ if (plural !== 'mcpservers') return super.createResource(plural, body, namespace)
+ const pruned = structuredClone(body) as CreateArgs[1] & {
+ spec?: { oauth?: Record<string, unknown> }
+ }
+ delete pruned.spec?.oauth?.asProfile
+ return super.createResource(plural, pruned, namespace)
+ }
+}
+
+let savedCallbackBaseUrl: string
+beforeEach(() => {
+ savedCallbackBaseUrl = config.oauthCallbackBaseUrl
+ config.oauthCallbackBaseUrl = 'https://control.example.com'
+})
+afterEach(() => {
+ config.oauthCallbackBaseUrl = savedCallbackBaseUrl
+})
+
+async function pruningGateway(): Promise<PruningGateway> {
+ const gw = new PruningGateway(NS)
+ await seedContext(gw, NS, CONTEXT)
+ return gw
+}
+
+function entraDiscovery(pilot: PilotFixture = ENTRA_WORKIQ_PILOT): Promise<DiscoveryResult> {
+ return discoverPilot(pilot, {
+ hasPreRegisteredClient: true,
+ authorizationServer: ENTRA_TENANT_AUTHORITY,
+ })
+}
+
+function preRegisteredBody(
+ pilot: PilotFixture,
+ extra: Record<string, unknown> = {}
+): Record<string, unknown> {
+ return {
+ serverName: 'workiq',
+ contextRef: CONTEXT,
+ baseUrl: pilot.mcpUrl,
+ mode: 'pre-registered',
+ clientId: '11111111-2222-3333-4444-555555555555',
+ clientSecret: 'entra-client-secret',
+ authorizationServer: ENTRA_TENANT_AUTHORITY,
+ scopes: [WORKIQ_GUID_SCOPE],
+ ...extra,
+ }
+}
+
+const CRD_OUTDATED = {
+ error: 'mcpserver_crd_outdated',
+ code: 'mcpserver_crd_outdated',
+}
+
+describe('remote install on an apiserver that prunes spec.oauth.asProfile', () => {
+ it('the profile is what makes the wire Entra-specific (why losing it matters)', async () => {
+ const discovery = await entraDiscovery()
+ expect(discovery.asProfile).toBe('entra-v2')
+ expect(remoteWireForProfile(discovery.asProfile)).not.toEqual(remoteWireForProfile(undefined))
+ })
+
+ it('pre-registered Entra → 409 mcpserver_crd_outdated; the fenced CR and Secret are removed', async () => {
+ const gw = await pruningGateway()
+ const created: Array<{ uid?: string; resourceVersion?: string }> = []
+ const createResource = gw.createResource.bind(gw)
+ vi.spyOn(gw, 'createResource').mockImplementation(async (plural, body, ns) => {
+ const out = (await createResource(plural, body, ns)) as {
+ metadata: { uid?: string; resourceVersion?: string }
+ }
+ if (plural === 'mcpservers') created.push({ ...out.metadata })
+ return out
+ })
+ const createSecret = gw.createSecret.bind(gw)
+ const secrets: Array<{ uid: string; resourceVersion: string }> = []
+ vi.spyOn(gw, 'createSecret').mockImplementation(async (req, opts) => {
+ const snap = await createSecret(req, opts)
+ secrets.push({ uid: snap.uid, resourceVersion: snap.resourceVersion })
+ return snap
+ })
+ const deleteResource = vi.spyOn(gw, 'deleteResource')
+ const deleteSecret = vi.spyOn(gw, 'deleteSecret')
+
+ const res = await installRemoteServer({
+ gateway: gw,
+ db: makeInMemoryDynamicClientsDb().db,
+ discovery: await entraDiscovery(),
+ body: preRegisteredBody(ENTRA_WORKIQ_PILOT),
+ })
+
+ expect(res.status).toBe(409)
+ expect(res.body).toMatchObject(CRD_OUTDATED)
+ expect(created).toHaveLength(1)
+ expect(deleteResource).toHaveBeenCalledWith('mcpservers', 'workiq', NS, {
+ uid: created[0].uid,
+ resourceVersion: created[0].resourceVersion,
+ })
+ expect(secrets).toHaveLength(1)
+ expect(deleteSecret).toHaveBeenCalledWith('workiq-oauth-client', NS, secrets[0])
+ await expect(gw.getResource('mcpservers', 'workiq', NS)).rejects.toThrow()
+ await expect(gw.getSecret('workiq-oauth-client', NS)).rejects.toThrow()
+ // Nothing was attached to the Context either.
+ const ctx = (await gw.getResource('contexts', CONTEXT, NS)) as {
+ spec: { mcpServers?: unknown[] }
+ }
+ expect(ctx.spec.mcpServers ?? []).toEqual([])
+ })
+
+ it('a failed rollback still answers 409 and names what was left behind', async () => {
+ const gw = await pruningGateway()
+ vi.spyOn(gw, 'deleteResource').mockRejectedValue(new Error('apiserver unavailable'))
+ const reqLog = { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }
+ const res = await installRemoteServer({
+ gateway: gw,
+ db: makeInMemoryDynamicClientsDb().db,
+ discovery: await entraDiscovery(),
+ body: preRegisteredBody(ENTRA_WORKIQ_PILOT),
+ reqLog,
+ })
+ expect(res.status).toBe(409)
+ expect(res.body).toMatchObject(CRD_OUTDATED)
+ expect(reqLog.error).toHaveBeenCalledWith(
+ expect.objectContaining({
+ event: 'remote_install_crd_outdated_rollback_failed',
+ serverName: 'workiq',
+ serverRemoved: false,
+ secretRemoved: true,
+ }),
+ expect.any(String)
+ )
+ })
+
+ it('the confirmed-unverified-scopes audit record is kept even when the install is refused', async () => {
+ // The CR existed, however briefly, and a rollback can fail: the record of who vouched
+ // for the audience must not depend on the install completing.
+ const gw = await pruningGateway()
+ const reqLog = { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }
+ const res = await installRemoteServer({
+ gateway: gw,
+ db: makeInMemoryDynamicClientsDb().db,
+ discovery: await entraDiscovery(ENTRA_UNKNOWN_GUID_PILOT),
+ body: preRegisteredBody(ENTRA_UNKNOWN_GUID_PILOT, {
+ scopes: [UNKNOWN_GUID_SCOPE],
+ confirmedUnverifiedScopes: [UNKNOWN_GUID_SCOPE],
+ }),
+ reqLog,
+ })
+ expect(res.status).toBe(409)
+ expect(reqLog.info).toHaveBeenCalledWith(
+ expect.objectContaining({
+ event: 'remote_install_unverified_scopes_confirmed',
+ scopes: [UNKNOWN_GUID_SCOPE],
+ }),
+ expect.any(String)
+ )
+ })
+
+ it('an install without a profile is unaffected by the outdated CRD → 201', async () => {
+ const gw = await pruningGateway()
+ const discovery = await discoverPilot(withRfc9207(PILOTS.sentry), {
+ hasPreRegisteredClient: true,
+ })
+ expect(discovery.asProfile).toBeUndefined()
+ const res = await installRemoteServer({
+ gateway: gw,
+ db: makeInMemoryDynamicClientsDb().db,
+ discovery,
+ body: {
+ serverName: 'sentry',
+ contextRef: CONTEXT,
+ baseUrl: PILOTS.sentry.mcpUrl,
+ mode: 'pre-registered',
+ clientId: 'client-abc',
+ clientSecret: 'client-secret',
+ },
+ })
+ expect(res.status).toBe(201)
+ await expect(gw.getResource('mcpservers', 'sentry', NS)).resolves.toBeTruthy()
+ })
+})
diff --git a/docs/crds/mcpserver.md b/docs/crds/mcpserver.md
index 5c62fc4fb..c807eb131 100644
--- a/docs/crds/mcpserver.md
+++ b/docs/crds/mcpserver.md
@@ -253,9 +253,12 @@ API translates to this explicit `egressClass: public-web` CRD shape.
- **AS-PROFILE-ISSUER:** when `spec.oauth.asProfile` is set, `spec.oauth.issuer` must be a Microsoft Entra ID tenant authority, matching `^https://login[.]microsoftonline[.]com/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/v2[.]0$` (lower-case tenant GUID, no trailing slash). Otherwise the write is rejected with `spec.oauth.asProfile entra-v2 requires spec.oauth.issuer to be a Microsoft Entra ID tenant authority`. A profile stops sending the RFC 8707 `resource`, so it must stay on the issuer it was recognised from. control-api applies the same check at runtime and refuses an object written before the rule was in the cluster (`409 remote_oauth_spec_incoherent`, reason `as_profile_issuer_mismatch`).
> **Apply the CRD before control-api.** An apiserver whose McpServer CRD does not
-> declare `asProfile` drops the field on write and still answers 200. The server
-> is then installed without the profile: control-api sends the RFC 8707
-> `resource`, which Entra rejects. See
+> declare `asProfile` drops the field on write and still answers 200. The install
+> route detects this, removes what it created and answers `409 mcpserver_crd_outdated`.
+> An object written another way (`kubectl`, GitOps) keeps the Entra issuer without
+> the profile; control-api refuses it at runtime (`409 remote_oauth_spec_incoherent`,
+> reason `as_profile_issuer_mismatch`), and because the profile cannot be added after
+> creation it must be deleted and recreated once the CRD is applied. See
> [Deploy order for additive schema fields](../../charts/clerum-crds/README.md#deploy-order-for-additive-schema-fields).
## Example
diff --git a/docs/how-to/connect-oauth-connector.md b/docs/how-to/connect-oauth-connector.md
index 093c522ee..076cdd7dc 100644
--- a/docs/how-to/connect-oauth-connector.md
+++ b/docs/how-to/connect-oauth-connector.md
@@ -500,6 +500,7 @@ out too, but the screen only tracks the current one.
| Install asks to confirm scopes (`scope_audience_unverified`) | A GUID scope names an API the platform cannot tie to the server | Confirm only if that API belongs to the server; otherwise change the scope |
| `409 oauth_client_id_in_use` | The app registration is already used by another server | Create a separate app registration for each server |
| Connect or token requests fail with `409 remote_oauth_spec_incoherent` | The McpServer was edited and breaks a rule (reason `scope_audience_mismatch` or `as_profile_issuer_mismatch`) | Fix `spec.oauth.scopes` or `spec.remote.baseUrl`, or reinstall the server |
+| Install returns `409 mcpserver_crd_outdated` | The cluster's McpServer CRD predates `spec.oauth.asProfile`, so Kubernetes dropped it | Apply the latest clerum-crds, then install again |
## Related