Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(hcc): partition completed create outcomes

Alfredo Lopez744e585bb9b8
diff --git a/host-context-controller/CREATE_OBSERVABILITY.md b/host-context-controller/CREATE_OBSERVABILITY.md
index 2e6cf5599..9935135a5 100644
--- a/host-context-controller/CREATE_OBSERVABILITY.md
+++ b/host-context-controller/CREATE_OBSERVABILITY.md
@@ -12,42 +12,51 @@ ConfigMap, PersistentVolumeClaim, ServiceAccount, Role, RoleBinding,
PodDisruptionBudget, and Secret. No resource names, namespaces, user identifiers,
request bodies, or exception messages become metric labels.
-| Counter | Outcome | Meaning |
-| ---------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------- |
-| `clerum_hcc_creates_total` | `issued` | One invocation of a Kubernetes create operation, including failed attempts. |
-| | `conflict` | An issued create rejected with HTTP 409. This is a subset of `issued`, not an exclusive outcome. |
-| | `skipped` | Reserved for a future presence-based create suppression; remains zero in this instrumentation change. |
-| `clerum_hcc_existence_reads_total` | `found` | An instrumented existence read resolved successfully. The wrapper does not validate or alter its result. |
-| | `absent` | An instrumented read rejected with HTTP 404; the original error still reaches the caller. |
-| | `error` | An instrumented read rejected for any other reason, including HTTP 409. |
-
-The bounded series are initialized at zero. Re-importing metrics must preserve
-the registry and existing values. A zero series alone is not proof that a path
-executed. Do not sum create outcomes to calculate request count.
-
-`issued - conflict` is **not successful creates**: it includes other failures
-(403, 5xx, network errors) and operations that have not settled. Starts and
-responses can also fall in different observation windows. A successful-create
-count needs matching audit success events and resource/business evidence; this
-counter does not supply it by subtraction.
-
-Select the outcome explicitly in queries. For a single scoped environment:
+| Counter | Outcome | Meaning |
+| ---------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------- |
+| `clerum_hcc_creates_total` | `created` | The create callback resolved successfully. |
+| | `conflict` | The create callback rejected with HTTP 409. |
+| | `error` | The create callback rejected for any other reason, including 403, 5xx, network errors, or synchronous client failure. |
+| | `skipped` | Reserved for presence-based suppression in the read-first stage; remains zero in this PR. |
+| `clerum_hcc_existence_reads_total` | `found` | An instrumented existence read resolved successfully; its result is not validated or altered. |
+| | `absent` | An instrumented read rejected with HTTP 404; the original error still reaches the caller. |
+| | `error` | An instrumented read rejected for any other reason, including HTTP 409. |
+
+Create outcomes are mutually exclusive and increment only after the callback
+settles. The former `issued` series is removed. Summing `created + conflict + error`
+counts completed attempts; `skipped` is not an invocation and is excluded from that
+sum. Operations still in flight are not counted. A create resolving to null or
+undefined still counts as resolved; telemetry does not assert object persistence.
+
+The bounded series are initialized at zero: 40 create series and 30 read series.
+Re-importing metrics must preserve the registry and existing values. A zero series
+alone is not proof that a path executed. For a single scoped environment:
```promql
-sum by (kind) (increase(clerum_hcc_creates_total{outcome="issued"}[5m]))
+sum by (kind) (increase(clerum_hcc_creates_total{outcome=~"created|conflict|error"}[5m]))
+sum by (kind) (increase(clerum_hcc_creates_total{outcome="created"}[5m]))
sum by (kind) (increase(clerum_hcc_creates_total{outcome="conflict"}[5m]))
+sum by (kind) (increase(clerum_hcc_creates_total{outcome="error"}[5m]))
```
-Add the actual environment/target filters from the monitoring configuration
-before comparing a deployment. These queries report attempts and conflicts,
-not successful creations or a count of all completed operations.
+Add the actual environment/target filters from the monitoring configuration.
+These are client outcomes, not proof that every operation reached the API server;
+a lost response can reject even if the server created an object. Validate server
+outcomes and resource/business state separately when that distinction matters.
+
+**Accepted trade-off:** if the process dies before a callback settles, that attempt
+has no outcome sample. The previous pre-call increment could have been observed
+before the crash. This loss is accepted in favor of an unambiguous partition;
+server audit events can provide additional evidence for requests that reached the
+server, but cannot reconstruct every client invocation. No crash-loss rate is
+claimed or measured here. This contract is changed before downstream dashboards
+and the read-first acceptance depend on it.
`observeCreate` and `observeExistenceRead` invoke their callbacks once and preserve
the returned value or rejected value. They do not swallow a 404, turn it into
`null`, retry a failed request, or convert an error into a create. Existing caller
-catch behavior remains responsible for those decisions. Synchronous client
-failures count as attempts/outcomes too, so these counters are client-operation
-observations rather than proof that every attempt reached the API server.
+catch behavior remains responsible for those decisions. Synchronous failures
+count as `error` when the wrapper catches them; the original rejection is retained.
## Coverage
@@ -124,7 +133,7 @@ command exits and retained-row counts; missing data is unknown, not zero. Local
tests and a historical re-query do not replace simultaneous deployed validation.
For create conflicts, compare the `conflict` subset with the audit's conflict
-result, not with all issued operations. The audit uses `google.rpc.Code=10`,
+result, not with all completed create outcomes. The audit uses `google.rpc.Code=10`,
which represents conflict, while client errors use HTTP 409. Historical audit
data did not contain GET records; no API-server read rate is inferred from that
absence. The read counter only covers the explicit inventory above.
diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.json
index 7d1d15364..aff8312f9 100644
--- a/host-context-controller/package-lock.json
+++ b/host-context-controller/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.301",
+ "version": "0.1.302",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-host-context-controller",
- "version": "0.1.301",
+ "version": "0.1.302",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"@clerum/codex-catalog-projection": "file:../packages/codex-catalog-projection",
diff --git a/host-context-controller/package.json b/host-context-controller/package.json
index 808f1a816..9a482303d 100644
--- a/host-context-controller/package.json
+++ b/host-context-controller/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-host-context-controller",
- "version": "0.1.301",
+ "version": "0.1.302",
"description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API",
"main": "dist/main.js",
"engines": {
diff --git a/host-context-controller/src/createMetrics.test.ts b/host-context-controller/src/createMetrics.test.ts
index f72c72086..1ac18eb60 100644
--- a/host-context-controller/src/createMetrics.test.ts
+++ b/host-context-controller/src/createMetrics.test.ts
@@ -15,8 +15,13 @@ async function count(kind: string, outcome: string): Promise<number> {
it('initializes create samples in the real registry before any test seeding', async () => {
expect(registry.getSingleMetric('clerum_hcc_creates_total')).toBe(createsTotal)
+ const snapshot = await createsTotal.get()
+ expect(snapshot.values).toHaveLength(CREATE_KINDS.length * 4)
+ expect(new Set(snapshot.values.map(sample => sample.labels.outcome))).toEqual(
+ new Set(['created', 'conflict', 'error', 'skipped'])
+ )
for (const kind of CREATE_KINDS) {
- for (const outcome of ['issued', 'conflict', 'skipped']) {
+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {
expect(await count(kind, outcome)).toBe(0)
}
}
@@ -26,7 +31,7 @@ describe('Kubernetes create instrumentation', () => {
beforeEach(() => {
createsTotal.reset()
for (const kind of CREATE_KINDS) {
- for (const outcome of ['issued', 'conflict', 'skipped']) {
+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {
createsTotal.inc({ kind, outcome }, 0)
}
}
@@ -80,8 +85,10 @@ describe('Kubernetes create instrumentation', () => {
})
await expect(observeCreate('Service', create)).rejects.toBe(error)
expect(create).toHaveBeenCalledTimes(1)
- expect(await count('Service', 'issued')).toBe(1)
+ expect(await count('Service', 'error')).toBe(1)
+ expect(await count('Service', 'created')).toBe(0)
expect(await count('Service', 'conflict')).toBe(0)
+ expect(await count('Service', 'skipped')).toBe(0)
})
it('preserves real registry samples and nonzero counts across re-import', async () => {
@@ -95,50 +102,117 @@ describe('Kubernetes create instrumentation', () => {
expect(reloaded.createsTotal).toBe(createsTotal)
const exposition = await registry.metrics()
for (const kind of CREATE_KINDS) {
- for (const outcome of ['issued', 'conflict', 'skipped']) {
+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {
expect(exposition).toContain(`clerum_hcc_creates_total{kind="${kind}",outcome="${outcome}"`)
- expect(await count(kind, outcome)).toBe(kind === 'Service' && outcome === 'issued' ? 1 : 0)
+ expect(await count(kind, outcome)).toBe(kind === 'Service' && outcome === 'created' ? 1 : 0)
}
}
})
- it('issues exactly once and returns the original server response', async () => {
+ it('counts a resolved create once and returns the original server response', async () => {
const response = { metadata: { uid: 'created-policy', resourceVersion: '17' } }
const create = vi.fn(async () => {
- expect(await count('NetworkPolicy', 'issued')).toBe(1)
+ expect(await count('NetworkPolicy', 'created')).toBe(0)
return response
})
expect(await observeCreate('NetworkPolicy', create)).toBe(response)
expect(create).toHaveBeenCalledTimes(1)
- expect(await count('NetworkPolicy', 'issued')).toBe(1)
+ expect(await count('NetworkPolicy', 'created')).toBe(1)
expect(await count('NetworkPolicy', 'conflict')).toBe(0)
+ expect(await count('NetworkPolicy', 'error')).toBe(0)
expect(await count('NetworkPolicy', 'skipped')).toBe(0)
})
+ it('partitions settled callbacks without counting operations still in flight', async () => {
+ const response = { metadata: { uid: 'settled-create' } }
+ const conflict = { code: 409 }
+ const forbidden = { code: 403 }
+ let resolveCreate!: (value: typeof response) => void
+ let rejectConflict!: (reason: unknown) => void
+ let rejectForbidden!: (reason: unknown) => void
+ const success = new Promise<typeof response>(resolve => {
+ resolveCreate = resolve
+ })
+ const conflicting = new Promise<never>((_resolve, reject) => {
+ rejectConflict = reject
+ })
+ const denied = new Promise<never>((_resolve, reject) => {
+ rejectForbidden = reject
+ })
+ const create = vi
+ .fn()
+ .mockReturnValueOnce(success)
+ .mockReturnValueOnce(conflicting)
+ .mockReturnValueOnce(denied)
+ const observed = [
+ observeCreate('Service', create),
+ observeCreate('Service', create),
+ observeCreate('Service', create),
+ ]
+ // Attach rejection handlers immediately, including if a mutant resolves early.
+ const settled = Promise.allSettled(observed)
+ const values = () =>
+ Promise.all(
+ ['created', 'conflict', 'error', 'skipped'].map(outcome => count('Service', outcome))
+ )
+ expect(create).toHaveBeenCalledTimes(3)
+ expect(await values()).toEqual([0, 0, 0, 0])
+
+ resolveCreate(response)
+ expect(await observed[0]).toBe(response)
+ expect(await values()).toEqual([1, 0, 0, 0])
+
+ const conflictCheck = expect(observed[1]).rejects.toBe(conflict)
+ rejectConflict(conflict)
+ await conflictCheck
+ expect(await values()).toEqual([1, 1, 0, 0])
+
+ const forbiddenCheck = expect(observed[2]).rejects.toBe(forbidden)
+ rejectForbidden(forbidden)
+ await forbiddenCheck
+ expect(await values()).toEqual([1, 1, 1, 0])
+ expect((await settled).map(result => result.status)).toEqual([
+ 'fulfilled',
+ 'rejected',
+ 'rejected',
+ ])
+ const completed = await Promise.all(
+ ['created', 'conflict', 'error'].map(outcome => count('Service', outcome))
+ )
+ expect(completed.reduce((sum, value) => sum + value, 0)).toBe(create.mock.calls.length)
+ })
+
// Minimal fixtures isolate getErrorCode's supported layouts: ApiException.code
// from the installed client and response.statusCode for compatibility.
it.each([{ code: 409 }, { response: { statusCode: 409 } }])(
- 'counts a conflict as a subset of attempts and preserves the error: %j',
+ 'counts only conflict and preserves the original rejection: %j',
async error => {
const create = vi.fn().mockRejectedValue(error)
await expect(observeCreate('Service', create)).rejects.toBe(error)
expect(create).toHaveBeenCalledTimes(1)
- expect(await count('Service', 'issued')).toBe(1)
+ expect(await count('Service', 'created')).toBe(0)
+ expect(await count('Service', 'error')).toBe(0)
expect(await count('Service', 'conflict')).toBe(1)
expect(await count('Service', 'skipped')).toBe(0)
}
)
- it.each([{ code: 403 }, { code: 500 }, { code: 'ECONNRESET' }, null, undefined])(
- 'propagates non-conflict errors without retrying or counting a conflict: %j',
- async error => {
- const create = vi.fn().mockRejectedValue(error)
- await expect(observeCreate('Deployment', create)).rejects.toBe(error)
- expect(create).toHaveBeenCalledTimes(1)
- expect(await count('Deployment', 'issued')).toBe(1)
- expect(await count('Deployment', 'conflict')).toBe(0)
- }
- )
+ it.each([
+ { code: 403 },
+ { response: { statusCode: 403 } },
+ { code: 500 },
+ { code: 'ECONNRESET' },
+ null,
+ undefined,
+ ])('propagates non-conflict errors without retrying or counting a conflict: %j', async error => {
+ const create = vi.fn().mockRejectedValue(error)
+ await expect(observeCreate('Deployment', create)).rejects.toBe(error)
+ expect(create).toHaveBeenCalledTimes(1)
+ expect(await count('Deployment', 'error')).toBe(1)
+ expect(await count('Deployment', 'created')).toBe(0)
+ expect(await count('Deployment', 'conflict')).toBe(0)
+ expect(await count('Deployment', 'skipped')).toBe(0)
+ })
it('observes the POST-first conflict path without replacing an unchanged policy', async () => {
const policy: k8s.V1NetworkPolicy = {
@@ -160,7 +234,8 @@ describe('Kubernetes create instrumentation', () => {
expect(api.createNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
expect(api.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)
expect(api.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()
- expect(await count('NetworkPolicy', 'issued')).toBe(1)
+ expect(await count('NetworkPolicy', 'created')).toBe(0)
+ expect(await count('NetworkPolicy', 'error')).toBe(0)
expect(await count('NetworkPolicy', 'conflict')).toBe(1)
expect(await count('NetworkPolicy', 'skipped')).toBe(0)
})
@@ -178,7 +253,9 @@ describe('Kubernetes create instrumentation', () => {
)
expect(allowed).toHaveBeenCalledTimes(1)
expect(create).not.toHaveBeenCalled()
- expect(await count('NetworkPolicy', 'issued')).toBe(0)
+ expect(await count('NetworkPolicy', 'created')).toBe(0)
+ expect(await count('NetworkPolicy', 'conflict')).toBe(0)
+ expect(await count('NetworkPolicy', 'error')).toBe(0)
expect(await count('NetworkPolicy', 'skipped')).toBe(0)
})
})
diff --git a/host-context-controller/src/existenceReadMetrics.test.ts b/host-context-controller/src/existenceReadMetrics.test.ts
index afa809687..cf9f87337 100644
--- a/host-context-controller/src/existenceReadMetrics.test.ts
+++ b/host-context-controller/src/existenceReadMetrics.test.ts
@@ -81,7 +81,7 @@ describe('Kubernetes existence-read instrumentation', () => {
for (const kind of CREATE_KINDS) {
for (const outcome of ['found', 'absent', 'error'])
existenceReadsTotal.inc({ kind, outcome }, 0)
- for (const outcome of ['issued', 'conflict', 'skipped'])
+ for (const outcome of ['created', 'conflict', 'error', 'skipped'])
createsTotal.inc({ kind, outcome }, 0)
}
})
@@ -199,8 +199,9 @@ describe('Kubernetes existence-read instrumentation', () => {
sample => sample.labels.kind === 'NetworkPolicy'
)
expect(samples.map(sample => [sample.labels.outcome, sample.value])).toEqual([
- ['issued', 1],
+ ['created', 0],
['conflict', 1],
+ ['error', 0],
['skipped', 0],
])
})
diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.ts
index 6579e377f..2361471ba 100644
--- a/host-context-controller/src/metrics.ts
+++ b/host-context-controller/src/metrics.ts
@@ -298,8 +298,8 @@ export const hostFleetLifecycleCatchTotal = counter({
labelNames: ['decision'] as const,
})
-// Events overlap: every conflict is also an issued request. Never sum outcomes
-// for request totals. The bounded kind inventory includes the read-first Secret.
+// Outcomes partition completed creates; sum created, conflict, and error for
+// completed attempts. The bounded kind inventory includes the read-first Secret.
export const CREATE_KINDS = [
'NetworkPolicy',
'Service',
@@ -315,11 +315,11 @@ export const CREATE_KINDS = [
export type CreateKind = (typeof CREATE_KINDS)[number]
export const createsTotal = counter({
name: 'clerum_hcc_creates_total',
- help: 'Kubernetes create events: issued attempts, conflict responses (a subset of issued), and skipped (reserved; zero in this instrumentation stage), by kind.',
+ help: 'Completed Kubernetes create outcomes: created (resolved), conflict (409), or error; skipped is reserved and zero in this instrumentation stage, by kind.',
labelNames: ['kind', 'outcome'] as const,
})
for (const kind of CREATE_KINDS) {
- for (const outcome of ['issued', 'conflict', 'skipped']) {
+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {
createsTotal.inc({ kind, outcome }, 0)
}
}
diff --git a/host-context-controller/src/utils.ts b/host-context-controller/src/utils.ts
index fbf8c7dd3..c69ee9a06 100644
--- a/host-context-controller/src/utils.ts
+++ b/host-context-controller/src/utils.ts
@@ -16,13 +16,13 @@ export function getErrorCode(error: unknown): number | undefined {
/** Observe only the create request, preserving its response and error identity. */
export async function observeCreate<T>(kind: CreateKind, create: () => Promise<T>): Promise<T> {
- createsTotal.inc({ kind, outcome: 'issued' })
try {
- return await create()
+ const result = await create()
+ createsTotal.inc({ kind, outcome: 'created' })
+ return result
} catch (error) {
- if (error != null && getErrorCode(error) === 409) {
- createsTotal.inc({ kind, outcome: 'conflict' })
- }
+ const outcome = error != null && getErrorCode(error) === 409 ? 'conflict' : 'error'
+ createsTotal.inc({ kind, outcome })
throw error
}
}