Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(hcc): partition completed create outcomes
Alfredo Lopez744e585bb9b8
diff --git a/host-context-controller/CREATE_OBSERVABILITY.md b/host-context-controller/CREATE_OBSERVABILITY.mdindex 2e6cf5599..9935135a5 100644--- a/host-context-controller/CREATE_OBSERVABILITY.md+++ b/host-context-controller/CREATE_OBSERVABILITY.md@@ -12,42 +12,51 @@ ConfigMap, PersistentVolumeClaim, ServiceAccount, Role, RoleBinding,PodDisruptionBudget, and Secret. No resource names, namespaces, user identifiers,request bodies, or exception messages become metric labels.-| Counter | Outcome | Meaning |-| ---------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------- |-| `clerum_hcc_creates_total` | `issued` | One invocation of a Kubernetes create operation, including failed attempts. |-| | `conflict` | An issued create rejected with HTTP 409. This is a subset of `issued`, not an exclusive outcome. |-| | `skipped` | Reserved for a future presence-based create suppression; remains zero in this instrumentation change. |-| `clerum_hcc_existence_reads_total` | `found` | An instrumented existence read resolved successfully. The wrapper does not validate or alter its result. |-| | `absent` | An instrumented read rejected with HTTP 404; the original error still reaches the caller. |-| | `error` | An instrumented read rejected for any other reason, including HTTP 409. |--The bounded series are initialized at zero. Re-importing metrics must preserve-the registry and existing values. A zero series alone is not proof that a path-executed. Do not sum create outcomes to calculate request count.--`issued - conflict` is **not successful creates**: it includes other failures-(403, 5xx, network errors) and operations that have not settled. Starts and-responses can also fall in different observation windows. A successful-create-count needs matching audit success events and resource/business evidence; this-counter does not supply it by subtraction.--Select the outcome explicitly in queries. For a single scoped environment:+| Counter | Outcome | Meaning |+| ---------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------- |+| `clerum_hcc_creates_total` | `created` | The create callback resolved successfully. |+| | `conflict` | The create callback rejected with HTTP 409. |+| | `error` | The create callback rejected for any other reason, including 403, 5xx, network errors, or synchronous client failure. |+| | `skipped` | Reserved for presence-based suppression in the read-first stage; remains zero in this PR. |+| `clerum_hcc_existence_reads_total` | `found` | An instrumented existence read resolved successfully; its result is not validated or altered. |+| | `absent` | An instrumented read rejected with HTTP 404; the original error still reaches the caller. |+| | `error` | An instrumented read rejected for any other reason, including HTTP 409. |++Create outcomes are mutually exclusive and increment only after the callback+settles. The former `issued` series is removed. Summing `created + conflict + error`+counts completed attempts; `skipped` is not an invocation and is excluded from that+sum. Operations still in flight are not counted. A create resolving to null or+undefined still counts as resolved; telemetry does not assert object persistence.++The bounded series are initialized at zero: 40 create series and 30 read series.+Re-importing metrics must preserve the registry and existing values. A zero series+alone is not proof that a path executed. For a single scoped environment:```promql-sum by (kind) (increase(clerum_hcc_creates_total{outcome="issued"}[5m]))+sum by (kind) (increase(clerum_hcc_creates_total{outcome=~"created|conflict|error"}[5m]))+sum by (kind) (increase(clerum_hcc_creates_total{outcome="created"}[5m]))sum by (kind) (increase(clerum_hcc_creates_total{outcome="conflict"}[5m]))+sum by (kind) (increase(clerum_hcc_creates_total{outcome="error"}[5m]))```-Add the actual environment/target filters from the monitoring configuration-before comparing a deployment. These queries report attempts and conflicts,-not successful creations or a count of all completed operations.+Add the actual environment/target filters from the monitoring configuration.+These are client outcomes, not proof that every operation reached the API server;+a lost response can reject even if the server created an object. Validate server+outcomes and resource/business state separately when that distinction matters.++**Accepted trade-off:** if the process dies before a callback settles, that attempt+has no outcome sample. The previous pre-call increment could have been observed+before the crash. This loss is accepted in favor of an unambiguous partition;+server audit events can provide additional evidence for requests that reached the+server, but cannot reconstruct every client invocation. No crash-loss rate is+claimed or measured here. This contract is changed before downstream dashboards+and the read-first acceptance depend on it.`observeCreate` and `observeExistenceRead` invoke their callbacks once and preservethe returned value or rejected value. They do not swallow a 404, turn it into`null`, retry a failed request, or convert an error into a create. Existing caller-catch behavior remains responsible for those decisions. Synchronous client-failures count as attempts/outcomes too, so these counters are client-operation-observations rather than proof that every attempt reached the API server.+catch behavior remains responsible for those decisions. Synchronous failures+count as `error` when the wrapper catches them; the original rejection is retained.## Coverage@@ -124,7 +133,7 @@ command exits and retained-row counts; missing data is unknown, not zero. Localtests and a historical re-query do not replace simultaneous deployed validation.For create conflicts, compare the `conflict` subset with the audit's conflict-result, not with all issued operations. The audit uses `google.rpc.Code=10`,+result, not with all completed create outcomes. The audit uses `google.rpc.Code=10`,which represents conflict, while client errors use HTTP 409. Historical auditdata did not contain GET records; no API-server read rate is inferred from thatabsence. The read counter only covers the explicit inventory above.diff --git a/host-context-controller/package-lock.json b/host-context-controller/package-lock.jsonindex 7d1d15364..aff8312f9 100644--- a/host-context-controller/package-lock.json+++ b/host-context-controller/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-host-context-controller",- "version": "0.1.301",+ "version": "0.1.302","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-host-context-controller",- "version": "0.1.301",+ "version": "0.1.302","license": "SEE LICENSE IN LICENSE","dependencies": {"@clerum/codex-catalog-projection": "file:../packages/codex-catalog-projection",diff --git a/host-context-controller/package.json b/host-context-controller/package.jsonindex 808f1a816..9a482303d 100644--- a/host-context-controller/package.json+++ b/host-context-controller/package.json@@ -1,6 +1,6 @@{"name": "clerum-host-context-controller",- "version": "0.1.301",+ "version": "0.1.302","description": "Host Context Controller - provides McpServer CRDs to mcp-host via REST API","main": "dist/main.js","engines": {diff --git a/host-context-controller/src/createMetrics.test.ts b/host-context-controller/src/createMetrics.test.tsindex f72c72086..1ac18eb60 100644--- a/host-context-controller/src/createMetrics.test.ts+++ b/host-context-controller/src/createMetrics.test.ts@@ -15,8 +15,13 @@ async function count(kind: string, outcome: string): Promise<number> {it('initializes create samples in the real registry before any test seeding', async () => {expect(registry.getSingleMetric('clerum_hcc_creates_total')).toBe(createsTotal)+ const snapshot = await createsTotal.get()+ expect(snapshot.values).toHaveLength(CREATE_KINDS.length * 4)+ expect(new Set(snapshot.values.map(sample => sample.labels.outcome))).toEqual(+ new Set(['created', 'conflict', 'error', 'skipped'])+ )for (const kind of CREATE_KINDS) {- for (const outcome of ['issued', 'conflict', 'skipped']) {+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {expect(await count(kind, outcome)).toBe(0)}}@@ -26,7 +31,7 @@ describe('Kubernetes create instrumentation', () => {beforeEach(() => {createsTotal.reset()for (const kind of CREATE_KINDS) {- for (const outcome of ['issued', 'conflict', 'skipped']) {+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {createsTotal.inc({ kind, outcome }, 0)}}@@ -80,8 +85,10 @@ describe('Kubernetes create instrumentation', () => {})await expect(observeCreate('Service', create)).rejects.toBe(error)expect(create).toHaveBeenCalledTimes(1)- expect(await count('Service', 'issued')).toBe(1)+ expect(await count('Service', 'error')).toBe(1)+ expect(await count('Service', 'created')).toBe(0)expect(await count('Service', 'conflict')).toBe(0)+ expect(await count('Service', 'skipped')).toBe(0)})it('preserves real registry samples and nonzero counts across re-import', async () => {@@ -95,50 +102,117 @@ describe('Kubernetes create instrumentation', () => {expect(reloaded.createsTotal).toBe(createsTotal)const exposition = await registry.metrics()for (const kind of CREATE_KINDS) {- for (const outcome of ['issued', 'conflict', 'skipped']) {+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {expect(exposition).toContain(`clerum_hcc_creates_total{kind="${kind}",outcome="${outcome}"`)- expect(await count(kind, outcome)).toBe(kind === 'Service' && outcome === 'issued' ? 1 : 0)+ expect(await count(kind, outcome)).toBe(kind === 'Service' && outcome === 'created' ? 1 : 0)}}})- it('issues exactly once and returns the original server response', async () => {+ it('counts a resolved create once and returns the original server response', async () => {const response = { metadata: { uid: 'created-policy', resourceVersion: '17' } }const create = vi.fn(async () => {- expect(await count('NetworkPolicy', 'issued')).toBe(1)+ expect(await count('NetworkPolicy', 'created')).toBe(0)return response})expect(await observeCreate('NetworkPolicy', create)).toBe(response)expect(create).toHaveBeenCalledTimes(1)- expect(await count('NetworkPolicy', 'issued')).toBe(1)+ expect(await count('NetworkPolicy', 'created')).toBe(1)expect(await count('NetworkPolicy', 'conflict')).toBe(0)+ expect(await count('NetworkPolicy', 'error')).toBe(0)expect(await count('NetworkPolicy', 'skipped')).toBe(0)})+ it('partitions settled callbacks without counting operations still in flight', async () => {+ const response = { metadata: { uid: 'settled-create' } }+ const conflict = { code: 409 }+ const forbidden = { code: 403 }+ let resolveCreate!: (value: typeof response) => void+ let rejectConflict!: (reason: unknown) => void+ let rejectForbidden!: (reason: unknown) => void+ const success = new Promise<typeof response>(resolve => {+ resolveCreate = resolve+ })+ const conflicting = new Promise<never>((_resolve, reject) => {+ rejectConflict = reject+ })+ const denied = new Promise<never>((_resolve, reject) => {+ rejectForbidden = reject+ })+ const create = vi+ .fn()+ .mockReturnValueOnce(success)+ .mockReturnValueOnce(conflicting)+ .mockReturnValueOnce(denied)+ const observed = [+ observeCreate('Service', create),+ observeCreate('Service', create),+ observeCreate('Service', create),+ ]+ // Attach rejection handlers immediately, including if a mutant resolves early.+ const settled = Promise.allSettled(observed)+ const values = () =>+ Promise.all(+ ['created', 'conflict', 'error', 'skipped'].map(outcome => count('Service', outcome))+ )+ expect(create).toHaveBeenCalledTimes(3)+ expect(await values()).toEqual([0, 0, 0, 0])++ resolveCreate(response)+ expect(await observed[0]).toBe(response)+ expect(await values()).toEqual([1, 0, 0, 0])++ const conflictCheck = expect(observed[1]).rejects.toBe(conflict)+ rejectConflict(conflict)+ await conflictCheck+ expect(await values()).toEqual([1, 1, 0, 0])++ const forbiddenCheck = expect(observed[2]).rejects.toBe(forbidden)+ rejectForbidden(forbidden)+ await forbiddenCheck+ expect(await values()).toEqual([1, 1, 1, 0])+ expect((await settled).map(result => result.status)).toEqual([+ 'fulfilled',+ 'rejected',+ 'rejected',+ ])+ const completed = await Promise.all(+ ['created', 'conflict', 'error'].map(outcome => count('Service', outcome))+ )+ expect(completed.reduce((sum, value) => sum + value, 0)).toBe(create.mock.calls.length)+ })+// Minimal fixtures isolate getErrorCode's supported layouts: ApiException.code// from the installed client and response.statusCode for compatibility.it.each([{ code: 409 }, { response: { statusCode: 409 } }])(- 'counts a conflict as a subset of attempts and preserves the error: %j',+ 'counts only conflict and preserves the original rejection: %j',async error => {const create = vi.fn().mockRejectedValue(error)await expect(observeCreate('Service', create)).rejects.toBe(error)expect(create).toHaveBeenCalledTimes(1)- expect(await count('Service', 'issued')).toBe(1)+ expect(await count('Service', 'created')).toBe(0)+ expect(await count('Service', 'error')).toBe(0)expect(await count('Service', 'conflict')).toBe(1)expect(await count('Service', 'skipped')).toBe(0)})- it.each([{ code: 403 }, { code: 500 }, { code: 'ECONNRESET' }, null, undefined])(- 'propagates non-conflict errors without retrying or counting a conflict: %j',- async error => {- const create = vi.fn().mockRejectedValue(error)- await expect(observeCreate('Deployment', create)).rejects.toBe(error)- expect(create).toHaveBeenCalledTimes(1)- expect(await count('Deployment', 'issued')).toBe(1)- expect(await count('Deployment', 'conflict')).toBe(0)- }- )+ it.each([+ { code: 403 },+ { response: { statusCode: 403 } },+ { code: 500 },+ { code: 'ECONNRESET' },+ null,+ undefined,+ ])('propagates non-conflict errors without retrying or counting a conflict: %j', async error => {+ const create = vi.fn().mockRejectedValue(error)+ await expect(observeCreate('Deployment', create)).rejects.toBe(error)+ expect(create).toHaveBeenCalledTimes(1)+ expect(await count('Deployment', 'error')).toBe(1)+ expect(await count('Deployment', 'created')).toBe(0)+ expect(await count('Deployment', 'conflict')).toBe(0)+ expect(await count('Deployment', 'skipped')).toBe(0)+ })it('observes the POST-first conflict path without replacing an unchanged policy', async () => {const policy: k8s.V1NetworkPolicy = {@@ -160,7 +234,8 @@ describe('Kubernetes create instrumentation', () => {expect(api.createNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)expect(api.readNamespacedNetworkPolicy).toHaveBeenCalledTimes(1)expect(api.replaceNamespacedNetworkPolicy).not.toHaveBeenCalled()- expect(await count('NetworkPolicy', 'issued')).toBe(1)+ expect(await count('NetworkPolicy', 'created')).toBe(0)+ expect(await count('NetworkPolicy', 'error')).toBe(0)expect(await count('NetworkPolicy', 'conflict')).toBe(1)expect(await count('NetworkPolicy', 'skipped')).toBe(0)})@@ -178,7 +253,9 @@ describe('Kubernetes create instrumentation', () => {)expect(allowed).toHaveBeenCalledTimes(1)expect(create).not.toHaveBeenCalled()- expect(await count('NetworkPolicy', 'issued')).toBe(0)+ expect(await count('NetworkPolicy', 'created')).toBe(0)+ expect(await count('NetworkPolicy', 'conflict')).toBe(0)+ expect(await count('NetworkPolicy', 'error')).toBe(0)expect(await count('NetworkPolicy', 'skipped')).toBe(0)})})diff --git a/host-context-controller/src/existenceReadMetrics.test.ts b/host-context-controller/src/existenceReadMetrics.test.tsindex afa809687..cf9f87337 100644--- a/host-context-controller/src/existenceReadMetrics.test.ts+++ b/host-context-controller/src/existenceReadMetrics.test.ts@@ -81,7 +81,7 @@ describe('Kubernetes existence-read instrumentation', () => {for (const kind of CREATE_KINDS) {for (const outcome of ['found', 'absent', 'error'])existenceReadsTotal.inc({ kind, outcome }, 0)- for (const outcome of ['issued', 'conflict', 'skipped'])+ for (const outcome of ['created', 'conflict', 'error', 'skipped'])createsTotal.inc({ kind, outcome }, 0)}})@@ -199,8 +199,9 @@ describe('Kubernetes existence-read instrumentation', () => {sample => sample.labels.kind === 'NetworkPolicy')expect(samples.map(sample => [sample.labels.outcome, sample.value])).toEqual([- ['issued', 1],+ ['created', 0],['conflict', 1],+ ['error', 0],['skipped', 0],])})diff --git a/host-context-controller/src/metrics.ts b/host-context-controller/src/metrics.tsindex 6579e377f..2361471ba 100644--- a/host-context-controller/src/metrics.ts+++ b/host-context-controller/src/metrics.ts@@ -298,8 +298,8 @@ export const hostFleetLifecycleCatchTotal = counter({labelNames: ['decision'] as const,})-// Events overlap: every conflict is also an issued request. Never sum outcomes-// for request totals. The bounded kind inventory includes the read-first Secret.+// Outcomes partition completed creates; sum created, conflict, and error for+// completed attempts. The bounded kind inventory includes the read-first Secret.export const CREATE_KINDS = ['NetworkPolicy','Service',@@ -315,11 +315,11 @@ export const CREATE_KINDS = [export type CreateKind = (typeof CREATE_KINDS)[number]export const createsTotal = counter({name: 'clerum_hcc_creates_total',- help: 'Kubernetes create events: issued attempts, conflict responses (a subset of issued), and skipped (reserved; zero in this instrumentation stage), by kind.',+ help: 'Completed Kubernetes create outcomes: created (resolved), conflict (409), or error; skipped is reserved and zero in this instrumentation stage, by kind.',labelNames: ['kind', 'outcome'] as const,})for (const kind of CREATE_KINDS) {- for (const outcome of ['issued', 'conflict', 'skipped']) {+ for (const outcome of ['created', 'conflict', 'error', 'skipped']) {createsTotal.inc({ kind, outcome }, 0)}}diff --git a/host-context-controller/src/utils.ts b/host-context-controller/src/utils.tsindex fbf8c7dd3..c69ee9a06 100644--- a/host-context-controller/src/utils.ts+++ b/host-context-controller/src/utils.ts@@ -16,13 +16,13 @@ export function getErrorCode(error: unknown): number | undefined {/** Observe only the create request, preserving its response and error identity. */export async function observeCreate<T>(kind: CreateKind, create: () => Promise<T>): Promise<T> {- createsTotal.inc({ kind, outcome: 'issued' })try {- return await create()+ const result = await create()+ createsTotal.inc({ kind, outcome: 'created' })+ return result} catch (error) {- if (error != null && getErrorCode(error) === 409) {- createsTotal.inc({ kind, outcome: 'conflict' })- }+ const outcome = error != null && getErrorCode(error) === 409 ? 'conflict' : 'error'+ createsTotal.inc({ kind, outcome })throw error}}