Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
Stateless-Lifecycle Invariants
Enforceable guardrails distilled from the stateless-agents work (PR #735). The eight stateless-lifecycle bugs found live on-branch (five in one E2E round, plus three from the review/kaizen passes) collapse into five reusable anti-patterns. Four of the five share one root cause: a decision made on evidence that had already aged between its producer and its consumer. Encode these as guardrails so the class does not reappear.
This doc is the "how to not reintroduce the class" companion to the
spec.lifecycle.stateless field and status.lifecycle states in the
Host CRD reference.
How to enforce (applies to AP-1, AP-2, AP-3, AP-5)
Before any state-mutating status write in a controller/tracker path:
- Fresh-read-before-status-write. Re-read the object from the API
server (not the informer cache) immediately before the write that
commits a lifecycle decision. The canonical helper is
StatelessLifecycleExecutor.readFreshHost()(host-context-controller/src/statelessLifecycleExecutor.ts:567); every heartbeat core reads it before touching status (suspend:640, suspend-blocked publisher:772, draining:854, cancel-drain:934), and the reconcile-path replicas guard reads it before any Deployment scale derives (resolveWakeBeforeScaleDown,:1382). - Gate the write with a
resourceVersionprecondition where the platform supports it, so a concurrent flip loses instead of silently clobbering. When the write is a status subresource patch, remember it bumpsresourceVersionand fires its own MODIFIED watch event — do not treat that self-echo as new work (statelessLifecycleExecutor.ts:71,sharedFileSystemReconciler.ts:104). - On a fresh-read failure, keep the conservative verdict for that beat;
never fall through to the cached optimistic one (see AP-2's failure
branch,
statelessLifecycleTracker.ts:453).
AP-1 — Decide on the freshest evidence at the commit point
Rule. A state-mutating status write must be decided on a value read at the commit point, not on a snapshot captured earlier in the reconcile. Never patch status from a cached CR snapshot that could have been overturned between capture and write.
Bug it came from. A stale drained report suspended a Host that had
just served a turn — the suspend decision rode a snapshot older than the
activity that invalidated it (fix 88b456416, drained-report superseded by
fresh activity; hardened by the fresh-read drained-pre-scale guard in
8f1fff5c3).
Follow. Re-read immediately before the write:
// statelessLifecycleExecutor.ts — FRESH GET guards against the CURRENT
// server-side status, not the snapshot the reconcile started with.
const fresh = await this.readFreshHost(host)
// ...decide suspend/scale from `fresh`, then write.
Every state-mutating status writer recomputes both the guard AND the
written value from fresh:
suspendHostFromHeartbeatCore— writessuspendedonly when fresh isdraining; the writtenwakeHandledGenerationcomes from fresh. Freshdrainingalone is NOT sufficient: an aged drain:true verdict can re-persistdraining(the poller'smarkHostDrainingFromHeartbeat) after a wake already cancelled the drain and was handled — and with requested==handled nothing would revive the suspended Host. The commit therefore also carries an AP-1 generation EPOCH — thewakeHandledGenerationthe tracker observed when it DECIDED the suspend — and no-ops when the fresh generation advanced past it (phase=drained_report_stale reason=wake_handled_since) or when a wake is pending in fresh (reason=wake_pending). The epoch, not "fresh state is active", is the discriminator: every normal suspension commits overdraining. The commit also REPORTS its outcome to the tracker (SuspendFromHeartbeatOutcome): askipped_stalecommit makes the tracker answer the drained poddrain:false(phase=drained_suspend_skipped_stale) so the pod UN-FENCES — a stale skip must never leave the emitter fenced behind evidence the commit itself rejected. Only the idempotent fresh-suspendedretry (and a landed commit) keeps answeringdrain:true.markHostDrainingFromHeartbeatCore— writesdrainingonly when fresh isactive; generation from fresh. Freshactivealone is NOT sufficient:activeis exactly the state a just-handled wake produces, so an AGED drain:true verdict (decided before the wake, persisted after) would re-fence the woken pod — and its consumed wake (requested==handled) revives nothing, stranding any pending intake. The write therefore carries the SAME AP-1 generation EPOCH as the suspend commit — thewakeHandledGenerationthe tracker observed when it DECIDED the drain, threaded throughHeartbeatVerdict.entryWakeHandledGenerationand the poller's persist call — and no-ops when the fresh generation advanced past it (phase=draining_write_stale reason=wake_handled_since) or when a wake is pending in fresh (reason=wake_pending). This closes the aged-writer loop end to end: BOTH the write that fences a pod (draining) and the one that parks it (suspended) are epoch-guarded, and the drained answer (drain:falseon a stale skip) un-fences instead of re-fencing.markHostActiveFromHeartbeatCore— cancel-drain toactiveonly when fresh isdraining; generation from fresh.handleWakeFastPath— the wake transition writer (the 10th costume). The cachedrequested > handledcomparison only DISCOVERS a potentially-pending wake; the commit is re-decided from a FRESH read inside the D3 precondition callback: it recomputes the requested generation (fresh annotation) andwakeHandledGeneration(fresh lifecycle) and SKIPS — no write, no duplicate wake-phase logs — when the fresh generation is already handled (a sibling or a prior fast-path pass won). Otherwise it emits ONLY a targeted/status/lifecycleop with `{ state: 'active', wakeHandledGeneration:
} — never the previous whole-/statusspread of the cached snapshot, which could resurrect staleconditions/reasonover a hardened sibling's fresher write. The stale cachedreasonis dropped on the wake transition (mirroringmarkHostActiveFromHeartbeatCoreand resolveWakeBeforeScaleDown; the change-only suspend-blocked publisher re-stamps a still-valid reason on the next heartbeat), and fresh conditions` can no longer be clobbered because the targeted op never
touches them. On a 409 the helper re-reads fresh and re-decides, so a
racing writer's newer decision is respected instead of overwritten. The
durable flip still happens FIRST (before the replicas=1 scale patch), and
the path stays bounded: one fresh GET + one status write + one scale patch
per wake event.
publishSuspendBlockedReasonCore— the D8 suspend-blocked reason ANNOTATOR. It must NEVER change state, only stamp the reason on top of the CURRENT server-side state. It re-sourcesstate+wakeHandledGenerationfromfreshinside the D3 precondition callback and GUARDS: a fresh state that is no longeractive(freshsuspended/draining— the Host is no longer being kept active/blocked) is a no-op, so a suspend/drain that landed underneath is never resurrected. It emits a targeted/status/lifecycleop (never a whole-/statusspread) and re-evaluates its change-only reason short-circuit against fresh.writeLifecycleStatusToCluster— the reconcile-loop accepted/rejected writer. Itsassessment.lifecycleis derived from the CACHED snapshot, and in the accepted path itsstate/reasonare a pass-through ECHO of that snapshot. It is subject to AP-1 like the heartbeat cores even though the D3 resourceVersion precondition passes trivially (the serialized per-host chain makes a fresh GET immediately before the PATCH always match the current resourceVersion) — the precondition proves nothing changed AFTER the fresh read, not that the DECIDED value is fresh. So inside thebuildcallback it re-sources the ECHOED lifecycle fromfresh: when the assessment'sstateEQUALS the cached snapshot's state it is an echo -> prefer fresh (a heartbeat suspend that landed since the snapshot is preserved;wakeHandledGenerationstays monotonic against fresh); when it DIFFERS the assessment INTENDED a transition (kill-switch/rejection forcing active, or aresolveWakeBeforeScaleDownwake transition) and that override wins.
Avoid. Deciding a suspend from the host argument the reconcile was
entered with, when a wake or a fresh turn may have flipped
status.lifecycle in between. In particular, never write a status VALUE
sourced from the pre-fresh-read snapshot (assessment.lifecycle verbatim) on
the strength of a resourceVersion precondition alone -- the precondition
guards ordering, not the freshness of the decided value.
AP-2 — The informer cache discovers work, it never decides it
Rule. The informer cache is allowed to trigger a reconcile (discovery) but must never be the evidence a lifecycle verdict is committed on. A missed or coalesced watch event leaves the cache stale until the periodic resync; a decision made on that stale entry is wrong for the whole resync window.
Bug it came from. The tracker's wakePending read the informer cache;
a missed MODIFIED event left a stale pending/not-pending verdict, blocking
re-suspension (cost leak) or risking a suspend over a landed wake until the
5-minute resync (fix cfa47ace1; the symmetric grace-expiry path was fixed
in KZ-R1 1b3c2b35a).
Follow. Cache lookup discovers; a fresh read decides:
// statelessLifecycleTracker.ts:232 — cache lookup: DISCOVERY only.
/** Host CRD lookup from the informer cache (McpServerWatcher.getHost). */
// statelessLifecycleTracker.ts:421-457 — resolve the cached-pending verdict
// against a FRESH read before it gates a drain:
if (wakePending) {
// The informer-cached Host can be STALE right here...
const fresh = await this.readFreshHost(...) // server-side truth
if (/* wake generation already handled */) wakePending = false
}
Avoid. Committing wakePending (or any drain/suspend gate) straight from
getHost() cache state without the fresh-read resolution.
Pod identity is evidence too. "Never-seen podUid ⇒ newer pod" is the same
cache-shaped assumption and breaks on out-of-order replay: after an HCC
restart, a straggler beat of the OLD pod replayed after the new pod's first
beat would adopt the straggler as current and retire the LIVE pod — every
live beat discarded, lifecycle wedged until the straggler dies. Adoption of a
never-seen podUid over an existing current pod is tie-broken on the pods'
immutable creationTimestamps (resolvePodOrdering,
statelessLifecycleTracker.ts:620): a strictly-older newcomer is itself
retired with a loud log; an unresolvable order (host unknown, lookup failure,
newcomer invisible) keeps the current pod and retires nobody.
AP-3 — A timeout is valid evidence only if the actor is inside your clock
Rule. A timeout only proves inaction if the thing you are timing shares
your clock. If the grace/deadline window is shorter than the actor's own beat
interval, expiry proves nothing — the actor may simply not have reported yet.
Size any grace as > actor_beat_interval and make expiry ack-aware.
Bug it came from. Drain-grace (20 s) fired before the emitter's 30 s beat
could report the drain, so the grace killed a pod mid-turn and the client saw
a 504 (fix e00b0baf5, grace expiry is ack-aware).
Follow. Only force-suspend on expiry when the emitter actually acked the drain or provably went silent; if beats kept flowing, re-arm:
// statelessLifecycleTracker.ts:36-47
// Drain-grace expiry is ACK-AWARE: after a `drain: true` verdict the grace
// only force-suspends when the emitter ACKED the drain (last beat reported
// 'draining'/'drained' but the drained report never landed) or went silent.
// If beats kept flowing the grace RE-ARMS instead of killing in-flight work.
Silence must be attributed to the right actor. "No beat since arming" is
read from bookkeeping that only successful polls feed — a control-api/poller
outage spanning the grace makes a LIVE, beating emitter look silent. The
expiry may treat silence as evidence against the EMITTER only when the feed
provably flowed across the window: the poller reports every fully-successful
poll (HeartbeatPoller → noteSuccessfulPoll,
statelessLifecycleTracker.ts:335), and an expiry that saw no successful
poll strictly after arming (and recent enough, within poll-interval × 2)
RE-ARMS with phase=grace_rearmed reason=feed_outage instead of
force-suspending. Feed health is part of the timeout's evidence, exactly as
the actor's beat interval is.
The expiry's identity evidence ages across its own awaits. The grace
callback re-validates — after its fresh read and immediately before
delegating the suspend — that the current pod is still the pod it armed for
and that no new grace was re-armed underneath
(statelessLifecycleTracker.ts:780-795); a pod replacement landing during
the await would otherwise force-suspend the live replacement pod on
misattributed silence (its beat bookkeeping was reset by the roll).
Avoid. A fixed grace shorter than CLERUM_STATELESS_HEARTBEAT_INTERVAL_MS
that force-suspends purely on wall-clock expiry, ignoring whether the emitter
was ever in a position to report — or whether anything was in a position to
OBSERVE the emitter at all.
AP-4 — Dedup on delivery identity, not a per-attempt id
Rule. Idempotency keys must be assigned to the delivery, not to each transport attempt. A retry of the same logical delivery must carry the same id so the second arrival replays instead of re-executing (and its MCP tool side-effects do not fire twice). Conversely, two genuinely distinct sends must get different ids even if their content is identical — so the durable identity should be a per-request nonce, not a content hash.
Bug it came from. rpc-proxy re-forwarded an already-answered request 83 s
later with no messageId; mcp-host minted a fresh uuid and the queue
re-executed it, double-firing tool side-effects (fix 686c97630 delivery
dedup + ff54367c2 wake-hold latch; the inert-on-wake-retry path was P1-1
1b3c2b35a).
Follow. Stamp one stable id per delivery and carry it across the retry:
// rpc-proxy stamps a stable messageId ONCE and reuses it across the wake
// retry (wake-and-hold-route.test.ts:114) so mcp-host dedups the retry;
// two distinct sends with identical content still get DIFFERENT ids
// (wake-and-hold-route.test.ts:143, "D1").
Avoid. Minting a new id per forward attempt (causes double-execution on
retry). Residual to close: the current stable id is
sha256(sender|hostRef|threadId|content) — a content hash, so two truly
identical messages inside the dedup TTL replay rather than execute. The
correct long-term identity is a per-request nonce (e.g. a client-supplied
idempotency key from the desktop app), which distinguishes deliveries by
intent rather than by bytes.
AP-5 — "Restart" is not "invalid"
Rule. A process restart is not evidence that its derived state is stale. Reuse healthy derived state across a restart; re-mint (rotate, regenerate, re-issue) only when a new consumer will actually read the new value — otherwise a restart needlessly churns the fleet and invalidates live state.
Bug it came from. Every HCC restart re-issued runtime tokens, which
rolled every stateless pod — killing in-flight turns and resetting the
idle anchor (fix 3f6381f0b, reuse the healthy Secret, rotate only near
expiry). The same principle makes replicas derived from Host.status on
every reconcile, so a routine reconcile or HCC restart never resurrects a
suspended Host. The writer of that derivation is
ensureDeployment/buildDeployment (hostReconciler.ts), whose replicas
come from the reconcile assessment — and the assessment ECHOES the CACHED
reconcile payload, so the claim only actually holds because
resolveWakeBeforeScaleDown (statelessLifecycleExecutor.ts:1382) guards
EVERY stateless assessment with a fresh read at the commit point: a cached
suspended over a fresh active (wake fully handled, requested==handled)
keeps replicas=1 instead of killing the live pod; a cached active over a
fresh suspended keeps replicas=0 instead of resurrecting a suspended Host
with no wake; a fresh-read failure skips the Deployment scale for the whole
pass (loud log, the resync retries). Replicas are a function of FRESH state
whenever fresh and cache disagree — never of the event payload.
Follow. Reuse the HCC-owned Secret while a serving pod holds the tokens in memory; rotate only inside the near-expiry window:
// hostReconciler.ts:642 — "Reuse is only safe for Secrets HCC itself wrote"
// hostReconciler.ts:686-694 — rotate ONLY when nowMs >= exp - rotateBeforeMs
// (reason: 'refresh_token_near_expiry')
// hostReconciler.ts:751-758 — reuse is safe precisely when a serving pod
// rotates tokens in memory (deploymentReady && replicas>=1).
Avoid. Unconditionally re-issuing credentials (or re-minting any derived
artifact) on startup, and deriving replica count from anything other than
the durable Host.status.lifecycle state.
AP-6 — The spec plane needs the reader's version at the commit point
Rule. An admin write that replaces spec must carry the
resourceVersion of the read the HUMAN's edit was built from, and a
conflict must surface to that human — never be retried away with the same
stale payload. Platform-owned clerum.io/* annotation keys must survive
admin writes that did not explicitly touch them.
Bug it came from. resourceService.updateResource read the current
object ONLY to harvest the server's resourceVersion (and annotations),
replaced spec wholesale with the caller's payload, and on 409 re-read and
re-applied the SAME stale payload up to 3×, actively defeating optimistic
concurrency. Trace: operator opens the host edit form at T0; at T0+5m
someone enables spec.lifecycle.stateless via kubectl; the form saves at
T0+10m → the stale echo replaces spec → lifecycle silently stripped → HCC
kill-switch, zero signal. The sibling metadata bug: spreading
current.annotations then body.annotations REPLACED the whole map,
erasing the clerum.io/wake-requested projection (write-only per
hostWakeService.ts) whenever an admin PUT carried any annotations map.
Follow. Reader's-version precondition end-to-end:
- Control UI captures
metadata.resourceVersionat form load — NOT from the pre-save re-fetch, which would only guard milliseconds instead of the human edit window (control-ui/app/hosts/[name]/page.tsx,formResourceVersionRef) — and sends it on PUT. resourceService.updateResourceuses the caller-provided version as the replace precondition instead of overwriting it with the server's current one, and maps 409 toK8sConflictErrorwith NO retry (control-api/src/services/resourceService.ts).- The admin facade answers
409 {error:'conflict', reason:'resource_changed'}so the UI can tell the operator to reload (control-api/src/routes/admin/resources.ts). - Annotations are merged per key: platform-owned
clerum.io/*keys present on the server survive unless the caller explicitly sets that exact key (mergeAnnotationsForReplace, applied by bothupdateResourceandmutateResource). - The spec-echo contract in
control-api/src/routes/admin/hostSpecValidation.tsdocuments how the echo and the precondition compose: the echo preserves fields the form does not edit; the precondition rejects the echo when it is stale.
Avoid. Re-reading only to harvest a fresh resourceVersion for a stale
payload; retry loops around full replaces that carry client intent; spreads
that replace whole metadata maps a projection writer co-owns. Callers that
legitimately want last-write-wins (API-only automation) simply omit
metadata.resourceVersion — that legacy path is pinned by
control-api/test/services.resourceService.test.ts.