Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure
TypeScript
fix(control-api): bound the Forbid skip-streak map and correct its docs
Jose Ramirez76a8299147db
diff --git a/control-api/package-lock.json b/control-api/package-lock.jsonindex 440d29703..6b4c1ff56 100644--- a/control-api/package-lock.json+++ b/control-api/package-lock.json@@ -1,12 +1,12 @@{"name": "clerum-control-api",- "version": "0.1.742",+ "version": "0.1.743","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "clerum-control-api",- "version": "0.1.742",+ "version": "0.1.743","dependencies": {"@clerum/display-field": "file:../packages/display-field","@clerum/grok-provider-attempt-contract": "file:../packages/grok-provider-attempt-contract",diff --git a/control-api/package.json b/control-api/package.jsonindex 0036854a4..f6c177d2c 100644--- a/control-api/package.json+++ b/control-api/package.json@@ -1,6 +1,6 @@{"name": "clerum-control-api",- "version": "0.1.742",+ "version": "0.1.743","description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets","main": "dist/main.js","engines": {diff --git a/control-api/src/services/workflowScheduleWorkerService.ts b/control-api/src/services/workflowScheduleWorkerService.tsindex 94a73672e..7ec4eff01 100644--- a/control-api/src/services/workflowScheduleWorkerService.ts+++ b/control-api/src/services/workflowScheduleWorkerService.ts@@ -119,6 +119,9 @@ export const FORBID_RUNNING_GRACE_SECONDS = 5 * 60/** Consecutive Forbid skips of one schedule before a warn log + counter fire. */export const FORBID_SUSTAINED_SKIP_THRESHOLD = 6+/** Upper bound on per-replica skip-streak entries (see `forbiddenSkipStreaks`). */+export const FORBID_SKIP_STREAK_CAPACITY = 1000+/** A live non-terminal run (see the CHECK on `workflow_runs.phase`). */interface ActiveRecipeRun {run_id: string@@ -135,12 +138,18 @@ interface ActiveRecipeRun {* never duplicated. `expectedFireAt` is the next_fire_at this replica wrote* with its last skip; if the row arrives with any other slot (another replica* fired or skipped it, or the batch rolled back) the streak restarts at 1.- * Only a fire deletes an entry; a schedule skipped and then deleted, disabled- * or actor-gated keeps its entry until restart. The map is therefore bounded- * by the number of schedules.+ * A fire deletes the entry. A schedule skipped and then deleted, disabled or+ * actor-gated would otherwise keep its entry until restart, so the map holds at+ * most FORBID_SKIP_STREAK_CAPACITY entries and evicts the least recently+ * skipped one first. An evicted schedule that skips again restarts at 1.*/const forbiddenSkipStreaks = new Map<string, { count: number; expectedFireAt: string }>()+/** Test-only: number of schedules with a live skip streak on this replica. */+export function concurrencyForbiddenStreakCountForTests(): number {+ return forbiddenSkipStreaks.size+}+/** Test-only: forget every streak so cases do not leak into each other. */export function resetConcurrencyForbiddenStreaksForTests(): void {forbiddenSkipStreaks.clear()@@ -579,10 +588,16 @@ async function skipForbiddenTick(const previous = forbiddenSkipStreaks.get(row.schedule_id)const consecutiveSkips =previous && previous.expectedFireAt === currentFire.toISOString() ? previous.count + 1 : 1+ // Delete first so the entry moves to the end of the Map's insertion order.+ forbiddenSkipStreaks.delete(row.schedule_id)forbiddenSkipStreaks.set(row.schedule_id, {count: consecutiveSkips,expectedFireAt: skipNextFire.toISOString(),})+ for (const scheduleId of forbiddenSkipStreaks.keys()) {+ if (forbiddenSkipStreaks.size <= FORBID_SKIP_STREAK_CAPACITY) break+ forbiddenSkipStreaks.delete(scheduleId)+ }const activeSince = new Date(activeRun.started_at ?? activeRun.created_at)const fields = {diff --git a/control-api/test/services.workflowScheduleWorker.test.ts b/control-api/test/services.workflowScheduleWorker.test.tsindex 31158c58c..8ce02c4bf 100644--- a/control-api/test/services.workflowScheduleWorker.test.ts+++ b/control-api/test/services.workflowScheduleWorker.test.ts@@ -9,8 +9,10 @@ import {FORBID_PENDING_ACTIVE_BOUND_SECONDS,FORBID_RUNNING_GRACE_SECONDS,FORBID_RUNNING_MAX_DURATION_CAP_SECONDS,+ FORBID_SKIP_STREAK_CAPACITY,FORBID_SUSTAINED_SKIP_THRESHOLD,computeNextFire,+ concurrencyForbiddenStreakCountForTests,processMaturedSchedules,resetConcurrencyForbiddenStreaksForTests,} from '../src/services/workflowScheduleWorkerService.js'@@ -64,6 +66,7 @@ type FakeRun = {const activeRuns = new Map<string, FakeRun>()/** Test-configurable hook that can inject a failure into the active-run lookup. */let activeRunQueryError: Error | null = null+let nextFireUpdateError: Error | null = nullconst clientQuery = vi.fn(async (sql: unknown, params?: unknown[]): Promise<AnyQueryResult | LockQueryResult> => {@@ -124,6 +127,7 @@ const clientQuery = vi.fn(}if (/UPDATE workflow_schedules/i.test(text) && /next_fire_at/i.test(text)) {+ if (nextFireUpdateError) throw nextFireUpdateErrorconst nextFire = params?.[0] as Dateconst scheduleId = String(params?.[1] ?? '')updatedSchedules.push({ schedule_id: scheduleId, next_fire_at: nextFire })@@ -210,6 +214,7 @@ describe('processMaturedSchedules', () => {selectQueryError = nullactiveRuns.clear()activeRunQueryError = null+ nextFireUpdateError = nullclientQuery.mockClear()clientRelease.mockClear()mockConnect.mockClear()@@ -737,6 +742,33 @@ describe('processMaturedSchedules', () => {expect(forbiddenSkips).toEqual([1, 1])})+ it('caps the skip-streak map and evicts the least recently skipped schedule first', async () => {+ setActiveRun()+ const total = FORBID_SKIP_STREAK_CAPACITY + 1+ for (let i = 0; i < total; i += 1) maturedRows.push(scheduleRow({ schedule_id: `s-${i}` }))+ await processMaturedSchedules({ batchSize: total })+ expect(concurrencyForbiddenStreakCountForTests()).toBe(FORBID_SKIP_STREAK_CAPACITY)++ // s-0 was evicted, so its next consecutive skip restarts at 1; s-1 was kept and counts 2.+ vi.setSystemTime(new Date('2026-04-20T09:05:05Z'))+ vi.mocked(rootLogger.info).mockClear()+ const slot = new Date('2026-04-20T09:05:00Z')+ // s-1 goes first: re-adding s-0 evicts the then-oldest entry, which would be s-1.+ maturedRows.push(scheduleRow({ schedule_id: 's-1', next_fire_at: slot }))+ maturedRows.push(scheduleRow({ schedule_id: 's-0', next_fire_at: slot }))+ await processMaturedSchedules()++ const streaks = Object.fromEntries(+ vi+ .mocked(rootLogger.info)+ .mock.calls.map(([fields]) => fields as Record<string, unknown>)+ .filter(fields => fields?.event === 'workflow_schedule_worker_concurrency_forbidden')+ .map(fields => [fields.scheduleId, fields.consecutiveSkips])+ )+ expect(streaks).toEqual({ 's-0': 1, 's-1': 2 })+ expect(concurrencyForbiddenStreakCountForTests()).toBe(FORBID_SKIP_STREAK_CAPACITY)+ })+it('defaults a null concurrency_policy to Forbid', async () => {setActiveRun()maturedRows.push(scheduleRow({ concurrency_policy: null }))@@ -827,6 +859,20 @@ describe('processMaturedSchedules', () => {expect(txEvents).toEqual(['BEGIN', 'ROLLBACK'])expect(clientRelease).toHaveBeenCalledTimes(1)})++ it('rolls back without firing when the skip write fails', async () => {+ setActiveRun()+ nextFireUpdateError = new Error('could not serialize access')+ maturedRows.push(scheduleRow())++ await expect(processMaturedSchedules()).rejects.toThrow('could not serialize access')++ expect(createRunMock).not.toHaveBeenCalled()+ expect(activeRunLookups()).toHaveLength(1)+ expect(txEvents).toEqual(['BEGIN', 'ROLLBACK'])+ expect(clientRelease).toHaveBeenCalledTimes(1)+ expect(concurrencyForbiddenStreakCountForTests()).toBe(0)+ })})})diff --git a/docs/crds/workflowrecipe.md b/docs/crds/workflowrecipe.mdindex 28c7737fc..d23598fbb 100644--- a/docs/crds/workflowrecipe.md+++ b/docs/crds/workflowrecipe.md@@ -1183,11 +1183,11 @@ spec:**`concurrencyPolicy`** (and `spec.scheduling.concurrencyPolicy` on the legacy path, §3.14) applies to scheduled ticks only; on-demand and autonomous triggers are never gated by it.-- `Forbid` (default): a tick is skipped while a run of the recipe is still live, whatever started it. Live means `Pending` for less than 15 minutes, or `Running` for less than `maxRunDurationSeconds` plus a 5-minute grace. The schedule worker caps that duration at 24 hours (the controller maximum) and assumes 24 hours when it is unset; the WRC's own default run duration is 1 hour, so this is only the blocking bound, not how long a child runs. The schedule advances to its next future window; skipped windows are not replayed.+- `Forbid` (default): a tick is skipped while a run of the recipe is still live, whatever started it. Live means `Pending` for less than 15 minutes, or `Running` for less than `maxRunDurationSeconds` plus a 5-minute grace. The schedule worker caps that duration at 24 hours (the controller maximum) and assumes 24 hours when it is unset. That is only the Forbid blocking bound: a run with no `maxRunDurationSeconds` stores no duration, and the timeout reaper never fails it, so nothing else ends it on a deadline. The schedule advances to its next future window; skipped windows are not replayed.- `Allow`: every tick fires, and runs may overlap.- `Replace`: accepted but **not implemented**. It currently behaves exactly like `Allow`.-A run that never reaches a terminal phase stops blocking once it passes that bound, so a stuck run cannot silence a schedule forever. After the bound, one new scheduled run may overlap the stuck one. Set `spec.runRetention.maxRunDurationSeconds` so a stuck run is failed (and the schedule resumes) sooner than the 24 hours the worker assumes when it is unset. Every skip is logged as `workflow_schedule_worker_concurrency_forbidden` with the blocking run id and age. Six consecutive skips of one schedule also log `workflow_schedule_worker_concurrency_forbidden_sustained` (warn) and increment `wrc_schedule_worker_concurrency_forbidden_sustained_total`.+A run that never reaches a terminal phase stops blocking once it passes that bound, so a stuck run cannot silence a schedule forever. After the bound, one new scheduled run may overlap the stuck one. Set `spec.runRetention.maxRunDurationSeconds` so the reaper fails a stuck run (and the schedule resumes) sooner than the 24 hours the worker assumes when it is unset. Every skip is logged as `workflow_schedule_worker_concurrency_forbidden` with the blocking run id and age. Six consecutive skips of one schedule on the same control-api replica also log `workflow_schedule_worker_concurrency_forbidden_sustained` (warn) and increment `wrc_schedule_worker_concurrency_forbidden_sustained_total`. The streak is kept in memory per replica, so it restarts at 1 when another replica takes the schedule lock, the process restarts, or the replica already tracks 1,000 skipped schedules. Treat the warning as best effort and alert on the per-skip log as well.### 3.14 Scheduling (`spec.scheduling`) — legacy