Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript

fix(control-api): bound the Forbid skip-streak map and correct its docs

Jose Ramirez76a8299147db
diff --git a/control-api/package-lock.json b/control-api/package-lock.json
index 440d29703..6b4c1ff56 100644
--- a/control-api/package-lock.json
+++ b/control-api/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "clerum-control-api",
- "version": "0.1.742",
+ "version": "0.1.743",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "clerum-control-api",
- "version": "0.1.742",
+ "version": "0.1.743",
"dependencies": {
"@clerum/display-field": "file:../packages/display-field",
"@clerum/grok-provider-attempt-contract": "file:../packages/grok-provider-attempt-contract",
diff --git a/control-api/package.json b/control-api/package.json
index 0036854a4..f6c177d2c 100644
--- a/control-api/package.json
+++ b/control-api/package.json
@@ -1,6 +1,6 @@
{
"name": "clerum-control-api",
- "version": "0.1.742",
+ "version": "0.1.743",
"description": "Internal Clerum control plane REST API + UI for managing CRDs and secrets",
"main": "dist/main.js",
"engines": {
diff --git a/control-api/src/services/workflowScheduleWorkerService.ts b/control-api/src/services/workflowScheduleWorkerService.ts
index 94a73672e..7ec4eff01 100644
--- a/control-api/src/services/workflowScheduleWorkerService.ts
+++ b/control-api/src/services/workflowScheduleWorkerService.ts
@@ -119,6 +119,9 @@ export const FORBID_RUNNING_GRACE_SECONDS = 5 * 60
/** Consecutive Forbid skips of one schedule before a warn log + counter fire. */
export const FORBID_SUSTAINED_SKIP_THRESHOLD = 6
+/** Upper bound on per-replica skip-streak entries (see `forbiddenSkipStreaks`). */
+export const FORBID_SKIP_STREAK_CAPACITY = 1000
+
/** A live non-terminal run (see the CHECK on `workflow_runs.phase`). */
interface ActiveRecipeRun {
run_id: string
@@ -135,12 +138,18 @@ interface ActiveRecipeRun {
* never duplicated. `expectedFireAt` is the next_fire_at this replica wrote
* with its last skip; if the row arrives with any other slot (another replica
* fired or skipped it, or the batch rolled back) the streak restarts at 1.
- * Only a fire deletes an entry; a schedule skipped and then deleted, disabled
- * or actor-gated keeps its entry until restart. The map is therefore bounded
- * by the number of schedules.
+ * A fire deletes the entry. A schedule skipped and then deleted, disabled or
+ * actor-gated would otherwise keep its entry until restart, so the map holds at
+ * most FORBID_SKIP_STREAK_CAPACITY entries and evicts the least recently
+ * skipped one first. An evicted schedule that skips again restarts at 1.
*/
const forbiddenSkipStreaks = new Map<string, { count: number; expectedFireAt: string }>()
+/** Test-only: number of schedules with a live skip streak on this replica. */
+export function concurrencyForbiddenStreakCountForTests(): number {
+ return forbiddenSkipStreaks.size
+}
+
/** Test-only: forget every streak so cases do not leak into each other. */
export function resetConcurrencyForbiddenStreaksForTests(): void {
forbiddenSkipStreaks.clear()
@@ -579,10 +588,16 @@ async function skipForbiddenTick(
const previous = forbiddenSkipStreaks.get(row.schedule_id)
const consecutiveSkips =
previous && previous.expectedFireAt === currentFire.toISOString() ? previous.count + 1 : 1
+ // Delete first so the entry moves to the end of the Map's insertion order.
+ forbiddenSkipStreaks.delete(row.schedule_id)
forbiddenSkipStreaks.set(row.schedule_id, {
count: consecutiveSkips,
expectedFireAt: skipNextFire.toISOString(),
})
+ for (const scheduleId of forbiddenSkipStreaks.keys()) {
+ if (forbiddenSkipStreaks.size <= FORBID_SKIP_STREAK_CAPACITY) break
+ forbiddenSkipStreaks.delete(scheduleId)
+ }
const activeSince = new Date(activeRun.started_at ?? activeRun.created_at)
const fields = {
diff --git a/control-api/test/services.workflowScheduleWorker.test.ts b/control-api/test/services.workflowScheduleWorker.test.ts
index 31158c58c..8ce02c4bf 100644
--- a/control-api/test/services.workflowScheduleWorker.test.ts
+++ b/control-api/test/services.workflowScheduleWorker.test.ts
@@ -9,8 +9,10 @@ import {
FORBID_PENDING_ACTIVE_BOUND_SECONDS,
FORBID_RUNNING_GRACE_SECONDS,
FORBID_RUNNING_MAX_DURATION_CAP_SECONDS,
+ FORBID_SKIP_STREAK_CAPACITY,
FORBID_SUSTAINED_SKIP_THRESHOLD,
computeNextFire,
+ concurrencyForbiddenStreakCountForTests,
processMaturedSchedules,
resetConcurrencyForbiddenStreaksForTests,
} from '../src/services/workflowScheduleWorkerService.js'
@@ -64,6 +66,7 @@ type FakeRun = {
const activeRuns = new Map<string, FakeRun>()
/** Test-configurable hook that can inject a failure into the active-run lookup. */
let activeRunQueryError: Error | null = null
+let nextFireUpdateError: Error | null = null
const clientQuery = vi.fn(
async (sql: unknown, params?: unknown[]): Promise<AnyQueryResult | LockQueryResult> => {
@@ -124,6 +127,7 @@ const clientQuery = vi.fn(
}
if (/UPDATE workflow_schedules/i.test(text) && /next_fire_at/i.test(text)) {
+ if (nextFireUpdateError) throw nextFireUpdateError
const nextFire = params?.[0] as Date
const scheduleId = String(params?.[1] ?? '')
updatedSchedules.push({ schedule_id: scheduleId, next_fire_at: nextFire })
@@ -210,6 +214,7 @@ describe('processMaturedSchedules', () => {
selectQueryError = null
activeRuns.clear()
activeRunQueryError = null
+ nextFireUpdateError = null
clientQuery.mockClear()
clientRelease.mockClear()
mockConnect.mockClear()
@@ -737,6 +742,33 @@ describe('processMaturedSchedules', () => {
expect(forbiddenSkips).toEqual([1, 1])
})
+ it('caps the skip-streak map and evicts the least recently skipped schedule first', async () => {
+ setActiveRun()
+ const total = FORBID_SKIP_STREAK_CAPACITY + 1
+ for (let i = 0; i < total; i += 1) maturedRows.push(scheduleRow({ schedule_id: `s-${i}` }))
+ await processMaturedSchedules({ batchSize: total })
+ expect(concurrencyForbiddenStreakCountForTests()).toBe(FORBID_SKIP_STREAK_CAPACITY)
+
+ // s-0 was evicted, so its next consecutive skip restarts at 1; s-1 was kept and counts 2.
+ vi.setSystemTime(new Date('2026-04-20T09:05:05Z'))
+ vi.mocked(rootLogger.info).mockClear()
+ const slot = new Date('2026-04-20T09:05:00Z')
+ // s-1 goes first: re-adding s-0 evicts the then-oldest entry, which would be s-1.
+ maturedRows.push(scheduleRow({ schedule_id: 's-1', next_fire_at: slot }))
+ maturedRows.push(scheduleRow({ schedule_id: 's-0', next_fire_at: slot }))
+ await processMaturedSchedules()
+
+ const streaks = Object.fromEntries(
+ vi
+ .mocked(rootLogger.info)
+ .mock.calls.map(([fields]) => fields as Record<string, unknown>)
+ .filter(fields => fields?.event === 'workflow_schedule_worker_concurrency_forbidden')
+ .map(fields => [fields.scheduleId, fields.consecutiveSkips])
+ )
+ expect(streaks).toEqual({ 's-0': 1, 's-1': 2 })
+ expect(concurrencyForbiddenStreakCountForTests()).toBe(FORBID_SKIP_STREAK_CAPACITY)
+ })
+
it('defaults a null concurrency_policy to Forbid', async () => {
setActiveRun()
maturedRows.push(scheduleRow({ concurrency_policy: null }))
@@ -827,6 +859,20 @@ describe('processMaturedSchedules', () => {
expect(txEvents).toEqual(['BEGIN', 'ROLLBACK'])
expect(clientRelease).toHaveBeenCalledTimes(1)
})
+
+ it('rolls back without firing when the skip write fails', async () => {
+ setActiveRun()
+ nextFireUpdateError = new Error('could not serialize access')
+ maturedRows.push(scheduleRow())
+
+ await expect(processMaturedSchedules()).rejects.toThrow('could not serialize access')
+
+ expect(createRunMock).not.toHaveBeenCalled()
+ expect(activeRunLookups()).toHaveLength(1)
+ expect(txEvents).toEqual(['BEGIN', 'ROLLBACK'])
+ expect(clientRelease).toHaveBeenCalledTimes(1)
+ expect(concurrencyForbiddenStreakCountForTests()).toBe(0)
+ })
})
})
diff --git a/docs/crds/workflowrecipe.md b/docs/crds/workflowrecipe.md
index 28c7737fc..d23598fbb 100644
--- a/docs/crds/workflowrecipe.md
+++ b/docs/crds/workflowrecipe.md
@@ -1183,11 +1183,11 @@ spec:
**`concurrencyPolicy`** (and `spec.scheduling.concurrencyPolicy` on the legacy path, §3.14) applies to scheduled ticks only; on-demand and autonomous triggers are never gated by it.
-- `Forbid` (default): a tick is skipped while a run of the recipe is still live, whatever started it. Live means `Pending` for less than 15 minutes, or `Running` for less than `maxRunDurationSeconds` plus a 5-minute grace. The schedule worker caps that duration at 24 hours (the controller maximum) and assumes 24 hours when it is unset; the WRC's own default run duration is 1 hour, so this is only the blocking bound, not how long a child runs. The schedule advances to its next future window; skipped windows are not replayed.
+- `Forbid` (default): a tick is skipped while a run of the recipe is still live, whatever started it. Live means `Pending` for less than 15 minutes, or `Running` for less than `maxRunDurationSeconds` plus a 5-minute grace. The schedule worker caps that duration at 24 hours (the controller maximum) and assumes 24 hours when it is unset. That is only the Forbid blocking bound: a run with no `maxRunDurationSeconds` stores no duration, and the timeout reaper never fails it, so nothing else ends it on a deadline. The schedule advances to its next future window; skipped windows are not replayed.
- `Allow`: every tick fires, and runs may overlap.
- `Replace`: accepted but **not implemented**. It currently behaves exactly like `Allow`.
-A run that never reaches a terminal phase stops blocking once it passes that bound, so a stuck run cannot silence a schedule forever. After the bound, one new scheduled run may overlap the stuck one. Set `spec.runRetention.maxRunDurationSeconds` so a stuck run is failed (and the schedule resumes) sooner than the 24 hours the worker assumes when it is unset. Every skip is logged as `workflow_schedule_worker_concurrency_forbidden` with the blocking run id and age. Six consecutive skips of one schedule also log `workflow_schedule_worker_concurrency_forbidden_sustained` (warn) and increment `wrc_schedule_worker_concurrency_forbidden_sustained_total`.
+A run that never reaches a terminal phase stops blocking once it passes that bound, so a stuck run cannot silence a schedule forever. After the bound, one new scheduled run may overlap the stuck one. Set `spec.runRetention.maxRunDurationSeconds` so the reaper fails a stuck run (and the schedule resumes) sooner than the 24 hours the worker assumes when it is unset. Every skip is logged as `workflow_schedule_worker_concurrency_forbidden` with the blocking run id and age. Six consecutive skips of one schedule on the same control-api replica also log `workflow_schedule_worker_concurrency_forbidden_sustained` (warn) and increment `wrc_schedule_worker_concurrency_forbidden_sustained_total`. The streak is kept in memory per replica, so it restarts at 1 when another replica takes the schedule lock, the process restarts, or the replica already tracks 1,000 skipped schedules. Treat the warning as best effort and alert on the per-skip log as well.
### 3.14 Scheduling (`spec.scheduling`) — legacy