Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.
TypeScript acmeagentsai-agentscertificate-managementdevsecopsenvironment-variableskubernetesopen-sourcepampkipostgresprivate-caprivileged-access-managementsecret-managementsecret-managersecretssecrets-managementsecuritysecurity-toolsvault
feat(cron): implement jitter and concurrency controls for cron jobs
Victor Hugo dos Santos716f2015710b
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.mdindex 0e8ad6df59..faf5034558 100644--- a/backend/CLAUDE.md+++ b/backend/CLAUDE.md@@ -662,6 +662,9 @@ Recurring work runs through the cron manager in `src/lib/cron/cron-job.ts` (`cro- Handlers must be idempotent at the boundary of `handlerTimeoutMs` (default 5 min). A timeout marks the run failed-final and waits for the next fire — it does NOT retry the same fire, because the timed-out handler may still be running.- Failures (non-timeout) retry with exponential backoff (base 30 s, max 5 min) up to `maxAttempts`, but only if the retry would still fit before the next scheduled fire. Otherwise the next fire is treated as the natural retry.- Long-running handlers should override `handlerTimeoutMs` / `leaseDurationMs` per-entry (must satisfy `handlerTimeoutMs <= leaseDurationMs`).+- **A fire is not picked up at its scheduled time; it is picked up at a deterministic offset past it.** Cron patterns cluster hard, so without a spread one pod claims a dozen handlers in a single tick. Two knobs bound that, both in `cronJobFactory`:+ - **Jitter** spreads pickup, and is *derived, never configured*: a job's window is `JITTER_INTERVAL_FRACTION` (0.25) of its own cron interval, capped by the factory's `maxJitterMs` (15 min), so a `*/5` job gets 75s and anything hourly or rarer gets the full 15 min. A fraction below 1 is what makes it safe by construction: a run can never reach its own next fire, and the first retry always fits (worst case `0.25 x interval + 30s backoff + 1s`, under `interval` for anything above ~41s, and cron cannot fire faster than every 60s). The offset within the window is `sha256(name) % window`, never random: every pod computes a run's eligibility independently and the run id is keyed on the *unjittered* fire, so a per-pod offset would break the Redis enqueue dedup and the lease logic. Hashing also spreads jobs permanently instead of letting them re-collide next fire. Only the pending-zset score carries the offset; the run id and `scheduled_at` are unchanged. `register` rejects a `runHashTtlS` that does not outlive the job's own window, since the hash would expire before pickup and the run would be reaped silently. Development wiring passes `maxJitterMs: 0` so a job runs when its pattern says.+ - **`maxConcurrentHandlers`** bounds accumulation across *overlapping* ticks, the failure mode when a handler runs long. **It does not bound a burst**, because the `processTick` loop is serial (`await processCandidate` per id), so one tick works through its whole batch back to back on a single pod whatever the cap is. `PROCESS_BATCH_SIZE` still bounds the zset scan, not execution.**When to use cron vs. queue**:- Scheduled/recurring (every N minutes, daily at X, cron pattern) → `cronJob.register(...)`.diff --git a/backend/src/lib/cron/cron-job.test.ts b/backend/src/lib/cron/cron-job.test.tsindex 2a671dc00f..023f405adf 100644--- a/backend/src/lib/cron/cron-job.test.ts+++ b/backend/src/lib/cron/cron-job.test.ts@@ -729,3 +729,310 @@ describe("stop", () => {expect(redis.eval.mock.calls.some(isSlotReleaseEval)).toBe(true);});});+// Cron patterns cluster: the every-5, every-10 and every-15 minute families all+// coincide on the hour and the half hour, so a single pod can claim a dozen+// handlers in one tick and block the event loop the HTTP API shares. Jitter+// spreads the pickups; the concurrency cap below bounds them outright.+describe("derived jitter", () => {+ const SCHEDULED_AT = Date.parse("2024-01-01T00:00:00Z");+ // Mid-interval, so prevFire is SCHEDULED_AT for every pattern used below.+ const NOW = new Date("2024-01-01T00:00:30Z");+ const MAX_JITTER_MS = 15 * 60_000; // factory default+ const FRACTION = 0.25;+ const BACKOFF_MS = 30_000; // retryBackoffBaseMs default+ const NEXT_FIRE_BUFFER_MS = 1_000;++ // What the manager should derive for a given interval. Kept as an independent+ // expression of the rule rather than a copy of the implementation's `Math.min`.+ const expectedWindowMs = (intervalMs: number, maxJitterMs = MAX_JITTER_MS) =>+ Math.min(intervalMs * FRACTION, maxJitterMs, intervalMs - NEXT_FIRE_BUFFER_MS);++ // ENQUEUE_RUN_LUA is the only two-key script:+ // eval(script, 2, runKey, pendingZset, name, scheduledAt, ttl, id, enqueuedAtMs, eligibleAt)+ const enqueuedRuns = (redis: ReturnType<typeof makeRedis>) =>+ redis.eval.mock.calls+ .filter((c) => c[1] === 2)+ .map((c) => ({+ name: String(c[4]),+ scheduledAt: Number(c[5]),+ id: String(c[7]),+ eligibleAt: Number(c[9])+ }));++ // Boots a manager, drives one enqueue tick, and returns what it wrote.+ const enqueueOnce = async (+ entries: Array<{ name: string; pattern: string; runHashTtlS?: number }>,+ maxJitterMs?: number+ ) => {+ vi.setSystemTime(NOW);+ const redis = makeRedis();+ redis.set.mockResolvedValue("OK"); // slot claim succeeds+ redis.eval.mockResolvedValue(1); // enqueue wins the race+ const f = cronJobFactory({+ redis: redis as never,+ redlock: makeRedlock() as never,+ slotRefreshMs: 50,+ enqueueIntervalMs: 100,+ processIntervalMs: 100,+ slotTtlMs: 200,+ leaseDurationMs: 1000,+ handlerTimeoutMs: 1000,+ ...(maxJitterMs === undefined ? {} : { maxJitterMs })+ });+ entries.forEach(({ name, pattern, runHashTtlS }) =>+ f.register({ name, pattern, handler: vi.fn(), runHashTtlS: runHashTtlS ?? 3 * 24 * 60 * 60 })+ );+ f.start();+ await vi.advanceTimersByTimeAsync(150);+ await f.stop();+ return enqueuedRuns(redis);+ };++ // The whole point of deriving rather than configuring: nothing is passed at the+ // registration, and the window still tracks how often the job runs.+ test("window is a fraction of the job's own interval, capped", async () => {+ const cases: Array<[string, number]> = [+ ["*/5 * * * *", 5 * 60_000],+ ["*/10 * * * *", 10 * 60_000],+ ["*/15 * * * *", 15 * 60_000],+ ["0 * * * *", 60 * 60_000],+ ["0 0 * * *", 24 * 60 * 60_000]+ ];+ // One registration per manager so each name's offset is read against its own pattern.+ for (const [pattern, intervalMs] of cases) {+ // eslint-disable-next-line no-await-in-loop+ const [run] = await enqueueOnce([{ name: "x", pattern }]);+ const offset = run.eligibleAt - run.scheduledAt;+ expect(offset).toBeGreaterThanOrEqual(0);+ expect(offset).toBeLessThan(expectedWindowMs(intervalMs));+ }+ });++ // An hourly job and a daily job both sit on the cap, so past ~1h the window stops growing.+ test("cap bounds the window for long intervals", async () => {+ const [hourly] = await enqueueOnce([{ name: "x", pattern: "0 * * * *" }]);+ const [daily] = await enqueueOnce([{ name: "x", pattern: "0 0 * * *" }]);+ expect(hourly.eligibleAt - hourly.scheduledAt).toBeLessThan(MAX_JITTER_MS);+ expect(daily.eligibleAt - daily.scheduledAt).toBeLessThan(MAX_JITTER_MS);+ });++ // The offset has to come from a stable hash of the job name, never Math.random():+ // every pod computes a run's eligibility time independently, so a per-pod offset+ // would make pods disagree and break the run-id dedup and the lease logic.+ test("two instances derive the same offset for the same job name", async () => {+ const [first] = await enqueueOnce([{ name: "telemetry-aggregated-events", pattern: "*/10 * * * *" }]);+ const [second] = await enqueueOnce([{ name: "telemetry-aggregated-events", pattern: "*/10 * * * *" }]);++ expect(first.eligibleAt).toBe(second.eligibleAt);+ expect(first.eligibleAt).toBeGreaterThan(SCHEDULED_AT);+ });++ // A shared-but-constant offset would satisfy determinism while spreading nothing.+ test("different job names land on different offsets", async () => {+ const runs = await enqueueOnce([+ { name: "telemetry-aggregated-events", pattern: "*/10 * * * *" },+ { name: "daily-resource-cleanup", pattern: "*/10 * * * *" },+ { name: "certificate-cleanup", pattern: "*/10 * * * *" },+ { name: "pki-sync-health-check", pattern: "*/10 * * * *" },+ { name: "health-alert", pattern: "*/10 * * * *" }+ ]);++ const offsets = runs.map((r) => r.eligibleAt - r.scheduledAt);+ expect(offsets).toHaveLength(5);+ expect(new Set(offsets).size).toBe(5);+ });++ // A run due at or past its own next fire would invert the run order and+ // invalidate `fitsBeforeNextFire`, which measures a retry against the raw next fire.+ test("a run is never eligible at or past its own next fire", async () => {+ const [run] = await enqueueOnce([{ name: "x", pattern: "* * * * *", runHashTtlS: 60 * 60 }]);+ const nextFire = Date.parse("2024-01-01T00:01:00Z");++ expect(run.scheduledAt).toBe(SCHEDULED_AT);+ expect(run.eligibleAt).toBeGreaterThanOrEqual(SCHEDULED_AT);+ expect(run.eligibleAt).toBeLessThanOrEqual(nextFire - NEXT_FIRE_BUFFER_MS);+ });++ // The reason JITTER_INTERVAL_FRACTION is 0.25 and not something larger: the offset is+ // spent from the same interval the retry backoff needs, so a wide window on a short+ // pattern silently converts every failure into failed-final. Cron cannot fire faster+ // than once a minute, so checking the minute case checks the worst case.+ test("the first retry still fits before the next fire, at every interval", async () => {+ const intervals: Array<[string, number]> = [+ ["* * * * *", 60_000],+ ["*/5 * * * *", 5 * 60_000],+ ["*/15 * * * *", 15 * 60_000],+ ["0 * * * *", 60 * 60_000],+ ["0 0 * * *", 24 * 60 * 60_000]+ ];+ for (const [, intervalMs] of intervals) {+ const worstOffset = expectedWindowMs(intervalMs);+ expect(worstOffset + BACKOFF_MS + NEXT_FIRE_BUFFER_MS).toBeLessThan(intervalMs);+ }++ // And the derived offset really does stay under that bound in practice.+ const [run] = await enqueueOnce([{ name: "x", pattern: "* * * * *", runHashTtlS: 60 * 60 }]);+ const offset = run.eligibleAt - run.scheduledAt;+ expect(offset + BACKOFF_MS + NEXT_FIRE_BUFFER_MS).toBeLessThan(60_000);+ });++ // Run identity is what the Redis dedup and the Redlock lease key are built+ // from, so it must stay keyed on the unjittered fire.+ test("run id and scheduled_at stay keyed on the unjittered fire", async () => {+ const [run] = await enqueueOnce([{ name: "x", pattern: "*/10 * * * *" }]);++ expect(run.id).toBe(`x:${SCHEDULED_AT}`);+ expect(run.scheduledAt).toBe(SCHEDULED_AT);+ });++ test("maxJitterMs: 0 makes every run eligible at its exact scheduled fire", async () => {+ const runs = await enqueueOnce(+ [+ { name: "x", pattern: "*/10 * * * *" },+ { name: "y", pattern: "0 0 * * *" }+ ],+ 0+ );++ expect(runs).toHaveLength(2);+ runs.forEach((run) => expect(run.eligibleAt).toBe(run.scheduledAt));+ });++ test("factory rejects a negative maxJitterMs", () => {+ expect(() =>+ cronJobFactory({ redis: makeRedis() as never, redlock: makeRedlock() as never, maxJitterMs: -1 })+ ).toThrow(/maxJitterMs/);+ });++ // A run hash that expires before its jittered pickup is reaped as an orphan,+ // so the job would silently never run. Catch that at registration.+ test("register rejects a runHashTtlS shorter than the derived window", () => {+ const { register } = makeFactory();+ // Daily pattern, so the window is the 15-minute cap; a 60s hash cannot cover it.+ expect(() => register({ name: "x", pattern: "0 0 * * *", handler: vi.fn(), runHashTtlS: 60 })).toThrow(+ /runHashTtlS.*must exceed its jitter window/+ );+ });++ // The guard measures the job's own window, not the cap, so a short-interval job+ // with a modest TTL is not rejected for a window it will never be given.+ test("register accepts a short runHashTtlS when the job's own window is small", () => {+ const { register } = makeFactory();+ // */15 gives a 225s window, comfortably under a 900s hash.+ expect(() =>+ register({ name: "x", pattern: "*/15 * * * *", handler: vi.fn(), runHashTtlS: 15 * 60 })+ ).not.toThrow();+ });+});++// Jitter is probabilistic. The per-pod cap is the deterministic guarantee that+// one pod never executes a burst of handlers on the event loop it shares with+// the HTTP API.+describe("handler concurrency cap", () => {+ const SCHEDULED_AT = Date.parse("2024-01-01T00:00:00Z");++ const setupPendingRuns = (redis: ReturnType<typeof makeRedis>, count: number) => {+ redis.set.mockResolvedValue("OK");+ redis.eval.mockResolvedValue(1);+ redis.zrangebyscore.mockResolvedValue(Array.from({ length: count }, (_, i) => `x:${SCHEDULED_AT - i * 60_000}`));+ redis.hgetall.mockResolvedValue({+ name: "x",+ status: "pending",+ attempts: "0",+ enqueued_at_ms: "0"+ });+ };++ // Handlers that hang until the test releases them, so in-flight count is+ // fully under the test's control.+ const makeBlockingHandler = () => {+ const resolvers: Array<() => void> = [];+ const handler = vi.fn().mockImplementation(+ () =>+ new Promise<void>((resolve) => {+ resolvers.push(resolve);+ })+ );+ return { handler, resolvers };+ };++ const makeCappedFactory = (+ redis: ReturnType<typeof makeRedis>,+ redlock: ReturnType<typeof makeRedlock>,+ maxConcurrentHandlers: number+ ) =>+ cronJobFactory({+ redis: redis as never,+ redlock: redlock as never,+ slotRefreshMs: 50,+ enqueueIntervalMs: 100,+ processIntervalMs: 100,+ slotTtlMs: 200,+ leaseDurationMs: 60_000,+ handlerTimeoutMs: 60_000,+ drainTimeoutMs: 50,+ maxConcurrentHandlers+ });++ test("stops claiming at the cap and leaves the surplus in the pending zset", async () => {+ vi.setSystemTime(new Date("2024-01-01T00:00:30Z"));+ const redis = makeRedis();+ setupPendingRuns(redis, 4);+ const { handler, resolvers } = makeBlockingHandler();+ const redlock = makeRedlock();++ const f = makeCappedFactory(redis, redlock, 1);+ f.register({ name: "x", pattern: "* * * * *", handler, runHashTtlS: 3600 });+ f.start();++ // Several process ticks see four due runs, but only one may be in flight.+ await vi.advanceTimersByTimeAsync(600);+ expect(redlock.using).toHaveBeenCalledTimes(1);++ // Deferred, not dropped: nothing was removed from the pending zset, so+ // another pod or a later tick can still take these runs.+ expect(redis.zrem).not.toHaveBeenCalled();++ // Freeing the slot lets the work resume rather than being lost.+ resolvers.forEach((resolve) => resolve());+ await vi.advanceTimersByTimeAsync(300);+ expect(redlock.using.mock.calls.length).toBeGreaterThan(1);++ resolvers.forEach((resolve) => resolve());+ const stopPromise = f.stop();+ await vi.advanceTimersByTimeAsync(100);+ await stopPromise;+ });++ // Guards against the cap collapsing to "one handler at a time".+ test("runs up to maxConcurrentHandlers at once", async () => {+ vi.setSystemTime(new Date("2024-01-01T00:00:30Z"));+ const redis = makeRedis();+ setupPendingRuns(redis, 4);+ const { handler, resolvers } = makeBlockingHandler();+ const redlock = makeRedlock();++ const f = makeCappedFactory(redis, redlock, 2);+ f.register({ name: "x", pattern: "* * * * *", handler, runHashTtlS: 3600 });+ f.start();++ await vi.advanceTimersByTimeAsync(600);+ expect(redlock.using).toHaveBeenCalledTimes(2);+ expect(redis.zrem).not.toHaveBeenCalled();++ resolvers.forEach((resolve) => resolve());+ const stopPromise = f.stop();+ await vi.advanceTimersByTimeAsync(100);+ await stopPromise;+ });++ test("factory rejects a cap below 1", () => {+ expect(() =>+ cronJobFactory({+ redis: makeRedis() as never,+ redlock: makeRedlock() as never,+ maxConcurrentHandlers: 0+ })+ ).toThrow(/maxConcurrentHandlers/);+ });+});diff --git a/backend/src/lib/cron/cron-job.ts b/backend/src/lib/cron/cron-job.tsindex 2f7096946d..84886e7483 100644--- a/backend/src/lib/cron/cron-job.ts+++ b/backend/src/lib/cron/cron-job.ts@@ -1,4 +1,4 @@-import { randomUUID } from "node:crypto";+import { createHash, randomUUID } from "node:crypto";import { CronExpressionParser } from "cron-parser";import { Cluster, Redis } from "ioredis";@@ -55,6 +55,16 @@ export const CronJobName = {// ── tuning constants ──────────────────────────────────────────────────────────+// Jitter is derived from the pattern, never configured per job: how long a job may be held+// back should follow how often it runs. A job's window is this fraction of its own interval,+// capped by `maxJitterMs`. A fraction below 1 is what makes the offset safe by construction,+// so it is not a free parameter:+// - a run can never reach its own next fire, which `fitsBeforeNextFire` relies on;+// - the first retry always fits. Worst case is 0.25 x interval + 30s backoff + 1s, under+// `interval` for any interval above ~41s, and cron cannot fire faster than every 60s.+// Raising this past ~0.48 would start costing short-interval jobs their retry.+const JITTER_INTERVAL_FRACTION = 0.25;+const PARTICIPANT_SLOTS = 5;const PROCESS_BATCH_SIZE = 50;// Safety buffer in `fitsBeforeNextFire`: if the retry's nextAttemptAt is within@@ -73,7 +83,9 @@ const DEFAULTS = {handlerTimeoutMs: 5 * 60_000,retryBackoffBaseMs: 30_000,retryBackoffMaxMs: 5 * 60_000,- drainTimeoutMs: 25_000+ drainTimeoutMs: 25_000,+ maxJitterMs: 15 * 60_000,+ maxConcurrentHandlers: 4} as const;// ── redis schema ──────────────────────────────────────────────────────────────@@ -108,6 +120,9 @@ const F = {// ── lua scripts ───────────────────────────────────────────────────────────────// Atomic "enqueue this run hash + zset entry if no other pod beat us to it".+// ARGV[2] is the unjittered scheduled fire, recorded on the hash as the run's+// identity. ARGV[6] is the jittered eligibility time, and the zset is scored by+// it so processTick's zrangebyscore holds the run back until its offset elapses.const ENQUEUE_RUN_LUA = `if redis.call('exists', KEYS[1]) == 0 thenredis.call('hset', KEYS[1],@@ -117,7 +132,7 @@ const ENQUEUE_RUN_LUA = `'attempts', 0,'enqueued_at_ms', ARGV[5])redis.call('expire', KEYS[1], ARGV[3])- redis.call('zadd', KEYS[2], ARGV[2], ARGV[4])+ redis.call('zadd', KEYS[2], ARGV[6], ARGV[4])return 1endreturn 0@@ -174,6 +189,8 @@ export const cronJobFactory = ({retryBackoffBaseMs = DEFAULTS.retryBackoffBaseMs,retryBackoffMaxMs = DEFAULTS.retryBackoffMaxMs,drainTimeoutMs = DEFAULTS.drainTimeoutMs,+ maxJitterMs = DEFAULTS.maxJitterMs,+ maxConcurrentHandlers = DEFAULTS.maxConcurrentHandlers,keyPrefix = KEY_HASH_TAG,schedulingEnabled = true}: {@@ -189,6 +206,19 @@ export const cronJobFactory = ({retryBackoffBaseMs?: number;retryBackoffMaxMs?: number;drainTimeoutMs?: number;+ /**+ * Ceiling on the derived jitter window. Without it a daily job would be spread over six+ * hours; with it, every job past a ~1h interval shares the same widest window. Set it to 0+ * to disable jitter, which is what development wiring does so a job runs when its pattern+ * says it should.+ */+ maxJitterMs?: number;+ /**+ * Ceiling on how many handlers this pod runs at once. At capacity the pod stops claiming for+ * the tick and leaves the runs in the pending zset, so another pod or a later tick picks them+ * up; nothing is dropped.+ */+ maxConcurrentHandlers?: number;/*** Namespace for every Redis key this manager owns. Defaults to the production* `{cron}` namespace. Tests override it so a test-owned manager and the@@ -203,6 +233,12 @@ export const cronJobFactory = ({schedulingEnabled?: boolean;}) => {assertHashTagged(keyPrefix);+ if (!Number.isFinite(maxJitterMs) || maxJitterMs < 0) {+ throw new Error(`cron: maxJitterMs (${maxJitterMs}) must be a non-negative number`);+ }+ if (!Number.isInteger(maxConcurrentHandlers) || maxConcurrentHandlers < 1) {+ throw new Error(`cron: maxConcurrentHandlers (${maxConcurrentHandlers}) must be an integer >= 1`);+ }const SLOT_KEY = (i: number) => `${keyPrefix}:slot:${i}`;const RUN_KEY = (id: string) => `${keyPrefix}:run:${id}`;@@ -231,6 +267,49 @@ export const cronJobFactory = ({const nextFireMs = (pattern: string) => CronExpressionParser.parse(pattern, { tz: "UTC" }).next().toDate().getTime();+ // How far past its scheduled fire a job with this interval may be held back. Proportional+ // to the interval so a frequent job is delayed proportionally less than a rare one, capped+ // so a daily job is not spread across the whole day. NEXT_FIRE_BUFFER_MS is belt-and-braces:+ // JITTER_INTERVAL_FRACTION already keeps the window well inside the interval, but the clamp+ // means the next-fire property survives someone raising the fraction.+ const jitterWindowMs = (intervalMs: number) =>+ Math.min(intervalMs * JITTER_INTERVAL_FRACTION, maxJitterMs, Math.max(0, intervalMs - NEXT_FIRE_BUFFER_MS));++ // The window for a pattern, sampled at registration. A pattern with uneven intervals (say+ // quarterly) gets its window recomputed per fire at enqueue; this sample only has to be good+ // enough for the run-hash TTL check.+ const jitterWindowForPattern = (pattern: string) => {+ const it = CronExpressionParser.parse(pattern, { tz: "UTC" });+ const next = it.next().toDate().getTime();+ return jitterWindowMs(it.next().toDate().getTime() - next);+ };++ // Per-job offset into the jitter window, derived from a stable hash of the job+ // name. It MUST NOT be random: the run id is keyed on the scheduled fire time+ // and every pod computes a run's eligibility time independently, so a per-pod offset+ // would make pods disagree on run identity and break the enqueue dedup and the+ // lease logic. Hashing also spreads the jobs permanently, where random offsets+ // would re-collide on the next fire.+ const jitterOffsetMs = (name: string, windowMs: number) => {+ if (windowMs <= 0) return 0;+ // Six bytes keeps the value inside Number.MAX_SAFE_INTEGER.+ return createHash("sha256").update(name).digest().readUIntBE(0, 6) % windowMs;+ };++ // The earliest a fire may be picked up: its scheduled time plus the job's+ // offset. Distinct from the run hash's `started_at`, which is when a handler+ // actually began. The window is clamped so an offset can never push a run to+ // or past its own next fire: the two would then become eligible in the wrong+ // order, and `fitsBeforeNextFire`, which measures a retry against the raw next+ // fire, would stop bounding the retry correctly. NEXT_FIRE_BUFFER_MS is the+ // same margin that check uses.+ const eligibleAtMs = (entry: CronEntry, scheduledAt: number, nextFire: number) => {+ const windowMs = jitterWindowMs(nextFire - scheduledAt);+ return scheduledAt + jitterOffsetMs(entry.name, windowMs);+ };++ const atHandlerCapacity = () => inFlight.size >= maxConcurrentHandlers;+const shuffleInPlace = <T>(arr: T[]): void => {for (let i = arr.length - 1; i > 0; i -= 1) {const j = Math.floor(Math.random() * (i + 1));@@ -345,6 +424,18 @@ export const cronJobFactory = ({if (entries.has(name)) throw new Error(`cron[${name}] already registered`);CronExpressionParser.parse(pattern, { tz: "UTC" }); // validate at registration+ // The run hash has to outlive the offset, or it expires before its own pickup and+ // processCandidate reaps the orphaned zset entry with no log and no failed status: the+ // job would silently never run. Measured against this pattern's own window.+ const windowMs = jitterWindowForPattern(pattern);+ if (runHashTtlS * 1000 <= windowMs) {+ throw new Error(+ `cron[${name}] runHashTtlS (${runHashTtlS}s) must exceed its jitter window (${Math.round(+ windowMs / 1000+ )}s), otherwise the run hash expires before the run is picked up`+ );+ }+const effectiveHandlerTimeoutMs = entryHandlerTimeoutMs ?? handlerTimeoutMs;const effectiveLeaseDurationMs = entryLeaseDurationMs ?? leaseDurationMs;if (effectiveHandlerTimeoutMs > effectiveLeaseDurationMs) {@@ -362,7 +453,7 @@ export const cronJobFactory = ({handlerTimeoutMs: entryHandlerTimeoutMs,leaseDurationMs: entryLeaseDurationMs});- logger.info(`cron[${name}]: registered (pattern="${pattern}")`);+ logger.info(`cron[${name}]: registered (pattern="${pattern}") [jitter_window_ms=${windowMs}]`);};// ── slot election ───────────────────────────────────────────────────────────@@ -412,6 +503,10 @@ export const cronJobFactory = ({// and atomically inserts a run hash + pending-zset entry if not already// present. The Lua script makes the existence check + write a single op so// multiple pods racing to enqueue the same fire produce exactly one run.+ //+ // The run id and the hash's `scheduled_at` stay keyed on the unjittered fire,+ // so run identity is exactly what it was before jitter existed. Only the zset+ // score carries the offset, which is what holds the run back from processTick.const enqueueDueFires = async () => {for (const entry of entries.values()) {const scheduledAt = prevFireMs(entry.pattern);@@ -419,6 +514,7 @@ export const cronJobFactory = ({if (lastEnqueuedAt.get(entry.name) === scheduledAt) continue;const id = `${entry.name}:${scheduledAt}`;+ const eligibleAt = eligibleAtMs(entry, scheduledAt, nextFireMs(entry.pattern));// eslint-disable-next-line no-await-in-loopconst initialized = await redis.eval(ENQUEUE_RUN_LUA,@@ -429,11 +525,14 @@ export const cronJobFactory = ({String(scheduledAt),String(entry.runHashTtlS),id,- String(Date.now())+ String(Date.now()),+ String(eligibleAt));if (initialized)logger.info(- `cron[${entry.name}]: enqueued run [id=${id}] [scheduled_at=${new Date(scheduledAt).toISOString()}]`+ `cron[${entry.name}]: enqueued run [id=${id}] [scheduled_at=${new Date(+ scheduledAt+ ).toISOString()}] [eligible_at=${new Date(eligibleAt).toISOString()}]`);lastEnqueuedAt.set(entry.name, scheduledAt);}@@ -546,6 +645,8 @@ export const cronJobFactory = ({logger.info(`cron[${data.name}]: re-claiming stalled run [id=${id}] [previous_worker=${data.worker_id}]`);}+ if (atHandlerCapacity()) return;+// Track the in-flight run so stop() can wait for it to settle before// tearing down. Lock-contention rejections settle within a tick, so they// don't measurably delay shutdown drain.@@ -576,12 +677,20 @@ export const cronJobFactory = ({const ids = await redis.zrangebyscore(PENDING_ZSET, "-inf", Date.now(), "LIMIT", 0, PROCESS_BATCH_SIZE);shuffleInPlace(ids);- for (const id of ids) {+ for (let i = 0; i < ids.length; i += 1) {+ if (atHandlerCapacity()) {+ logger.info(+ `cron: at handler capacity (${maxConcurrentHandlers}), deferring ${+ ids.length - i+ } due run(s) [worker=${workerId}]`+ );+ break;+ }try {// eslint-disable-next-line no-await-in-loop- await processCandidate(id);+ await processCandidate(ids[i]);} catch (err) {- logger.error({ err, id }, "cron: processCandidate failed");+ logger.error({ err, id: ids[i] }, "cron: processCandidate failed");}}};diff --git a/backend/src/server/routes/index.ts b/backend/src/server/routes/index.tsindex 7898b66f38..32fcf279b7 100644--- a/backend/src/server/routes/index.ts+++ b/backend/src/server/routes/index.ts@@ -606,7 +606,12 @@ export const registerRoutes = async (const appCfg = getConfig();const redlock = new Redlock([redis], { retryCount: 0 });- const cronJob = cronJobFactory({ redis, redlock, schedulingEnabled: envConfig.isGeneralWorkerRunModeEnabled });+ const cronJob = cronJobFactory({+ redis,+ redlock,+ schedulingEnabled: envConfig.isGeneralWorkerRunModeEnabled,+ maxJitterMs: envConfig.isDevelopmentMode ? 0 : undefined+ });if (envConfig.isGeneralWorkerRunModeEnabled) {cronJob.start();}