Skip to content

Infisical is the open-source platform for securing secrets, certificates, access, and AI Agents.

feat(cron): implement jitter and concurrency controls for cron jobs

Victor Hugo dos Santos716f2015710b
diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md
index 0e8ad6df59..faf5034558 100644
--- a/backend/CLAUDE.md
+++ b/backend/CLAUDE.md
@@ -662,6 +662,9 @@ Recurring work runs through the cron manager in `src/lib/cron/cron-job.ts` (`cro
- Handlers must be idempotent at the boundary of `handlerTimeoutMs` (default 5 min). A timeout marks the run failed-final and waits for the next fire — it does NOT retry the same fire, because the timed-out handler may still be running.
- Failures (non-timeout) retry with exponential backoff (base 30 s, max 5 min) up to `maxAttempts`, but only if the retry would still fit before the next scheduled fire. Otherwise the next fire is treated as the natural retry.
- Long-running handlers should override `handlerTimeoutMs` / `leaseDurationMs` per-entry (must satisfy `handlerTimeoutMs <= leaseDurationMs`).
+- **A fire is not picked up at its scheduled time; it is picked up at a deterministic offset past it.** Cron patterns cluster hard, so without a spread one pod claims a dozen handlers in a single tick. Two knobs bound that, both in `cronJobFactory`:
+ - **Jitter** spreads pickup, and is *derived, never configured*: a job's window is `JITTER_INTERVAL_FRACTION` (0.25) of its own cron interval, capped by the factory's `maxJitterMs` (15 min), so a `*/5` job gets 75s and anything hourly or rarer gets the full 15 min. A fraction below 1 is what makes it safe by construction: a run can never reach its own next fire, and the first retry always fits (worst case `0.25 x interval + 30s backoff + 1s`, under `interval` for anything above ~41s, and cron cannot fire faster than every 60s). The offset within the window is `sha256(name) % window`, never random: every pod computes a run's eligibility independently and the run id is keyed on the *unjittered* fire, so a per-pod offset would break the Redis enqueue dedup and the lease logic. Hashing also spreads jobs permanently instead of letting them re-collide next fire. Only the pending-zset score carries the offset; the run id and `scheduled_at` are unchanged. `register` rejects a `runHashTtlS` that does not outlive the job's own window, since the hash would expire before pickup and the run would be reaped silently. Development wiring passes `maxJitterMs: 0` so a job runs when its pattern says.
+ - **`maxConcurrentHandlers`** bounds accumulation across *overlapping* ticks, the failure mode when a handler runs long. **It does not bound a burst**, because the `processTick` loop is serial (`await processCandidate` per id), so one tick works through its whole batch back to back on a single pod whatever the cap is. `PROCESS_BATCH_SIZE` still bounds the zset scan, not execution.
**When to use cron vs. queue**:
- Scheduled/recurring (every N minutes, daily at X, cron pattern) → `cronJob.register(...)`.
diff --git a/backend/src/lib/cron/cron-job.test.ts b/backend/src/lib/cron/cron-job.test.ts
index 2a671dc00f..023f405adf 100644
--- a/backend/src/lib/cron/cron-job.test.ts
+++ b/backend/src/lib/cron/cron-job.test.ts
@@ -729,3 +729,310 @@ describe("stop", () => {
expect(redis.eval.mock.calls.some(isSlotReleaseEval)).toBe(true);
});
});
+// Cron patterns cluster: the every-5, every-10 and every-15 minute families all
+// coincide on the hour and the half hour, so a single pod can claim a dozen
+// handlers in one tick and block the event loop the HTTP API shares. Jitter
+// spreads the pickups; the concurrency cap below bounds them outright.
+describe("derived jitter", () => {
+ const SCHEDULED_AT = Date.parse("2024-01-01T00:00:00Z");
+ // Mid-interval, so prevFire is SCHEDULED_AT for every pattern used below.
+ const NOW = new Date("2024-01-01T00:00:30Z");
+ const MAX_JITTER_MS = 15 * 60_000; // factory default
+ const FRACTION = 0.25;
+ const BACKOFF_MS = 30_000; // retryBackoffBaseMs default
+ const NEXT_FIRE_BUFFER_MS = 1_000;
+
+ // What the manager should derive for a given interval. Kept as an independent
+ // expression of the rule rather than a copy of the implementation's `Math.min`.
+ const expectedWindowMs = (intervalMs: number, maxJitterMs = MAX_JITTER_MS) =>
+ Math.min(intervalMs * FRACTION, maxJitterMs, intervalMs - NEXT_FIRE_BUFFER_MS);
+
+ // ENQUEUE_RUN_LUA is the only two-key script:
+ // eval(script, 2, runKey, pendingZset, name, scheduledAt, ttl, id, enqueuedAtMs, eligibleAt)
+ const enqueuedRuns = (redis: ReturnType<typeof makeRedis>) =>
+ redis.eval.mock.calls
+ .filter((c) => c[1] === 2)
+ .map((c) => ({
+ name: String(c[4]),
+ scheduledAt: Number(c[5]),
+ id: String(c[7]),
+ eligibleAt: Number(c[9])
+ }));
+
+ // Boots a manager, drives one enqueue tick, and returns what it wrote.
+ const enqueueOnce = async (
+ entries: Array<{ name: string; pattern: string; runHashTtlS?: number }>,
+ maxJitterMs?: number
+ ) => {
+ vi.setSystemTime(NOW);
+ const redis = makeRedis();
+ redis.set.mockResolvedValue("OK"); // slot claim succeeds
+ redis.eval.mockResolvedValue(1); // enqueue wins the race
+ const f = cronJobFactory({
+ redis: redis as never,
+ redlock: makeRedlock() as never,
+ slotRefreshMs: 50,
+ enqueueIntervalMs: 100,
+ processIntervalMs: 100,
+ slotTtlMs: 200,
+ leaseDurationMs: 1000,
+ handlerTimeoutMs: 1000,
+ ...(maxJitterMs === undefined ? {} : { maxJitterMs })
+ });
+ entries.forEach(({ name, pattern, runHashTtlS }) =>
+ f.register({ name, pattern, handler: vi.fn(), runHashTtlS: runHashTtlS ?? 3 * 24 * 60 * 60 })
+ );
+ f.start();
+ await vi.advanceTimersByTimeAsync(150);
+ await f.stop();
+ return enqueuedRuns(redis);
+ };
+
+ // The whole point of deriving rather than configuring: nothing is passed at the
+ // registration, and the window still tracks how often the job runs.
+ test("window is a fraction of the job's own interval, capped", async () => {
+ const cases: Array<[string, number]> = [
+ ["*/5 * * * *", 5 * 60_000],
+ ["*/10 * * * *", 10 * 60_000],
+ ["*/15 * * * *", 15 * 60_000],
+ ["0 * * * *", 60 * 60_000],
+ ["0 0 * * *", 24 * 60 * 60_000]
+ ];
+ // One registration per manager so each name's offset is read against its own pattern.
+ for (const [pattern, intervalMs] of cases) {
+ // eslint-disable-next-line no-await-in-loop
+ const [run] = await enqueueOnce([{ name: "x", pattern }]);
+ const offset = run.eligibleAt - run.scheduledAt;
+ expect(offset).toBeGreaterThanOrEqual(0);
+ expect(offset).toBeLessThan(expectedWindowMs(intervalMs));
+ }
+ });
+
+ // An hourly job and a daily job both sit on the cap, so past ~1h the window stops growing.
+ test("cap bounds the window for long intervals", async () => {
+ const [hourly] = await enqueueOnce([{ name: "x", pattern: "0 * * * *" }]);
+ const [daily] = await enqueueOnce([{ name: "x", pattern: "0 0 * * *" }]);
+ expect(hourly.eligibleAt - hourly.scheduledAt).toBeLessThan(MAX_JITTER_MS);
+ expect(daily.eligibleAt - daily.scheduledAt).toBeLessThan(MAX_JITTER_MS);
+ });
+
+ // The offset has to come from a stable hash of the job name, never Math.random():
+ // every pod computes a run's eligibility time independently, so a per-pod offset
+ // would make pods disagree and break the run-id dedup and the lease logic.
+ test("two instances derive the same offset for the same job name", async () => {
+ const [first] = await enqueueOnce([{ name: "telemetry-aggregated-events", pattern: "*/10 * * * *" }]);
+ const [second] = await enqueueOnce([{ name: "telemetry-aggregated-events", pattern: "*/10 * * * *" }]);
+
+ expect(first.eligibleAt).toBe(second.eligibleAt);
+ expect(first.eligibleAt).toBeGreaterThan(SCHEDULED_AT);
+ });
+
+ // A shared-but-constant offset would satisfy determinism while spreading nothing.
+ test("different job names land on different offsets", async () => {
+ const runs = await enqueueOnce([
+ { name: "telemetry-aggregated-events", pattern: "*/10 * * * *" },
+ { name: "daily-resource-cleanup", pattern: "*/10 * * * *" },
+ { name: "certificate-cleanup", pattern: "*/10 * * * *" },
+ { name: "pki-sync-health-check", pattern: "*/10 * * * *" },
+ { name: "health-alert", pattern: "*/10 * * * *" }
+ ]);
+
+ const offsets = runs.map((r) => r.eligibleAt - r.scheduledAt);
+ expect(offsets).toHaveLength(5);
+ expect(new Set(offsets).size).toBe(5);
+ });
+
+ // A run due at or past its own next fire would invert the run order and
+ // invalidate `fitsBeforeNextFire`, which measures a retry against the raw next fire.
+ test("a run is never eligible at or past its own next fire", async () => {
+ const [run] = await enqueueOnce([{ name: "x", pattern: "* * * * *", runHashTtlS: 60 * 60 }]);
+ const nextFire = Date.parse("2024-01-01T00:01:00Z");
+
+ expect(run.scheduledAt).toBe(SCHEDULED_AT);
+ expect(run.eligibleAt).toBeGreaterThanOrEqual(SCHEDULED_AT);
+ expect(run.eligibleAt).toBeLessThanOrEqual(nextFire - NEXT_FIRE_BUFFER_MS);
+ });
+
+ // The reason JITTER_INTERVAL_FRACTION is 0.25 and not something larger: the offset is
+ // spent from the same interval the retry backoff needs, so a wide window on a short
+ // pattern silently converts every failure into failed-final. Cron cannot fire faster
+ // than once a minute, so checking the minute case checks the worst case.
+ test("the first retry still fits before the next fire, at every interval", async () => {
+ const intervals: Array<[string, number]> = [
+ ["* * * * *", 60_000],
+ ["*/5 * * * *", 5 * 60_000],
+ ["*/15 * * * *", 15 * 60_000],
+ ["0 * * * *", 60 * 60_000],
+ ["0 0 * * *", 24 * 60 * 60_000]
+ ];
+ for (const [, intervalMs] of intervals) {
+ const worstOffset = expectedWindowMs(intervalMs);
+ expect(worstOffset + BACKOFF_MS + NEXT_FIRE_BUFFER_MS).toBeLessThan(intervalMs);
+ }
+
+ // And the derived offset really does stay under that bound in practice.
+ const [run] = await enqueueOnce([{ name: "x", pattern: "* * * * *", runHashTtlS: 60 * 60 }]);
+ const offset = run.eligibleAt - run.scheduledAt;
+ expect(offset + BACKOFF_MS + NEXT_FIRE_BUFFER_MS).toBeLessThan(60_000);
+ });
+
+ // Run identity is what the Redis dedup and the Redlock lease key are built
+ // from, so it must stay keyed on the unjittered fire.
+ test("run id and scheduled_at stay keyed on the unjittered fire", async () => {
+ const [run] = await enqueueOnce([{ name: "x", pattern: "*/10 * * * *" }]);
+
+ expect(run.id).toBe(`x:${SCHEDULED_AT}`);
+ expect(run.scheduledAt).toBe(SCHEDULED_AT);
+ });
+
+ test("maxJitterMs: 0 makes every run eligible at its exact scheduled fire", async () => {
+ const runs = await enqueueOnce(
+ [
+ { name: "x", pattern: "*/10 * * * *" },
+ { name: "y", pattern: "0 0 * * *" }
+ ],
+ 0
+ );
+
+ expect(runs).toHaveLength(2);
+ runs.forEach((run) => expect(run.eligibleAt).toBe(run.scheduledAt));
+ });
+
+ test("factory rejects a negative maxJitterMs", () => {
+ expect(() =>
+ cronJobFactory({ redis: makeRedis() as never, redlock: makeRedlock() as never, maxJitterMs: -1 })
+ ).toThrow(/maxJitterMs/);
+ });
+
+ // A run hash that expires before its jittered pickup is reaped as an orphan,
+ // so the job would silently never run. Catch that at registration.
+ test("register rejects a runHashTtlS shorter than the derived window", () => {
+ const { register } = makeFactory();
+ // Daily pattern, so the window is the 15-minute cap; a 60s hash cannot cover it.
+ expect(() => register({ name: "x", pattern: "0 0 * * *", handler: vi.fn(), runHashTtlS: 60 })).toThrow(
+ /runHashTtlS.*must exceed its jitter window/
+ );
+ });
+
+ // The guard measures the job's own window, not the cap, so a short-interval job
+ // with a modest TTL is not rejected for a window it will never be given.
+ test("register accepts a short runHashTtlS when the job's own window is small", () => {
+ const { register } = makeFactory();
+ // */15 gives a 225s window, comfortably under a 900s hash.
+ expect(() =>
+ register({ name: "x", pattern: "*/15 * * * *", handler: vi.fn(), runHashTtlS: 15 * 60 })
+ ).not.toThrow();
+ });
+});
+
+// Jitter is probabilistic. The per-pod cap is the deterministic guarantee that
+// one pod never executes a burst of handlers on the event loop it shares with
+// the HTTP API.
+describe("handler concurrency cap", () => {
+ const SCHEDULED_AT = Date.parse("2024-01-01T00:00:00Z");
+
+ const setupPendingRuns = (redis: ReturnType<typeof makeRedis>, count: number) => {
+ redis.set.mockResolvedValue("OK");
+ redis.eval.mockResolvedValue(1);
+ redis.zrangebyscore.mockResolvedValue(Array.from({ length: count }, (_, i) => `x:${SCHEDULED_AT - i * 60_000}`));
+ redis.hgetall.mockResolvedValue({
+ name: "x",
+ status: "pending",
+ attempts: "0",
+ enqueued_at_ms: "0"
+ });
+ };
+
+ // Handlers that hang until the test releases them, so in-flight count is
+ // fully under the test's control.
+ const makeBlockingHandler = () => {
+ const resolvers: Array<() => void> = [];
+ const handler = vi.fn().mockImplementation(
+ () =>
+ new Promise<void>((resolve) => {
+ resolvers.push(resolve);
+ })
+ );
+ return { handler, resolvers };
+ };
+
+ const makeCappedFactory = (
+ redis: ReturnType<typeof makeRedis>,
+ redlock: ReturnType<typeof makeRedlock>,
+ maxConcurrentHandlers: number
+ ) =>
+ cronJobFactory({
+ redis: redis as never,
+ redlock: redlock as never,
+ slotRefreshMs: 50,
+ enqueueIntervalMs: 100,
+ processIntervalMs: 100,
+ slotTtlMs: 200,
+ leaseDurationMs: 60_000,
+ handlerTimeoutMs: 60_000,
+ drainTimeoutMs: 50,
+ maxConcurrentHandlers
+ });
+
+ test("stops claiming at the cap and leaves the surplus in the pending zset", async () => {
+ vi.setSystemTime(new Date("2024-01-01T00:00:30Z"));
+ const redis = makeRedis();
+ setupPendingRuns(redis, 4);
+ const { handler, resolvers } = makeBlockingHandler();
+ const redlock = makeRedlock();
+
+ const f = makeCappedFactory(redis, redlock, 1);
+ f.register({ name: "x", pattern: "* * * * *", handler, runHashTtlS: 3600 });
+ f.start();
+
+ // Several process ticks see four due runs, but only one may be in flight.
+ await vi.advanceTimersByTimeAsync(600);
+ expect(redlock.using).toHaveBeenCalledTimes(1);
+
+ // Deferred, not dropped: nothing was removed from the pending zset, so
+ // another pod or a later tick can still take these runs.
+ expect(redis.zrem).not.toHaveBeenCalled();
+
+ // Freeing the slot lets the work resume rather than being lost.
+ resolvers.forEach((resolve) => resolve());
+ await vi.advanceTimersByTimeAsync(300);
+ expect(redlock.using.mock.calls.length).toBeGreaterThan(1);
+
+ resolvers.forEach((resolve) => resolve());
+ const stopPromise = f.stop();
+ await vi.advanceTimersByTimeAsync(100);
+ await stopPromise;
+ });
+
+ // Guards against the cap collapsing to "one handler at a time".
+ test("runs up to maxConcurrentHandlers at once", async () => {
+ vi.setSystemTime(new Date("2024-01-01T00:00:30Z"));
+ const redis = makeRedis();
+ setupPendingRuns(redis, 4);
+ const { handler, resolvers } = makeBlockingHandler();
+ const redlock = makeRedlock();
+
+ const f = makeCappedFactory(redis, redlock, 2);
+ f.register({ name: "x", pattern: "* * * * *", handler, runHashTtlS: 3600 });
+ f.start();
+
+ await vi.advanceTimersByTimeAsync(600);
+ expect(redlock.using).toHaveBeenCalledTimes(2);
+ expect(redis.zrem).not.toHaveBeenCalled();
+
+ resolvers.forEach((resolve) => resolve());
+ const stopPromise = f.stop();
+ await vi.advanceTimersByTimeAsync(100);
+ await stopPromise;
+ });
+
+ test("factory rejects a cap below 1", () => {
+ expect(() =>
+ cronJobFactory({
+ redis: makeRedis() as never,
+ redlock: makeRedlock() as never,
+ maxConcurrentHandlers: 0
+ })
+ ).toThrow(/maxConcurrentHandlers/);
+ });
+});
diff --git a/backend/src/lib/cron/cron-job.ts b/backend/src/lib/cron/cron-job.ts
index 2f7096946d..84886e7483 100644
--- a/backend/src/lib/cron/cron-job.ts
+++ b/backend/src/lib/cron/cron-job.ts
@@ -1,4 +1,4 @@
-import { randomUUID } from "node:crypto";
+import { createHash, randomUUID } from "node:crypto";
import { CronExpressionParser } from "cron-parser";
import { Cluster, Redis } from "ioredis";
@@ -55,6 +55,16 @@ export const CronJobName = {
// ── tuning constants ──────────────────────────────────────────────────────────
+// Jitter is derived from the pattern, never configured per job: how long a job may be held
+// back should follow how often it runs. A job's window is this fraction of its own interval,
+// capped by `maxJitterMs`. A fraction below 1 is what makes the offset safe by construction,
+// so it is not a free parameter:
+// - a run can never reach its own next fire, which `fitsBeforeNextFire` relies on;
+// - the first retry always fits. Worst case is 0.25 x interval + 30s backoff + 1s, under
+// `interval` for any interval above ~41s, and cron cannot fire faster than every 60s.
+// Raising this past ~0.48 would start costing short-interval jobs their retry.
+const JITTER_INTERVAL_FRACTION = 0.25;
+
const PARTICIPANT_SLOTS = 5;
const PROCESS_BATCH_SIZE = 50;
// Safety buffer in `fitsBeforeNextFire`: if the retry's nextAttemptAt is within
@@ -73,7 +83,9 @@ const DEFAULTS = {
handlerTimeoutMs: 5 * 60_000,
retryBackoffBaseMs: 30_000,
retryBackoffMaxMs: 5 * 60_000,
- drainTimeoutMs: 25_000
+ drainTimeoutMs: 25_000,
+ maxJitterMs: 15 * 60_000,
+ maxConcurrentHandlers: 4
} as const;
// ── redis schema ──────────────────────────────────────────────────────────────
@@ -108,6 +120,9 @@ const F = {
// ── lua scripts ───────────────────────────────────────────────────────────────
// Atomic "enqueue this run hash + zset entry if no other pod beat us to it".
+// ARGV[2] is the unjittered scheduled fire, recorded on the hash as the run's
+// identity. ARGV[6] is the jittered eligibility time, and the zset is scored by
+// it so processTick's zrangebyscore holds the run back until its offset elapses.
const ENQUEUE_RUN_LUA = `
if redis.call('exists', KEYS[1]) == 0 then
redis.call('hset', KEYS[1],
@@ -117,7 +132,7 @@ const ENQUEUE_RUN_LUA = `
'attempts', 0,
'enqueued_at_ms', ARGV[5])
redis.call('expire', KEYS[1], ARGV[3])
- redis.call('zadd', KEYS[2], ARGV[2], ARGV[4])
+ redis.call('zadd', KEYS[2], ARGV[6], ARGV[4])
return 1
end
return 0
@@ -174,6 +189,8 @@ export const cronJobFactory = ({
retryBackoffBaseMs = DEFAULTS.retryBackoffBaseMs,
retryBackoffMaxMs = DEFAULTS.retryBackoffMaxMs,
drainTimeoutMs = DEFAULTS.drainTimeoutMs,
+ maxJitterMs = DEFAULTS.maxJitterMs,
+ maxConcurrentHandlers = DEFAULTS.maxConcurrentHandlers,
keyPrefix = KEY_HASH_TAG,
schedulingEnabled = true
}: {
@@ -189,6 +206,19 @@ export const cronJobFactory = ({
retryBackoffBaseMs?: number;
retryBackoffMaxMs?: number;
drainTimeoutMs?: number;
+ /**
+ * Ceiling on the derived jitter window. Without it a daily job would be spread over six
+ * hours; with it, every job past a ~1h interval shares the same widest window. Set it to 0
+ * to disable jitter, which is what development wiring does so a job runs when its pattern
+ * says it should.
+ */
+ maxJitterMs?: number;
+ /**
+ * Ceiling on how many handlers this pod runs at once. At capacity the pod stops claiming for
+ * the tick and leaves the runs in the pending zset, so another pod or a later tick picks them
+ * up; nothing is dropped.
+ */
+ maxConcurrentHandlers?: number;
/**
* Namespace for every Redis key this manager owns. Defaults to the production
* `{cron}` namespace. Tests override it so a test-owned manager and the
@@ -203,6 +233,12 @@ export const cronJobFactory = ({
schedulingEnabled?: boolean;
}) => {
assertHashTagged(keyPrefix);
+ if (!Number.isFinite(maxJitterMs) || maxJitterMs < 0) {
+ throw new Error(`cron: maxJitterMs (${maxJitterMs}) must be a non-negative number`);
+ }
+ if (!Number.isInteger(maxConcurrentHandlers) || maxConcurrentHandlers < 1) {
+ throw new Error(`cron: maxConcurrentHandlers (${maxConcurrentHandlers}) must be an integer >= 1`);
+ }
const SLOT_KEY = (i: number) => `${keyPrefix}:slot:${i}`;
const RUN_KEY = (id: string) => `${keyPrefix}:run:${id}`;
@@ -231,6 +267,49 @@ export const cronJobFactory = ({
const nextFireMs = (pattern: string) => CronExpressionParser.parse(pattern, { tz: "UTC" }).next().toDate().getTime();
+ // How far past its scheduled fire a job with this interval may be held back. Proportional
+ // to the interval so a frequent job is delayed proportionally less than a rare one, capped
+ // so a daily job is not spread across the whole day. NEXT_FIRE_BUFFER_MS is belt-and-braces:
+ // JITTER_INTERVAL_FRACTION already keeps the window well inside the interval, but the clamp
+ // means the next-fire property survives someone raising the fraction.
+ const jitterWindowMs = (intervalMs: number) =>
+ Math.min(intervalMs * JITTER_INTERVAL_FRACTION, maxJitterMs, Math.max(0, intervalMs - NEXT_FIRE_BUFFER_MS));
+
+ // The window for a pattern, sampled at registration. A pattern with uneven intervals (say
+ // quarterly) gets its window recomputed per fire at enqueue; this sample only has to be good
+ // enough for the run-hash TTL check.
+ const jitterWindowForPattern = (pattern: string) => {
+ const it = CronExpressionParser.parse(pattern, { tz: "UTC" });
+ const next = it.next().toDate().getTime();
+ return jitterWindowMs(it.next().toDate().getTime() - next);
+ };
+
+ // Per-job offset into the jitter window, derived from a stable hash of the job
+ // name. It MUST NOT be random: the run id is keyed on the scheduled fire time
+ // and every pod computes a run's eligibility time independently, so a per-pod offset
+ // would make pods disagree on run identity and break the enqueue dedup and the
+ // lease logic. Hashing also spreads the jobs permanently, where random offsets
+ // would re-collide on the next fire.
+ const jitterOffsetMs = (name: string, windowMs: number) => {
+ if (windowMs <= 0) return 0;
+ // Six bytes keeps the value inside Number.MAX_SAFE_INTEGER.
+ return createHash("sha256").update(name).digest().readUIntBE(0, 6) % windowMs;
+ };
+
+ // The earliest a fire may be picked up: its scheduled time plus the job's
+ // offset. Distinct from the run hash's `started_at`, which is when a handler
+ // actually began. The window is clamped so an offset can never push a run to
+ // or past its own next fire: the two would then become eligible in the wrong
+ // order, and `fitsBeforeNextFire`, which measures a retry against the raw next
+ // fire, would stop bounding the retry correctly. NEXT_FIRE_BUFFER_MS is the
+ // same margin that check uses.
+ const eligibleAtMs = (entry: CronEntry, scheduledAt: number, nextFire: number) => {
+ const windowMs = jitterWindowMs(nextFire - scheduledAt);
+ return scheduledAt + jitterOffsetMs(entry.name, windowMs);
+ };
+
+ const atHandlerCapacity = () => inFlight.size >= maxConcurrentHandlers;
+
const shuffleInPlace = <T>(arr: T[]): void => {
for (let i = arr.length - 1; i > 0; i -= 1) {
const j = Math.floor(Math.random() * (i + 1));
@@ -345,6 +424,18 @@ export const cronJobFactory = ({
if (entries.has(name)) throw new Error(`cron[${name}] already registered`);
CronExpressionParser.parse(pattern, { tz: "UTC" }); // validate at registration
+ // The run hash has to outlive the offset, or it expires before its own pickup and
+ // processCandidate reaps the orphaned zset entry with no log and no failed status: the
+ // job would silently never run. Measured against this pattern's own window.
+ const windowMs = jitterWindowForPattern(pattern);
+ if (runHashTtlS * 1000 <= windowMs) {
+ throw new Error(
+ `cron[${name}] runHashTtlS (${runHashTtlS}s) must exceed its jitter window (${Math.round(
+ windowMs / 1000
+ )}s), otherwise the run hash expires before the run is picked up`
+ );
+ }
+
const effectiveHandlerTimeoutMs = entryHandlerTimeoutMs ?? handlerTimeoutMs;
const effectiveLeaseDurationMs = entryLeaseDurationMs ?? leaseDurationMs;
if (effectiveHandlerTimeoutMs > effectiveLeaseDurationMs) {
@@ -362,7 +453,7 @@ export const cronJobFactory = ({
handlerTimeoutMs: entryHandlerTimeoutMs,
leaseDurationMs: entryLeaseDurationMs
});
- logger.info(`cron[${name}]: registered (pattern="${pattern}")`);
+ logger.info(`cron[${name}]: registered (pattern="${pattern}") [jitter_window_ms=${windowMs}]`);
};
// ── slot election ───────────────────────────────────────────────────────────
@@ -412,6 +503,10 @@ export const cronJobFactory = ({
// and atomically inserts a run hash + pending-zset entry if not already
// present. The Lua script makes the existence check + write a single op so
// multiple pods racing to enqueue the same fire produce exactly one run.
+ //
+ // The run id and the hash's `scheduled_at` stay keyed on the unjittered fire,
+ // so run identity is exactly what it was before jitter existed. Only the zset
+ // score carries the offset, which is what holds the run back from processTick.
const enqueueDueFires = async () => {
for (const entry of entries.values()) {
const scheduledAt = prevFireMs(entry.pattern);
@@ -419,6 +514,7 @@ export const cronJobFactory = ({
if (lastEnqueuedAt.get(entry.name) === scheduledAt) continue;
const id = `${entry.name}:${scheduledAt}`;
+ const eligibleAt = eligibleAtMs(entry, scheduledAt, nextFireMs(entry.pattern));
// eslint-disable-next-line no-await-in-loop
const initialized = await redis.eval(
ENQUEUE_RUN_LUA,
@@ -429,11 +525,14 @@ export const cronJobFactory = ({
String(scheduledAt),
String(entry.runHashTtlS),
id,
- String(Date.now())
+ String(Date.now()),
+ String(eligibleAt)
);
if (initialized)
logger.info(
- `cron[${entry.name}]: enqueued run [id=${id}] [scheduled_at=${new Date(scheduledAt).toISOString()}]`
+ `cron[${entry.name}]: enqueued run [id=${id}] [scheduled_at=${new Date(
+ scheduledAt
+ ).toISOString()}] [eligible_at=${new Date(eligibleAt).toISOString()}]`
);
lastEnqueuedAt.set(entry.name, scheduledAt);
}
@@ -546,6 +645,8 @@ export const cronJobFactory = ({
logger.info(`cron[${data.name}]: re-claiming stalled run [id=${id}] [previous_worker=${data.worker_id}]`);
}
+ if (atHandlerCapacity()) return;
+
// Track the in-flight run so stop() can wait for it to settle before
// tearing down. Lock-contention rejections settle within a tick, so they
// don't measurably delay shutdown drain.
@@ -576,12 +677,20 @@ export const cronJobFactory = ({
const ids = await redis.zrangebyscore(PENDING_ZSET, "-inf", Date.now(), "LIMIT", 0, PROCESS_BATCH_SIZE);
shuffleInPlace(ids);
- for (const id of ids) {
+ for (let i = 0; i < ids.length; i += 1) {
+ if (atHandlerCapacity()) {
+ logger.info(
+ `cron: at handler capacity (${maxConcurrentHandlers}), deferring ${
+ ids.length - i
+ } due run(s) [worker=${workerId}]`
+ );
+ break;
+ }
try {
// eslint-disable-next-line no-await-in-loop
- await processCandidate(id);
+ await processCandidate(ids[i]);
} catch (err) {
- logger.error({ err, id }, "cron: processCandidate failed");
+ logger.error({ err, id: ids[i] }, "cron: processCandidate failed");
}
}
};
diff --git a/backend/src/server/routes/index.ts b/backend/src/server/routes/index.ts
index 7898b66f38..32fcf279b7 100644
--- a/backend/src/server/routes/index.ts
+++ b/backend/src/server/routes/index.ts
@@ -606,7 +606,12 @@ export const registerRoutes = async (
const appCfg = getConfig();
const redlock = new Redlock([redis], { retryCount: 0 });
- const cronJob = cronJobFactory({ redis, redlock, schedulingEnabled: envConfig.isGeneralWorkerRunModeEnabled });
+ const cronJob = cronJobFactory({
+ redis,
+ redlock,
+ schedulingEnabled: envConfig.isGeneralWorkerRunModeEnabled,
+ maxJitterMs: envConfig.isDevelopmentMode ? 0 : undefined
+ });
if (envConfig.isGeneralWorkerRunModeEnabled) {
cronJob.start();
}