Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

25.2 KB
Raw Blame
/**
 * Turning "this routine is due" into exactly one firing, on a clock, across replicas.
 *
 * TWO HALVES, LIKE `server/src/work/culler.ts`, and separated for its reason: deciding what should
 * fire is not the same act as firing it. This half reads the ledger and puts an item on the shared
 * queue; the next one claims those items and dispatches them. So whichever replica noticed does not
 * have to be the one that carries it out, and a dispatch that dies halfway is picked up by whoever
 * claims it next rather than lost with the process that saw it was due.
 *
 * Two free functions over one options type rather than a factory, again like the culler, so the two
 * halves cannot drift about what a lease, an owner or a limit means: there is one description of the
 * things they share, and both read it.
 *
 * THE OFFER KEY IS THE WHOLE IDEMPOTENCE STORY. It carries the minute the firing was due, so three
 * replicas waking at 09:00 produce one work item and one run. That holds only while every replica
 * renders that minute identically, which is why the format is pinned in one function below and
 * asserted literally in `server/tests/routine-sweep.integration.test.ts`.
 *
 * NO CLAIM OR LEASE MACHINERY HERE, and none on the routines table. `server/src/work/queue.ts`
 * already owns `for update skip locked`, leases named on the database's clock and an attempt count.
 * A second half-right copy grown next to it is the duplicated firing mechanism #235 exists to
 * prevent.
 */
import { TURN_LIMITS } from "@openbot/toolkit/limits";
import { DEFAULT_MAX_ATTEMPTS, type WorkQueue } from "../work/queue";
import { RoutineRefusedError, type RoutineStore } from "./store";

export const ROUTINE_FIRE_KIND = "routine.fire";

/** How many due routines one pass will look at. Bounded, because a pass has to end. */
const DEFAULT_LIMIT = 50;

/**
 * How long a claimed firing is leased for, and renewed by, while it is being dispatched.
 *
 * A minute: a dispatch is one HTTP call to this deployment's own server, which either answers or
 * fails long before that, and a firing whose owner died is worth picking up again quickly. It is
 * renewed before every item, so the length bounds one dispatch rather than the whole batch.
 */
const DEFAULT_LEASE_MS = 60_000;

/**
 * How long a firing that could not be dispatched waits before anybody tries again.
 *
 * A minute, for the culler's reason: whatever refused this will probably refuse it again in the next
 * second, and the queue's attempt cap is what stops the waiting going on for ever.
 */
const DISPATCH_RETRY_DELAY_MS = 60_000;

/**
 * How late a firing may be and still be worth having.
 *
 * Ten minutes: several sweep intervals plus a slow pass, so a firing delayed by a deploy, a restart
 * or a busy queue is still delivered rather than silently dropped — and comfortably under the
 * fifteen-minute floor a routine's schedule may have (`MINIMUM_INTERVAL_MS` in `./schedule`), so the
 * window can never call two consecutive occurrences of one routine current at the same time.
 */
export const DEFAULT_GRACE_MS = 10 * 60_000;

/**
 * How long a run row may sit open with no outcome before a pass declares it abandoned.
 *
 * A server that dies mid-turn strands its run row for ever: the queue item was finished on the 202,
 * so no retry comes back for it, and nothing else writes that row — the routines page reads
 * "running now" for a run no process is running. Twice the turn timeout, so a slow-but-alive turn is
 * never closed out from under the server still running it. Derived rather than copied, so raising the
 * turn timeout cannot quietly make the sweep close turns that are still running. `limits` has no
 * imports, so the CronJob still does not drag the runtime's import graph into its process.
 */
const ABANDONED_RUN_MS = 2 * TURN_LIMITS.turnTimeoutMs;

export type RoutineSweepOptions = {
  routineStore: RoutineStore;
  /** The shared `work_items` queue. Not a second queue, and not a timer. */
  queue: WorkQueue;
  /** POST /internal/routines/run. Throws on anything that is not a 202. */
  dispatch: (routineRunId: string) => Promise<void>;
  /** Who this process is, for the lease. A name, so a stuck claim traces back to a pod. */
  owner: string;
  /** Lease for a claimed firing; phase two is what applies it. Default 60_000. */
  leaseMs?: number;
  /** How many goes one firing gets before it stops being offered. Default the queue's default attempt cap. */
  maxAttempts?: number;
  /** How many due routines one pass considers. Default 50. */
  limit?: number;
  /** How late a firing may be and still be offered. Default ten minutes; see the policy below. */
  graceMs?: number;
  now?: () => Date;
};

/**
 * What one pass of phase two did.
 *
 * `fired` is "a run was opened and the dispatch was accepted", not "the routine succeeded": the run
 * row in `routine_runs` owns the outcome from the moment the dispatch resolves, and this report is
 * the sweep's own account of its pass rather than a summary of anybody's turn.
 */
export type RoutineSweepReport = {
  considered: number;
  fired: string[];
  skipped: { routineId: string; reason: string }[];
};

/**
 * The minute a firing was due, rendered the same way by every replica.
 *
 * `2026-08-26T09:30Z`: ISO, truncated to the minute, always UTC, no seconds and no fractional part.
 * This string is half of the offer key, so THE FORMAT IS THE IDEMPOTENCE — two sweeps that render one
 * due moment differently offer two items and a person gets two runs of the same routine. Changing it
 * also orphans every key already in `work_items`, which is why the tests assert it literally rather
 * than recomputing it.
 *
 * Truncated rather than rounded, so a stamp never renders as a minute it is not in, and UTC by
 * construction: `toISOString` has no local component, so a replica in another zone cannot name the
 * same moment differently.
 */
function minuteKey(due: Date): string {
  // "2026-08-26T09:30:00.000Z" -> "2026-08-26T09:30" -> back with the zone it never left.
  return `${due.toISOString().slice(0, 16)}Z`;
}

/**
 * Phase one: due routines become idempotent work items, and `next_run_at` moves exactly once.
 *
 * WHICH CLOCK, given that everything around this names its moments in SQL. Both moments that decide
 * anything come from the database: `dueRoutines` compares `next_run_at <= now()` inside Postgres, so
 * what is due is Postgres's judgement, and the stamp this keys the offer on and hands the
 * compare-and-set is the value Postgres gave back. The one process-clock reading is `now` here, used
 * only to measure the width of the grace window below — a window minutes wide, against a stamp the
 * database chose, so sub-second skew cannot change an answer. A badly skewed node could admit or
 * suppress a firing near the boundary; it cannot double-fire one, because the key and the CAS are
 * both the database's. That is the difference between this and a lease, which is why a lease is
 * never computed here. The option exists so tests can put a stamp anywhere they like.
 */
export async function offerDueRoutines(
  options: RoutineSweepOptions,
): Promise<{ offered: string[] }> {
  const now = options.now?.() ?? new Date();
  const graceMs = options.graceMs ?? DEFAULT_GRACE_MS;

  try {
    await options.routineStore.recordSweep(options.owner);
  } catch (error) {
    console.warn(
      JSON.stringify({
        type: "routine-sweep-heartbeat-failed",
        reason: error instanceof Error ? error.message : String(error),
      }),
    );
  }

  const due = await options.routineStore.dueRoutines(
    options.limit ?? DEFAULT_LIMIT,
  );

  const offered: string[] = [];
  for (const routine of due) {
    /*
     * ONE ROUTINE'S FAILURE IS ONE ROUTINE'S FAILURE. A cron the parser cannot read — a row written
     * before a validation existed, a hand-edited value — makes `advanceNextRun` throw, and an
     * unguarded loop would take the whole pass down with it: everybody else's routines, on every
     * pass, for as long as the bad row exists. So each routine is its own attempt, and the pass
     * carries on.
     */
    try {
      /*
       * A STALE STAMP IS NOT A BACKLOG TO REPLAY. A routine whose stamp is a month behind — a
       * deployment that ran with no worker, a worker that was down — must not fire once per missed
       * occurrence when the worker comes back: turn it on after a quiet month and a person would get
       * thirty summaries of thirty days ago.
       *
       * So this loop offers only firings that are still worth having: a stamp within GRACE of now.
       * For anything later than that, advance WITHOUT offering and compute the next occurrence from
       * NOW rather than from the stale stamp, so one pass makes the clock current. Stepping one
       * occurrence per pass instead — the earlier shape — kept a fifteen-minute routine silent a
       * further fortnight after a month of downtime, because draining ~2,900 missed occurrences at
       * one per five-minute sweep is itself two weeks. The store deliberately does not decide this:
       * which stamps are worth firing, and where a stale clock should land, are this file's policy.
       */
      const lateBy = now.getTime() - routine.nextRunAt.getTime();
      if (lateBy <= graceMs) {
        /*
         * OFFERED BEFORE THE CLOCK MOVES. A crash between the two leaves the stamp where it was, so
         * the next pass reads the same stamp, renders the same key and collides: the firing happens
         * once and nothing is lost. Advancing first and offering second loses that firing outright —
         * the stamp is gone and nothing remembers what it was for.
         */
        await options.queue.offer({
          kind: ROUTINE_FIRE_KIND,
          key: `${routine.id}:${minuteKey(routine.nextRunAt)}`,
          payload: {
            routineId: routine.id,
            scheduledFor: routine.nextRunAt.toISOString(),
          },
        });
        offered.push(routine.id);
        // False means another sweep advanced it first, which is fine either way: the firing was
        // offered under the same key by both, so it still happens once.
        await options.routineStore.advanceNextRun(
          routine.id,
          routine.nextRunAt,
        );
      } else {
        // The CAS still compares against the stale stamp it read — only the landing point moves.
        await options.routineStore.advanceNextRun(
          routine.id,
          routine.nextRunAt,
          now,
        );
      }
    } catch (error) {
      /*
       * A refusal from the store here is the schedule's own: `advanceNextRun` recomputes the next
       * occurrence, and a cron the schedule module refuses — a row written before a validation
       * existed, a hand-edited value — throws on every pass for ever. Left alone, that routine's
       * clock never moves, it burns one of this pass's `limit` slots each time, and its owner sees a
       * routine that silently stopped. Deterministic refusals do not heal, so the routine is
       * switched off and the reason written where the routines page reads it — the sweep has no
       * channel to announce itself in, the way the runner's fatigue switch-off does. Queue and
       * database errors are NOT this: they heal, so those routines are left enabled for the next
       * pass to try again.
       */
      if (error instanceof RoutineRefusedError) {
        try {
          await options.routineStore.markUnschedulable(
            routine.id,
            error.message,
          );
        } catch (markError) {
          // Best-effort: a switch-off that failed leaves the loud warning below, and the next pass
          // will be back here to try the switch-off again.
          console.warn(
            JSON.stringify({
              type: "routine-sweep-disable-failed",
              routineId: routine.id,
              reason: String(markError),
            }),
          );
        }
      }
      // Said out loud, with the routine in it: a pass that swallowed this would look clean while
      // one routine was switched off, or failed to be.
      console.warn(
        JSON.stringify({
          type: "routine-sweep-offer-failed",
          routineId: routine.id,
          scheduledFor: routine.nextRunAt.toISOString(),
          reason:
            error instanceof Error ? error.message : "could not be offered",
        }),
      );
    }
  }

  return { offered };
}

/**
 * Phase two: claimed items become dispatched firings, with the queue's booleans honoured.
 *
 * EVERY BRANCH HERE IS ABOUT THE GAP BETWEEN THE OFFER AND THE FIRING. Another replica decided this
 * should fire, at another time, and by now the routine may be switched off, deleted, or the
 * occurrence may have gone stale while the item waited behind a backlog. So the world is re-read at
 * the moment of acting — the culler's discipline, for the culler's reason — and the queue's own
 * answers are believed: a `renew` that says no means the item is somebody else's, and a `finish` that
 * says no means it stopped being ours while we were working.
 *
 * `finish` and `release` mean different things and are not interchangeable. `release` is for a
 * dispatch that could have worked and might work next time; `finish` is for a firing that will never
 * be worth having, however many times it comes back.
 */
export async function dispatchClaimedRoutines(
  options: RoutineSweepOptions,
): Promise<RoutineSweepReport> {
  const leaseMs = options.leaseMs ?? DEFAULT_LEASE_MS;
  const graceMs = options.graceMs ?? DEFAULT_GRACE_MS;
  const maxAttempts = options.maxAttempts ?? DEFAULT_MAX_ATTEMPTS;
  const claimed = await options.queue.claim({
    kind: ROUTINE_FIRE_KIND,
    owner: options.owner,
    leaseMs,
    limit: options.limit ?? DEFAULT_LIMIT,
    // Passed through only when asked for, so the queue's own default stays the one default.
    ...(options.maxAttempts === undefined
      ? {}
      : { maxAttempts: options.maxAttempts }),
  });

  const report: RoutineSweepReport = {
    considered: claimed.length,
    fired: [],
    skipped: [],
  };

  /*
   * THE REAPER, before any firing is considered. It closes the run rows nothing in the system will
   * ever come back for: a server that died mid-turn (the queue item was finished on the 202, so no
   * retry returns for that row), and the rows opened by dispatch attempts that threw (the loop below
   * deliberately does not close those itself — see the comment at the dispatch). Without it those
   * rows read "running now" on the routines page for ever. Age-scoped rather than identity-scoped,
   * because age is the one signal that distinguishes an abandoned row from a turn some server is
   * still running; the cutoff sits above the turn timeout so a live turn always finishes its own row
   * first. Best-effort with its own catch, because a reaper that cannot run must not stop this pass
   * from firing what is due.
   */
  try {
    const reaped = await options.routineStore.reapAbandonedRuns(
      ABANDONED_RUN_MS,
      "the server never finished this run; it may have restarted mid-turn, or the run may never have been dispatched",
    );
    if (reaped > 0) {
      console.warn(JSON.stringify({ type: "routine-runs-reaped", reaped }));
    }
  } catch (error) {
    console.warn(
      JSON.stringify({
        type: "routine-run-reap-failed",
        reason: String(error),
      }),
    );
  }

  for (const item of claimed) {
    // The offer above is the only writer of these items and always names the routine in the
    // payload; the key is the last-resort stand-in for a row written by hand, the culler's idiom.
    const routineId = String(item.payload.routineId ?? item.key);

    /*
     * Renewed before acting, because the batch is many and the lease is one.
     *
     * This is the lesson at `server/src/work/culler.ts:141-151` verbatim: twenty API calls with
     * nothing renewed while they ran, the lease expiring part-way down the list, and another replica
     * claiming the tail this one was still working through. A lease nobody renews is a timer, and a
     * timer is what this queue exists not to be.
     *
     * False means it is already somebody else's, and then the only correct answer is to leave it
     * entirely alone: no dispatch, because that replica is already running this firing and a second
     * dispatch is a second message to a person; and no `finish`, because finishing somebody else's
     * item takes the lease away from a run that is still going.
     */
    if (
      !(await options.queue.renew({
        kind: ROUTINE_FIRE_KIND,
        key: item.key,
        owner: options.owner,
        leaseMs,
      }))
    ) {
      report.skipped.push({
        routineId,
        reason: "the lease went to another replica",
      });
      continue;
    }

    try {
      /*
       * RE-READ, because the offer was another replica's judgement at another time. A person who
       * switched a routine off a minute after it was offered, or deleted it, has said what they want;
       * firing it anyway posts a message they have asked not to receive.
       *
       * Finished rather than released in both cases: no number of retries will make a deleted routine
       * exist, and a switched-off one does not want its queued firing carried out later either. The
       * next occurrence is offered afresh if it is switched back on.
       */
      const routine = await options.routineStore.routineForFiring(routineId);
      if (!routine?.enabled) {
        const reason = routine
          ? "switched off between the offer and the firing"
          : "deleted between the offer and the firing";
        await finishOrSay(options, item.key, routineId, reason);
        report.skipped.push({ routineId, reason });
        continue;
      }

      /*
       * AND THE WINDOW AGAIN, HERE, before any run row exists.
       *
       * The offer already enforced this window at offer time, and that is not enough: the queue's
       * redelivery machinery can outlive it. A backlogged queue, or five releases at a minute each,
       * and the item is claimed well after the occurrence it names — "here is your morning summary",
       * in the afternoon, which is exactly what the stale-stamp policy above exists to prevent. So it
       * is re-checked at the moment of acting, which is the culler's precedent ("Somebody came back",
       * `culler.ts:~170`): the decision was made elsewhere and the world has moved since.
       *
       * BESIDE the deleted/disabled branch and before `insertRun`, so a skipped firing leaves no
       * `routine_runs` row: a run opened with no outcome and nothing coming to give it one shows on
       * the routines page as a firing that started and never ended.
       *
       * Finished, not released, for the same reason as above: re-delivery cannot make a past
       * occurrence current. A missing or unreadable stamp is not treated as stale — the offer is the
       * only writer of this payload and always writes one, so there is no window to enforce rather
       * than a window that has passed, and dropping the firing on a payload this file wrote would be
       * inventing a reason to lose it.
       */
      const now = options.now?.() ?? new Date();
      const stamp = item.payload.scheduledFor;
      const scheduledFor =
        typeof stamp === "string" ? new Date(stamp) : undefined;
      if (
        scheduledFor &&
        !Number.isNaN(scheduledFor.getTime()) &&
        now.getTime() - scheduledFor.getTime() > graceMs
      ) {
        const reason = "claimed too long after the occurrence it was due for";
        await finishOrSay(options, item.key, routineId, reason);
        report.skipped.push({ routineId, reason });
        continue;
      }

      /*
       * The run row first, then the dispatch, because the dispatch is told a run id and nothing else.
       * From the moment it resolves the run row owns the outcome: the queue's retries are for
       * DISPATCH failures only, and a turn that failed is final for this firing — the fatigue rule
       * owns that, not this loop.
       */
      const { runId } = await options.routineStore.insertRun(routineId);
      /*
       * A dispatch that throws leaves the row this attempt opened with no status, AND NOTHING HERE
       * CLOSES IT — the reaper above does, once the row is older than any turn could still be
       * running. That restraint is deliberate: a dispatch that timed out is not a dispatch that
       * failed, because the abort tears down the sweep's side of the call while the server may
       * already have accepted it and detached the turn — a turn that will come back minutes later
       * and finish this very row. `finishRun` finishes once, so closing the row now would turn that
       * turn's real outcome into a silent no-op; the earlier shape of this cleanup ("close every
       * open run of the routine at the give-up") mislabelled exactly such in-flight runs as failed.
       * Age is the only signal the sweep has that no server is coming back for a row, so age is the
       * scope the closing uses.
       */
      await options.dispatch(runId);
      if (
        !(await options.queue.finish({
          kind: ROUTINE_FIRE_KIND,
          key: item.key,
          owner: options.owner,
        }))
      ) {
        /*
         * The dispatch happened, so this is `fired` either way; but a `finish` that says no says the
         * lease lapsed while the call was in flight, which means another replica may claim this same
         * minute and dispatch it again. Said out loud, because it is the shape of a duplicate run and
         * nothing else in the system will mention it.
         */
        console.warn(
          JSON.stringify({
            type: "routine-fire-redelivery-possible",
            routineId,
            runId,
            reason:
              "the lease had gone by the time the dispatch came back, so the firing may be redelivered",
          }),
        );
      }
      report.fired.push(routineId);
    } catch (error) {
      /*
       * ONE FIRING'S FAILURE IS ONE FIRING'S FAILURE, exactly as in the offering half above: an
       * unguarded throw here would take everybody else's claimed firings down with it, on every
       * pass, for as long as the one bad item exists.
       *
       * Released rather than finished, and pushed out rather than retried in this pass: a server
       * that refused this dispatch will probably refuse it again in the next second, and the queue's
       * attempt cap is what bounds the retrying.
       */
      const reason =
        error instanceof Error ? error.message : "could not be dispatched";
      let released = false;
      try {
        released = await options.queue.release({
          kind: ROUTINE_FIRE_KIND,
          key: item.key,
          owner: options.owner,
          delayMs: DISPATCH_RETRY_DELAY_MS,
          reason,
        });
      } catch (releaseError) {
        // Best-effort: an item that could not even be released has a lease that will lapse on its
        // own, and the pass still has other firings to get through.
        console.warn(
          JSON.stringify({
            type: "routine-fire-release-failed",
            routineId,
            reason: String(releaseError),
          }),
        );
      }
      /*
       * Said out loud when it gives up, because otherwise it stops silently.
       *
       * At the cap the item is no longer claimed, so this loop simply never sees that routine again
       * and every sweep looks clean while one person's routine never fires. The row carries the count
       * and the reason for anybody who queries the table; this is for whoever reads the logs.
       */
      if (item.attempts >= maxAttempts) {
        // The run rows the attempts opened are NOT closed here: one of them may be a turn a wedged
        // server accepted after the dispatch timed out, and only age can tell (see the comment at
        // the dispatch). The reaper closes them on a later pass; this branch only has to say so.
        console.warn(
          JSON.stringify({
            type: "routine-fire-gave-up",
            routineId,
            key: item.key,
            attempts: item.attempts,
            reason,
          }),
        );
      } else if (!released) {
        // Not ours any more, which means somebody else holds it: worth a line, because a release that
        // did nothing leaves this pass's failure recorded nowhere on the row.
        console.warn(
          JSON.stringify({
            type: "routine-fire-release-lost",
            routineId,
            key: item.key,
            reason,
          }),
        );
      }
      report.skipped.push({ routineId, reason });
    }
  }

  return report;
}

/**
 * Finish a firing nobody wants, and say so if it was not ours to finish.
 *
 * The boolean is the truth about ownership rather than a formality: false means the lease went while
 * this pass was deciding, so the routine was NOT stopped from firing here — whoever holds it now will
 * make its own decision, and this one should say what it saw rather than retry into a race.
 */
async function finishOrSay(
  options: RoutineSweepOptions,
  key: string,
  routineId: string,
  reason: string,
): Promise<void> {
  const finished = await options.queue.finish({
    kind: ROUTINE_FIRE_KIND,
    key,
    owner: options.owner,
  });
  if (!finished) {
    console.warn(
      JSON.stringify({
        type: "routine-fire-finish-lost",
        routineId,
        key,
        reason,
      }),
    );
  }
}