Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Quota-burn runner — the single install-level loop.
*
* ONE loop lives in PortOS (not one scheduled task per managed app, which is
* what this replaced): every `checkIntervalMinutes` it reads the burn plan,
* takes a zero-token quota reading, and — only when a family's window is inside
* its reset horizon with headroom above its reserve — runs the next job in that
* family's ordered plan that reports pending work.
*
* The interval only STARTS a family's burn. Once a job is out, each finished burn
* agent triggers a fresh evaluation of that family (`onBurnAgentCompleted`), so
* the plan is walked one agent at a time until a gate closes — the window's
* dispatch cap, the reserve, or the reset horizon.
*
* "Next job" is literal — the walk resumes after the family's last dispatch
* rather than restarting at the top (`rotatePlanAfter`), so an N-job plan cycles
* through all N instead of re-running job #1 every time.
*
* At most one dispatch per FAMILY per cycle. Families do not share a budget:
* each one draws down its own subscription window, against its own reserve and
* its own `maxDispatchesPerWindow`. Stopping the whole cycle after the first
* dispatch (which is what this used to do) therefore didn't protect anything —
* it just meant the soonest-resetting family took every cycle, and a family with
* a longer window never burned at all while it was enabled. A `claude` window
* resetting in 2h beat an `agy` window resetting in 21h on every single tick, so
* the agy plan sat at "93% left · 0/5 used" indefinitely.
*
* AI-policy posture (AGENTS.md): this is a sanctioned scheduled automation —
* disabled by default, armed only by an explicit opt-in on the Quota Burn page,
* where the family, provider, model, and work are all named before anything
* runs. Boot ARMS the interval; it never dispatches. With `enabled: false` the
* tick returns before reading quota, so a fresh install is silent.
*
* Everything fails CLOSED and nothing here throws into the timer: a probe that
* errors, a job that declines, or an unreadable config all end the cycle with a
* logged reason instead of charging a dispatch or crashing the process.
*
* ACCOUNTING happens on acceptance, not on dispatch. The two synchronous lanes
* (a programmatic handler, a custom job) are accepted the moment they return, so
* this file charges them itself. A reference step goes out as an on-demand
* REQUEST an engine may still refuse, so it takes a reservation instead — which
* counts against the window cap and blocks a second dispatch of the same step
* until `quotaBurnAcceptance.js` settles it into exactly one charge, or releases
* it uncharged. Each cycle settles the previous one's requests before deciding
* what to ask for next.
*/
import { getProviderQuotas } from './providerUsage.js';
import { getQuotaBurnDispatches, evaluateFamilies, PLAN_COMPLETE_SKIP_REASON, recordQuotaBurnDispatch, selectBurnCandidates, withPendingDispatches } from './quotaBurn.js';
import { reconcileQuotaBurnReservations, reserveQuotaBurnDispatch } from './quotaBurnAcceptance.js';
import { getQuotaBurnCompletions, recordQuotaBurnJobCompletion } from './quotaBurnCompletions.js';
import { getActiveQuotaBurnBlocks, recordBurnAgentCompletion } from './quotaBurnDenials.js';
import { isQueuedContinuation } from '../lib/agentOutcome.js';
import { getQuotaBurnConfig, getQuotaBurnReservations, getQuotaBurnRuns, quotaBurnReservationKey, recordQuotaBurnRun } from './quotaBurnStore.js';
import { countQuotaBurnStepPending, getQuotaBurnTaskCatalog, invokeQuotaBurnStep } from './quotaBurnInvoke.js';
import { familyHasRunnableJobs, familyIsConfigured, jobIsSpent, quotaBurnJobKey } from '../lib/quotaBurnConfig.js';
import { applyQuotaBurnAvailability } from '../lib/quotaBurnTaskRef.js';
import { windowLabelOf } from '../lib/quotaWindows.js';
import { WAIT } from '../lib/staleWhileRevalidate.js';
import { cosEvents } from './cosEvents.js';
import { nextQuotaBurnSequenceJob, completeQuotaBurnSequenceStep } from './quotaBurnSequence.js';
import { burnPlanOwnsAgent } from '../lib/quotaBurnOrigin.js';
const TICK_MS = 60_000;
let tickTimer = null;
let running = false;
let lastRunAt = null;
// Completion continuations deferred while a cycle runs. Retain the completed
// task id: its task record can still say in_progress when the event arrives.
const deferredContinuations = new Map();
/**
* The jobs a cycle will consider, in plan order.
*
* `jobId` scopes to one named job. When it is named AND the run was forced, the
* job's own `enabled` checkbox — and a spent `run once` marker — are ignored:
* the user just clicked ▶ on that exact row, which is a more specific
* instruction than the checkbox they set earlier, and re-running a finished
* one-shot job on demand is exactly what that button is for. Without this,
* clicking ▶ on a paused (or already-run) job is a silent no-op reported as "no
* pending work".
*/
const selectJobs = (family, { jobId = null, force = false, completions = {} } = {}) => {
const targeted = Boolean(jobId) && force;
return (family.jobs || []).filter((job) => (!jobId || job.id === jobId)
&& (targeted || (job?.enabled !== false && !jobIsSpent(job, family.id, completions))));
};
/**
* The two fields of a pending-probe the page actually renders. An allowlist
* rather than an omit so a job type that starts returning something new has to
* opt into the wire deliberately.
*/
const wireShape = (pending) => ({ count: pending?.count ?? 0, detail: pending?.detail ?? '' });
/**
* What a step names, for the run log and the skip report: the scheduled task it
* references, or the legacy job type a plan that has not been converted yet is
* still carrying. `jobType` is null on a reference step, so recording it
* unconditionally wrote `jobType: undefined` on every one of them.
*/
const stepIdentity = (job) => (job?.taskRef
? { taskRef: job.taskRef }
: { jobType: job?.jobType ?? null });
/**
* ONE invocation path, whatever the step names.
*
* There is no second lane any more: the quota-only `JOB_MODULES` registry and
* its direct `agentPrompt` executor are retired (#6381), so a step still
* carrying a legacy `jobType` has no reference to run and resolves to
* `LEGACY_UNMIGRATED` inside the shared path — reported as "nothing pending"
* with its migration reason rather than dispatched down an executor that no
* longer exists. Keeping the refusal INSIDE the shared path is deliberate: the
* page and the runner then say the same thing about the same step.
*/
const probeStep = ({ job, family, catalog }) => countQuotaBurnStepPending({ step: job, family, catalog });
const runStep = ({ job, family, candidate, context, force, catalog }) =>
invokeQuotaBurnStep({ step: job, family, candidate, context, force, catalog });
/**
* The live task catalog, read at most once per cycle/status pass and only when a
* reference step is actually present — it reads the schedule, the app registry
* and the custom-job store, and a plan made entirely of legacy steps needs none
* of them, and neither does an unreadable one. Both come back EMPTY rather than
* as a third state: `resolveQuotaBurnStepAvailability` reads `{}` as "ask no
* catalog questions", so target scope is still judged (it is decidable from the
* reference alone), no step is mass-orphaned by a transient read failure, and a
* legacy-only plan resolves to exactly the verdict it already had.
*/
const catalogFor = async (jobs) => ((jobs || []).some((job) => job?.taskRef)
? await getQuotaBurnTaskCatalog().catch((err) => {
console.error(`❌ Quota-burn could not read the scheduled-task catalog: ${err.message}`);
return {};
})
: {});
/**
* The plan, re-anchored to start just AFTER the job this family last dispatched.
*
* A burn plan is an ordered ROTATION, not a priority list. Without this the walk
* restarts at index 0 every cycle, and since `agent-prompt` always reports work
* pending (its probe only checks that an app and a provider resolve), the first
* job in the plan won every dispatch forever — an eight-job agy plan spent its
* entire window re-running "Performance issues" and the other seven never ran
* once. Rotating preserves the "first job with pending work wins" semantics that
* matter for probing jobs like `universe-bible-images`; it only moves where the
* walk begins.
*
* An unknown cursor (first ever dispatch, a job since deleted from the plan, or
* one aged out of the capped run log) falls back to plan order.
*/
export function rotatePlanAfter(jobs, afterJobId) {
const at = afterJobId ? jobs.findIndex((job) => job.id === afterJobId) : -1;
return at < 0 ? jobs : [...jobs.slice(at + 1), ...jobs.slice(0, at + 1)];
}
/**
* Each family's most recently dispatched job id, read from the run log rather
* than a second persisted cursor file — `recordQuotaBurnRun` already stamps
* `familyId` + `jobId` on every dispatch, newest first, so the log IS the cursor.
*/
async function lastDispatchedJobByFamily() {
const runs = await getQuotaBurnRuns().catch(() => []);
const cursors = new Map();
for (const entry of runs) {
if (!entry?.dispatched || !entry.familyId || !entry.jobId) continue;
if (!cursors.has(entry.familyId)) cursors.set(entry.familyId, entry.jobId);
}
return cursors;
}
/**
* Walk a candidate's plan — starting after the family's last dispatch — and run
* the first job with pending work. Returns the dispatch outcome plus the per-job
* reasons, so a cycle that dispatched nothing still explains itself in the run
* log.
*
* The probe's `context` is handed to `run` rather than thrown away: a
* `universe-bible-images` probe reads every universe bible to count the backlog,
* and `run` needs the very same scan to know what to render. Without the
* passthrough that multi-megabyte read happened twice per dispatch.
*/
async function dispatchFromCandidate(candidate, { jobId = null, force = false, afterJobId = null, completions = {}, catalog = {}, reserved = new Set() } = {}) {
const attempts = [];
// A forced run of a NAMED job skips the pending probe entirely and calls the
// job directly. The probe exists to pick which job in the plan to run; when
// the user has already picked one, letting it veto the click reproduces the
// silent no-op the force path exists to fix — one gate later. It bites hardest
// on `universe-bible-images`, whose 6-hour in-flight cooldown makes the probe
// report zero for entries that are merely already queued, with no way to
// override it from the page. `force` is threaded into the job so it can relax
// its own cooldown too.
const targeted = force && jobId;
for (const job of rotatePlanAfter(selectJobs(candidate.family, { jobId, force, completions }), afterJobId)) {
// A burn of this step is already out and not yet accepted. Re-dispatching it
// would queue the same work twice and charge the window twice for one unit —
// the case `quotaBurnAcceptance.js` exists to close — so the walk moves on to
// the next step instead. Not bypassable by `force`: a duplicate is not a
// quota gate, it is the same work twice (see quotaBurnInvoke.js's header).
if (reserved.has(quotaBurnReservationKey(candidate.family.id, job.id))) {
attempts.push({ jobId: job.id, ...stepIdentity(job), skipped: 'a burn of this step is already awaiting acceptance' });
continue;
}
const pending = targeted ? null : await probeStep({ job, family: candidate.family, catalog });
if (pending && !(pending.count > 0)) {
attempts.push({ jobId: job.id, ...stepIdentity(job), skipped: pending.detail || 'no pending work' });
continue;
}
const result = await runStep({ job, family: candidate.family, candidate, context: pending?.context, force, catalog });
if (!result.dispatched) {
attempts.push({ jobId: job.id, ...stepIdentity(job), skipped: result.reason || 'declined' });
continue;
}
return { dispatched: true, job, result, attempts };
}
return { dispatched: false, attempts };
}
/**
* One evaluation. `trigger` is recorded in the run log so the page can tell a
* scheduled tick from a "Run now". A manual trigger ignores the master
* `enabled` switch (the user just clicked the button); an explicit `force`
* request also bypasses the reset window, reserve, dispatch cap, and denial
* gates.
*/
export async function runQuotaBurnCycle(options = {}) {
if (running) {
// A continuation that lands mid-cycle is REMEMBERED, not dropped. Two burn
// agents from different families finishing within the same cycle is ordinary
// — and a dropped continuation stalls that family's plan until the next
// interval tick, which at the 12-hour default is most of a day of the window
// this feature exists to spend. Keyed by family, so several completions from
// one family while it is mid-cycle collapse to the single re-evaluation they
// amount to.
if (options.trigger === 'continuation' && options.familyId) deferredContinuations.set(options.familyId, options.ignoreTaskId || null);
return { skipped: 'already-running' };
}
running = true;
// try/finally, not try/catch: this runs from a timer and from a route, and a
// throw anywhere below (an ENOSPC on the ledger write, say) would otherwise
// leave `running` stuck true and wedge the loop for the life of the process.
// The error itself is deliberately NOT swallowed — the timer logs it and the
// route surfaces it.
try {
return await evaluate(options);
} finally {
running = false;
await drainDeferredContinuations();
}
}
/**
* Run the continuations that arrived while a cycle was in flight, one at a time.
*
* Bounded by the same gate ladder every other dispatch faces: a drained family
* gets one cycle, which the reserve or the reset horizon can decline. A
* completion that lands DURING the drain re-enters the set legitimately — it is
* a different agent finishing — and faces that ladder too. Errors are swallowed here
* rather than allowed to replace the outer cycle's result: this runs from the
* caller's `finally`, so a throw would mask what that cycle actually decided.
*/
async function drainDeferredContinuations() {
while (deferredContinuations.size) {
const [[familyId, ignoreTaskId]] = deferredContinuations;
deferredContinuations.delete(familyId);
await runQuotaBurnCycle({ trigger: 'continuation', familyId, ignoreTaskId })
.catch((err) => console.error(`❌ Quota-burn deferred continuation for ${familyId} failed: ${err.message}`));
}
}
async function evaluate({ trigger = 'scheduled', familyId = null, jobId = null, force = false, ignoreTaskId = null }) {
const finish = async (entry) => {
// Only a SCHEDULED cycle advances the interval clock. A manual "Evaluate
// now" that reports "no burnable window" would otherwise push the next
// automatic cycle a full interval out — on a 12-hour interval, one curiosity
// click can defer the tick past the reset the feature exists to spend.
if (trigger === 'scheduled') lastRunAt = Date.now();
// Every cycle says what it decided. Previously ONLY the error paths logged,
// so clicking "Evaluate now" — which can spend a provider's quota — left no
// trace in the server log at all, and a cycle that correctly declined was
// indistinguishable from one that never ran.
console.log(entry.dispatched
? `🔥 Quota-burn ${trigger}: dispatched ${entry.familyId}/${entry.jobId} — ${entry.summary}`
: `💤 Quota-burn ${trigger}: nothing dispatched — ${entry.reason}`);
await recordQuotaBurnRun({ trigger, ...entry });
return entry;
};
const config = await getQuotaBurnConfig().catch((err) => {
console.error(`❌ Quota-burn could not read its config: ${err.message}`);
return null;
});
if (!config) return finish({ dispatched: false, reason: 'config unreadable' });
// A disabled tick writes NO run-log entry: a silent loop would otherwise fill
// the log with "disabled" rows and bury the last real run. Only a `manual`
// trigger — the user asking for this cycle by name — runs while the master
// switch is off; everything unattended (the interval tick AND the completion
// continuation it hands off to) stops. Phrased as "not manual" rather than an
// allowlist of automatic triggers so a trigger added later fails CLOSED.
if (!config.enabled && trigger !== 'manual') return { skipped: 'disabled' };
// Settle what the LAST dispatch asked for before deciding what this cycle may
// ask for. A reference burn's acceptance is asynchronous, so this is where a
// request that produced a task is charged and one that was refused releases
// its reservation — both of which change the ladder's answers below. Ahead of
// the run-once read for the same reason: a step accepted here is spent now.
const settlement = await reconcileQuotaBurnReservations();
if (settlement.accepted || settlement.refused) {
console.log(`🧾 Quota-burn settled ${settlement.accepted} accepted and ${settlement.refused} refused burn${settlement.accepted + settlement.refused === 1 ? '' : 's'}`);
}
// Read once per cycle and threaded through selection, the gate ladder, and the
// dispatch record, so a job spent mid-cycle can't be re-picked by a later
// family's walk.
//
// `null` means the ledger could not be read, which is NOT "nothing has run" —
// treating it as such would re-dispatch every `run once` job on the plan. Fails
// CLOSED: skip the cycle and say so rather than redoing work the user asked
// for exactly once.
const completions = await getQuotaBurnCompletions();
if (!completions) {
console.error('❌ Quota-burn could not read its run-once ledger — skipping this cycle');
return finish({ dispatched: false, reason: 'run-once ledger unreadable' });
}
// Check the plan BEFORE the quota read. `getProviderQuotas({ wait: WAIT.FRESH })`
// spawns a multi-second TUI scrape per enabled family; with no actionable
// family configured that scrape could never produce a dispatch, and it would
// otherwise run every `checkIntervalMinutes` forever. A plan made entirely of
// spent `run once` jobs is the same case, which is why `completions` is passed
// — without it a finished one-shot plan keeps paying for the scrape forever.
//
// A forced run of a NAMED job is exempt: this requires an enabled family with
// an enabled, unspent job, but ▶ on a paused job (or a paused family, or one
// that already ran) is exactly the case the force path exists to serve —
// gating it here would make the click a silent no-op before selection ever ran.
//
// One short-circuiting pass on the healthy path; the second only runs to pick
// the wording, and only when nothing is runnable. The two are kept distinct
// because a finished one-shot plan did what it was asked and wants Re-arm,
// which is not what an unset plan wants (add a job).
const targeted = force && familyId;
const families = Object.values(config.families);
if (!targeted && !families.some((family) => familyHasRunnableJobs(family, completions))) {
return finish({
dispatched: false,
reason: families.some(familyIsConfigured) ? PLAN_COMPLETE_SKIP_REASON : 'no families enabled',
});
}
// Scoped to the one family when the caller named it — a per-family "Run now",
// or a completion continuation. Every card this read returns for another family
// is discarded by the `familyId` filter below, and each one costs its own
// multi-second PTY scrape (WAIT.FRESH bypasses the cache by design). Unscoped
// that was two wasted scrapes a day; with the continuation it would be N per
// finished burn agent, on every link of the chain.
const quotas = await getProviderQuotas({ wait: WAIT.FRESH, family: familyId }).catch((err) => {
console.error(`❌ Quota-burn could not read provider quota: ${err.message}`);
return null;
});
if (!quotas) return finish({ dispatched: false, reason: 'provider quota read failed' });
const [charged, blocks, reservations] = await Promise.all([
getQuotaBurnDispatches(), getActiveQuotaBurnBlocks(), getQuotaBurnReservations(),
]);
// An unreadable dispatch ledger reads as "0 used this window", which would
// walk straight past `maxDispatchesPerWindow` and spend quota the user already
// spent (#4115). Same posture as the provider-quota read above: skip the cycle.
if (!charged) return finish({ dispatched: false, reason: 'dispatch ledger read failed' });
// And the same posture for the reservations: reading them as "nothing pending"
// would both re-queue a step already awaiting acceptance and re-open the share
// of the cap that step is holding.
if (!reservations) return finish({ dispatched: false, reason: 'pending reservations read failed' });
// What the ladder gates on: charges plus the burns already committed to this
// window and not yet accepted.
const dispatches = withPendingDispatches(charged, reservations);
const reserved = new Set(Object.keys(reservations));
// `force` is the page's explicit family/job "Run now" — the
// window/reserve/cap/denial gates that bound UNATTENDED burns don't apply to a
// run the user just asked for. It goes through the same selection, so the
// candidate still carries the family's real card and limit; it only comes
// back `charge: false`.
const candidates = selectBurnCandidates(quotas, config, {
dispatches, blocks, completions, bypassGatesFor: force ? familyId : null,
}).filter((candidate) => !familyId || candidate.family.id === familyId);
if (!candidates.length) {
// Scoped to the family the user asked about, and NOT filtered to enabled
// families: when someone force-runs a job on a family whose checkbox is
// off, "disabled" IS the answer — reporting a different family's verdict
// instead (or claiming it is "ready" when it did not run) leaves them with
// no path to the control they need.
const reasons = evaluateFamilies(quotas, config, { dispatches, blocks, completions })
.filter(({ family }) => (familyId ? family.id === familyId : family.enabled))
.map(({ family, skipReason }) => `${family.id}: ${skipReason || 'ready'}`);
return finish({
dispatched: false,
reason: reasons.length ? `no burnable window — ${reasons.join('; ')}` : 'no families enabled',
});
}
const attempts = [];
const dispatched = [];
// Where each family's plan walk resumes (see `rotatePlanAfter`). Read once per
// cycle, before any dispatch, so two families in the same cycle can't see each
// other's fresh run-log entries.
const cursors = await lastDispatchedJobByFamily();
// ONE catalog for the whole cycle. It reads the schedule, the app registry and
// the custom-job store, none of which vary per family — building it inside the
// loop paid all three once per enabled family, every tick.
const catalog = await catalogFor(candidates.flatMap((entry) => entry.family.jobs || []));
// Every eligible family gets its own dispatch — see the module header. The
// loop does NOT break on the first success: `candidates` is already the set
// that passed the full gate ladder, and each entry spends a different
// provider's window.
for (const candidate of candidates) {
let selectedJobId = jobId;
if (candidate.family.sequence) {
const next = await nextQuotaBurnSequenceJob(candidate.family, { completions, reservations, catalog, ignoreTaskId });
if (!next.job || (jobId && jobId !== next.job.id)) {
attempts.push({ familyId: candidate.family.id, jobId: jobId || 'sequence', skipped: next.reason || 'an earlier sequence step must finish first' });
continue;
}
selectedJobId = next.job.id;
}
const outcome = await dispatchFromCandidate(candidate, {
jobId: selectedJobId, force, completions, catalog, reserved, afterJobId: cursors.get(candidate.family.id) || null,
});
attempts.push(...outcome.attempts.map((entry) => ({ familyId: candidate.family.id, ...entry })));
if (!outcome.dispatched) continue;
// Charge the window only once work is actually ACCEPTED. A declined step
// reports rather than throwing and never reaches here, so the cap bounds
// real burns and not attempts — but a step whose lane accepts
// asynchronously (`awaiting`) is not accepted yet either. That one holds a
// reservation instead, which counts against the cap for the whole in-flight
// window and converts to exactly one charge when the request is joined to
// the task it produced (`quotaBurnAcceptance.js`). `charge` is false for a
// forced run, which the user asked for outside the automatic budget.
const requestId = outcome.result.awaiting?.requestId || null;
if (requestId) {
const held = await reserveQuotaBurnDispatch({
familyId: candidate.family.id,
stepId: outcome.job.id,
dispatchKey: candidate.dispatchKey,
charge: candidate.charge,
// A strict sequence finishes on successful agent completion (or a drained
// claim backlog), not merely on the request being accepted.
runOnce: outcome.job.runOnce === true && !candidate.family.sequence,
requestId,
});
// The request is already on the schedule and cannot be recalled, so this
// is a warning rather than a failure: the burn will run, it just will not
// be charged. Better an uncharged burn than a charge for work that may
// still be refused.
if (!held) console.error(`⚠️ Quota-burn could not reserve ${candidate.family.id}/${outcome.job.id} for request ${requestId} — this burn will go uncharged`);
} else {
if (candidate.charge) await recordQuotaBurnDispatch(candidate.dispatchKey);
// A `run once` job records its one dispatch even when the run was FORCED
// and therefore uncharged. The two ledgers answer different questions:
// `charge` is about this window's automatic budget, while `runOnce` is a
// statement about the WORK ("this only needs doing once") — and the work
// just happened, however it was triggered. The ▶ on the row stays the way
// back, since a forced run bypasses this gate too.
if (outcome.job.runOnce && !candidate.family.sequence) {
await recordQuotaBurnJobCompletion(candidate.family.id, outcome.job.id)
// A ledger failure must not fail a dispatch that already happened —
// the worst case is the job running one extra time next cycle, which
// is the pre-`runOnce` behavior.
.catch((err) => console.error(`⚠️ Quota-burn run-once ledger for ${candidate.family.id}/${outcome.job.id}: ${err.message}`));
}
}
// One run-log entry PER dispatch, recorded as it happens: the page's
// "Recent runs" list is how the user audits what their subscriptions were
// spent on, and folding three families into one row would hide two of them.
// A pending row is settled IN PLACE once its request is accepted or refused
// (`settleQuotaBurnRun`), so one burn stays one line in a capped feed.
dispatched.push(await finish({
dispatched: true,
familyId: candidate.family.id,
jobId: outcome.job.id,
...stepIdentity(outcome.job),
dispatchKey: candidate.dispatchKey,
requestId,
pending: Boolean(requestId),
taskId: outcome.result.detail?.taskId ?? null,
charged: candidate.charge && !requestId,
hoursUntilReset: Math.round(candidate.hoursUntilReset * 10) / 10,
percentRemaining: candidate.limit?.percentRemaining ?? null,
summary: outcome.result.summary || 'dispatched',
detail: outcome.result.detail || null,
}));
}
// Single dispatch keeps the flat entry shape the route, the run log, and the
// tests already speak. Several come back as one aggregate so the caller's
// toast names all of them rather than an arbitrary first.
if (dispatched.length === 1) return dispatched[0];
if (dispatched.length > 1) {
return {
dispatched: true,
summary: `Dispatched ${dispatched.length} jobs — ${dispatched.map((entry) => `${entry.familyId}: ${entry.summary}`).join('; ')}`,
dispatches: dispatched,
};
}
// Report what each job actually said. Collapsing every non-dispatch to "no
// job had pending work" asserts something usually false — a job that DID have
// work and declined ("an identical burn task is already running", "managed
// app app-x no longer exists", "no enabled CLI/TUI provider in the grok
// family") is the actionable case, and it was the one being thrown away.
// Prefix each reason with its family once more than one was evaluated —
// "j: no managed app selected" is unactionable when three plans were walked.
const label = (entry) => (candidates.length > 1 ? `${entry.familyId}/${entry.jobId}` : entry.jobId);
const detail = attempts.map((entry) => `${label(entry)}: ${entry.skipped}`).join('; ');
const plans = candidates.map((candidate) => candidate.family.id).join(', ');
return finish({
dispatched: false,
familyId: candidates[0].family.id,
reason: detail
? `nothing dispatched — ${detail}`
: `no enabled job in the ${plans} plan${candidates.length > 1 ? 's' : ''}`,
});
}
/**
* The config page's whole payload: the plan plus, per family, whether it would
* burn on the next tick (and if not, exactly why) and each job's pending count.
*
* Returns the config it loaded so the route doesn't read and normalize the same
* file a second time.
*
* NEVER blocks on a quota reading. A cached one is served immediately (and
* refreshed behind the response once stale); a cold cache comes back as a
* `pending: true` card while the scrape runs, which `evaluateFamily` reports as
* "reading provider quota…" and never turns into a burn. Waiting instead cost
* this page 20-30s per open — a PTY spawn per family — for numbers that move by
* single digits an hour. The page's explicit Refresh passes `refresh: true` and
* does wait, because that click IS the request for a live reading.
*/
export async function getQuotaBurnStatus({ refresh = false } = {}) {
// Independent reads: only `quotas` is slow (a PTY scrape on the Refresh path),
// and nothing else waits on it.
const configPromise = getQuotaBurnConfig();
const [config, quotas, charged, blocks, completions, runs, catalog, reservations] = await Promise.all([
configPromise,
getProviderQuotas({ wait: refresh ? WAIT.FRESH : WAIT.NEVER }).catch((err) => {
console.error(`❌ Quota-burn status could not read provider quota: ${err.message}`);
return [];
}),
// Degrades to "no dispatches counted" on an unreadable ledger for the same
// reason the completions read below does: on the STATUS path the cost of
// being wrong is a stale `N/M used` label, not re-spent quota. The cycle's
// read of the same ledger refuses to run instead.
getQuotaBurnDispatches().then((ledger) => ledger || {}),
getActiveQuotaBurnBlocks(),
// An unreadable ledger degrades to "no badges" here rather than failing the
// whole status read — the opposite of the CYCLE's posture, and deliberately
// so: the cost of being wrong on this path is a missing "Ran once" label,
// while on the cycle's path it is re-spending quota on finished work.
getQuotaBurnCompletions().then((ledger) => ledger || {}),
getQuotaBurnRuns(),
// Depends only on the config, never on `quotas` — so it overlaps the PTY
// scrape the Refresh path waits on instead of queueing behind it.
configPromise.then((cfg) => catalogFor(Object.values(cfg.families).flatMap((family) => family.jobs || []))),
// READ ONLY. The page shows a reserved burn as already used against the cap
// — it is committed to this window — but it never settles one: a probe read
// that charged quota would make opening the page a spend (AGENTS.md).
// Degrades to "nothing pending" for the same reason the two ledgers above
// do: on this path the cost of being wrong is a stale `N/M used` label,
// while the cycle refuses to run instead.
getQuotaBurnReservations().then((pending) => pending || {}),
]);
const dispatches = withPendingDispatches(charged, reservations);
const cards = new Map(quotas.map((card) => [card.family, card]));
// ONE pass over the gate ladder the runner uses — the page's "will burn" and
// its reason come from the same verdict, so they can't contradict each other.
const families = await Promise.all(evaluateFamilies(quotas, config, { dispatches, blocks, completions })
.map(async ({ family, candidate, skipReason, block }) => {
const card = cards.get(family.id);
return {
id: family.id,
label: candidate?.card?.label || card?.label || family.id,
percentRemaining: candidate?.limit?.percentRemaining ?? null,
hoursUntilReset: candidate ? Math.round(candidate.hoursUntilReset * 10) / 10 : null,
// WHICH window those two numbers describe. Without it the card reports a
// percentage and a countdown with no way to tell the weekly allowance
// from the 5-hour one — the exact ambiguity that hid the wrong window
// being selected in the first place.
windowLabel: candidate ? windowLabelOf(candidate.limit) : null,
dispatchesUsed: candidate?.dispatchesUsed ?? null,
willBurn: Boolean(candidate),
skipReason: skipReason || null,
// An observed provider refusal, surfaced whether or not it is the gate
// that closed — a family blocked AND out of window should say both.
blockedUntil: block?.until ? new Date(block.until).toISOString() : null,
blockedReason: block?.reason || null,
// The reading for this family is being taken right now — the page polls
// again instead of leaving a card that looks like it has no quota.
pending: Boolean(card?.pending),
// Probe pending work only for families the user actually enabled, and
// only for jobs a cycle could still pick — a probe is not free (the
// universe job reads every bible), and neither a disabled family's
// counts nor a spent `run once` job's are ever acted on. `ranAt` is what
// the row renders instead, and it is the only signal the page has that a
// step is finished rather than merely idle.
jobs: await Promise.all(family.jobs.map(async (job) => {
const ranAt = completions[quotaBurnJobKey(family.id, job.id)] || null;
// `jobIsSpent` re-derived from the value just read rather than called —
// it would rebuild the same key and re-index the same object. Not
// shipped on the wire either: the client gates its badge on the
// OPTIMISTIC config it holds, so a just-ticked checkbox reads as spent
// before the save round-trips, and a server copy would only be a second
// rule to keep in sync.
const spent = job.runOnce === true && Boolean(ranAt);
return {
id: job.id,
ranAt,
// `context` is stripped: it is the probe→run hand-off (see the job
// registry), and it holds whatever the probe already computed —
// for the universe jobs a picked-universe payload. The page renders
// `count` and `detail` only, so shipping the rest would put probe
// internals on the wire and grow with every job type.
pending: family.enabled && !spent ? wireShape(await probeStep({ job, family, catalog })) : null,
};
})),
};
}));
// Availability is a fact about the catalog RIGHT NOW — a reference whose task
// was disabled or deleted has to render its reason and lose its Run affordance
// — so it is stamped onto the config the page receives and never onto disk.
return { config: applyQuotaBurnAvailability(config, catalog), status: { running, families, runs } };
}
/**
* When the last SCHEDULED cycle ran, in ms — from memory once this process has
* run one, otherwise from the persisted run log.
*
* Reading the log is what makes `checkIntervalMinutes` mean anything across a
* restart. `lastRunAt` is in-process, so on a bare `null` the very first tick
* after boot is always "due": a PM2 restart, a self-update, or a crash-loop
* would each fire a full cycle (a multi-second TUI scrape per family, plus a
* possible dispatch) ~60s later, no matter how long the configured interval is
* — pacing a 12-hourly plan into minutes, bounded only by the window cap.
*/
async function lastScheduledRunAt() {
if (lastRunAt) return lastRunAt;
const runs = await getQuotaBurnRuns().catch(() => []);
const previous = runs.find((entry) => entry?.trigger === 'scheduled' && entry.at);
const parsed = previous ? Date.parse(previous.at) : NaN;
// Cache it so a restart pays the log read once, not every minute. An
// unreadable/absent log leaves it null and the next tick runs — the correct
// fail-open for "we have no idea when we last ran".
if (Number.isFinite(parsed)) lastRunAt = parsed;
return lastRunAt;
}
/**
* Continuation: when a burn agent finishes, evaluate its family again so the
* NEXT job in the plan goes out.
*
* Without this the plan advances only once per `checkIntervalMinutes` — at the
* 12-hour default, two dispatches a day against a window that resets every 5
* hours, so most of the allowance the feature exists to spend expired unspent.
* Completion is the right pacing signal: it runs the plan strictly one agent at
* a time and asks the gate ladder, every time, whether there is still quota to
* spend.
*
* It cannot run away. The chain is serial by construction — the next dispatch
* needs an agent to finish first — and every link re-reads the family's live
* quota, so the reserve, the reset horizon, and an observed provider refusal
* each close it as soon as the numbers say to. A `maxDispatchesPerWindow` other
* than the unlimited default bounds it earlier still, from the same ledger every
* continuation dispatch is charged to.
*/
function onBurnAgentCompleted(agent) {
const familyId = agent?.metadata?.taskQuotaBurnFamily;
if (!familyId) return null;
// A relaunch is not a completion, and both halves below would misread it as
// one. The denial ledger would file "Relaunched by user on <provider>" as an
// observed provider refusal and block the family on it — the exact opposite of
// what the user asked for, since swapping providers is usually how they get
// AROUND a limit. And the plan walk would dispatch the next job while the
// relaunched task is being re-spawned, breaking the one-agent-at-a-time pacing
// this chain depends on to stay closed. The continuation run emits its own
// `agent:completed`; that is the one that settles the family.
//
// Gated on `isQueuedContinuation`, not the bare `isAgentHandoff`: a stranded
// pause retirement (`retireStrandedPausedAgents`) also stamps `resumed: true`
// but queues nothing, so there is no continuation run to emit that later
// `agent:completed` — returning `null` here stalled the family until the next
// `checkIntervalMinutes` tick (#7469).
if (isQueuedContinuation(agent)) return null;
// Returned so tests can await the cycle. The emitter ignores it — this is a
// fire-and-forget listener, and the `.catch` is what keeps a rejection from
// reaching the emitter as an unhandled one.
//
// The denial ledger is folded in FIRST and awaited. If this run was refused,
// the family's short rolling window is spent and the very next thing this
// function does — dispatch the next job — would walk straight into the same
// wall. Recording after the cycle (or from a second `agent:completed`
// subscriber, whose ordering against this one is not guaranteed) is one wasted
// agent too late, every single time. A ledger failure must not stop the
// continuation, so it degrades to a logged warning.
// The plan is walked only for an agent the plan OWNS (`burnPlanOwnsAgent`):
// a manual maintenance run's agent spends this family's window — so the
// denial ledger still hears about it — but its completion advances that run,
// never this plan.
const deepAudit = agent?.result?.deepAudit || agent?.metadata?.deepAudit;
const walkPlan = () => (agent?.metadata?.taskAuditDepth === 'deep' || deepAudit) && deepAudit?.complete !== true
? Promise.resolve({ skipped: 'deep-audit-checkpoint', reason: deepAudit.reason })
: completeQuotaBurnSequenceStep(agent)
.then(() => runQuotaBurnCycle({ trigger: 'continuation', familyId, ignoreTaskId: agent?.result?.success ? agent?.taskId : null }));
return recordBurnAgentCompletion(agent)
.catch((err) => console.error(`⚠️ Quota-burn denial ledger for ${familyId}: ${err.message}`))
.then(() => (burnPlanOwnsAgent(agent) ? walkPlan() : { skipped: 'maintenance-run' }))
.catch((err) => console.error(`❌ Quota-burn continuation for ${familyId} failed: ${err.message}`));
}
async function tick() {
const config = await getQuotaBurnConfig().catch(() => null);
if (!config?.enabled) return;
const dueAt = (await lastScheduledRunAt() || 0) + config.checkIntervalMinutes * 60_000;
if (Date.now() < dueAt) return;
await runQuotaBurnCycle({ trigger: 'scheduled' });
}
/**
* Arm the loop. Called once at boot. The interval itself is unconditional and
* cheap (a config read per minute); the SPEND is gated inside `tick`, so
* toggling the feature on the page takes effect without a restart.
*/
export function startQuotaBurnScheduler() {
if (tickTimer) return;
// Every timer callback runs outside the request lifecycle — an uncaught throw
// here would take down the process, so the whole tick is caught and logged.
tickTimer = setInterval(() => {
// `running` is already released by runQuotaBurnCycle's finally — this only
// has to keep the throw from reaching the timer and killing the process.
tick().catch((err) => console.error(`❌ Quota-burn tick failed: ${err.message}`));
}, TICK_MS);
tickTimer.unref?.();
// Same "outside the request lifecycle" posture as the timer: an emitter
// callback that throws would take the process down, so `onBurnAgentCompleted`
// never awaits and never rethrows.
cosEvents.on('agent:completed', onBurnAgentCompleted);
console.log('🔥 Quota-burn loop armed (off unless enabled in Quota Burn settings)');
}
/** Test seam — one interval tick, without waiting on the timer. */
export const __tickQuotaBurn = tick;
/** Test seam — the completion continuation, without going through cosEvents. */
export const __onBurnAgentCompleted = onBurnAgentCompleted;
/** Test seam — disarms the timer/listener and clears the in-process cycle state. */
export function __resetQuotaBurnRunner() {
if (tickTimer) clearInterval(tickTimer);
tickTimer = null;
cosEvents.off('agent:completed', onBurnAgentCompleted);
deferredContinuations.clear();
running = false;
lastRunAt = null;
}