Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

9.2 KB
Raw Blame
/**
 * Provider cooldown policy — "this run failed; should the provider be benched,
 * and for how long?"
 *
 * One answer, shared by both paths that can observe a provider failing:
 *   - `promptRunner.js` (a one-shot prompt cascade), and
 *   - `agentFinalization.js` (a finished CoS agent run).
 *
 * They used to disagree: the prompt path had this table, and the agent path
 * benched only usage/rate limits — so an `agy` TUI agent blocked on account
 * verification left the provider marked healthy and the next dequeued task
 * picked it again and died the same way. Keeping the policy here means a
 * category benches for the same duration no matter which path noticed.
 *
 * Pure — no I/O, no provider-status handle. Callers own the actual marking.
 */

import { ERROR_CATEGORIES } from './aiToolkit/errorDetection.js';

// Cooldown per error category — how long the failed provider is marked
// unavailable so subsequent calls skip it and proactively use the fallback.
// USAGE_LIMIT is absent because `markUsageLimit` parses the wait time
// from the error body (e.g. "resets 5pm"). Values target the timescale
// the user can plausibly recover the underlying cause:
//   RATE_LIMIT      — 5m: provider-side counter typically clears in minutes
//   AUTH_ERROR      — 15m: usually a config issue that needs human action
//   MODEL_NOT_FOUND — 30m: also config; longer because retry is unlikely
//   QUOTA_EXCEEDED  — 60m: billing/credits; retry sooner is futile
//   NETWORK_ERROR   — 2m: usually a transient hiccup
//   RESOURCE_EXHAUSTED — 2m: a local GPU that just OOM'd needs the in-flight
//                     work to drain before it can serve the same context again;
//                     the endpoint itself is healthy, so this is a short bench
//   TIMEOUT/UNKNOWN — 1m: short enough to retry, long enough to skip
//                     while the immediate workload retries via fallback. For a
//                     TIMEOUT this is only the FLOOR — see benchMsForTimeout:
//                     a run that burned its whole budget benches for at least
//                     that budget, because a shorter bench guarantees the next
//                     caller repeats the same full-length stall.
export const COOLDOWN_MS_BY_CATEGORY = {
  [ERROR_CATEGORIES.RATE_LIMIT]: 5 * 60 * 1000,
  [ERROR_CATEGORIES.AUTH_ERROR]: 15 * 60 * 1000,
  [ERROR_CATEGORIES.MODEL_NOT_FOUND]: 30 * 60 * 1000,
  [ERROR_CATEGORIES.QUOTA_EXCEEDED]: 60 * 60 * 1000,
  [ERROR_CATEGORIES.NETWORK_ERROR]: 2 * 60 * 1000,
  [ERROR_CATEGORIES.RESOURCE_EXHAUSTED]: 2 * 60 * 1000,
  [ERROR_CATEGORIES.TIMEOUT]: 60 * 1000,
  [ERROR_CATEGORIES.UNKNOWN]: 60 * 1000,
};
export const DEFAULT_COOLDOWN_MS = 60 * 1000;
// A usage limit with no parsed reset window. Matches QUOTA_EXCEEDED's hour:
// both mean "the plan is spent", and retrying inside the hour is futile.
export const DEFAULT_USAGE_LIMIT_COOLDOWN_MS = 60 * 60 * 1000;
// The longest a bench may ever last. A provider-reported reset is trusted over
// the category table, but not unconditionally: a weekly window, or a value in a
// unit that merely happens to parse (bare epoch seconds read as a year), would
// otherwise take a provider offline for days with no path back.
export const MAX_BENCH_MS = 24 * 60 * 60 * 1000;

// Categories the tiered cascade classifies as schema/type (mirrors
// autoFixer.CATEGORY_TO_TIER's SCHEMA_TYPE entries; kept out of the CoS stack so
// promptRunner stays decoupled from it — see loadAutoFixer). promptRunner tags
// its synthetic response-schema failure 'parse-error' so it lands in this tier
// when it re-enters the cascade.
const SCHEMA_TYPE_CATEGORIES = new Set([
  'parse-error', 'bad-request', 'context-length', 'ollama-context-window', 'output-length',
  'build-error', 'lint-error',
]);
export const isSchemaTypeCategory = (category) => SCHEMA_TYPE_CATEGORIES.has(category);

// Every runner prints the budget it blew the same way: "CLI run timed out
// after 600000ms". Captured so `benchMsForTimeout` can bench for at least that.
const TIMEOUT_BUDGET_RE = /timed out after +([0-9]+) *ms/i;

/**
 * True when a failure is specific to the REQUEST or RESPONSE rather than the
 * provider, so benching the provider would take a healthy service offline for
 * every other caller.
 *
 * A content/safety refusal is prompt-specific — the provider is healthy and
 * other prompts still work. A model-not-found is likewise request-specific: the
 * request named a model id the (reachable) endpoint doesn't have, so benching
 * would take its OTHER valid models offline for the full cooldown (e.g. one bad
 * `codex-configured-default` vision call benching Ollama so a correct
 * `qwen2.5vl` call then proactively swaps to a non-vision fallback). A genuine
 * endpoint outage surfaces as NETWORK_ERROR, not MODEL_NOT_FOUND, so it is still
 * benched. A schema/type failure (#2350) is response-specific: the provider
 * returned HTTP 200 with content that just didn't match this caller's declared
 * schema. In every case the single failing call still falls back via its
 * caller's retry path.
 */
export function isRequestSpecificCategory(category) {
  return category === ERROR_CATEGORIES.CONTENT_REFUSAL
    || category === ERROR_CATEGORIES.MODEL_NOT_FOUND
    || isSchemaTypeCategory(category);
}

/**
 * How long to bench a run that died on its own wall-clock timeout.
 *
 * A timeout is the one category whose cost is stated in its own message: the
 * run consumed the provider’s entire configured budget before failing.
 * Benching it for less than that budget is self-defeating — the provider
 * becomes eligible again long before the workload that just stalled could have
 * finished, so the next caller pays the same full-length stall. Observed on a
 * pinned local backend whose prompt had outgrown what it could serve: ten
 * consecutive 600s timeouts, each followed by a 60s bench.
 *
 * So take the larger of the category floor and the timeout the failure names,
 * clamped to {@link MAX_BENCH_MS}. Every runner spells the budget the same way
 * ("… timed out after 600000ms" — services/runner.js, aiToolkit/runner.js,
 * lib/cliProviderRun.js); a timeout message with no parsable budget (the TUI
 * retry-exhaustion form) keeps the plain category floor.
 *
 * @param {string|null} message — the failure message
 * @param {number} floorMs — the category’s table value
 * @returns {number} milliseconds
 */
function benchMsForTimeout(message, floorMs) {
  const match = typeof message === 'string' ? message.match(TIMEOUT_BUDGET_RE) : null;
  const budgetMs = match ? Number(match[1]) : NaN;
  if (!Number.isFinite(budgetMs) || budgetMs <= 0) return floorMs;
  return Math.min(Math.max(floorMs, budgetMs), MAX_BENCH_MS);
}

/**
 * Decide how a failed provider should be benched.
 *
 * @param {object|null} analysis — an errorDetection / agentErrorAnalysis payload
 * @returns {null
 *   | { marker: 'usage-limit', category: string, message: string|null, waitTime: string|null }
 *   | { marker: 'unavailable', category: string, message: string|null, waitTimeMs: number }}
 *   `null` means "don't bench". The `usage-limit` marker is distinct because
 *   `markUsageLimit` parses its own wait window out of the provider's message.
 */
export function resolveProviderBench(analysis) {
  const category = analysis?.category || ERROR_CATEGORIES.UNKNOWN;
  if (category === ERROR_CATEGORIES.RUNTIME_BUDGET_EXHAUSTED || isRequestSpecificCategory(category)) return null;

  const message = analysis?.message || null;
  if (category === ERROR_CATEGORIES.USAGE_LIMIT) {
    return { marker: 'usage-limit', category, message, waitTime: analysis?.waitTime ?? null };
  }
  const floorMs = COOLDOWN_MS_BY_CATEGORY[category] ?? DEFAULT_COOLDOWN_MS;
  return {
    marker: 'unavailable',
    category,
    message,
    waitTimeMs: category === ERROR_CATEGORIES.TIMEOUT ? benchMsForTimeout(message, floorMs) : floorMs,
  };
}

/**
 * How long a bench should actually last, in milliseconds.
 *
 * `resolveProviderBench` deliberately returns the `usage-limit` marker WITHOUT a
 * duration, because `providerStatus.markUsageLimit` parses its own window out of
 * the provider's error text. A caller that benches something other than a whole
 * provider record — the ChatGPT-subscription text transport, say — has no such
 * parser, and needs a number.
 *
 * `resetsAt` is the provider's own stated recovery time when one is known (Codex
 * reports it on the rate-limit window). It wins over the category table because
 * it is the truth rather than an estimate, but only when it is a real future
 * timestamp: a past or unparseable value would otherwise resolve to "unbench
 * immediately", which is exactly the failure a bench exists to prevent. It is
 * also clamped to {@link MAX_BENCH_MS} at the other end.
 *
 * @param {ReturnType<typeof resolveProviderBench>} bench
 * @param {{ resetsAt?: string|number|Date|null, now?: number }} [options]
 * @returns {number} milliseconds; `0` when `bench` is null (don't bench)
 */
export function resolveBenchWaitMs(bench, { resetsAt = null, now = Date.now() } = {}) {
  if (!bench) return 0;
  const resetMs = resetsAt == null ? NaN : new Date(resetsAt).getTime();
  if (Number.isFinite(resetMs) && resetMs > now) return Math.min(resetMs - now, MAX_BENCH_MS);
  if (bench.marker === 'usage-limit') return DEFAULT_USAGE_LIMIT_COOLDOWN_MS;
  return bench.waitTimeMs ?? DEFAULT_COOLDOWN_MS;
}