Skip to content

Evenfire is an intelligence layer platform for running LLM agents that take real actions, on your infrastructure

TypeScript
2.6 KB
Raw Blame
import { LIMITS as GROK_LIMITS } from '@clerum/grok-provider-attempt-contract'
import { LIMITS } from '@clerum/llm-provider-attempt-contract'
import type { AuthorizeBodyAdmissionPolicy } from './llmProviderAttemptBodyAdmission.js'

// Ships as an explicit uncertified ceiling; see #813. The combined
// authorize/GFS workload has not yet selected and recorded a count, deadlines,
// heap policy and non-heap margin below 768 MiB in three fresh cgroups.
// The policy is process-wide; every Node worker needs its own measured budget.
//
// It governs only the retained path: bodies declared above the text authorize
// envelope, chunked bodies and unparseable lengths. Bodies declared at or below
// the envelope are bounded by their own parser limit and never take a unit (see
// the route). The split is by declared length, not by content: a V2 text-only
// body carries each text part twice (content and contentParts) and can exceed
// the envelope while inside the non-image budget; it then takes the unit like
// an image body.
//
// maxQueued, maxPerPrincipal and queueWaitMs bound latency and fairness, not
// memory: queued requests are paused before any body read. Two waiters allow
// one same-principal overlap plus one other principal; the running+queued
// share stops one principal holding all three positions. queueWaitMs is the
// longest a healthy holder can keep the unit (its read deadline plus its work
// deadline), so a waiter is refused only when the holder outlives both.
//
// retryAfterSeconds is the Retry-After on a capacity refusal. The Host retries
// once in the same provider after it, re-uploading the body, and the second
// refusal is terminal. It equals the read deadline, the bound on a slow
// upload, and stays below the Host's 30 s retry ceiling. Uncertified (#813).
//
// maxDiscardBodyBytes is the retained parser's limit. A capacity refusal reads
// and discards up to that many bytes before it answers, so the gateway reads
// the 503 instead of failing its write on a reset (502). Discarded bytes are
// never retained, so this costs network and CPU, not memory.
const READ_DEADLINE_MS = 10_000
const WORK_DEADLINE_MS = 30_000

export const LLM_PROVIDER_ATTEMPT_ADMISSION_POLICY: AuthorizeBodyAdmissionPolicy = Object.freeze({
  maxInFlight: 1,
  maxQueued: 2,
  maxPerPrincipal: 2,
  queueWaitMs: READ_DEADLINE_MS + WORK_DEADLINE_MS,
  readDeadlineMs: READ_DEADLINE_MS,
  workDeadlineMs: WORK_DEADLINE_MS,
  closeGraceMs: 250,
  retryAfterSeconds: READ_DEADLINE_MS / 1000,
  maxDiscardBodyBytes: Math.max(
    LIMITS.maxVisualRequestBodyBytes,
    GROK_LIMITS.maxVisualRequestBodyBytes
  ),
})