Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* CoS local-inference agent slots (issue #4834)
*
* `promptRunner`'s `withLocalConcurrencyGate` caps concurrent IN-FLIGHT calls
* per local endpoint, but it only sees requests PortOS itself makes. A CoS TUI
* agent runs a vendor CLI that opens its own connection to the local model
* server — PortOS never observes that traffic, so `dequeueNextTask` could
* dispatch two or three agents at one GPU and push it into an accelerator OOM.
*
* This module supplies the *avoidance* half, enforced at two altitudes:
*
* - `acquireLocalEndpointSpawnSlot` is the AUTHORITATIVE cap, called from
* subAgentSpawner's `task:ready` listener — the one chokepoint every
* emitter funnels through. It reserves the slot across the spawn window so
* two dispatches can't both read a pre-registration snapshot and pass.
* - `createLocalEndpointSlotContext` feeds the pure capacity tracker in
* cosDequeue.js, so `dequeueNextTask` simply never emits a task that would
* be held — cheaper, and it logs queued-no-slot against the right task.
*
* Deliberately separate accounting from promptRunner's in-process gate: one
* throttles requests PortOS makes, the other throttles agents PortOS launches.
* A shared counter across process boundaries isn't worth the coupling.
*/
import { LOCAL_LLM_MAX_CONCURRENCY } from '../lib/localLlmConcurrency.js';
import { SWARM_COUNT_MAX, SWARM_COUNT_MIN } from '../lib/validation.js';
import { localRuntimeForProvider, localEndpointPort, normalizeOpenAiBaseUrl } from '../lib/localProviderRuntime.js';
import { listProviders, getActiveProvider } from './providers.js';
import { isProviderAvailable, getFallbackProvider } from './providerStatus.js';
import { allowedModesFor } from '../lib/callerModePolicy.js';
/**
* The raw base URL a provider's inference actually goes to, local or not.
*
* `localRuntimeForProvider` already owns this question — it reads the base the
* provider ITSELF configures (an OpenCode `baseURL`, `ANTHROPIC_BASE_URL`, or
* `endpoint`), then the `OLLAMA_URL`/`OLLAMA_HOST`/`LM_STUDIO_URL` overrides the
* backend managers read, then the runtime's canonical default. That matters
* here: a user who relocates their daemon with `OLLAMA_HOST` would otherwise get
* two independent slot keys for one daemon — the exact double-booking this cap
* exists to prevent.
*
* It answers `null` for a provider that maps to no KNOWN local runtime, so fall
* back to the recorded endpoint: an arbitrary local server (the shipped
* `opencode-vllm-tui` on `127.0.0.1:18020`, or a user's own TUI provider) still
* occupies a GPU. Whether either is on THIS box is decided by
* `localEndpointPort` below, never by the provider's name or model id.
*/
export function providerBaseUrl(provider) {
return localRuntimeForProvider(provider)?.endpoint
|| provider?.endpoint
// Last resort, and REMOTE by construction — `localRuntimeForProvider` already
// took this value when it pointed at this box. It still has to be returned:
// an Ollama-backed CLI aimed at another host records its base ONLY here, and
// `endpointForAgent` treats an absent stamp as "re-resolve by id". Answering
// null would let that cloud agent start counting against a local endpoint the
// moment its provider record is re-pointed — the hole the stamp exists to close.
|| provider?.envVars?.ANTHROPIC_BASE_URL
|| null;
}
/**
* The slot key a provider's inference occupies, or null when it isn't on this
* machine. Host+port identifies the model server; the scheme, the path (`/v1`)
* and the host SPELLING do not — the shipped catalog seeds `lmstudio` at
* `localhost:1234` and everything else at `127.0.0.1`, so keying on the raw
* string would give one LM Studio process two independent caps.
*
* `localEndpointPort` supplies that normalization (every loopback spelling
* collapses, the port defaults by scheme) AND rejects a LAN/Tailscale peer that
* merely shares a port. Not gated on `provider.type === 'api'` (unlike
* promptRunner's request gate): a TUI provider pointed at a local LM Studio is
* the case this exists for.
*/
export function localEndpointOfProvider(provider) {
return localSlotKey(providerBaseUrl(provider));
}
/**
* Codex's multi-agent limit includes the root orchestrator. Cloud claim swarms
* therefore need `workers + 1` session threads to honor the configured worker
* count. A provider on this machine deliberately gets no override: local
* inference remains governed by its bounded GPU concurrency posture.
*/
export function cloudSwarmThreadCapacity(provider, swarmCount) {
const workers = Number(swarmCount);
if (!Number.isSafeInteger(workers) || workers < SWARM_COUNT_MIN || workers > SWARM_COUNT_MAX) return null;
return localEndpointOfProvider(provider) ? null : workers + 1;
}
// Normalize first so a schemeless value still parses — `localEndpointPort` goes
// through `new URL`, which reads `localhost:1234` as a SCHEME and yields null.
// Providers configured through the UI can carry a bare `host:port`.
const localSlotKey = (endpoint) => {
const port = localEndpointPort(normalizeOpenAiBaseUrl(endpoint));
return port ? `localhost:${port}` : null;
};
/**
* Build the per-cycle local-endpoint lookups from an ALREADY-FETCHED provider
* snapshot. Pure and synchronous — the capacity tracker consults it per
* candidate task, and tests drive it with fixtures instead of a live toolkit.
*
* - `endpointForAgent(agent)` — the endpoint a RUNNING agent is occupying,
* from the `providerId` agentLifecycle stamps onto its metadata.
* - `resolveLocalEndpoint(task)` — the endpoint a QUEUED task would land on.
* Mirrors `resolveAgentProviderAndModel`: a `metadata.provider` pin wins, an
* unknown pin falls back to the active provider, and an UNAVAILABLE provider
* is followed through the SAME `getFallbackProvider` spawn uses. Following
* the swap matters in both directions: gating on a benched provider would
* hold the task behind a GPU it never touches (nothing would clear the hold),
* while ignoring the swap entirely would drop the cap exactly when the
* endpoint is unhealthy — the shipped catalog has four providers sharing
* `127.0.0.1:18021`, so a fallback commonly lands on the same server. A
* RUNTIME fallback after this point can still move a run: this is avoidance,
* not a guarantee, and promptRunner's gate plus the OOM nudge/fail-over
* remain the recovery half.
*
* `isAvailable` / `resolveFallback` are injected (rather than imported) so this
* stays pure and a test can drive the real resolver without provider-status
* module state.
*/
export function createLocalEndpointSlotContext({
providers = [],
activeProvider = null,
// One knob governs both paths — no new config key. A box beefy enough to hold
// N model contexts lifts LOCAL_LLM_MAX_CONCURRENCY and gets N agent slots
// along with N in-flight API calls.
limit = LOCAL_LLM_MAX_CONCURRENCY,
isAvailable = () => true,
resolveFallback = () => null,
} = {}) {
const byId = new Map();
for (const provider of providers) {
if (provider?.id) byId.set(provider.id, provider);
}
if (activeProvider?.id && !byId.has(activeProvider.id)) byId.set(activeProvider.id, activeProvider);
const endpointById = (id) => localEndpointOfProvider(byId.get(id));
// `getFallbackProvider` indexes its providers arg BY ID, so hand it a map —
// not the array (mirrors the same note in promptRunner.js / agentProviderResolution.js).
const providersMap = Object.fromEntries(byId);
// The provider a queued task would be resolved onto, before the availability
// check below. An unknown pin falls through to the active provider, exactly
// as `resolveAgentProviderAndModel` does.
const providerForTask = (task) => {
const pinnedId = task?.metadata?.provider;
return (pinnedId && byId.get(pinnedId)) || activeProvider;
};
return {
limit,
endpointForAgent: (agent) => {
// A PRESENT stamp is authoritative in BOTH directions: it is what this
// agent's inference actually landed on, so a REMOTE stamp means "not on a
// local endpoint" — never "go re-resolve the provider". Falling through
// there would reintroduce the mid-run edit the stamp exists to prevent: a
// cloud agent would start counting against whatever endpoint its provider
// record now names, saturating a GPU that has no agents on it at all.
// Only an ABSENT stamp (a pre-#4834 record) falls back to the id lookup.
const stamped = agent?.metadata?.providerEndpoint;
if (stamped != null) return localSlotKey(stamped);
const providerId = agent?.metadata?.providerId || agent?.providerId;
return providerId ? endpointById(providerId) : null;
},
resolveLocalEndpoint: (task) => {
const primary = providerForTask(task);
if (!primary?.id) return null;
// An unavailable provider is NOT where this task lands — spawn swaps it
// for a fallback, which may be cloud (gating would starve the task behind
// a GPU it never touches) or may be another provider on the SAME local
// server (ignoring it would drop the cap when the endpoint is unhealthy).
// Follow the real resolver rather than guessing either way.
const effective = isAvailable(primary.id)
? primary
: resolveFallback(primary.id, providersMap, task);
return localEndpointOfProvider(effective);
},
};
}
/**
* Fetch the provider snapshot and build the lookups above.
*
* Never throws: `listProviders` already swallows a failed read into `[]`, and
* an uninitialized toolkit yields a null active provider — which resolves every
* task to null (ungated) rather than stalling the queue.
*/
// Deliberately NOT memoized. One dispatch builds this 2-3 times, but the
// toolkit already caches the underlying provider read, so a local snapshot would
// only save a Map and an object allocation — while adding a staleness window the
// toolkit does not have: it invalidates on a provider edit, and nothing here can
// observe that, so an edited endpoint would keep gating on its old value.
export async function buildLocalEndpointSlotContext() {
const providers = await listProviders();
const activeProvider = await getActiveProvider().catch(() => null);
return createLocalEndpointSlotContext({
providers,
activeProvider,
isAvailable: isProviderAvailable,
// The REAL resolver spawn uses, so the prediction can't drift from it.
resolveFallback: (primaryId, providersMap, task) => getFallbackProvider(
primaryId,
providersMap,
task?.metadata?.fallbackProvider ?? null,
task?.metadata?.fallbackModel ?? null,
// Same caller mode policy `resolveAgentProviderAndModel` sends, or this
// prediction would follow a swap onto a route spawn will refuse.
{ allowedModes: allowedModesFor('agent-harness') }
)?.provider ?? null,
});
}
// ── In-flight spawn reservations ───────────────────────────────────────────
// A dispatched task is invisible to the running-agent tally until
// its agent record reaches `running` — a window several awaits wide (provider
// resolution, prompt build, worktree setup, PTY spawn). Two `task:ready`
// dispatches landing inside that window would both read the same snapshot and
// both pass the cap, which is the exact over-dispatch #4834 exists to stop.
//
// So the chokepoint reserves the slot up front and releases it once the spawn
// settles — by which point the agent record carries the load. This is a simple
// in-process re-entrancy guard over one server's own dispatch loop, not a
// defense against competing actors (see the Security Model in AGENTS.md).
// Mirrors `spawningJobIds` in cosJobScheduler.js, which bridges the same gap.
//
// Each reservation carries its task id, because `registerAgent` flips the record
// to `running` well BEFORE `spawnAgentForTask` returns: for the config + PTY
// launch window one agent would otherwise be counted twice (running tally AND
// reservation) and under-admit at limits above 1. Counting only reservations
// whose task has no running agent yet removes the overlap exactly, without
// releasing early and reopening the window the reservation exists to close.
const pendingSpawnsByEndpoint = new Map(); // endpoint -> Map<token, taskId|null>
let reservationSeq = 0;
const NOOP_RELEASE = () => {};
/**
* Reserve an in-flight spawn slot on `endpoint` for `taskId`. Release is
* idempotent because the spawner calls it from a `finally` that a throw can
* reach twice — a second decrement would free a slot still occupied.
*/
export function reserveLocalEndpointSpawn(endpoint, taskId = null) {
if (!endpoint) return NOOP_RELEASE;
const token = ++reservationSeq;
let held = pendingSpawnsByEndpoint.get(endpoint);
if (!held) {
held = new Map();
pendingSpawnsByEndpoint.set(endpoint, held);
}
held.set(token, taskId);
let released = false;
return () => {
if (released) return;
released = true;
const current = pendingSpawnsByEndpoint.get(endpoint);
if (!current) return;
current.delete(token);
if (current.size === 0) pendingSpawnsByEndpoint.delete(endpoint);
};
}
/** Spawns dispatched at `endpoint` that have not yet reached `running`. */
export function pendingLocalEndpointSpawns(endpoint, { excludeTaskIds = null } = {}) {
const held = endpoint ? pendingSpawnsByEndpoint.get(endpoint) : null;
if (!held) return 0;
if (!excludeTaskIds?.size) return held.size;
let count = 0;
for (const taskId of held.values()) {
if (!taskId || !excludeTaskIds.has(taskId)) count++;
}
return count;
}
/** Test hook — drop every outstanding reservation. */
export function __resetLocalEndpointSpawnReservations() {
pendingSpawnsByEndpoint.clear();
}
/**
* Claim a local-endpoint spawn slot for `task`, or report why it must wait.
*
* This is the AUTHORITATIVE cap. The scheduler-side check in `dequeueNextTask`
* only avoids emitting a `task:ready` that would be held anyway; six other
* emitters (evaluateTasks, forceSpawnTask, cosJobScheduler, the Creative
* Director bridge, …) reach the spawner without passing through it, so the gate
* has to live where all of them funnel — subAgentSpawner's `task:ready`
* listener.
*
* `{ ok: true }` with a no-op release when the task has no local endpoint.
* Callers MUST invoke `release()` in a `finally` once the spawn settles.
*
* @param {object} task
* @param {object} agents - `state.agents`, the running-agent map
* @returns {Promise<{ ok: true, release: () => void } | { ok: false, reason: string }>}
*/
export async function acquireLocalEndpointSpawnSlot(task, agents) {
const slots = await buildLocalEndpointSlotContext();
const endpoint = slots.resolveLocalEndpoint(task);
if (!endpoint) return { ok: true, release: NOOP_RELEASE };
const { atCapacity, inFlight, limit } = readEndpointCapacity(endpoint, agents, slots, { ignoreTaskId: task?.id });
if (atCapacity) {
return { ok: false, reason: `local endpoint ${endpoint} is at capacity (${inFlight}/${limit})` };
}
return { ok: true, release: reserveLocalEndpointSpawn(endpoint, task?.id ?? null) };
}
/**
* Reserve the same local inference capacity for a non-agent provider call.
*
* Persistent-mind turns use PortOS's text-provider interface directly rather
* than spawning a file-writing task agent. They still occupy the same model
* server and must therefore share this accounting instead of inventing a
* parallel GPU counter. Cloud providers return a no-op release.
*/
export async function acquireLocalEndpointProviderSlot(provider, agents, reservationId = null) {
const endpoint = localEndpointOfProvider(provider);
if (!endpoint) return { ok: true, release: NOOP_RELEASE };
const slots = await buildLocalEndpointSlotContext();
const { atCapacity, inFlight, limit } = readEndpointCapacity(endpoint, agents, slots);
if (atCapacity) {
return { ok: false, reason: `local endpoint ${endpoint} is at capacity (${inFlight}/${limit})` };
}
return { ok: true, release: reserveLocalEndpointSpawn(endpoint, reservationId) };
}
/**
* Running agents plus in-flight reservations on `endpoint`, against the cap.
*
* `ignoreTaskId` excludes agents belonging to the task being dispatched. A task
* never competes with itself, and counting its own agent would break the two
* paths that deliberately re-dispatch one: `forceSpawnTask` supersedes a stale
* `running` holder (the documented "Run now" recovery for a zombie agent whose
* PTY died), and a retry re-runs the same task. Without this, that recovery is
* unreachable at a limit of 1 — the zombie it is meant to replace fills the slot.
*/
export function readEndpointCapacity(endpoint, agents, slots, { ignoreTaskId = null } = {}) {
const runningTaskIds = new Set();
let running = 0;
for (const agent of Object.values(agents || {})) {
if (agent.status !== 'running') continue;
if (ignoreTaskId && agent.taskId === ignoreTaskId) continue;
if (slots.endpointForAgent(agent) !== endpoint) continue;
running++;
if (agent.taskId) runningTaskIds.add(agent.taskId);
}
const inFlight = running + pendingLocalEndpointSpawns(endpoint, { excludeTaskIds: runningTaskIds });
return { inFlight, limit: slots.limit, atCapacity: inFlight >= slots.limit };
}
/**
* Why a spawn on the ALREADY-RESOLVED `provider` must wait, or null when it may
* proceed.
*
* `forceSpawnTask` (the user's explicit "Run now") returns synchronously while
* the spawn happens later in a `task:ready` listener, so without this it would
* answer `{ success: true }` and toast "Spawning" for a dispatch the chokepoint
* immediately holds — the same lie the provider-resolution pre-check upstream of
* it exists to prevent. Takes the post-fallback provider, so it is strictly more
* accurate than the queued-task prediction in `resolveLocalEndpoint`.
*/
export async function localEndpointCapacityError(provider, agents, taskId = null) {
const endpoint = localEndpointOfProvider(provider);
if (!endpoint) return null;
const slots = await buildLocalEndpointSlotContext();
const { atCapacity, inFlight, limit } = readEndpointCapacity(endpoint, agents, slots, { ignoreTaskId: taskId });
if (!atCapacity) return null;
return `Local inference endpoint ${endpoint} is at capacity (${inFlight}/${limit}) — wait for a running agent to finish`;
}