Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: prevent managed daemon idle and memory pressure eviction during active requests

Stephen Hilderbrandcd63c314e0c1
diff --git a/server/lib/README.md b/server/lib/README.md
index 2e56bd79f1..b4f9013b91 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -356,7 +356,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `localReviewLifecycle.js` | `localReviewLifecycle` — durable pre-PR phase completion bound to the branch, origin, base and exact PR delivery; preserves its original reviewed head through later pushes. |
| `agentExecutionProfiles.js` | Named agent execution postures shared by lifecycle dispatch, CLI/TUI spawners, and child-environment filtering. A stage declares a PROFILE (`PUBLIC_REVIEW_EXECUTION_PROFILE`, `PUBLIC_REVIEW_GATE_EXECUTION_PROFILE`, `PUBLIC_REVIEW_ACTIONS_EXECUTION_PROFILE`); `publicReviewPostureForProfile` maps it to the enforceable POSTURE a provider must carry a maintained recipe for (`no-tool` or `sandboxed-actions`), which is how a pipeline stage stays vendor-agnostic. `isPublicReviewRestrictedProfile` covers both postures; `isPublicReviewNoToolProfile` / `isPublicReviewActionsProfile` separate them for the callers whose rule differs by posture (credential-bootstrap wrapping, child-env allowlist). |
| `agentCompletionMode.js` | The ONE completion-contract decision behind every agent prompt: `resolveCompletionMode(flags)` returns a single `COMPLETION_MODES` key (`tool-free`, `sentinel-payload`, `action-output`, `external-repository`, `discard-worktree`, `claim-flow`, `read-only`, `review-loop-follow-up`, `release-flow`, `audit-flow`, `reconcile-flow`, `checkout-recovery`, `tui-slashdo-free`, `tui`, `portos-merges`, `worktree-no-push`, `commit-and-push`) from an ordered rule list, plus `portosMergesBranchOnExit`. Replaces the five independent flag ladders in `agentPromptBuilder.js` that each re-derived it in a different order — two had drifted far enough to render a prompt telling a tool-free or read-only task both to commit and that it had no git at all (#6616). Call sites branch on the key and render per mode; they never re-test the flags. |
-| `localLlmConcurrency.js` | `LOCAL_LLM_MAX_CONCURRENCY` — the per-endpoint local-LLM concurrency cap (env read at load); a leaf so `cosLocalEndpointSlots.js` avoids importing `promptRunner.js`, which re-exports it. |
+| `localLlmConcurrency.js` | `LOCAL_LLM_MAX_CONCURRENCY` plus the per-endpoint local-LLM slot gate (`acquireLocalSlot`, `releaseLocalSlot`), in-flight activity predicate (`isLocalEndpointBusy`, matching exact endpoint or loopback port), and settle observer (`onLocalEndpointSettled`) — a dependency-free leaf so `cosLocalEndpointSlots.js` and `managedDaemon.js` avoid importing `promptRunner.js`. |
| `localEndpoint.js` | Dependency-light local-instance URL predicates (`isLocalInstanceHost`, `isLocalInstanceEndpoint`, scheme-optional `isLoopbackEndpoint` — the one "is this a local daemon" answer, `localEndpointPort`) shared by provider safety policy and local-runtime classification without importing backend configuration or daemon managers. |
| `credentialBootstrap.js` | resolveBootstrapEnv(provider, { safetyProfile }) runs an optional `envCommand` argv for ordinary provider runs or a no-tool review, parses KEY=value output without evaluation, and caches credentials in memory for one minute — the alternative for a bootstrap that PRINTS credentials rather than exec'ing the harness; a record that only names the wrap form is credentialed by the wrap and gets `{}`. Generic support for a CLI/TUI provider whose auth is provisioned by an external CLI at spawn time (e.g. a short-lived token for a proxy) rather than a static `apiKey` PortOS stores — a `provider.credentialBootstrap: { setupCommand?, command, args?, harnessId?, argsSeparator? }` record. `hasCredentialBootstrap(provider)` and `applyCredentialBootstrap(provider, command, args)` (pure) return the command+args PortOS should actually spawn: `<command> <args...> <harnessId ?? command> [<argsSeparator>] <args...>` — the bootstrap CLI in front of the harness invocation, naming the harness as `harnessId` when the bootstrap CLI uses its own identifier rather than the binary name, and inserting `argsSeparator` (e.g. `--`) before the harness's own args when the bootstrap CLI needs its flags kept apart from the wrapped program's. `resolveCliSpawn(provider, command, args, childEnv)` composes that with Windows shim resolution (`bufferedSpawn.js`) for the three headless spawn sites that repeated the same sequence (`runner.js`, `cliProviderRun.js`, `visionCli.js`) — the argv analog of why `cliChildEnv.js`'s `buildCliChildEnv` exists. Every call site keeps resolving `provider.command`/`args` unchanged for vendor detection, prompt-delivery convention, and the TUI's own ready-text heuristics, and only asks this module for the pair handed to the literal `spawn`/`pty.spawn` call — plus anything that observes the launching shell or PATH (missing-binary detection, readiness probes), which must read the SPAWNED pair. The `sandboxed-actions` posture is never wrapped — it executes contributor-supplied code inside a sandbox spelled entirely in the recipe's argv, so nothing may sit in front of the harness and rewrite it; the `no-tool` postures (the tool-free reviewer among them) ARE wrapped, because spawned bare they carry no credential at all (#7720). The split lives in the helper, keyed on `safetyProfile`. The pure argv composition lives in `aiToolkit/internal/credentialBootstrap.js` (`composeBootstrapSpawn` / `hasCredentialBootstrap`, dependency-free), re-exported through here so the vendored toolkit's own catalog probes compose the same shape without importing out. Every provider spawn site goes through here — a bare-harness spawn runs with no credential and falls through to ambient vendor auth. Both also return `wrapped`, because a wrapped spawn's direct child is the SUPERVISING wrapper rather than the harness: `needsProcessGroup(wrapped)` says whether to spawn `detached` and kill with `{ processGroup: true }` (POSIX only — Windows' `taskkill /T` is already tree-wide), and `processGroupKillable(child, processGroup)` adapts a child for the aiToolkit runner's external-run registry, whose vendored `killProcessTree` has no group option. Without them a stop/timeout/cancel signals the wrapper alone and leaves the harness running. `trackDetachedGroup(child, processGroup)` / `signalDetachedGroups(signal, logFailure)` are the other side of that trade: detaching also moves a child OUT of the server's own process group, so the shutdown handler sweeps this registry to keep a signal aimed at that group (Ctrl-C at `npm start`) from orphaning one. |
| `cliProviderArgs.js` | Per-CLI argv conventions (`buildCliArgs`) for stdin prompt delivery — dependency-light extraction from runner.js so out-of-process callers (autofixer) can import it. |
diff --git a/server/lib/localLlmConcurrency.js b/server/lib/localLlmConcurrency.js
index 7580992ae9..da706bd49f 100644
--- a/server/lib/localLlmConcurrency.js
+++ b/server/lib/localLlmConcurrency.js
@@ -1,3 +1,81 @@
-// Per-endpoint cap on concurrent local-LLM work. Own leaf so CoS slot
-// accounting can read it without loading the prompt runner's import graph.
+// Per-endpoint cap and in-flight slot tracking for concurrent local-LLM work.
+// Own dependency-free leaf so CoS slot accounting and the managed-daemon idle
+// reaper can read it without loading the prompt runner's import graph.
export const LOCAL_LLM_MAX_CONCURRENCY = Math.max(1, Number(process.env.LOCAL_LLM_MAX_CONCURRENCY) || 1);
+
+const localEndpointGates = new Map(); // endpoint -> { active: number, queue: (()=>void)[] }
+const settledListeners = new Set();
+
+function endpointPortKey(endpoint) {
+ if (typeof endpoint !== 'string' || !endpoint.trim()) return null;
+ const trimmed = endpoint.trim();
+ const candidate = /^[a-z][a-z0-9+.-]*:\/\//i.test(trimmed) ? trimmed : `http://${trimmed}`;
+ if (!URL.canParse(candidate)) return null;
+ const url = new URL(candidate);
+ return url.port || (url.protocol === 'https:' ? '443' : '80');
+}
+
+function notifyEndpointActivity(endpoint) {
+ for (const listener of settledListeners) {
+ try {
+ listener(endpoint);
+ } catch {
+ // Best-effort observer notification.
+ }
+ }
+}
+
+export function acquireLocalSlot(endpoint) {
+ let gate = localEndpointGates.get(endpoint);
+ if (!gate) {
+ gate = { active: 0, queue: [] };
+ localEndpointGates.set(endpoint, gate);
+ }
+ notifyEndpointActivity(endpoint);
+ if (gate.active < LOCAL_LLM_MAX_CONCURRENCY) {
+ gate.active += 1;
+ return Promise.resolve();
+ }
+ return new Promise((resolve) => {
+ gate.queue.push(() => {
+ notifyEndpointActivity(endpoint);
+ resolve();
+ });
+ });
+}
+
+export function releaseLocalSlot(endpoint) {
+ const gate = localEndpointGates.get(endpoint);
+ if (!gate) return;
+ const next = gate.queue.shift();
+ if (next) next(); // hand the slot straight to the next waiter (active unchanged)
+ else gate.active = Math.max(0, gate.active - 1);
+ notifyEndpointActivity(endpoint);
+}
+
+/**
+ * True when any local-LLM run is currently in flight (or queued) against
+ * `endpoint` or a loopback alias sharing its port.
+ */
+export function isLocalEndpointBusy(endpoint) {
+ if (typeof endpoint !== 'string' || !endpoint.trim()) return false;
+ const targetPort = endpointPortKey(endpoint);
+ for (const [key, gate] of localEndpointGates) {
+ if (!gate || (gate.active <= 0 && gate.queue.length === 0)) continue;
+ if (key === endpoint) return true;
+ if (targetPort && endpointPortKey(key) === targetPort) return true;
+ }
+ return false;
+}
+
+/**
+ * Register a listener invoked when a local endpoint slot is acquired or
+ * released, so a managed daemon's idle clock refreshes both on dispatch and
+ * when a long-running inference finishes.
+ */
+export function onLocalEndpointSettled(listener) {
+ if (typeof listener !== 'function') return () => {};
+ settledListeners.add(listener);
+ return () => settledListeners.delete(listener);
+}
+
diff --git a/server/lib/managedDaemon.js b/server/lib/managedDaemon.js
index d72700bcf6..728b5c2964 100644
--- a/server/lib/managedDaemon.js
+++ b/server/lib/managedDaemon.js
@@ -1,5 +1,6 @@
import { getMemoryStats } from './memoryStats.js';
import { localEndpointPort } from './localEndpoint.js';
+import { isLocalEndpointBusy, onLocalEndpointSettled } from './localLlmConcurrency.js';
import { createSingleFlight } from './singleFlight.js';
/**
@@ -288,12 +289,24 @@ export function createOnDemandDaemon({
return Boolean(section?.keepLoaded ?? section?.pinned);
};
+ let inFlightStarts = 0;
+ const currentEndpoint = () => endpointFor(getConfig());
+ const isBusy = () => inFlightStarts > 0 || isLocalEndpointBusy(currentEndpoint());
+
+ onLocalEndpointSettled((endpoint) => {
+ const port = localEndpointPort(currentEndpoint());
+ if (port && localEndpointPort(endpoint) === port) {
+ markDaemonUsed(appName);
+ }
+ });
+
const registerIdle = () => {
registerIdleDaemon({
name: appName,
getIdleMs: async () => idleWindowMs(await configuredIdleMinutes()),
isPinned: async () => configuredKeepLoaded(),
isRunning: async () => Boolean((await getStatusStrict())?.status === 'online'),
+ isBusy,
stop: () => stop(),
});
};
@@ -360,11 +373,19 @@ export function createOnDemandDaemon({
*/
const starts = createSingleFlight({ exclusive: true });
- const ensureRunning = async () => {
+ const trackStart = async (key, fn) => {
+ inFlightStarts += 1;
markUsed();
- return starts.run('wake', wake);
+ try {
+ return await starts.run(key, fn);
+ } finally {
+ inFlightStarts = Math.max(0, inFlightStarts - 1);
+ markUsed();
+ }
};
+ const ensureRunning = () => trackStart('wake', wake);
+
/**
* Run an explicit start/stop sequence (a user-clicked Start, a tuning
* relaunch) in the daemon's exclusive start slot. Each call gets its own key,
@@ -377,7 +398,7 @@ export function createOnDemandDaemon({
* @param {() => Promise<T>} fn
* @returns {Promise<T>}
*/
- const runStart = (fn) => starts.run(Symbol('start'), fn);
+ const runStart = (fn) => trackStart(Symbol('start'), fn);
const wake = async () => {
const pm2Status = await getStatusStrict();
@@ -536,18 +557,21 @@ export function idleWindowMs(minutes) {
* getIdleMs: () => Promise<number|null>|number|null,
* isPinned?: () => Promise<boolean>|boolean,
* isRunning?: () => Promise<boolean>|boolean,
+ * isBusy?: () => Promise<boolean>|boolean,
* stop: () => Promise<unknown>
* }} daemon
* `getIdleMs` resolves the CURRENT configured window on every sweep; `null`/`0` = never stop.
* `isPinned` returns true if user pinned the server ("keep loaded") — pinned servers are never stopped.
* `isRunning` checks if the daemon process is online.
+ * `isBusy` returns true while a wake/start or inference request is in flight — busy servers are never stopped.
*/
-export function registerIdleDaemon({ name, getIdleMs, isPinned, isRunning, stop }) {
+export function registerIdleDaemon({ name, getIdleMs, isPinned, isRunning, isBusy, stop }) {
const existing = idleDaemons.get(name);
idleDaemons.set(name, {
getIdleMs,
isPinned: typeof isPinned === 'function' ? isPinned : () => Boolean(isPinned),
isRunning: typeof isRunning === 'function' ? isRunning : null,
+ isBusy: typeof isBusy === 'function' ? isBusy : () => Boolean(isBusy),
stop,
lastUsedAt: existing?.lastUsedAt ?? Date.now(),
releaseReason: existing?.releaseReason ?? null,
@@ -685,6 +709,7 @@ export function evaluateMemoryPressurePolicy({
const eligible = list.filter((d) => (
d
&& d.running !== false
+ && !d.busy
&& !d.pinned
&& !d.keepLoaded
));
@@ -723,12 +748,15 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
// 1. Normal idle timeout pass
for (const [name, entry] of idleDaemons) {
- // Fail-safe to pinned on a transient read error, matching the pressure-aware
- // pass below — assuming "not pinned" here would risk stopping a keepLoaded
- // daemon on a flaky settings read.
+ // Fail-safe to pinned/busy on a transient read error, matching the pressure-aware
+ // pass below — assuming "not pinned" or "not busy" here would risk stopping a
+ // keepLoaded or actively serving daemon on a flaky read.
const isPinned = await Promise.resolve(entry.isPinned?.()).catch(() => true);
if (isPinned) continue; // Pinned servers are exempt
+ const isBusy = await Promise.resolve(entry.isBusy?.()).catch(() => true);
+ if (isBusy) continue; // Active wake/start or in-flight request is exempt
+
// Already stopped (by the pressure-aware pass, or externally) — skip so this
// pass doesn't overwrite its release reason and doesn't preempt the
// pressure-aware pass below via the early return once its idle window re-elapses.
@@ -778,6 +806,7 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
const daemonList = [];
for (const [name, entry] of idleDaemons) {
const isPinned = await Promise.resolve(entry.isPinned?.()).catch(() => true);
+ const isBusy = await Promise.resolve(entry.isBusy?.()).catch(() => true);
const isRunning = entry.isRunning
? await Promise.resolve(entry.isRunning()).catch(() => false)
: true;
@@ -786,6 +815,7 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
entry,
lastUsedAt: entry.lastUsedAt,
pinned: isPinned,
+ busy: isBusy,
running: isRunning,
});
}
diff --git a/server/lib/managedDaemon.pressure.test.js b/server/lib/managedDaemon.pressure.test.js
index edcd69a759..6f89983570 100644
--- a/server/lib/managedDaemon.pressure.test.js
+++ b/server/lib/managedDaemon.pressure.test.js
@@ -12,7 +12,9 @@ import {
DEFAULT_PRESSURE_CALM_DOWN_MS,
DEFAULT_SUSTAINED_PRESSURE_MS,
createDaemonWatcher,
+ createOnDemandDaemon,
} from './managedDaemon.js';
+import { acquireLocalSlot, releaseLocalSlot } from './localLlmConcurrency.js';
const MINUTE = 60_000;
const GB = 1024 * 1024 * 1024;
@@ -209,6 +211,35 @@ describe('managedDaemon memory pressure policy', () => {
expect(result.target.name).toBe('slotstream');
});
+ it('never releases a daemon that is currently busy', () => {
+ const daemons = [
+ { name: 'mtplx', lastUsedAt: now - 60_000, busy: true },
+ ];
+ const result = evaluateMemoryPressurePolicy({
+ daemons,
+ memoryStats: { total: 64 * GB, used: 62 * GB, free: 2 * GB },
+ history: sustainedHistory,
+ now,
+ });
+ expect(result.shouldRelease).toBe(false);
+ expect(result.reason).toBe('no eligible daemons to release');
+ });
+
+ it('skips busy daemon and releases the next least recently used idle daemon', () => {
+ const daemons = [
+ { name: 'mtplx', lastUsedAt: now - 60_000, busy: true },
+ { name: 'slotstream', lastUsedAt: now - 20_000, busy: false },
+ ];
+ const result = evaluateMemoryPressurePolicy({
+ daemons,
+ memoryStats: { total: 64 * GB, used: 62 * GB, free: 2 * GB },
+ history: sustainedHistory,
+ now,
+ });
+ expect(result.shouldRelease).toBe(true);
+ expect(result.target.name).toBe('slotstream');
+ });
+
it('honors sustainedDurationMs: 0 for instant evaluation', () => {
const result = evaluateMemoryPressurePolicy({
daemons: [{ name: 'mtplx', lastUsedAt: now - 1000 }],
@@ -406,5 +437,98 @@ describe('managedDaemon memory pressure policy', () => {
const onlineStatus = await watcher.getStatusBase({ installed: true });
expect(onlineStatus.releaseReason).toBeNull();
});
+
+ it('never stops a busy daemon during idle-timeout or memory-pressure passes, and fail-safes on isBusy errors', async () => {
+ const stopBusy = vi.fn().mockResolvedValue(undefined);
+ const stopFlaky = vi.fn().mockResolvedValue(undefined);
+ registerIdleDaemon({
+ name: 'daemon-busy',
+ getIdleMs: () => 5 * MINUTE,
+ isBusy: () => true,
+ stop: stopBusy,
+ });
+ registerIdleDaemon({
+ name: 'daemon-flaky-busy',
+ getIdleMs: () => 5 * MINUTE,
+ isBusy: () => Promise.reject(new Error('transient busy check error')),
+ stop: stopFlaky,
+ });
+
+ const now = Date.now() + 10 * MINUTE;
+ const stopped = await reapIdleDaemons(now, {
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now - 40_000, free: 2 * GB }, { at: now, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+
+ expect(stopped).toEqual([]);
+ expect(stopBusy).not.toHaveBeenCalled();
+ expect(stopFlaky).not.toHaveBeenCalled();
+ });
+
+ it('protects createOnDemandDaemon from pressure eviction while a wake or local-LLM slot is active on its port', async () => {
+ const stop = vi.fn().mockResolvedValue(undefined);
+ let resolveWakeStart;
+ const startPromise = new Promise((r) => { resolveWakeStart = r; });
+ let pm2Status = { status: 'stopped' };
+
+ const onDemand = createOnDemandDaemon({
+ appName: 'portos-mtplx',
+ label: 'MTPLX server',
+ emoji: '⚡',
+ readSection: async () => ({ idleMinutes: 30, keepLoaded: false, launch: { model: 'test-model', port: 8092 } }),
+ endpointFor: (cfg) => `http://127.0.0.1:${cfg?.port || 8092}/v1`,
+ probe: async () => false,
+ start: () => startPromise,
+ stop,
+ getStatusStrict: async () => pm2Status,
+ resolveLaunch: (current, saved) => current || { model: saved.model, port: saved.port || 8092 },
+ getConfig: () => ({ model: 'test-model', port: 8092 }),
+ sleep: async () => {},
+ getRelaunchReadyTimeoutMs: () => 5000,
+ getRelaunchPollMs: () => 10,
+ });
+ onDemand.registerIdle();
+
+ // 1. In-flight wake/load: daemon is online in PM2 while weights load, but ensureRunning has not finished.
+ const wakeCall = onDemand.ensureRunning();
+ pm2Status = { status: 'online' };
+ const now1 = Date.now();
+ const stoppedDuringWake = await reapIdleDaemons(now1, {
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now1 - 40_000, free: 2 * GB }, { at: now1, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+ expect(stoppedDuringWake).toEqual([]);
+ expect(stop).not.toHaveBeenCalled();
+
+ resolveWakeStart({ online: true, endpoint: 'http://127.0.0.1:8092/v1' });
+ await expect(wakeCall).resolves.toEqual({ ready: true, reason: null });
+
+ // 2. In-flight inference slot on the daemon's port (even via localhost alias):
+ await acquireLocalSlot('http://localhost:8092/v1');
+ try {
+ const now2 = Date.now() + 1000;
+ const stoppedDuringInference = await reapIdleDaemons(now2, {
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now1 - 40_000, free: 2 * GB }, { at: now2, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+ expect(stoppedDuringInference).toEqual([]);
+ expect(stop).not.toHaveBeenCalled();
+ } finally {
+ releaseLocalSlot('http://localhost:8092/v1');
+ }
+
+ // 3. Once the slot is released, sustained memory pressure can evict the now-idle daemon.
+ const now3 = Date.now() + 2000;
+ const stoppedAfterInference = await reapIdleDaemons(now3, {
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now1 - 40_000, free: 2 * GB }, { at: now3, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+ expect(stoppedAfterInference).toEqual(['portos-mtplx']);
+ expect(stop).toHaveBeenCalledTimes(1);
+ });
});
});
diff --git a/server/services/promptRunner.js b/server/services/promptRunner.js
index 47ed3b8622..3e2a7f37b2 100644
--- a/server/services/promptRunner.js
+++ b/server/services/promptRunner.js
@@ -28,7 +28,7 @@ import { maintenance } from '../lib/maintenanceAdmission.js';
* stay honest about what the runner actually executed.
*/
-import { LOCAL_LLM_MAX_CONCURRENCY } from '../lib/localLlmConcurrency.js';
+import { LOCAL_LLM_MAX_CONCURRENCY, acquireLocalSlot, releaseLocalSlot } from '../lib/localLlmConcurrency.js';
import { isVisionCapableCodexTuiProvider } from '../lib/codex.js';
import { createRun, executeApiRun, executeCliRun, extractBakedModel, hasModelFlag, stopRun, patchRunMetadata, finalizeRunRecord } from './runner.js';
import { getActiveProvider, getProviderById, getAllProviders } from './providers.js';
@@ -148,25 +148,6 @@ export function buildRequestCapabilities({ prompt, screenshots, outputReserveTok
export const isLocalEndpoint = isLoopbackEndpoint;
export { LOCAL_LLM_MAX_CONCURRENCY };
-const localEndpointGates = new Map(); // endpoint -> { active: number, queue: (()=>void)[] }
-
-function acquireLocalSlot(endpoint) {
- let gate = localEndpointGates.get(endpoint);
- if (!gate) { gate = { active: 0, queue: [] }; localEndpointGates.set(endpoint, gate); }
- if (gate.active < LOCAL_LLM_MAX_CONCURRENCY) {
- gate.active += 1;
- return Promise.resolve();
- }
- return new Promise((resolve) => { gate.queue.push(resolve); });
-}
-
-function releaseLocalSlot(endpoint) {
- const gate = localEndpointGates.get(endpoint);
- if (!gate) return;
- const next = gate.queue.shift();
- if (next) next(); // hand the slot straight to the next waiter (active unchanged)
- else gate.active = Math.max(0, gate.active - 1);
-}
// Run `fn` under the local-endpoint gate when `provider` is a local API or TUI
// backend; otherwise run it immediately (cloud / non-local CLI/TUI are