Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: keep loading and custom-port daemons out of the idle reaper

[._.]/ Adam Eivy003e4cdd16c9
diff --git a/server/lib/README.md b/server/lib/README.md
index b4f9013b91..0701fdf03e 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -356,7 +356,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `localReviewLifecycle.js` | `localReviewLifecycle` — durable pre-PR phase completion bound to the branch, origin, base and exact PR delivery; preserves its original reviewed head through later pushes. |
| `agentExecutionProfiles.js` | Named agent execution postures shared by lifecycle dispatch, CLI/TUI spawners, and child-environment filtering. A stage declares a PROFILE (`PUBLIC_REVIEW_EXECUTION_PROFILE`, `PUBLIC_REVIEW_GATE_EXECUTION_PROFILE`, `PUBLIC_REVIEW_ACTIONS_EXECUTION_PROFILE`); `publicReviewPostureForProfile` maps it to the enforceable POSTURE a provider must carry a maintained recipe for (`no-tool` or `sandboxed-actions`), which is how a pipeline stage stays vendor-agnostic. `isPublicReviewRestrictedProfile` covers both postures; `isPublicReviewNoToolProfile` / `isPublicReviewActionsProfile` separate them for the callers whose rule differs by posture (credential-bootstrap wrapping, child-env allowlist). |
| `agentCompletionMode.js` | The ONE completion-contract decision behind every agent prompt: `resolveCompletionMode(flags)` returns a single `COMPLETION_MODES` key (`tool-free`, `sentinel-payload`, `action-output`, `external-repository`, `discard-worktree`, `claim-flow`, `read-only`, `review-loop-follow-up`, `release-flow`, `audit-flow`, `reconcile-flow`, `checkout-recovery`, `tui-slashdo-free`, `tui`, `portos-merges`, `worktree-no-push`, `commit-and-push`) from an ordered rule list, plus `portosMergesBranchOnExit`. Replaces the five independent flag ladders in `agentPromptBuilder.js` that each re-derived it in a different order — two had drifted far enough to render a prompt telling a tool-free or read-only task both to commit and that it had no git at all (#6616). Call sites branch on the key and render per mode; they never re-test the flags. |
-| `localLlmConcurrency.js` | `LOCAL_LLM_MAX_CONCURRENCY` plus the per-endpoint local-LLM slot gate (`acquireLocalSlot`, `releaseLocalSlot`), in-flight activity predicate (`isLocalEndpointBusy`, matching exact endpoint or loopback port), and settle observer (`onLocalEndpointSettled`) — a dependency-free leaf so `cosLocalEndpointSlots.js` and `managedDaemon.js` avoid importing `promptRunner.js`. |
+| `localLlmConcurrency.js` | `LOCAL_LLM_MAX_CONCURRENCY` plus the per-endpoint local-LLM slot gate (`acquireLocalSlot`, `releaseLocalSlot`), in-flight activity predicate (`isLocalEndpointBusy`), scheme-optional port match (`localEndpointsSharePort`), and settle observer (`onLocalEndpointSettled`) — a dependency-free leaf so `cosLocalEndpointSlots.js` and `managedDaemon.js` avoid importing `promptRunner.js`. |
| `localEndpoint.js` | Dependency-light local-instance URL predicates (`isLocalInstanceHost`, `isLocalInstanceEndpoint`, scheme-optional `isLoopbackEndpoint` — the one "is this a local daemon" answer, `localEndpointPort`) shared by provider safety policy and local-runtime classification without importing backend configuration or daemon managers. |
| `credentialBootstrap.js` | resolveBootstrapEnv(provider, { safetyProfile }) runs an optional `envCommand` argv for ordinary provider runs or a no-tool review, parses KEY=value output without evaluation, and caches credentials in memory for one minute — the alternative for a bootstrap that PRINTS credentials rather than exec'ing the harness; a record that only names the wrap form is credentialed by the wrap and gets `{}`. Generic support for a CLI/TUI provider whose auth is provisioned by an external CLI at spawn time (e.g. a short-lived token for a proxy) rather than a static `apiKey` PortOS stores — a `provider.credentialBootstrap: { setupCommand?, command, args?, harnessId?, argsSeparator? }` record. `hasCredentialBootstrap(provider)` and `applyCredentialBootstrap(provider, command, args)` (pure) return the command+args PortOS should actually spawn: `<command> <args...> <harnessId ?? command> [<argsSeparator>] <args...>` — the bootstrap CLI in front of the harness invocation, naming the harness as `harnessId` when the bootstrap CLI uses its own identifier rather than the binary name, and inserting `argsSeparator` (e.g. `--`) before the harness's own args when the bootstrap CLI needs its flags kept apart from the wrapped program's. `resolveCliSpawn(provider, command, args, childEnv)` composes that with Windows shim resolution (`bufferedSpawn.js`) for the three headless spawn sites that repeated the same sequence (`runner.js`, `cliProviderRun.js`, `visionCli.js`) — the argv analog of why `cliChildEnv.js`'s `buildCliChildEnv` exists. Every call site keeps resolving `provider.command`/`args` unchanged for vendor detection, prompt-delivery convention, and the TUI's own ready-text heuristics, and only asks this module for the pair handed to the literal `spawn`/`pty.spawn` call — plus anything that observes the launching shell or PATH (missing-binary detection, readiness probes), which must read the SPAWNED pair. The `sandboxed-actions` posture is never wrapped — it executes contributor-supplied code inside a sandbox spelled entirely in the recipe's argv, so nothing may sit in front of the harness and rewrite it; the `no-tool` postures (the tool-free reviewer among them) ARE wrapped, because spawned bare they carry no credential at all (#7720). The split lives in the helper, keyed on `safetyProfile`. The pure argv composition lives in `aiToolkit/internal/credentialBootstrap.js` (`composeBootstrapSpawn` / `hasCredentialBootstrap`, dependency-free), re-exported through here so the vendored toolkit's own catalog probes compose the same shape without importing out. Every provider spawn site goes through here — a bare-harness spawn runs with no credential and falls through to ambient vendor auth. Both also return `wrapped`, because a wrapped spawn's direct child is the SUPERVISING wrapper rather than the harness: `needsProcessGroup(wrapped)` says whether to spawn `detached` and kill with `{ processGroup: true }` (POSIX only — Windows' `taskkill /T` is already tree-wide), and `processGroupKillable(child, processGroup)` adapts a child for the aiToolkit runner's external-run registry, whose vendored `killProcessTree` has no group option. Without them a stop/timeout/cancel signals the wrapper alone and leaves the harness running. `trackDetachedGroup(child, processGroup)` / `signalDetachedGroups(signal, logFailure)` are the other side of that trade: detaching also moves a child OUT of the server's own process group, so the shutdown handler sweeps this registry to keep a signal aimed at that group (Ctrl-C at `npm start`) from orphaning one. |
| `cliProviderArgs.js` | Per-CLI argv conventions (`buildCliArgs`) for stdin prompt delivery — dependency-light extraction from runner.js so out-of-process callers (autofixer) can import it. |
diff --git a/server/lib/localLlmConcurrency.js b/server/lib/localLlmConcurrency.js
index da706bd49f..d4a55c7b55 100644
--- a/server/lib/localLlmConcurrency.js
+++ b/server/lib/localLlmConcurrency.js
@@ -31,9 +31,9 @@ export function acquireLocalSlot(endpoint) {
gate = { active: 0, queue: [] };
localEndpointGates.set(endpoint, gate);
}
- notifyEndpointActivity(endpoint);
if (gate.active < LOCAL_LLM_MAX_CONCURRENCY) {
gate.active += 1;
+ notifyEndpointActivity(endpoint);
return Promise.resolve();
}
return new Promise((resolve) => {
@@ -55,7 +55,8 @@ export function releaseLocalSlot(endpoint) {
/**
* True when any local-LLM run is currently in flight (or queued) against
- * `endpoint` or a loopback alias sharing its port.
+ * `endpoint` or another endpoint string sharing its port. Scheme-less
+ * (`localhost:8010/v1`) and absolute (`http://127.0.0.1:8010/v1`) forms match.
*/
export function isLocalEndpointBusy(endpoint) {
if (typeof endpoint !== 'string' || !endpoint.trim()) return false;
@@ -68,6 +69,17 @@ export function isLocalEndpointBusy(endpoint) {
return false;
}
+/** True when both endpoint strings parse to the same port, scheme optional. */
+export function localEndpointsSharePort(left, right) {
+ const leftPort = endpointPortKey(left);
+ return Boolean(leftPort) && leftPort === endpointPortKey(right);
+}
+
+/** Drop slot gates. Test isolation only — production state is the live process. */
+export function _resetLocalLlmSlotsForTests() {
+ localEndpointGates.clear();
+}
+
/**
* Register a listener invoked when a local endpoint slot is acquired or
* released, so a managed daemon's idle clock refreshes both on dispatch and
diff --git a/server/lib/managedDaemon.js b/server/lib/managedDaemon.js
index 728b5c2964..0b3e6692a0 100644
--- a/server/lib/managedDaemon.js
+++ b/server/lib/managedDaemon.js
@@ -1,6 +1,6 @@
import { getMemoryStats } from './memoryStats.js';
import { localEndpointPort } from './localEndpoint.js';
-import { isLocalEndpointBusy, onLocalEndpointSettled } from './localLlmConcurrency.js';
+import { isLocalEndpointBusy, localEndpointsSharePort, onLocalEndpointSettled } from './localLlmConcurrency.js';
import { createSingleFlight } from './singleFlight.js';
/**
@@ -245,6 +245,7 @@ export function createPm2ExitTail({ appName, execPm2, appendLog, lines = 15, tai
* exitTail?: (status: string) => Promise<string>,
* resolveLaunch: (current: object|null, saved: {model: string|null, port: number|null, raw: object}) => object,
* getConfig?: () => object|null,
+ * parseConfigFromArgs?: (args: string[]|string) => object|null,
* sleep: (ms: number) => Promise<void>,
* getRelaunchReadyTimeoutMs: () => number,
* getRelaunchPollMs: () => number,
@@ -264,6 +265,7 @@ export function createOnDemandDaemon({
exitTail,
resolveLaunch,
getConfig = () => null,
+ parseConfigFromArgs = () => null,
sleep,
getRelaunchReadyTimeoutMs,
getRelaunchPollMs,
@@ -290,26 +292,20 @@ export function createOnDemandDaemon({
};
let inFlightStarts = 0;
- const currentEndpoint = () => endpointFor(getConfig());
- const isBusy = () => inFlightStarts > 0 || isLocalEndpointBusy(currentEndpoint());
+ // Explicit starts return after a short HTTP budget while weights are still
+ // loading. This stays raised until the endpoint answers or PM2 leaves online.
+ let inFlightLoads = 0;
+ // In-memory launch config is null after a PortOS restart until something
+ // recovers argv. The provider slot is the port the process is actually on.
+ let boundEndpoint = null;
- onLocalEndpointSettled((endpoint) => {
- const port = localEndpointPort(currentEndpoint());
- if (port && localEndpointPort(endpoint) === port) {
- markDaemonUsed(appName);
- }
- });
+ const portEndpoint = (config) => (
+ Number.isFinite(Number(config?.port)) ? endpointFor({ port: Number(config.port) }) : null
+ );
+ const busyEndpoint = () => portEndpoint(getConfig()) || boundEndpoint || endpointFor(getConfig());
+ const isBusy = () => inFlightStarts > 0 || inFlightLoads > 0 || isLocalEndpointBusy(busyEndpoint());
- const registerIdle = () => {
- registerIdleDaemon({
- name: appName,
- getIdleMs: async () => idleWindowMs(await configuredIdleMinutes()),
- isPinned: async () => configuredKeepLoaded(),
- isRunning: async () => Boolean((await getStatusStrict())?.status === 'online'),
- isBusy,
- stop: () => stop(),
- });
- };
+ let unsubscribeSettled = () => {};
/** The launch line last saved on the settings card, sanitized. */
const savedLaunchConfig = async () => {
@@ -325,6 +321,51 @@ export function createOnDemandDaemon({
};
};
+ const refreshBoundEndpoint = async (knownStatus) => {
+ const live = portEndpoint(getConfig());
+ if (live) {
+ boundEndpoint = live;
+ return;
+ }
+ const status = knownStatus ?? await Promise.resolve(getStatusStrict()).catch(() => null);
+ if (status?.status === 'online' && status.args) {
+ try {
+ const fromArgs = portEndpoint(parseConfigFromArgs(status.args));
+ if (fromArgs) {
+ boundEndpoint = fromArgs;
+ return;
+ }
+ } catch {
+ // Unparseable argv is not a port. The saved launch is the next source.
+ }
+ }
+ const saved = await savedLaunchConfig().catch(() => null);
+ if (saved?.port != null) boundEndpoint = endpointFor({ port: saved.port });
+ };
+
+ // Resolves once, then stays sync: the reaper's last busy check cannot await.
+ const ensureBoundEndpoint = () => (
+ portEndpoint(getConfig()) || boundEndpoint ? Promise.resolve() : refreshBoundEndpoint()
+ );
+
+ const registerIdle = () => {
+ unsubscribeSettled();
+ settledUnsubscribers.delete(unsubscribeSettled);
+ unsubscribeSettled = onLocalEndpointSettled((endpoint) => {
+ if (localEndpointsSharePort(endpoint, busyEndpoint())) markDaemonUsed(appName);
+ });
+ settledUnsubscribers.add(unsubscribeSettled);
+ registerIdleDaemon({
+ name: appName,
+ getIdleMs: async () => idleWindowMs(await configuredIdleMinutes()),
+ isPinned: async () => configuredKeepLoaded(),
+ isRunning: async () => Boolean((await getStatusStrict())?.status === 'online'),
+ isBusy,
+ warm: ensureBoundEndpoint,
+ stop: () => stop(),
+ });
+ };
+
/**
* Block until the relaunched daemon answers, or until it is proven dead.
* MTPLX's `waitForRelaunchedEndpoint`, generalized: poll PM2 alongside the
@@ -386,6 +427,18 @@ export function createOnDemandDaemon({
const ensureRunning = () => trackStart('wake', wake);
+ const armLoadWatch = (endpoint) => {
+ inFlightLoads += 1;
+ const settle = () => {
+ inFlightLoads = Math.max(0, inFlightLoads - 1);
+ markUsed();
+ };
+ // Two-arm then, not finally: a floating finally re-raises on a promise
+ // nobody awaits. The watch must not hold the caller's start open — launch*
+ // returns after its short budget while a cold checkpoint is still loading.
+ waitForReady(endpoint).then(settle, settle);
+ };
+
/**
* Run an explicit start/stop sequence (a user-clicked Start, a tuning
* relaunch) in the daemon's exclusive start slot. Each call gets its own key,
@@ -398,11 +451,20 @@ export function createOnDemandDaemon({
* @param {() => Promise<T>} fn
* @returns {Promise<T>}
*/
- const runStart = (fn) => trackStart(Symbol('start'), fn);
+ const runStart = (fn) => trackStart(Symbol('start'), async () => {
+ const result = await fn();
+ if (result?.online === false && typeof result.endpoint === 'string') armLoadWatch(result.endpoint);
+ return result;
+ });
const wake = async () => {
const pm2Status = await getStatusStrict();
- if (pm2Status?.status === 'online') return { ready: true, reason: null };
+ if (pm2Status?.status === 'online') {
+ // Reuse the status just read. A second PM2 read here would sit on every
+ // request, and getConfig() is still null until argv is recovered.
+ if (!portEndpoint(getConfig())) await refreshBoundEndpoint(pm2Status);
+ return { ready: true, reason: null };
+ }
// Resolve the launch line BEFORE probing, so the probe below asks about
// the port this start would actually bind. `resolveLaunch` sets the
@@ -410,6 +472,7 @@ export function createOnDemandDaemon({
// launch the user saved, then the daemon's own default.
const saved = await savedLaunchConfig();
const config = resolveLaunch(getConfig(), saved);
+ if (Number.isFinite(Number(config?.port))) boundEndpoint = endpointFor({ port: Number(config.port) });
// Something else is already serving that port — a daemon the user started
// outside PortOS, or another process entirely. Either way this is not
@@ -492,8 +555,10 @@ export function formatReleaseTime(timestamp = Date.now()) {
return `${hh}:${mm}`;
}
-/** name → `{ getIdleMs, isPinned, isRunning, stop, lastUsedAt, releaseReason, releasedAt }`. */
+/** name → `{ getIdleMs, isPinned, isRunning, isBusy, warm, stop, lastUsedAt, releaseReason, releasedAt }`. */
const idleDaemons = new Map();
+/** Unsubscribes for per-daemon slot listeners. Cleared with the idle map in tests. */
+const settledUnsubscribers = new Set();
let reaperTimer = null;
let pressureHistory = [];
@@ -558,20 +623,23 @@ export function idleWindowMs(minutes) {
* isPinned?: () => Promise<boolean>|boolean,
* isRunning?: () => Promise<boolean>|boolean,
* isBusy?: () => Promise<boolean>|boolean,
+ * warm?: () => Promise<void>|void,
* stop: () => Promise<unknown>
* }} daemon
* `getIdleMs` resolves the CURRENT configured window on every sweep; `null`/`0` = never stop.
* `isPinned` returns true if user pinned the server ("keep loaded") — pinned servers are never stopped.
* `isRunning` checks if the daemon process is online.
- * `isBusy` returns true while a wake/start or inference request is in flight — busy servers are never stopped.
+ * `isBusy` returns true while a wake/start, a still-loading checkpoint, or an inference request is in flight — busy servers are never stopped.
+ * `warm` may resolve the port `isBusy` reads before the synchronous re-check. It must not itself decide to stop.
*/
-export function registerIdleDaemon({ name, getIdleMs, isPinned, isRunning, isBusy, stop }) {
+export function registerIdleDaemon({ name, getIdleMs, isPinned, isRunning, isBusy, warm, stop }) {
const existing = idleDaemons.get(name);
idleDaemons.set(name, {
getIdleMs,
isPinned: typeof isPinned === 'function' ? isPinned : () => Boolean(isPinned),
isRunning: typeof isRunning === 'function' ? isRunning : null,
isBusy: typeof isBusy === 'function' ? isBusy : () => Boolean(isBusy),
+ warm: typeof warm === 'function' ? warm : null,
stop,
lastUsedAt: existing?.lastUsedAt ?? Date.now(),
releaseReason: existing?.releaseReason ?? null,
@@ -735,6 +803,22 @@ export function evaluateMemoryPressurePolicy({
};
}
+// Synchronous on purpose. Awaiting here is the yield in which a request can
+// start after the snapshot and still be stopped. A thenable fails safe to busy;
+// its rejection is swallowed so the second call cannot become unhandled.
+function readBusySync(entry) {
+ try {
+ const value = entry?.isBusy?.();
+ if (value && typeof value.then === 'function') {
+ value.catch(() => {});
+ return true;
+ }
+ return Boolean(value);
+ } catch {
+ return true;
+ }
+}
+
/**
* Sweep: stop daemons whose idle window has elapsed, and run pressure-aware pass
* to release the least recently used unpinned daemon under sustained host memory pressure.
@@ -751,6 +835,7 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
// Fail-safe to pinned/busy on a transient read error, matching the pressure-aware
// pass below — assuming "not pinned" or "not busy" here would risk stopping a
// keepLoaded or actively serving daemon on a flaky read.
+ await Promise.resolve(entry.warm?.()).catch(() => {});
const isPinned = await Promise.resolve(entry.isPinned?.()).catch(() => true);
if (isPinned) continue; // Pinned servers are exempt
@@ -770,6 +855,7 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
const windowMs = await Promise.resolve(entry.getIdleMs()).catch(() => null);
if (!windowMs || windowMs <= 0) continue;
if (now - entry.lastUsedAt < windowMs) continue;
+ if (readBusySync(entry)) continue;
const idleMin = Math.round((now - entry.lastUsedAt) / 60_000);
console.log(`💤 Stopping ${name} — idle ${idleMin}m (window ${Math.round(windowMs / 60_000)}m)`);
@@ -805,6 +891,7 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
const daemonList = [];
for (const [name, entry] of idleDaemons) {
+ await Promise.resolve(entry.warm?.()).catch(() => {});
const isPinned = await Promise.resolve(entry.isPinned?.()).catch(() => true);
const isBusy = await Promise.resolve(entry.isBusy?.()).catch(() => true);
const isRunning = entry.isRunning
@@ -830,7 +917,7 @@ export async function reapIdleDaemons(now = Date.now(), options = {}) {
options: policyOptions,
});
- if (decision.shouldRelease && decision.target) {
+ if (decision.shouldRelease && decision.target && !readBusySync(decision.target.entry)) {
const { target } = decision;
const freeMb = Math.round((memoryStats.free || 0) / (1024 * 1024));
console.log(`⚠️ Stopping ${target.name} — host memory pressure (free ${freeMb}MB)`);
@@ -884,6 +971,8 @@ export function stopIdleReaper() {
/** Test seam: drop every registration, reset pressure state, and disarm the timer. */
export function _resetIdleDaemonsForTests() {
stopIdleReaper();
+ for (const unsubscribe of settledUnsubscribers) unsubscribe();
+ settledUnsubscribers.clear();
idleDaemons.clear();
pressureHistory = [];
lastPressureReleaseAt = null;
diff --git a/server/lib/managedDaemon.pressure.test.js b/server/lib/managedDaemon.pressure.test.js
index 6f89983570..6875ceb2c5 100644
--- a/server/lib/managedDaemon.pressure.test.js
+++ b/server/lib/managedDaemon.pressure.test.js
@@ -13,8 +13,9 @@ import {
DEFAULT_SUSTAINED_PRESSURE_MS,
createDaemonWatcher,
createOnDemandDaemon,
+ daemonLastUsedAt,
} from './managedDaemon.js';
-import { acquireLocalSlot, releaseLocalSlot } from './localLlmConcurrency.js';
+import { acquireLocalSlot, isLocalEndpointBusy, releaseLocalSlot, _resetLocalLlmSlotsForTests } from './localLlmConcurrency.js';
const MINUTE = 60_000;
const GB = 1024 * 1024 * 1024;
@@ -27,6 +28,7 @@ describe('managedDaemon memory pressure policy', () => {
});
afterEach(() => {
_resetIdleDaemonsForTests();
+ _resetLocalLlmSlotsForTests();
vi.restoreAllMocks();
});
@@ -530,5 +532,154 @@ describe('managedDaemon memory pressure policy', () => {
expect(stoppedAfterInference).toEqual(['portos-mtplx']);
expect(stop).toHaveBeenCalledTimes(1);
});
+
+ it('treats a restarted custom-port daemon as busy for a scheme-less slot', async () => {
+ const stop = vi.fn().mockResolvedValue(undefined);
+ const onDemand = createOnDemandDaemon({
+ appName: 'portos-mtplx',
+ label: 'MTPLX server',
+ emoji: '⚡',
+ readSection: async () => ({ idleMinutes: 30, keepLoaded: false, launch: { port: 8000 } }),
+ endpointFor: (cfg) => `http://127.0.0.1:${cfg?.port ?? 8000}/v1`,
+ probe: async () => true,
+ start: async () => ({ online: true }),
+ stop,
+ getStatusStrict: async () => ({ status: 'online', args: ['serve', '--port', '8010'] }),
+ parseConfigFromArgs: (args) => {
+ const index = args.indexOf('--port');
+ return index === -1 ? null : { port: Number(args[index + 1]) };
+ },
+ resolveLaunch: (current, saved) => ({ port: current?.port ?? saved.port ?? 8000 }),
+ getConfig: () => null,
+ sleep: async () => {},
+ getRelaunchReadyTimeoutMs: () => 50,
+ getRelaunchPollMs: () => 0,
+ });
+ onDemand.registerIdle();
+ await onDemand.ensureRunning();
+ const seeded = daemonLastUsedAt('portos-mtplx');
+ await new Promise((resolve) => { setTimeout(resolve, 5); });
+ await acquireLocalSlot('localhost:8010/v1');
+ try {
+ expect(daemonLastUsedAt('portos-mtplx')).toBeGreaterThan(seeded);
+ const now = Date.now() + 10 * MINUTE;
+ const stopped = await reapIdleDaemons(now, {
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now - 40_000, free: 2 * GB }, { at: now, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+ expect(stopped).toEqual([]);
+ expect(stop).not.toHaveBeenCalled();
+ } finally {
+ releaseLocalSlot('localhost:8010/v1');
+ }
+ });
+
+ it('does not stop a daemon whose request starts while status is being read', async () => {
+ const stop = vi.fn().mockResolvedValue(undefined);
+ let runningReads = 0;
+ const endpoint = 'http://127.0.0.1:8093/v1';
+ registerIdleDaemon({
+ name: 'daemon-racy',
+ getIdleMs: () => 30 * MINUTE,
+ isBusy: () => isLocalEndpointBusy(endpoint),
+ isRunning: async () => {
+ runningReads += 1;
+ // The idle pass reads once. The pressure pass reads again after it
+ // has already snapshotted busy=false.
+ if (runningReads === 2) await acquireLocalSlot(endpoint);
+ return true;
+ },
+ stop,
+ });
+ const now = Date.now();
+ try {
+ const stopped = await reapIdleDaemons(now, {
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now - 40_000, free: 2 * GB }, { at: now, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+ expect(runningReads).toBeGreaterThanOrEqual(2);
+ expect(stopped).toEqual([]);
+ expect(stop).not.toHaveBeenCalled();
+ } finally {
+ releaseLocalSlot(endpoint);
+ }
+ });
+
+ it('keeps an explicit start busy after launch returns while the checkpoint is still loading', async () => {
+ const stop = vi.fn().mockResolvedValue(undefined);
+ let reachable = false;
+ const onDemand = createOnDemandDaemon({
+ appName: 'portos-loading',
+ label: 'MTPLX server',
+ emoji: '⚡',
+ readSection: async () => ({ idleMinutes: 30, keepLoaded: false }),
+ endpointFor: (cfg) => `http://127.0.0.1:${cfg?.port ?? 8092}/v1`,
+ probe: async () => reachable,
+ start: async () => ({ online: false, endpoint: 'http://127.0.0.1:8092/v1' }),
+ stop,
+ getStatusStrict: async () => ({ status: 'online' }),
+ resolveLaunch: (current) => current || { port: 8092 },
+ getConfig: () => ({ port: 8092 }),
+ sleep: () => new Promise((resolve) => { setTimeout(resolve, 1); }),
+ getRelaunchReadyTimeoutMs: () => 5_000,
+ getRelaunchPollMs: () => 0,
+ });
+ onDemand.registerIdle();
+ const pressure = (now) => ({
+ memoryStats: { total: 32 * GB, used: 30 * GB, free: 2 * GB },
+ history: [{ at: now - 40_000, free: 2 * GB }, { at: now, free: 2 * GB }],
+ sustainedDurationMs: 30_000,
+ });
+
+ await onDemand.runStart(async () => ({ online: false, endpoint: 'http://127.0.0.1:8092/v1' }));
+ const duringLoad = Date.now();
+ const stoppedDuringLoad = await reapIdleDaemons(duringLoad, pressure(duringLoad));
+ expect(stoppedDuringLoad).toEqual([]);
+ expect(stop).not.toHaveBeenCalled();
+
+ reachable = true;
+ await vi.waitFor(async () => {
+ const now = Date.now();
+ const stopped = await reapIdleDaemons(now, pressure(now));
+ expect(stopped).toEqual(['portos-loading']);
+ });
+ expect(stop).toHaveBeenCalledTimes(1);
+ });
+
+ it('drops slot listeners when idle registrations are reset', async () => {
+ const onDemand = createOnDemandDaemon({
+ appName: 'portos-mtplx',
+ label: 'MTPLX server',
+ emoji: '⚡',
+ readSection: async () => ({ idleMinutes: 30, keepLoaded: false }),
+ endpointFor: () => 'http://127.0.0.1:8092/v1',
+ probe: async () => false,
+ start: async () => ({ online: true }),
+ stop: async () => {},
+ getStatusStrict: async () => ({ status: 'stopped' }),
+ resolveLaunch: () => ({ port: 8092 }),
+ getConfig: () => null,
+ sleep: async () => {},
+ getRelaunchReadyTimeoutMs: () => 50,
+ getRelaunchPollMs: () => 0,
+ });
+ onDemand.registerIdle();
+ _resetIdleDaemonsForTests();
+ registerIdleDaemon({
+ name: 'portos-mtplx',
+ getIdleMs: () => 60 * MINUTE,
+ stop: vi.fn(),
+ });
+ const seeded = daemonLastUsedAt('portos-mtplx');
+ await new Promise((resolve) => { setTimeout(resolve, 5); });
+ await acquireLocalSlot('http://127.0.0.1:8092/v1');
+ try {
+ expect(daemonLastUsedAt('portos-mtplx')).toBe(seeded);
+ } finally {
+ releaseLocalSlot('http://127.0.0.1:8092/v1');
+ }
+ });
});
});
diff --git a/server/lib/managedDaemon.test.js b/server/lib/managedDaemon.test.js
index bc2596e9ff..14452e37a0 100644
--- a/server/lib/managedDaemon.test.js
+++ b/server/lib/managedDaemon.test.js
@@ -1,4 +1,4 @@
-import { describe, expect, it, vi } from 'vitest';
+import { beforeEach, describe, expect, it, vi } from 'vitest';
import { createDaemonWatcher, createOnDemandDaemon, createPm2ExitTail, reapIdleDaemons, _resetIdleDaemonsForTests } from './managedDaemon.js';
const makeWatcher = (overrides = {}) => {
@@ -102,6 +102,10 @@ describe('createPm2ExitTail', () => {
});
describe('createOnDemandDaemon', () => {
+ beforeEach(() => {
+ _resetIdleDaemonsForTests();
+ });
+
const makeOnDemand = (overrides = {}) => {
let config = overrides.config ?? null;
let section = overrides.section ?? {};
diff --git a/server/services/mtplxServerManager.js b/server/services/mtplxServerManager.js
index 4a5b566436..9d8f8aa2f8 100644
--- a/server/services/mtplxServerManager.js
+++ b/server/services/mtplxServerManager.js
@@ -226,6 +226,7 @@ const onDemand = createOnDemandDaemon({
tuning: current?.tuning ?? normalizeTuning('mtplx', saved.raw?.tuning),
}),
getConfig: () => currentConfig,
+ parseConfigFromArgs,
sleep,
getRelaunchReadyTimeoutMs: () => relaunchReadyTimeoutMs,
getRelaunchPollMs: () => relaunchPollMs,
diff --git a/server/services/slotstreamServerManager.js b/server/services/slotstreamServerManager.js
index 6e3f7625db..b3594fc627 100644
--- a/server/services/slotstreamServerManager.js
+++ b/server/services/slotstreamServerManager.js
@@ -138,6 +138,7 @@ const onDemand = createOnDemandDaemon({
memoryGb: current?.memoryGb ?? (Number.isFinite(Number(saved.raw?.memoryGb)) ? Number(saved.raw.memoryGb) : null),
}),
getConfig: () => currentConfig,
+ parseConfigFromArgs,
sleep,
getRelaunchReadyTimeoutMs: () => relaunchReadyTimeoutMs,
getRelaunchPollMs: () => relaunchPollMs,