Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix([issue-12723]): queue explicit MTPLX/Slotstream starts behind an in-flight on-demand wake

[._.]/ Adam Eivy7373f9ebe34b
diff --git a/server/lib/README.md b/server/lib/README.md
index 34efdf3324..cda62a507d 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -309,7 +309,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `opencodeCatalogCache.js` | Primes the on-disk catalog `opencode models` prints from — `primeOpencodeCatalogCache()` fetches OpenCode's `api.json` with Node's fetch and atomically writes `$XDG_CACHE_HOME/opencode/models.json` (`~/.cache` when unset). OpenCode refreshes that file from a forked task whose failures it swallows (`opencode models --refresh` still prints `Models cache refreshed`) and its HTTP client has no Happy Eyeballs, so a host advertising an unreachable IPv6 default route freezes the catalog indefinitely while other machines on the same account list newer models. Refuses to fetch or write when `OPENCODE_MODELS_PATH` / a custom `OPENCODE_MODELS_URL` / `OPENCODE_DISABLE_MODELS_FETCH` means PortOS cannot be sure which file OpenCode reads, when the file is under five minutes old, or when the body did not parse as a catalog — a stale list beats an empty picker. Never throws; the caller probes either way. |
| `opencodeConfig.js` | OpenCode config builder — `buildOpencodeEnvVars(provider, model)` builds dynamic `OPENCODE_CONFIG_CONTENT` declaring model ids under the namespace the provider's marker selects: a local runtime (`ollama` / `mtplx` / `llama` / `vllm` / `sglang`, bare ids) or a hosted gateway from `providerGateways.js` (`vendor/model` ids kept whole). Fixes --model rejection. Also attaches the key for a key-bearing namespace, and pins `small_model` to the run model for a gateway so OpenCode's own side calls (titles, summarization) can't land on its built-in default — a billed model the operator never chose. Under a `no-tool` public-review profile it also applies `hardenOpencodeConfigForNoTool` — root `permission: deny`, an emptied tool map on every agent, `tool_call: false` on every declared model, and no MCP/plugins/share/autoupdate — which IS OpenCode's enforced tool-free recipe, since it ships no read-only argv flag (`providerVendors.js` pairs it with `run --agent` + `OPENCODE_PUBLIC_REVIEW_AGENT`). The harden step also copies `agent.build`'s generation settings onto that agent, so the stage's configured thinking effort reaches the model that actually runs. |
| `localProviderRuntime.js` | Which LOCAL daemon a provider talks to, and where — `LOCAL_RUNTIMES` (llama.cpp / Ollama / LM Studio / MTPLX / vLLM: label, binary, canonical base URL read from `opencodeConfig.js` rather than re-typed, manage/docs links, model-download hint), `localEndpointPort` + `isLocalInstanceHost`, and re-exports of `localBackendForProvider` / `localRuntimeKind` / `modelPinIsOffered` — all three now live in the browser-safe leaf `modelPinMembership.js` (a picker needs the pin rule and must not pull the endpoint graph below to get it), so existing importers here are untouched — `localRuntimeForProvider(provider)` → the row with the endpoint the provider ITSELF configures (`OPENCODE_CONFIG_CONTENT`'s `baseURL`, `ANTHROPIC_BASE_URL`, or `endpoint`), then the `OLLAMA_URL`/`OLLAMA_HOST`/`LM_STUDIO_URL` override the backend managers read, then the canonical default — and `null` when that resolved endpoint fails `isLocalInstanceEndpoint` (an API provider on another machine has no local daemon to check, whatever its name says) — plus `normalizeOpenAiBaseUrl`. Pure; the probing half is `services/providerReadiness.js`. Optional `setupStateDetail` overrides `providerReadiness`'s per-state prose for a runtime whose local setup is not a model cache (vLLM's is a compose project); `standbyWhenStopped` marks an installed runtime such as llama.cpp whose stopped state is intentional standby rather than incomplete setup. |
-| `managedDaemon.js` | Shared mechanism for the local daemons PortOS runs as optional PM2 processes (`services/llamaServerManager.js` → `portos-llama-server`, `services/mtplxServerManager.js` → `portos-mtplx`, `services/slotstreamServerManager.js` → `portos-slotstream`). Owns their PM2 process names — `LLAMA_APP`, `MTPLX_APP`, and `SLOTSTREAM_APP`; the managers re-export those names. `createDaemonWatcher({...})` supplies the common PM2 launch-line re-adoption, endpoint probe, status skeleton, bounded log view, and port-release wait while managers retain daemon-specific parsing and lifecycle policy. `createDaemonLogBuffer({maxLines?})` is the bounded timestamped ring buffer of what PortOS logged around a launch, plus `withPm2Logs(output)` → that buffer followed by anything `pm2 logs` has which it doesn't already hold, deduped and re-capped (PM2's lines are a VIEW, never folded into the buffer — PM2 owns them and re-reads them every status call). `pm2ArgValue(args, flag)` reads one value back out of a PM2 process's recorded argv so a manager can recover a still-online daemon's launch config after a PortOS restart; `null` means the flag was absent, which a relaunch must leave off rather than defaulting. Also the shared **idle reaper**, for a daemon that cannot release its weights any other way: `registerIdleDaemon({name, getIdleMs, isPinned?, isRunning?, stop})` (seeds `lastUsedAt` to NOW, so a hand-started daemon gets a full window; `isPinned` exempts user-pinned servers), `markDaemonUsed(name)` — call on real traffic, NEVER on a status poll — `daemonLastUsedAt(name)`, `daemonReleaseReason(name)`, `idleWindowMs(minutes)` (minutes → ms; `0` = never, `null` = not configured, kept distinct), `evaluateMemoryPressurePolicy({...})` (pure policy function evaluating current state, pressure reading, and recent history), `reapIdleDaemons(now?, options?)` → the names stopped (stops idle daemons whose window elapsed, and runs a pressure-aware pass releasing the least recently used unpinned daemon under sustained host memory pressure; at most one daemon stopped per tick), and `startIdleReaper({intervalMs?})` / `stopIdleReaper()` (ONE interval for all registrants, `unref`'d, idempotent). `mtplxServerManager` and `slotstreamServerManager` register: llama.cpp releases its checkpoint in place via `--sleep-idle-seconds` and must NOT be stopped for it. `createOnDemandDaemon({appName, label, emoji, readSection, endpointFor, probe, start, stop, getStatusStrict, clearStatusCache?, exitTail?, resolveLaunch, getConfig?, sleep, getRelaunchReadyTimeoutMs, getRelaunchPollMs})` is the shared ON-DEMAND WAKE mechanism (#8105) — `ensureRunning()` (mark used → already online? → resolve the launch line via the manager's `resolveLaunch(current, saved)` → probe → start → wait), `registerIdle()` (wraps `registerIdleDaemon` with the idle-minutes/keep-loaded reads, including the legacy `pinned` fallback and their own `set*OverrideForTests` seams), `waitForReady(endpoint)` (the death-detecting readiness poll, with `exitTail` folded in), `servesPort(provider, managedPort)` (the numeric port-match arm — `Number(localEndpointPort(...))`, fixing a `===` string/number mismatch that made the MTPLX arm permanently dead), and `idleMinutes` / `keepLoaded` / `savedLaunch` for a status card's reads. `createPm2ExitTail({appName, execPm2, appendLog, lines?, tailLines?})` is the shared "what did the dead process print" diagnosis both `waitForReady` and a manager's own startup-window failure can share. Both mechanisms only — what a launch line means and when a daemon may start is exactly what differs between the two. |
+| `managedDaemon.js` | Shared mechanism for the local daemons PortOS runs as optional PM2 processes (`services/llamaServerManager.js` → `portos-llama-server`, `services/mtplxServerManager.js` → `portos-mtplx`, `services/slotstreamServerManager.js` → `portos-slotstream`). Owns their PM2 process names — `LLAMA_APP`, `MTPLX_APP`, and `SLOTSTREAM_APP`; the managers re-export those names. `createDaemonWatcher({...})` supplies the common PM2 launch-line re-adoption, endpoint probe, status skeleton, bounded log view, and port-release wait while managers retain daemon-specific parsing and lifecycle policy. `createDaemonLogBuffer({maxLines?})` is the bounded timestamped ring buffer of what PortOS logged around a launch, plus `withPm2Logs(output)` → that buffer followed by anything `pm2 logs` has which it doesn't already hold, deduped and re-capped (PM2's lines are a VIEW, never folded into the buffer — PM2 owns them and re-reads them every status call). `pm2ArgValue(args, flag)` reads one value back out of a PM2 process's recorded argv so a manager can recover a still-online daemon's launch config after a PortOS restart; `null` means the flag was absent, which a relaunch must leave off rather than defaulting. Also the shared **idle reaper**, for a daemon that cannot release its weights any other way: `registerIdleDaemon({name, getIdleMs, isPinned?, isRunning?, stop})` (seeds `lastUsedAt` to NOW, so a hand-started daemon gets a full window; `isPinned` exempts user-pinned servers), `markDaemonUsed(name)` — call on real traffic, NEVER on a status poll — `daemonLastUsedAt(name)`, `daemonReleaseReason(name)`, `idleWindowMs(minutes)` (minutes → ms; `0` = never, `null` = not configured, kept distinct), `evaluateMemoryPressurePolicy({...})` (pure policy function evaluating current state, pressure reading, and recent history), `reapIdleDaemons(now?, options?)` → the names stopped (stops idle daemons whose window elapsed, and runs a pressure-aware pass releasing the least recently used unpinned daemon under sustained host memory pressure; at most one daemon stopped per tick), and `startIdleReaper({intervalMs?})` / `stopIdleReaper()` (ONE interval for all registrants, `unref`'d, idempotent). `mtplxServerManager` and `slotstreamServerManager` register: llama.cpp releases its checkpoint in place via `--sleep-idle-seconds` and must NOT be stopped for it. `createOnDemandDaemon({appName, label, emoji, readSection, endpointFor, probe, start, stop, getStatusStrict, clearStatusCache?, exitTail?, resolveLaunch, getConfig?, sleep, getRelaunchReadyTimeoutMs, getRelaunchPollMs})` is the shared ON-DEMAND WAKE mechanism (#8105) — `ensureRunning()` (mark used → already online? → resolve the launch line via the manager's `resolveLaunch(current, saved)` → probe → start → wait), `runStart(fn)` (runs an explicit start/stop sequence in the same exclusive start slot as the wake, so a user-clicked Start or tuning relaunch queues behind an in-flight wake instead of deleting it, #12723; `fn` calls the raw start, never `ensureRunning`/`runStart`), `registerIdle()` (wraps `registerIdleDaemon` with the idle-minutes/keep-loaded reads, including the legacy `pinned` fallback and their own `set*OverrideForTests` seams), `waitForReady(endpoint)` (the death-detecting readiness poll, with `exitTail` folded in), `servesPort(provider, managedPort)` (the numeric port-match arm — `Number(localEndpointPort(...))`, fixing a `===` string/number mismatch that made the MTPLX arm permanently dead), and `idleMinutes` / `keepLoaded` / `savedLaunch` for a status card's reads. `createPm2ExitTail({appName, execPm2, appendLog, lines?, tailLines?})` is the shared "what did the dead process print" diagnosis both `waitForReady` and a manager's own startup-window failure can share. Both mechanisms only — what a launch line means and when a daemon may start is exactly what differs between the two. |
| `mtplxModels.js` | `services/localCachedModels.js` owns provider-refresh discovery and gates it on runtime readiness without loading the process manager. `listMtplxCachedModels({command?})` → `{models, error}` from `mtplx models --json` (walks local directories — pulls no weights, loads no model, but see `mtplxRuntime.js`: on an un-warmed Homebrew wrapper the spawn ITSELF is a several-hundred-megabyte runtime download, so poll callers must gate on `describeMtplxRuntime().ready` first) `servableMtplxCachedModels(models)` → the rows MTPLX itself calls complete, and `pickMtplxCachedModel(models)` → the one repo id to hand `mtplx serve --model`. `models: null` means the cache could not be READ (no binary, command failed, unparseable) and is deliberately distinct from `[]` (read, and empty), because `services/localRuntimeSetup.js` starts MTPLX on its own default in the first case and refuses with the `mtplx pull` command in the second. Exists because `mtplx serve` defaults `--model` to one hard-coded checkpoint and exits 1 before binding when that repo is not cached — even on a host holding a different MTP model that serves fine. Picks only entries MTPLX itself calls complete (`validation.ok !== false`, so a half-finished pull is not served), preferring one with a recorded `mtplx_runtime.json` exactness contract. `describeMtplxCache(cache)` → `{state: 'unknown'\|'empty'\|'partial'\|'ready', model, count, error}` folds both into the one value `services/providerReadiness.js` puts on the checklist and `describeRuntimeSetup` picks a button from — so an empty cache is named up front instead of only inside the failure of a Start that could never work. |
| `mtplxRuntime.js` | `describeMtplxRuntime(binaryPath, {env?})` → `{ready, wrapper, venvPath}` — is MTPLX's own Python runtime on disk, decided by READING the binary rather than running it. Homebrew's `mtplx` is a shell wrapper that lazily bootstraps a version-keyed Python venv (a multi-hundred-megabyte pip install) on its first invocation, and `brew upgrade` re-arms it, so a status poll or an 8s-judged PM2 start is really a package download. This parses the wrapper's own `VENV=` assignment, honours `$MTPLX_BREW_VENV` the way `${MTPLX_BREW_VENV:-…}` does, and tests `<venv>/bin/mtplx` for executability — the wrapper's own `[ ! -x … ]` guard, so the two cannot disagree. Anything unrecognisable (a pip install, a compiled binary, an unparseable script) reports `ready: true, wrapper: false`, i.e. the status quo — never a block over a parse failure. |
| `slotstreamCatalog.js` | `SLOTSTREAM_CATALOG` — the curated mixture-of-experts checkpoints worth streaming from SSD (the technique only pays off on MoE: a dense model of the same size would stream every layer for every token). `resolveSlotstreamRepo(idOrRepo)` maps a catalog id or an `owner/name` repo id to a repo (null for anything else), `slotstreamModelDirName(repo)` flattens it to the single-segment cache directory whose NAME is the id a start hands `--model`, and `selectSlotstreamRepoFiles(siblings)` picks the `.safetensors`/config/tokenizer files a checkpoint is made of while dropping mirrored `original/`, GGUF, ONNX, and PyTorch copies of the same weights (which can double a 100 GB+ pull) and any name that is not a plain relative path. |
diff --git a/server/lib/managedDaemon.js b/server/lib/managedDaemon.js
index b617a8a45b..d72700bcf6 100644
--- a/server/lib/managedDaemon.js
+++ b/server/lib/managedDaemon.js
@@ -352,14 +352,33 @@ export function createOnDemandDaemon({
* "exited while loading". Joining is safe because the launch line comes from
* live/saved state, not from the caller. The slot clears on settle, so a later
* wake after another stop starts the daemon again.
+ *
+ * The slot is exclusive and shared with `runStart` (#12723): an explicit start
+ * or a tuning relaunch would otherwise `pm2 delete` an in-flight wake's
+ * process, the same race between a different pair of callers. A wake that
+ * arrives during one waits for it, then finds the daemon online.
*/
- const wakes = createSingleFlight();
+ const starts = createSingleFlight({ exclusive: true });
const ensureRunning = async () => {
markUsed();
- return wakes.run('wake', wake);
+ return starts.run('wake', wake);
};
+ /**
+ * Run an explicit start/stop sequence (a user-clicked Start, a tuning
+ * relaunch) in the daemon's exclusive start slot. Each call gets its own key,
+ * so it queues behind an in-flight wake or start rather than joining it: its
+ * launch config can differ from the one being loaded. `fn` must call the
+ * manager's raw start, never `ensureRunning` or `runStart` — either would
+ * wait on the slot `fn` itself holds.
+ *
+ * @template T
+ * @param {() => Promise<T>} fn
+ * @returns {Promise<T>}
+ */
+ const runStart = (fn) => starts.run(Symbol('start'), fn);
+
const wake = async () => {
const pm2Status = await getStatusStrict();
if (pm2Status?.status === 'online') return { ready: true, reason: null };
@@ -394,6 +413,7 @@ export function createOnDemandDaemon({
return {
ensureRunning,
+ runStart,
markUsed,
registerIdle,
waitForReady,
diff --git a/server/lib/managedDaemon.test.js b/server/lib/managedDaemon.test.js
index 549e8cdce3..bc2596e9ff 100644
--- a/server/lib/managedDaemon.test.js
+++ b/server/lib/managedDaemon.test.js
@@ -192,6 +192,49 @@ describe('createOnDemandDaemon', () => {
expect(await third).toEqual({ ready: true, reason: null });
});
+ // #12723: a user-clicked Start used to `pm2 delete` a wake's still-loading
+ // process. It must queue, not join: its launch config can differ.
+ it('runs an explicit start only after an in-flight wake settles', async () => {
+ let loading = false;
+ let finishWake;
+ const start = vi.fn(() => {
+ loading = true;
+ return new Promise((resolve) => { finishWake = (value) => { loading = false; resolve(value); }; });
+ });
+ const { onDemand, stop } = makeOnDemand({ start });
+
+ const wake = onDemand.ensureRunning();
+ await vi.waitFor(() => expect(start).toHaveBeenCalledTimes(1));
+ const stoppedWhileLoading = [];
+ const queued = onDemand.runStart(async () => {
+ stoppedWhileLoading.push(loading);
+ await stop();
+ return 'restarted';
+ });
+
+ finishWake({ online: true });
+ expect(await wake).toEqual({ ready: true, reason: null });
+ expect(await queued).toBe('restarted');
+ expect(stoppedWhileLoading).toEqual([false]);
+ });
+
+ it('makes a wake issued during an explicit relaunch wait for it, then find the daemon online', async () => {
+ const { onDemand, start, setStatus } = makeOnDemand();
+ const order = [];
+ let finishRelaunch;
+ const relaunch = onDemand.runStart(() => new Promise((resolve) => {
+ finishRelaunch = () => { order.push('relaunch'); resolve('relaunched'); };
+ }));
+ const wake = onDemand.ensureRunning().then((result) => { order.push('wake'); return result; });
+
+ setStatus({ status: 'online' });
+ finishRelaunch();
+ expect(await relaunch).toBe('relaunched');
+ expect(await wake).toEqual({ ready: true, reason: null });
+ expect(order).toEqual(['relaunch', 'wake']);
+ expect(start).not.toHaveBeenCalled();
+ });
+
it('waitForReady reports the shared exit-tail diagnosis when the process dies before answering', async () => {
const { onDemand, setStatus } = makeOnDemand({
start: vi.fn(async () => ({ online: false, endpoint: 'http://127.0.0.1:9000/v1' })),
diff --git a/server/services/mtplxServerManager.js b/server/services/mtplxServerManager.js
index 15a1920697..4a5b566436 100644
--- a/server/services/mtplxServerManager.js
+++ b/server/services/mtplxServerManager.js
@@ -215,7 +215,7 @@ const onDemand = createOnDemandDaemon({
readSection: () => readSettings().then((settings) => settings?.localLlm?.mtplx ?? null),
endpointFor,
probe: probeEndpoint,
- start: (config) => startMtplxServer(config),
+ start: (config) => launchMtplxServer(config),
stop: () => stopMtplxServer(),
getStatusStrict: () => getAppStatusStrict(MTPLX_APP),
clearStatusCache: () => clearJlistCache(),
@@ -474,7 +474,17 @@ async function resolveStartModel(requested, emit) {
}
/**
- * Start `mtplx serve` under PM2.
+ * Start `mtplx serve` under PM2, queued behind any in-flight on-demand wake,
+ * explicit start, or tuning relaunch (`onDemand.runStart`, #12723) so it cannot
+ * `pm2 delete` a launch that is still loading.
+ *
+ * @param {Parameters<typeof launchMtplxServer>[0]} [options]
+ */
+export const startMtplxServer = (options = {}) => onDemand.runStart(() => launchMtplxServer(options));
+
+/**
+ * The unserialized start behind `startMtplxServer`. Only code already holding
+ * the start slot calls it directly: the on-demand wake and the tuning relaunch.
*
* `waitMs` overrides how long this blocks before handing back `online: false`
* for a server that is still loading. The LLMs-page launcher takes the short
@@ -496,7 +506,7 @@ async function resolveStartModel(requested, emit) {
*
* @param {{port?: number, model?: string, tuning?: object, waitMs?: number, onProgress?: (line: string) => void}} options
*/
-export async function startMtplxServer(options = {}) {
+async function launchMtplxServer(options = {}) {
const { port = DEFAULT_PORT, model: requestedModel = null, tuning = null, waitMs, onProgress = () => {} } = options;
const normalizedTuning = normalizeTuning('mtplx', tuning);
const emit = (line) => { appendLog(line); onProgress(line); };
@@ -706,6 +716,12 @@ export async function relaunchMtplxServerWithTuning(tuning = {}) {
return { applied: false, reason: 'that tuning renders no `mtplx serve` flag, so there is nothing to relaunch with', config: currentConfig };
}
+ // The status read sits inside the slot too, so a relaunch queued behind a
+ // wake reads the configuration that wake brought up (#12723).
+ return onDemand.runStart(() => relaunchWithTuning(normalized, knobs));
+}
+
+async function relaunchWithTuning(normalized, knobs) {
const status = await getMtplxServerStatus();
if (!status.running) {
return { applied: false, reason: 'MTPLX is not running, so PortOS has no checkpoint to relaunch with', config: null };
@@ -752,7 +768,7 @@ export async function relaunchMtplxServerWithTuning(tuning = {}) {
await stopMtplxServer();
await waitForPortRelease(port);
- const started = await startMtplxServer(next).catch((err) => ({ failure: err }));
+ const started = await launchMtplxServer(next).catch((err) => ({ failure: err }));
if (started.failure) {
const reason = launchFailureReason(started.failure);
console.error(`❌ MTPLX: tuning launch failed (${started.failure.message}) — restoring the previous configuration`);
@@ -803,7 +819,7 @@ const launchFailureReason = (err) => (err?.code === 'MTPLX_EXITED'
*/
async function restorePrevious(previous, failure) {
await waitForPortRelease(previous.port ?? DEFAULT_PORT);
- const restored = await startMtplxServer(previous).catch((err) => {
+ const restored = await launchMtplxServer(previous).catch((err) => {
console.error(`❌ MTPLX: could not restore the previous configuration: ${err.message}`);
return null;
});
diff --git a/server/services/mtplxServerManager.test.js b/server/services/mtplxServerManager.test.js
index 4f329ee086..14b480373c 100644
--- a/server/services/mtplxServerManager.test.js
+++ b/server/services/mtplxServerManager.test.js
@@ -964,6 +964,86 @@ describe('mtplxServerManager', () => {
});
});
+ // #12723: every start of the daemon shares one exclusive slot, so a direct
+ // start or a tuning relaunch cannot `pm2 delete` a wake that is still loading.
+ describe('serializing starts against an on-demand wake (#12723)', () => {
+ const count = (verb) => execPm2Calls.filter((c) => c[0] === verb).length;
+
+ // Holds the next `pm2 start` until released, leaving that launch in flight.
+ const holdNextPm2Start = () => {
+ const record = pm2Module.execPm2.getMockImplementation();
+ let release;
+ const held = new Promise((resolve) => { release = resolve; });
+ const state = { reached: false, release: () => release() };
+ pm2Module.execPm2.mockImplementation(async (args) => {
+ if (args[0] === 'start' && !state.reached) {
+ state.reached = true;
+ await held;
+ }
+ return record(args);
+ });
+ return state;
+ };
+
+ let findBinary;
+ beforeEach(() => {
+ findBinary = vi.spyOn(processEnv, 'findCommandOnPath').mockReturnValue(BINARY);
+ });
+
+ it('runs a direct start only after an in-flight wake settles, deleting nothing meanwhile', async () => {
+ const hold = holdNextPm2Start();
+ const wake = ensureMtplxRunning();
+ await vi.waitFor(() => expect(hold.reached).toBe(true));
+ const deletesBeforeStart = count('delete');
+
+ // An unqueued start resolves the binary synchronously, before its first await.
+ const binaryLookups = findBinary.mock.calls.length;
+ const direct = startMtplxServer({ port: 8000 }).catch((err) => err);
+ expect(findBinary.mock.calls.length).toBe(binaryLookups);
+
+ hold.release();
+ expect(await wake).toEqual({ ready: true, reason: null });
+ // Queued behind the wake, the direct start finds it online and refuses
+ // instead of replacing it.
+ expect((await direct).message).toMatch(/already running/);
+ expect(count('delete')).toBe(deletesBeforeStart);
+ expect(count('start')).toBe(1);
+ expect(pm2State.status).toBe('online');
+ });
+
+ it('resolves a wake issued during a tuning relaunch after it, with no second start', async () => {
+ await startMtplxServer({ port: 8010 });
+ execPm2Calls = [];
+ const hold = holdNextPm2Start();
+ const relaunch = relaunchMtplxServerWithTuning({ contextWindow: 32768 });
+ // Held after the relaunch's stop: the daemon is down, which a wake
+ // outside the slot reads synchronously as "start it".
+ await vi.waitFor(() => expect(hold.reached).toBe(true));
+ expect(pm2State).toBeNull();
+
+ // The relaunched checkpoint takes a couple of probes to load, so a wake
+ // outside the slot cannot settle by finding the port already answering.
+ let loadingProbes = 2;
+ vi.spyOn(openAiModelsProbe, 'probeOpenAiModels').mockImplementation(async () => {
+ if (pm2State?.status !== 'online') return { reachable: false };
+ if (loadingProbes > 0) {
+ loadingProbes -= 1;
+ return { reachable: false };
+ }
+ return { reachable: true };
+ });
+
+ const wake = ensureMtplxRunning();
+ hold.release();
+
+ expect((await relaunch).applied).toBe(true);
+ expect(await wake).toEqual({ ready: true, reason: null });
+ expect(count('start')).toBe(1);
+ const launch = execPm2Calls.find((c) => c[0] === 'start');
+ expect(launch[launch.indexOf('--context-window') + 1]).toBe('32768');
+ });
+ });
+
describe('surviving an idle-reaper stop (#8105)', () => {
// Bug #2 from the issue: `stopMtplxServer` clears `currentConfig`, and the
// saved launch line used to hold only `model`/`port` — an assessment's
diff --git a/server/services/slotstreamServerManager.js b/server/services/slotstreamServerManager.js
index 48022c02db..6e3f7625db 100644
--- a/server/services/slotstreamServerManager.js
+++ b/server/services/slotstreamServerManager.js
@@ -127,7 +127,7 @@ const onDemand = createOnDemandDaemon({
readSection: () => readSettings().then((settings) => settings?.localLlm?.slotstream ?? null),
endpointFor,
probe: probeEndpoint,
- start: (config) => startSlotstreamServer(config),
+ start: (config) => launchSlotstreamServer(config),
stop: () => stopSlotstreamServer(),
getStatusStrict: () => getAppStatusStrict(SLOTSTREAM_APP),
clearStatusCache: () => clearJlistCache(),
@@ -284,11 +284,21 @@ async function resolveStartModel(requested, emit) {
}
/**
- * Start `slotstream serve` under PM2.
+ * Start `slotstream serve` under PM2, queued behind any in-flight on-demand
+ * wake or explicit start (`onDemand.runStart`, #12723) so it cannot
+ * `pm2 delete` a launch that is still loading.
+ *
+ * @param {Parameters<typeof launchSlotstreamServer>[0]} [options]
+ */
+export const startSlotstreamServer = (options = {}) => onDemand.runStart(() => launchSlotstreamServer(options));
+
+/**
+ * The unserialized start behind `startSlotstreamServer`; only the on-demand
+ * wake, which already holds the start slot, calls it directly.
*
* @param {{port?: number, model?: string, memoryGb?: number, waitMs?: number, onProgress?: (line: string) => void}} options
*/
-export async function startSlotstreamServer(options = {}) {
+async function launchSlotstreamServer(options = {}) {
const { port: requestedPort, model: requestedModel = null, memoryGb = null, waitMs, onProgress = () => {} } = options;
const port = resolveStartPort(requestedPort ?? DEFAULT_PORT);
const emit = (line) => { appendLog(line); onProgress(line); };