Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

46.5 KB
Raw Blame
/**
 * MTPLX process manager
 *
 * Lifecycle management (status probe, install, start, stop, recent logs) for the
 * local `mtplx serve` OpenAI-compatible API server, managed as an optional PM2
 * process (`portos-mtplx`) — deliberately the same shape as
 * `llamaServerManager.js`.
 *
 * PM2 rather than a bare detached spawn, because that is what makes the daemon
 * *manageable*: `pm2 list` shows it next to the rest of the install, `pm2 logs`
 * has its output, restart-on-boot is a `pm2 save` away, and a PortOS restart can
 * re-adopt a server it started earlier instead of losing track of a detached
 * pid. A detached child gets none of that.
 *
 * Two limits carried over from `docs/features/mtplx.md`:
 *   - **A start never downloads weights.** `mtplx serve` is started on a
 *     checkpoint ALREADY in MTPLX's cache; an empty cache is reported rather
 *     than silently filled. Fetching one is its own explicit, user-pressed
 *     action in `mtplxModelManager.js` — in the UI, never in a terminal.
 *   - **MTPLX's privileged paths are never touched.** Upstream's optional
 *     `mtplx max --install` fan-control helper stays an explicit operator action
 *     outside PortOS.
 */

import { commandExists } from '../lib/commandExists.js';
import { writeFile } from 'fs/promises';
import { tmpdir } from 'os';
import { join } from 'path';
import { sleep } from '../lib/fileUtils.js';
import { ServerError } from '../lib/errorHandler.js';
import { launchArgs, normalizeTuning, tuningSpecsFor } from '../lib/localModelTuning.js';
import { LOCAL_RUNTIMES, localEndpointPort, localRuntimeKind, isLocalInstanceEndpoint } from '../lib/localProviderRuntime.js';
import { listMtplxCachedModels, pickMtplxCachedModel } from '../lib/mtplxModels.js';
import { describeMtplxRuntime } from '../lib/mtplxRuntime.js';
import { probeOpenAiModels } from '../lib/openAiModelsProbe.js';
import { createDaemonWatcher, createOnDemandDaemon, createPm2ExitTail, pm2ArgValue, MTPLX_APP } from '../lib/managedDaemon.js';
import { persistDaemonLaunchConfig } from './localDaemonLaunchConfig.js';
// `settings.js` is lazy-imported at its call sites below, never statically: it
// eagerly resolves `fileUtils.PATHS` at module load, which drags PATHS into the
// module graph of every consumer of this manager and breaks the many suites that
// partial-mock fileUtils without it. Same reason `services/aiProvider.js` defers
// `localModelHealing`.

import { isAppleSilicon, isPortInUse } from '../lib/platform.js';
import { findCommandOnPath } from '../lib/processEnv.js';
import { runStreamingCommand } from '../lib/streamingSpawn.js';
import { execPm2, getAppStatusStrict, clearJlistCache, getSavedProcessNames } from './pm2.js';

export { MTPLX_APP };
// Compatibility export; provider cache discovery owns no daemon lifecycle state.
export { mtplxCachedModelIds } from './localCachedModels.js';

const PROBE_TIMEOUT_MS = 1500;
/**
 * How long `startMtplxServer` waits before returning.
 *
 * This is NOT how long MTPLX gets to come up — it is how long the HTTP request
 * blocks. MTPLX loads a multi-gigabyte MLX checkpoint before it binds, which
 * routinely outlasts any sane request timeout, so the wait exists only to catch
 * a server that dies immediately (an empty cache, a bad checkpoint). Still
 * loading after this returns `online: false` and the status poll takes over —
 * exactly what `llamaServerManager` does with its own four-second beat.
 *
 * Mutable only through the test seam below, so a suite asserting the
 * still-loading path doesn't have to sit through the real budget.
 */
let startupWaitMs = 8_000;
/**
 * Beat between startup polls. Mutable through the same test seam as the budget
 * above: a suite that pins the endpoint probe unreachable pays this delay in
 * FULL on every lifecycle test, which is what made this file the slowest server
 * suite — a 1.5s floor per start, several starts per relaunch case, close enough
 * to the 10s per-test timeout to flake under CI load.
 */
let startupPollMs = 1500;
/**
 * How long to let the stopped daemon give the port up before starting the next
 * one. Six times llama.cpp's five seconds, because what is being torn down is a
 * multi-gigabyte MLX process and giving up early is the expensive mistake here:
 * `startMtplxServer` refuses a bound port BEFORE it launches anything, so a
 * relaunch that jumps the gun fails the tuned start AND the restore behind it,
 * leaving the install's `mtplx` provider down over a race rather than over
 * anything the user tuned. Mutable through the test seam below, like the other
 * budgets here, so a suite exercising a still-held port does not sit it out.
 */
let portReleaseTimeoutMs = 30_000;
/**
 * How long a TUNING relaunch waits for the new launch line to answer before
 * calling it wedged and putting the previous configuration back.
 *
 * Far longer than `startupWaitMs`, and deliberately longer than llama.cpp's
 * equivalent: `startMtplxServer` only proves the process survived its first
 * seconds, and a cold multi-gigabyte MLX checkpoint routinely takes minutes to
 * load. Judging it at eight seconds would restore the previous configuration
 * over a server that was merely still loading. Mutable through the test seam so
 * a suite asserting the never-answered path doesn't sit through the real budget.
 */
let relaunchReadyTimeoutMs = 300_000;
// Readiness poll cadence. Kept separate from the budget so lifecycle tests can
// exercise retries and timer-driven PM2 transitions without one real second per
// loop.
let relaunchPollMs = 1000;
/** Package-manager installs routinely run for minutes. */
const INSTALL_TIMEOUT_MS = 20 * 60 * 1000;

/**
 * The default loopback endpoint the shipped MTPLX provider presets point at.
 * Read from the runtime table rather than re-declared, so the launcher and the
 * readiness probe cannot drift onto different ports.
 */
const DEFAULT_HOST = '127.0.0.1';
const DEFAULT_PORT = Number(localEndpointPort(LOCAL_RUNTIMES.mtplx.defaultBaseUrl)) || 8000;

/** MTPLX is an Apple-Silicon MLX runtime — nothing to install anywhere else. */
export const MTPLX_UNSUPPORTED_REASON = 'MTPLX runs only on macOS with Apple Silicon.';

/**
 * Why an installed MTPLX still cannot be started. Weights are a multi-gigabyte
 * download and stay the user's decision — so a start never fetches them
 * silently, and this names the in-app control that does instead of a terminal
 * command (`services/mtplxModelManager.js`).
 */
const MTPLX_NO_MODEL_ERROR = 'MTPLX has no model weights cached, so its server exits before it binds a port. Use "Download default checkpoint" on the MTPLX card (or search for another MTP model there) to fetch one — a multi-gigabyte download PortOS will not start without you asking — then start MTPLX again.';

const MTPLX_RUNTIME_BOOTSTRAP_ERROR = 'MTPLX\'s own Python runtime failed to bootstrap because Homebrew\'s Python does not provide a working `ensurepip`. Try `brew reinstall [email protected]` (or `brew reinstall --build-from-source youssofal/mtplx/mtplx` to force a rebuild against a working interpreter), then start MTPLX again.';

/**
 * Why an installed MTPLX still cannot be started, one layer below a missing
 * checkpoint: the Homebrew wrapper on PATH has not built its Python venv yet, so
 * its first act would be a multi-hundred-megabyte pip install — inside an
 * eight-second startup window, reported afterwards as a crashed daemon. Same
 * contract as MTPLX_NO_MODEL_ERROR: a large download is a button the user
 * presses, never something a start does behind them.
 */
const MTPLX_RUNTIME_NOT_BOOTSTRAPPED_ERROR = `MTPLX's Python runtime has not been downloaded yet — Homebrew installs a wrapper that fetches it (several hundred megabytes) on first run, and PortOS will not do that inside a server start. Use "Download MTPLX runtime" on the MTPLX card to run it as its own step, then start MTPLX again.`;

/**
 * Keep MTPLX startup logs raw while scoping each diagnosis to the current
 * launch. PM2 appends to its configured files across process deletion, so
 * stable, app-specific paths are truncated before every start.
 */
const DEFAULT_MTPLX_LOG_FILES = {
  stdout: join(tmpdir(), 'portos-mtplx-out.log'),
  stderr: join(tmpdir(), 'portos-mtplx-error.log'),
};

let mtplxLogFiles = DEFAULT_MTPLX_LOG_FILES;

const resetMtplxLogs = async () => {
  const results = await Promise.all(
    Object.entries(mtplxLogFiles).map(([stream, path]) => (
      writeFile(path, '')
        .then(() => true)
        .catch((error) => {
          console.error(`❌ MTPLX: could not reset ${stream} startup log (${error?.code || 'unknown'}); bootstrap diagnosis disabled for this launch`);
          return false;
        })
    )),
  );
  return results.every(Boolean);
};

const isMtplxRuntimeBootstrapFailure = (output) => {
  const text = String(output ?? '');
  return /runtime is not installed|bootstrapping with pip/i.test(text)
    && /ensurepip/i.test(text)
    && /returned non-zero exit status|no module named ensurepip|ensurepip is not available|failed to bootstrap/i.test(text);
};

let currentConfig = null;
let lastExitError = null;

const probeEndpoint = async (endpoint) =>
  (await probeOpenAiModels(endpoint, { timeoutMs: PROBE_TIMEOUT_MS })).reachable;

const endpointFor = (config) =>
  `http://${DEFAULT_HOST}:${config?.port ?? DEFAULT_PORT}/v1`;

// `settings.js` stays a dynamic import here too — see the note at the top of
// this file on why it is never imported statically.
const readSettings = () => import('./settings.js').then((m) => m.getSettings()).catch(() => null);

const daemon = createDaemonWatcher({
  appName: MTPLX_APP,
  defaultHost: DEFAULT_HOST,
  defaultPort: DEFAULT_PORT,
  endpointFor,
  parseConfigFromArgs,
  probe: probeEndpoint,
  isPortInUse: (...args) => isPortInUse(...args),
  sleep,
  getConfig: () => currentConfig,
  setConfig: (config) => { currentConfig = config; },
  getLastExitError: () => lastExitError,
  getAppStatus: (...args) => getAppStatusStrict(...args),
  getSavedProcessNames: (...args) => getSavedProcessNames(...args),
  execPm2: (...args) => execPm2(...args),
  getPortReleaseTimeoutMs: () => portReleaseTimeoutMs,
});
const appendLog = daemon.appendLog;

/**
 * The shared on-demand wake mechanism (#8105) — idle-window/keep-loaded reads,
 * the relaunch readiness poll (with the PM2 exit-log tail), and the numeric
 * port-match arm `isMtplxProvider` layers its `localRuntimeKind` check on top
 * of. What a launch line MEANS stays here, in `resolveLaunch`: the precedence
 * is the live process's recovered config, then the launch line last saved on
 * the MTPLX card (now including `tuning`, so an idle stop no longer forgets
 * an assessment relaunch's tuning), then the daemon's own default.
 */
const onDemand = createOnDemandDaemon({
  appName: MTPLX_APP,
  label: 'MTPLX',
  emoji: '🚄',
  readSection: () => readSettings().then((settings) => settings?.localLlm?.mtplx ?? null),
  endpointFor,
  probe: probeEndpoint,
  start: (config) => launchMtplxServer(config),
  stop: () => stopMtplxServer(),
  getStatusStrict: () => getAppStatusStrict(MTPLX_APP),
  clearStatusCache: () => clearJlistCache(),
  exitTail: createPm2ExitTail({ appName: MTPLX_APP, execPm2: (...args) => execPm2(...args), appendLog }),
  resolveLaunch: (current, saved) => ({
    port: current?.port ?? saved.port ?? DEFAULT_PORT,
    model: current?.model ?? saved.model ?? null,
    tuning: current?.tuning ?? normalizeTuning('mtplx', saved.raw?.tuning),
  }),
  getConfig: () => currentConfig,
  parseConfigFromArgs,
  sleep,
  getRelaunchReadyTimeoutMs: () => relaunchReadyTimeoutMs,
  getRelaunchPollMs: () => relaunchPollMs,
});
onDemand.registerIdle();

/**
 * Reconstructs the launch config from PM2 process args when PortOS restarted
 * while the PM2 process stayed online — same recovery `llamaServerManager` does,
 * so a restart doesn't report a server it started as "external".
 */
function parseConfigFromArgs(args) {
  if (!args) return null;
  const list = Array.isArray(args) ? args : String(args).split(' ');
  if (!list.includes('serve')) return null;
  const port = pm2ArgValue(list, '--port');
  return {
    port: port ? Number(port) : DEFAULT_PORT,
    // Absent means MTPLX was started on its OWN default checkpoint — don't
    // invent a repo id the launch line never carried.
    model: pm2ArgValue(list, '--model'),
    tuning: parseTuningFromArgs(list),
  };
}

/**
 * Recover the tuning knobs from a launch line PortOS is re-adopting.
 *
 * Walks the CATALOG rather than the argv, so a flag PortOS no longer declares
 * cannot come back as a knob, and every value goes through `normalizeTuning` —
 * argv holds strings, while the status payload and the restore path both expect
 * the coerced shape. Empty when nothing was tuned, which reads the same as a
 * server started before tuning existed.
 *
 * This is a READ of a running process, so a value the catalog cannot represent
 * is DROPPED, never clamped. `normalizeTuning` clamps, which is right for user
 * input (the form offered the range) and wrong here: installs upgrade on their
 * own schedule, so a daemon started before a release that tightened a bound is
 * exactly the case this hits — and clamping would report `--depth 4` on the card
 * and hand `--depth 4` to a restore, while the process is demonstrably running
 * `--depth 8`. Absent at least means "PortOS cannot name this", which is true.
 */
function parseTuningFromArgs(list) {
  const raw = {};
  for (const spec of tuningSpecsFor('mtplx')) {
    if (!spec.cli) continue;
    // `launchArgs` renders a boolean as a BARE flag, so its inverse is presence,
    // not the next token — reading one would recover the NEXT flag's name as
    // this knob's value. No MTPLX knob is boolean today; this is here so adding
    // one cannot silently break re-adoption.
    if (spec.type === 'boolean') {
      if (list.includes(spec.cli)) raw[spec.id] = true;
      continue;
    }
    const value = pm2ArgValue(list, spec.cli);
    if (value === null) continue;
    // Outside the declared range: drop rather than let `normalizeTuning` clamp
    // it into a number the daemon is not running with. (An enum value the
    // catalog no longer lists is already dropped there, for the same reason.)
    if (spec.type === 'number' && !withinDeclaredRange(spec, value)) continue;
    raw[spec.id] = value;
  }
  return normalizeTuning('mtplx', raw);
}

const withinDeclaredRange = (spec, raw) => {
  const num = Number(raw);
  return Number.isFinite(num) && num >= (spec.min ?? -Infinity) && num <= (spec.max ?? Infinity);
};

const waitForPortRelease = daemon.waitForPortRelease;

// `waitForRelaunchedEndpoint` (the death-detecting readiness poll) and its
// `exitTail` PM2 log-tail helper now live in `lib/managedDaemon.js` as
// `onDemand.waitForReady` — shared with `slotstreamServerManager.js`, whose
// copy of the same loop reported only a bare PM2 status word (#8105).

/** Resolve the `mtplx` executable on the child-process PATH. */
const resolveMtplxBinary = () => findCommandOnPath('mtplx');

/** Just the base URL MTPLX is serving on — no endpoint probe, no PM2 log fetch. */
export async function getMtplxServerEndpoint() {
  if (!currentConfig) {
    await daemon.readLaunch();
  }
  return daemon.endpoint();
}

/**
 * Current MTPLX state: binary availability, runtime readiness, process state,
 * endpoint, config, logs, and what is in its model cache.
 *
 * The cache listing (`mtplx models --json`) reads a local directory and pulls
 * nothing over the network — but it does so by INVOKING `mtplx`, and the
 * Homebrew `mtplx` is a wrapper whose first invocation on a given version
 * bootstraps a several-hundred-megabyte Python venv (`lib/mtplxRuntime.js`).
 * On a host where that has not happened yet, this poll WAS the bootstrap: it
 * outran the 30s cache-query budget, got killed, reported `cacheError`, and the
 * next poll started the download over. So the cache is only read once
 * `runtimeReady` says the wrapper will exec straight through to the real
 * binary; until then `cachedModels` stays empty with `cacheError: null` —
 * "not read", never "read and empty".
 */
export async function getMtplxServerStatus() {
  const binaryPath = resolveMtplxBinary();
  const installed = Boolean(binaryPath);
  // The platform gate is about what PortOS may INSTALL, never about what is
  // already there: a binary on PATH (or a server answering) is proof enough that
  // this host runs it, and reporting "macOS only" over a working install would
  // be a false negative.
  const supported = installed || isAppleSilicon();

  const base = await daemon.getStatusBase({ installed });

  // Reading the wrapper, never running it — see `lib/mtplxRuntime.js`.
  const runtimeReady = installed ? (await describeMtplxRuntime(binaryPath)).ready : false;

  // `models: null` means the cache could NOT be read — deliberately not the same
  // as `[]` (read, and empty). Only the latter blocks a start.
  const cache = installed && runtimeReady ? await listMtplxCachedModels() : { models: null, error: null };

  return {
    ...base,
    supported,
    unsupportedReason: supported ? null : MTPLX_UNSUPPORTED_REASON,
    // Installed ≠ runnable: the Homebrew binary is a wrapper that downloads the
    // real runtime on first use. The card needs the two apart so it offers the
    // download instead of a Start button that is guaranteed to fail.
    runtimeReady,
    // The tuning flags the running daemon was LAUNCHED with, rendered by the
    // catalog that owns the transport rather than re-derived in the UI. A
    // measured assessment relaunches this daemon and leaves its flags on, so a
    // status that reported only the model would show a server as plain
    // "running" while it serves with, say, MTP decoding switched off. Empty for
    // an untuned server, and for one PortOS does not manage — it cannot read
    // another process's launch line.
    tuningFlags: base.managed === true ? launchArgs('mtplx', currentConfig?.tuning) : [],
    idleMinutes: await onDemand.idleMinutes(),
    keepLoaded: await onDemand.keepLoaded(),
    // What a lazy start will launch on, so the card's fields show the saved
    // choice rather than resetting to "Auto" on every page load.
    launch: await onDemand.savedLaunch().then(({ model, port }) => ({ model, port })),
    cachedModels: (cache.models || []).map((m) => m?.repo_id).filter(Boolean),
    // The same cache, with what the manage-checkpoints UI needs to let a user
    // free the disk: how big each pack is, and whether it is actually servable
    // (an interrupted pull leaves a directory that lists but cannot load).
    cachedModelRows: (cache.models || [])
      .filter((m) => m?.repo_id)
      .map((m) => ({
        repo: m.repo_id,
        sizeBytes: Number.isFinite(Number(m.size_bytes)) ? Number(m.size_bytes) : null,
        hasRuntimeContract: m.has_runtime_contract === true,
        // `validation.ok` absent means an older `mtplx models` that did not
        // report one — treat that as usable rather than flagging every row.
        valid: m.validation?.ok !== false,
      })),
    cacheError: cache.error,
  };
}

/**
 * Run the Homebrew wrapper's deferred bootstrap HERE, where the user pressed a
 * button and the budget fits.
 *
 * `brew install` finishes with only a shell shim on PATH; the actual MTPLX —
 * fastapi, huggingface_hub, numpy/scipy, the MLX stack — is downloaded by the
 * wrapper's first invocation (`lib/mtplxRuntime.js`). Left alone, that lands on
 * whatever runs `mtplx` next: a 30s status poll or an 8s-judged PM2 start,
 * neither of which can survive it, both of which misreport it. `mtplx --version`
 * is the cheapest invocation that trips the same guard, and it streams into the
 * install modal under the 20-minute install budget.
 *
 * A failure here FAILS the install: returning success would leave a card
 * reporting `installed: true` for something that cannot serve a request.
 *
 * Skipped when the runtime is already present — a re-run of the install (the
 * card's "Download MTPLX runtime" button drives this same flow) then costs a
 * `brew install` no-op rather than another invocation.
 */
async function warmMtplxRuntime(emit) {
  const binaryPath = resolveMtplxBinary();
  if (!binaryPath) return;
  const runtime = await describeMtplxRuntime(binaryPath);
  if (runtime.ready) return;

  emit('MTPLX bootstraps its own Python runtime on first run — downloading it now. This can take several minutes.');
  const result = await runStreamingCommand(binaryPath, ['--version'], emit, { timeoutMs: INSTALL_TIMEOUT_MS });
  if (!result.success) {
    throw new ServerError(`MTPLX installed, but its Python runtime failed to bootstrap: ${result.error}`, { status: 500, code: 'MTPLX_RUNTIME_BOOTSTRAP_FAILED' });
  }
}

/**
 * Install MTPLX from upstream's Homebrew tap, falling back to pip on a host
 * without Homebrew. Both install the same `mtplx` binary, and neither runs the
 * optional privileged fan-control helper.
 */
export async function installMtplx({ onProgress = () => {} } = {}) {
  const emit = (message) => onProgress({ event: 'progress', message });
  if (!isAppleSilicon()) {
    throw new ServerError(MTPLX_UNSUPPORTED_REASON, { status: 400, code: 'MTPLX_UNSUPPORTED_PLATFORM' });
  }
  if (await commandExists('brew', ['--version'])) {
    emit('Installing MTPLX via Homebrew (youssofal/mtplx/mtplx)…');
    const result = await runStreamingCommand('brew', ['install', 'youssofal/mtplx/mtplx'], emit, { timeoutMs: INSTALL_TIMEOUT_MS });
    if (!result.success) throw new ServerError(`MTPLX install failed: ${result.error}`, { status: 500 });
    await warmMtplxRuntime(emit);
    return { success: true, message: 'MTPLX installed' };
  }
  if (await commandExists('python3', ['--version'])) {
    // No warm-up here: pip installs the REAL package, not a wrapper, so there is
    // no deferred runtime left to fetch.
    emit('Homebrew was not found — installing MTPLX with pip instead…');
    const result = await runStreamingCommand('python3', ['-m', 'pip', 'install', '--upgrade', 'mtplx'], emit, { timeoutMs: INSTALL_TIMEOUT_MS });
    if (!result.success) throw new ServerError(`MTPLX install failed: ${result.error}`, { status: 500 });
    return { success: true, message: 'MTPLX installed' };
  }
  throw new ServerError('Neither Homebrew nor python3 is available. Install Homebrew from https://brew.sh, then try again.', { status: 400 });
}

/**
 * Resolve which cached checkpoint `mtplx serve` should be started on.
 *
 * `mtplx serve` defaults `--model` to ONE hard-coded repo id and exits 1 before
 * binding when that repo is not cached — even on a machine holding a different
 * MTP model that would have served fine. So ask the cache first and name what is
 * actually there.
 *
 * Returns `{ model }` (possibly `null` = fall through to MTPLX's own default
 * because the cache could not be READ) or `{ error }`.
 */
async function resolveStartModel(requested, emit) {
  if (requested) return { model: requested };
  const cache = await listMtplxCachedModels();
  if (cache.models === null) {
    emit(`Could not read MTPLX's model cache (${cache.error}) — starting with its default model.`);
    return { model: null };
  }
  if (cache.models.length === 0) return { error: MTPLX_NO_MODEL_ERROR };
  const model = pickMtplxCachedModel(cache.models);
  if (!model) {
    const count = cache.models.length;
    return { error: `MTPLX's cache holds ${count} model${count === 1 ? '' : 's'}, but none passed its own file check — an interrupted download leaves a partial pack behind. Remove it on the MTPLX card and download the checkpoint again, then try again.` };
  }
  return { model };
}

/**
 * Start `mtplx serve` under PM2, queued behind any in-flight on-demand wake,
 * explicit start, or tuning relaunch (`onDemand.runStart`, #12723) so it cannot
 * `pm2 delete` a launch that is still loading.
 *
 * @param {Parameters<typeof launchMtplxServer>[0]} [options]
 */
export const startMtplxServer = (options = {}) => onDemand.runStart(() => launchMtplxServer(options));

/**
 * The unserialized start behind `startMtplxServer`. Only code already holding
 * the start slot calls it directly: the on-demand wake and the tuning relaunch.
 *
 * `waitMs` overrides how long this blocks before handing back `online: false`
 * for a server that is still loading. The LLMs-page launcher takes the short
 * default and lets its status poll finish the story; the provider-readiness
 * checklist passes its own longer budget, because that flow's contract is "the
 * endpoint answers when this returns".
 *
 * There is deliberately no `host` option: MTPLX is a loopback daemon
 * (`docs/features/mtplx.md`), every shipped provider preset points at
 * 127.0.0.1, and accepting a host PortOS never puts on the launch line would
 * report an endpoint the server is not bound to.
 *
 * `tuning` is a knob set from `lib/localModelTuning.js` (`TUNING_SPECS.mtplx`),
 * rendered to `mtplx serve` flags by the catalog rather than by a flag map here,
 * so a knob cannot reach the launch line without also being declared, described,
 * and offered in the form. It is kept on `currentConfig` because that is what a
 * failed tuning relaunch puts back: without it the restore would bring the
 * daemon up untuned and call that "the previous configuration".
 *
 * @param {{port?: number, model?: string, tuning?: object, waitMs?: number, onProgress?: (line: string) => void}} options
 */
async function launchMtplxServer(options = {}) {
  const { port = DEFAULT_PORT, model: requestedModel = null, tuning = null, waitMs, onProgress = () => {} } = options;
  const normalizedTuning = normalizeTuning('mtplx', tuning);
  const emit = (line) => { appendLog(line); onProgress(line); };

  const binaryPath = resolveMtplxBinary();
  if (!binaryPath) {
    throw new ServerError(
      'The `mtplx` binary was not found on PATH. Install it from this card (Homebrew tap, or pip as a fallback), then try again.',
      { status: 400, code: 'MTPLX_NOT_INSTALLED' }
    );
  }

  // A start never performs a large silent download — the same limit that keeps
  // it off model weights. Launching the wrapper before its venv exists would
  // hand PM2 a process whose first act is a pip install, judged (and killed) by
  // an eight-second startup window and reported as a crashed daemon.
  if (!(await describeMtplxRuntime(binaryPath)).ready) {
    throw new ServerError(MTPLX_RUNTIME_NOT_BOOTSTRAPPED_ERROR, { status: 400, code: 'MTPLX_RUNTIME_NOT_BOOTSTRAPPED' });
  }

  const pm2Status = await getAppStatusStrict(MTPLX_APP);
  if (pm2Status && pm2Status.status === 'online') {
    throw new ServerError(`MTPLX is already running with PID ${pm2Status.pid}`, { status: 409 });
  }

  const endpoint = `http://${DEFAULT_HOST}:${port}/v1`;
  if (await probeEndpoint(endpoint)) {
    throw new ServerError(`Port ${port} is already in use by an active server at ${endpoint}`, { status: 409 });
  }
  if (await isPortInUse(port)) {
    throw new ServerError(
      `Port ${port} is already in use on ${DEFAULT_HOST}. Point MTPLX at a different port before starting it.`,
      { status: 409, code: 'MTPLX_PORT_IN_USE' }
    );
  }

  daemon.resetLogs();
  lastExitError = null;

  const resolved = await resolveStartModel(requestedModel, emit);
  if (resolved.error) throw new ServerError(resolved.error, { status: 400, code: 'MTPLX_NO_CACHED_MODEL' });
  const model = resolved.model;
  if (model) emit(`Serving the cached MTPLX model ${model}.`);

  // `mtplx start` is interactive (it prompts for a model); `serve` is the
  // API-only server, which is the half PortOS talks to. The daemon must bind
  // where the PROVIDER points — a user who moved MTPLX to 8010 would otherwise
  // get a second server on 8000 that nothing talks to.
  const tuningArgs = launchArgs('mtplx', normalizedTuning);
  const args = ['serve', '--port', String(port), ...(model ? ['--model', model] : []), ...tuningArgs];
  currentConfig = { port, model, tuning: normalizedTuning };
  appendLog(`Starting: mtplx ${args.join(' ')}`);

  // Delete any stale PM2 entry so our own previous instance doesn't count as a collision.
  await execPm2(['delete', MTPLX_APP]).catch(() => {});
  clearJlistCache();

  console.log(`🚄 MTPLX starting on ${DEFAULT_HOST}:${port}${model ? ` (model ${model})` : ' (MTPLX default model)'}${tuningArgs.length ? ` with ${tuningArgs.join(' ')}` : ''}`);
  const logsReset = await resetMtplxLogs();
  await execPm2([
    'start', binaryPath,
    '--name', MTPLX_APP,
    '--interpreter', 'none',
    '--no-autorestart',
    '--output', mtplxLogFiles.stdout,
    '--error', mtplxLogFiles.stderr,
    '--',
    ...args,
  ]);
  clearJlistCache();

  const deadline = Date.now() + (Number.isFinite(waitMs) ? waitMs : startupWaitMs);
  let online = false;
  let currentProc = null;
  while (Date.now() < deadline) {
    await sleep(startupPollMs);
    clearJlistCache();
    currentProc = await getAppStatusStrict(MTPLX_APP);
    if (currentProc && ['errored', 'stopped', 'not_found'].includes(currentProc.status)) break;
    online = await probeEndpoint(endpoint);
    if (online) break;
  }

  if (currentProc && ['errored', 'stopped', 'not_found'].includes(currentProc.status)) {
    const pm2Logs = await execPm2(['logs', MTPLX_APP, '--nostream', '--lines', '15']).catch(() => null);
    const lines = `${pm2Logs?.stderr || pm2Logs?.stdout || ''}`.split('\n').map((l) => l.trimEnd()).filter(Boolean);
    const currentLines = logsReset ? lines : [];
    for (const line of currentLines) appendLog(line);
    const tail = (currentLines.length ? currentLines : daemon.snapshotLogs()).slice(-4).join(' | ');
    lastExitError = `PM2 status: ${currentProc.status}`;
    const message = logsReset && isMtplxRuntimeBootstrapFailure(currentLines.join('\n'))
      ? `${MTPLX_RUNTIME_BOOTSTRAP_ERROR}${tail ? ` Last output: ${tail}` : ''}`
      : `MTPLX exited immediately (${lastExitError}).${tail ? ` Last output: ${tail}` : ''}`;

    await execPm2(['delete', MTPLX_APP]).catch(() => {});
    clearJlistCache();
    currentConfig = null;
    throw new ServerError(
      message,
      { status: 500, code: 'MTPLX_EXITED' }
    );
  }

  // Persist the launch line every successful start used — including `tuning` —
  // so the next on-demand restart (after an idle stop) replays exactly this
  // configuration instead of falling back to a resolved default. Additive
  // field: an older install's saved launch simply has no `tuning` key, which
  // `onDemand`'s `resolveLaunch` reads as untuned. Fixes the bug Slotstream's
  // own `persistLaunchConfig` did not have: MTPLX used to forget a tuning
  // applied through an assessment relaunch the moment the idle reaper stopped
  // it (#8105).
  await persistDaemonLaunchConfig({ key: 'mtplx', label: 'MTPLX', launch: { port, model, tuning: normalizedTuning } });

  const finalProc = await getAppStatusStrict(MTPLX_APP);
  return {
    success: true,
    running: true,
    managed: true,
    pid: finalProc?.pid || null,
    endpoint,
    online,
    config: currentConfig,
  };
}

/** Stop the managed MTPLX process. */
export async function stopMtplxServer() {
  const pm2Status = await getAppStatusStrict(MTPLX_APP);
  const isManaged = Boolean(pm2Status && pm2Status.status === 'online');

  if (!isManaged) {
    const endpoint = endpointFor(currentConfig);
    if (await probeEndpoint(endpoint)) {
      return {
        success: false,
        message: `An external process is listening on ${endpoint}. It was not started by PortOS and cannot be stopped here.`,
      };
    }
    return { success: true, message: 'MTPLX is not running' };
  }

  appendLog(`Stopping ${MTPLX_APP}`);
  await execPm2(['delete', MTPLX_APP]).catch((err) => {
    throw new ServerError(`Failed to stop MTPLX: ${err.message}`, { status: 500 });
  });
  clearJlistCache();
  currentConfig = null;
  return { success: true, message: 'MTPLX stopped' };
}

/**
 * Relaunch `mtplx serve` with a different tuning, keeping the checkpoint and
 * port it is already serving.
 *
 * This is MTPLX's half of the measured-assessment feature: a sweep across MTP
 * depths or KV-quantization modes is only possible if something can put those
 * flags on the launch line between runs. It returns the same result shape as
 * `llamaServerManager.relaunchLlamaServerWithTuning`, because the assessment
 * runner treats every runtime's applier identically.
 *
 * **It is NOT a line-for-line copy of that function, and syncing the two from
 * this comment would reintroduce bugs.** Two deliberate divergences:
 *   - llama MERGES the request onto the flags already set (`{...previous,
 *     ...tuning}`); this REPLACES them, so the launch line is exactly the knob
 *     set the record is labelled with. See the note at the merge point below.
 *   - llama returns as soon as the restore's `startLlamaServer` returns; this
 *     waits for the restored daemon to actually answer. See `restorePrevious`.
 *
 * It refuses rather than guesses in the four cases where it cannot know what to
 * relaunch, or must not:
 *   - nothing is running, so there is no checkpoint to reuse;
 *   - PM2 could not be read, so PortOS cannot prove it owns the process;
 *   - something IS listening but PortOS did not start it, so stopping it would
 *     kill a process the user owns;
 *   - the launch line names no `--model`, so the running server is on MTPLX's
 *     own hard-coded default. Restarting with `model: null` sends the request
 *     back through `resolveStartModel`, which now picks from the cache — a
 *     DIFFERENT checkpoint from the one being measured, filed under the model
 *     id of the one that was.
 *
 * Every one of those returns `{ applied: false, reason }` instead of throwing:
 * the caller can still measure whatever is actually serving and record that the
 * tuning was NOT applied, which is far more useful than failing the whole run.
 *
 * The restore path is not optional. A tuning sweep EXPECTS launch lines that do
 * not work — a `--context-window` past what unified memory holds, a `--depth`
 * the checkpoint's MTP sidecar will not draft. `mtplx serve` exits before it
 * binds and `startMtplxServer` throws; leaving the daemon down would break the
 * whole install's `mtplx` provider, not just the measurement. So the previous
 * configuration goes back up before the failure is reported.
 *
 * @param {object} tuning launch knobs from `lib/localModelTuning.js`
 * @returns {Promise<{applied: boolean, reason: string|null, config: object|null}>}
 */
export async function relaunchMtplxServerWithTuning(tuning = {}) {
  const normalized = normalizeTuning('mtplx', tuning);
  const knobs = Object.entries(normalized);
  if (knobs.length === 0) {
    return { applied: false, reason: 'no launch knobs were requested', config: currentConfig };
  }
  // Gated on the RENDERED flags, not the knob count: a knob can normalize to a
  // value that renders nothing (a boolean set to false — a CLI has no spelling
  // for "explicitly off"). Relaunching on that would run daemon defaults while
  // the caller records `tuningApplied: true`, which is the un-applied-but-claimed
  // reading this whole path exists to prevent.
  if (launchArgs('mtplx', normalized).length === 0) {
    return { applied: false, reason: 'that tuning renders no `mtplx serve` flag, so there is nothing to relaunch with', config: currentConfig };
  }

  // The status read sits inside the slot too, so a relaunch queued behind a
  // wake reads the configuration that wake brought up (#12723).
  return onDemand.runStart(() => relaunchWithTuning(normalized, knobs));
}

async function relaunchWithTuning(normalized, knobs) {
  const status = await getMtplxServerStatus();
  if (!status.running) {
    return { applied: false, reason: 'MTPLX is not running, so PortOS has no checkpoint to relaunch with', config: null };
  }
  // `null` is "PM2 could not be read", NOT "someone else started it". Collapsing
  // the two would tell a user their own managed daemon is external — a
  // misdiagnosis that points them at the wrong fix. Refusing is still right:
  // PortOS must not stop a process it cannot prove it owns.
  if (status.managed === null) {
    return {
      applied: false,
      reason: 'PortOS could not read PM2, so it cannot tell whether it owns this MTPLX server',
      config: status.config || null,
    };
  }
  if (!status.managed) {
    return {
      applied: false,
      reason: 'MTPLX was started outside PortOS — start it from the LLMs page to let PortOS apply tuning',
      config: status.config || null,
    };
  }
  if (!status.config?.model) {
    return {
      applied: false,
      reason: 'MTPLX is serving its own default checkpoint, which PortOS cannot name — restart it on a chosen checkpoint from the LLMs page to let PortOS apply tuning',
      config: status.config || null,
    };
  }

  const previous = status.config;
  // REPLACES the tuning on the launch line rather than merging onto it, so the
  // flags `mtplx serve` runs with are exactly the knob set the caller named —
  // and the assessment's `tuningKey`/`tuningLabel`, which describe only that
  // set, describe the whole configuration. Merging would leave the second run
  // of a sweep carrying the first run's flags while labelled with only its own,
  // which makes `compareTunings` rank two readings that were not what they say.
  // Replacing is exact here because an absent `mtplx serve` flag IS the daemon
  // default — the same sentinel contract `lib/localModelTuning.js` documents.
  const next = { ...previous, tuning: normalized };
  const port = next.port ?? DEFAULT_PORT;
  console.log(`🚄 MTPLX: relaunching to apply tuning (${knobs.map(([k, v]) => `${k}=${v}`).join(', ')})`);

  await stopMtplxServer();
  await waitForPortRelease(port);

  const started = await launchMtplxServer(next).catch((err) => ({ failure: err }));
  if (started.failure) {
    const reason = launchFailureReason(started.failure);
    console.error(`❌ MTPLX: tuning launch failed (${started.failure.message}) — restoring the previous configuration`);
    return { applied: false, reason, config: await restorePrevious(previous, reason) };
  }

  // `startMtplxServer` returning is not the same as the endpoint answering — it
  // waits only long enough to catch a server that dies immediately, and a cold
  // MLX checkpoint takes far longer than that to load. So `online: false` here
  // is "not ready YET", not "wedged"; give it the real readiness budget first.
  const ready = started.online ? { ready: true, reason: null } : await onDemand.waitForReady(started.endpoint);
  if (ready.ready) return { applied: true, reason: null, config: started.config };

  // Treat it exactly like a rejected launch line: put the previous configuration
  // back, so the install's mtplx provider is not left pointing at a process that
  // never serves. Without this the caller would go on to measure a dead endpoint
  // and record the timeouts as evidence for this tuning.
  console.error(`❌ MTPLX: ${ready.reason} — restoring the previous configuration`);
  await stopMtplxServer().catch(() => {});
  return { applied: false, reason: ready.reason, config: await restorePrevious(previous, ready.reason) };
}

/**
 * Why the tuned launch failed, in the caller's words.
 *
 * "MTPLX rejected that tuning" is only true when `mtplx serve` actually ran and
 * exited. `startMtplxServer` also throws from guards that fire BEFORE the
 * process starts — most realistically the port still being held by the daemon
 * that was just stopped — and reporting those as a rejected tuning sends the
 * user looking for a bad flag that was never even passed.
 */
const launchFailureReason = (err) => (err?.code === 'MTPLX_EXITED'
  ? `MTPLX rejected that tuning: ${err.message}`
  : `PortOS could not start MTPLX with that tuning: ${err?.message || 'relaunch failed'}`);

/**
 * Put the configuration MTPLX was serving before back up, and WAIT for it.
 *
 * Returning as soon as `startMtplxServer` does would hand back a config the
 * daemon is still minutes from honouring — and the caller goes straight on to
 * measure, so every sample times out and a junk `does-not-fit` record is stored
 * against the model. That record then counts as "assessed", dropping the model
 * out of the unassessed list and the sweep's `unmeasured` scope for good.
 *
 * `null` when it could not be brought back at all, which is the honest answer:
 * the caller reports the tuning as not applied either way, and a config here
 * would claim a daemon that is not running.
 */
async function restorePrevious(previous, failure) {
  await waitForPortRelease(previous.port ?? DEFAULT_PORT);
  const restored = await launchMtplxServer(previous).catch((err) => {
    console.error(`❌ MTPLX: could not restore the previous configuration: ${err.message}`);
    return null;
  });
  // `startMtplxServer` clears the log buffer and `lastExitError` for the server
  // it is about to launch — correct in general, but here it would erase the ONLY
  // trace of why the tuning failed. The card would then show a healthy daemon
  // with empty logs, and the reason would survive only inside the assessment
  // record. Put the failure back so the LLMs page can still explain it.
  if (failure) {
    lastExitError = failure;
    appendLog(`Tuning launch failed (${failure}) — restored the previous configuration.`);
  }
  if (!restored) return null;
  if (restored.online) return restored.config;
  const back = await onDemand.waitForReady(restored.endpoint);
  if (back.ready) return restored.config;
  console.error(`❌ MTPLX: the restored configuration ${back.reason}`);
  return null;
}

/** Clears in-memory state (used by test suites). */
export function _resetMtplxServerStateForTests({
  startupWait,
  startupPoll,
  portRelease,
  relaunchReadyTimeout,
  relaunchPoll,
  idleMinutes = 0,
  keepLoaded = null,
  logFiles,
} = {}) {
  onDemand.setIdleMinutesOverrideForTests(idleMinutes);
  onDemand.setKeepLoadedOverrideForTests(keepLoaded);
  mtplxLogFiles = logFiles ? {
    stdout: logFiles.stdout || DEFAULT_MTPLX_LOG_FILES.stdout,
    stderr: logFiles.stderr || DEFAULT_MTPLX_LOG_FILES.stderr,
  } : DEFAULT_MTPLX_LOG_FILES;
  currentConfig = null;
  daemon.resetLogs();
  lastExitError = null;
  // Restored to the production budget unless a suite asks for a shorter one.
  startupWaitMs = Number.isFinite(startupWait) ? startupWait : 8_000;
  startupPollMs = Number.isFinite(startupPoll) ? startupPoll : 1500;
  portReleaseTimeoutMs = Number.isFinite(portRelease) ? portRelease : 30_000;
  relaunchReadyTimeoutMs = Number.isFinite(relaunchReadyTimeout) ? relaunchReadyTimeout : 300_000;
  relaunchPollMs = Number.isFinite(relaunchPoll) ? relaunchPoll : 1000;
}

// Test hook for pinning
export function _setMtplxKeepLoadedOverrideForTests(val) {
  onDemand.setKeepLoadedOverrideForTests(val);
}

// =============================================================================
// IDLE STOP + LAZY START
// =============================================================================

/**
 * Why MTPLX is stopped on idle while llama-server is not.
 *
 * MTPLX holds its whole MLX checkpoint — 20GB is ordinary — for as long as the
 * process is up, and it has no way to put that down in place: its
 * `--retrieval-idle-timeout` unloads RETRIEVAL models (embedding/rerank) only,
 * never the main checkpoint, and `mtplx settings set` covers live tunables like
 * depth, not residency. Stopping the process is therefore the only lever, and
 * lazy-start is what makes stopping it safe to do automatically.
 *
 * `llamaServerManager` deliberately takes the other path (`--sleep-idle-seconds`),
 * because llama.cpp CAN unload in place — see `supportsSleepIdle` there.
 *
 * The idle-minutes/keep-loaded reads, the `registerIdleDaemon` registration,
 * `savedLaunchConfig`, `markMtplxUsed`, and `ensureMtplxRunning` all now live in
 * `onDemand` (`lib/managedDaemon.js`'s `createOnDemandDaemon`) — see #8105.
 */

/** Record real MTPLX traffic. Never call this from a status poll — see `markDaemonUsed`. */
export const markMtplxUsed = () => onDemand.markUsed();

/**
 * Bring MTPLX up if the idle reaper (or the user) stopped it, and mark it used
 * either way. See `onDemand.ensureRunning` for the shared mechanism; MTPLX's
 * own contribution is the `resolveLaunch` hook wired above, which carries
 * `tuning` alongside `port`/`model`.
 *
 * @returns {Promise<{ready: boolean, reason: string|null}>}
 */
export const ensureMtplxRunning = () => onDemand.ensureRunning();

/**
 * Is this provider served by the MTPLX daemon PortOS manages?
 *
 * Mirrors `ollamaManager.isOllamaProvider` so `services/aiProvider.js` gates both
 * local daemons the same way. Only a LOCAL endpoint counts: an MTPLX on a
 * tailnet peer is someone else's process, and neither starting nor idle-stopping
 * it is this install's business.
 */
export function isMtplxProvider(provider) {
  // API providers reach MTPLX directly, while OpenCode TUI providers carry
  // the same local endpoint and need the exact same lazy wake-up before the
  // TUI is spawned.
  if (!provider || !['api', 'tui'].includes(provider.type)) return false;
  if (!isLocalInstanceEndpoint(provider.endpoint)) return false;
  // `localRuntimeKind`, not `localBackendForProvider` — the latter only ever
  // answers 'ollama'/'lmstudio' (it maps those two catalog ports), so it reports
  // every MTPLX provider as having no local backend at all. This one call now
  // covers both marker-backed wrappers (`mtplxBacked`) and the shipped bare API
  // record (`id === 'mtplx'`) — #6466 collapsed the two hand-rolled checks that
  // used to live here into `localRuntimeKind` itself.
  if (localRuntimeKind(provider) === 'mtplx') return true;
  // ...and an endpoint-only provider aimed at the port THIS daemon is serving.
  // Deliberately compared against the live launch config rather than treating
  // :8000 as "must be MTPLX" — 8000 is a generic port, and claiming any local
  // server on it would lazily start MTPLX for someone else's API. This arm
  // stays here rather than moving into `localRuntimeKind`: it needs the
  // MANAGED daemon's live port, which only this module tracks, and reading it
  // from `localProviderRuntime.js` (a side-effect-free module every readiness
  // check imports) would be a circular import back to this one.
  //
  // `onDemand.servesPort` compares numerically (#8105) — `localEndpointPort`
  // returns a STRING, and comparing that with `===` against the numeric
  // `managedPort` was permanently false, so this arm never matched.
  return onDemand.servesPort(provider, currentConfig?.port);
}

/**
 * The `ensureProviderReady` shape `services/aiProvider.js` expects, for MTPLX.
 *
 * This is the one call site that both refreshes the idle clock and lazily
 * restarts — deliberately on the INFERENCE path rather than in
 * `getMtplxServerEndpoint`, which status polls call every few seconds and which
 * would therefore keep a 20GB checkpoint resident for a UI nobody is watching.
 */
export async function ensureMtplxProviderReady(provider) {
  if (!isMtplxProvider(provider)) return { success: true };
  const { ready, reason } = await ensureMtplxRunning();
  return ready ? { success: true } : { success: false, error: reason };
}