Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

26.5 KB
Raw Blame
import { invalidateModelObservation } from './modelObservation.js';
/**
 * "Is everything this provider needs actually installed and running?"
 *
 * The Providers page could already tell you whether the CLI binary a provider
 * shells out to exists (`providerRuntimeInstaller.js`). For a provider backed by
 * a LOCAL daemon that is only half the story: `opencode` can be perfectly
 * installed and the run still ends at
 *
 *     Cannot connect to API: Unable to connect. Is the computer able to access the url
 *
 * because `llama-server` was never installed, or was installed but never
 * started, or is running but serving a different model alias than the provider
 * asks for. Those are three distinct fixes, and none of them was visible from
 * the provider card — the user had to read a failed agent transcript to learn
 * that a second piece of software (and a multi-gigabyte model download) was
 * still missing.
 *
 * This module turns that into a per-provider requirements checklist:
 *
 *   1. runtime  — the daemon's binary is on PortOS's PATH (or something is
 *                 already answering, which proves it another way)
 *   2. server   — the endpoint the provider points at answers `GET /v1/models`
 *   3. model    — the provider's default model is one the endpoint serves
 *   4. catalog  — every OTHER model the provider offers is one the endpoint
 *                 serves, so a task or pipeline stage cannot pin an id that
 *                 only fails once the agent has already spawned
 *
 * Checks are reported in fix order, each with what to do about it. Only
 * local-daemon providers get a report at all (`localProviderRuntime.js` decides
 * which those are); everything else returns `null` and renders nothing. That
 * deliberately excludes a provider pointed at ANOTHER machine's OpenAI-compatible
 * server — an external endpoint is somebody else's install to run, so PortOS
 * neither probes this host for it nor offers to start it here.
 *
 * No LLM call is ever made here — `GET /v1/models` is a listing, so this is safe
 * to poll from a settings page under the no-cold-bootstrap policy in AGENTS.md.
 */

import { localRuntimeForProvider } from '../lib/localProviderRuntime.js';
import { expandPageToken, getNavPageForPath } from '../lib/navManifest.js';
import { bareLocalModelId } from '../lib/providerModels.js';
import { probeOpenAiModelsCached, resetOpenAiModelsProbeCache } from '../lib/openAiModelsProbeCache.js';
import { aliasServedContextWindows } from '../lib/providerContextWindows.js';
import { managedOllamaContextLength } from '../lib/ollamaContext.js';
import { findCommandOnPath } from '../lib/processEnv.js';
import { actionCovers, describeRuntimeSetup, readRuntimeWeights, weightsBlockStart } from './localRuntimeSetup.js';
import { isAppInstalled as isLmStudioAppInstalled } from './lmStudioManager.js';

/**
 * Same 60s TTL and reasoning as `providerRuntimeInstaller.js`'s status cache:
 * availability changes only when someone installs or removes a binary, and
 * `findCommandOnPath` is a SYNCHRONOUS PATH walk (a stat per directory), so an
 * uncached miss blocks the event loop on every 20s poll from every open tab.
 */
const BINARY_TTL_MS = 60_000;

/**
 * A `mtplx models --json` listing walks the cache directories to size them, so
 * it is far too expensive to repeat on every 20s page poll — and the answer
 * only changes when a `mtplx pull` finishes, which is minutes of downloading
 * away. Longer than the probe TTL for exactly that reason.
 */
const WEIGHTS_TTL_MS = 60_000;

// command → { at, path }
const binaryCache = new Map();
// runtime kind → { at, promise } — the local model cache behind a daemon that
// is NOT running, which is the only time PortOS cannot ask `/v1/models`.
const weightsCache = new Map();

/**
 * What a runtime's own model cache holds, for the runtimes that have one PortOS
 * can read without starting them (today: MTPLX).
 *
 * This is the fact that turned the MTPLX checklist into a catch-22: "installed
 * ✓ / server not responding ✗ — use Start MTPLX, PortOS does this for you",
 * and Start then failed with "no model weights are cached, so its server exits
 * before it binds a port". Reading the cache HERE is what lets the checklist
 * say so up front and offer the download instead of the start.
 *
 * A cache that could not be read reports `'unknown'` — never `'empty'`, which
 * would claim a fact PortOS does not have, and so does a runtime with no cache
 * PortOS can read offline (`readRuntimeWeights` answers those without spending
 * a subprocess).
 *
 * TTL-cached — see WEIGHTS_TTL_MS.
 *
 * @returns {Promise<'unknown'|'empty'|'partial'|'ready'>}
 */
function readWeightsStateCached(kind) {
  const now = Date.now();
  const cached = weightsCache.get(kind);
  if (cached && now - cached.at < WEIGHTS_TTL_MS) return cached.promise;
  // Cached BEFORE the await so concurrent providers on one runtime share the
  // subprocess; dropped on an unexpected throw so a transient failure cannot
  // poison the entry for its whole TTL (same shape as the probe cache above).
  const promise = readRuntimeWeights(kind).catch((err) => {
    weightsCache.delete(kind);
    throw err;
  });
  weightsCache.set(kind, { at: now, promise });
  return promise;
}

/** TTL-cached PATH lookup for the handful of daemon binaries — see BINARY_TTL_MS. */
function findCommandCached(command) {
  const cached = binaryCache.get(command);
  if (cached && Date.now() - cached.at < BINARY_TTL_MS) return cached.path;
  const path = findCommandOnPath(command);
  binaryCache.set(command, { at: Date.now(), path });
  return path;
}

/**
 * The model id the endpoint would be asked for — the provider's default.
 * Returns null when the provider selects no specific model.
 */
export function servedModelId(provider, kind) {
  return bareLocalModelId(provider?.defaultModel, kind);
}

/**
 * The model ids this provider OFFERS beyond its pinned default — the catalog a
 * task or a pipeline stage picks a model out of.
 *
 * The pin is excluded because `modelCheck` already covers it; what is left is
 * the part of the provider nothing was checking. A provider that pins NO model
 * (the shipped OpenCode local wrappers all ship `defaultModel: null`) had no
 * model check at all, so its whole catalog could name ids the daemon has not
 * served since the day they were pulled — and every run dispatched onto one
 * died at spawn with no output instead of failing resolution up front (#6125).
 */
function offeredModelIds(provider, kind) {
  const pinned = servedModelId(provider, kind);
  const ids = (Array.isArray(provider?.models) ? provider.models : [])
    .map((model) => bareLocalModelId(model, kind))
    .filter((id) => id !== null && id !== pinned);
  return [...new Set(ids)];
}

/**
 * The fix hint for a check the one-click setup button already covers.
 *
 * `setup.actionLabel` names a button rendered right below the checklist, so the
 * hint points AT it rather than repeating a terminal recipe — and a runtime this
 * host cannot set up at all says why instead of sending the user to a doc that
 * would not help them either.
 *
 * `null` means "no button covers this", and the caller falls back to its own
 * prose. `covers` is the action the check needs: an install-only button does
 * nothing for a `server` check.
 */
function setupHint(setup, capability) {
  if (!setup) return null;
  if (setup.blockedReason) return setup.blockedReason;
  // `capability` is one of `installs` / `starts` / `provisions`, asked of the
  // action table rather than matched against the action's spelling. The old
  // substring test worked only because every action name happened to contain
  // the word — `provision-start` does not contain `pull`, and a hint that
  // vanishes when an action is renamed fails no test.
  return actionCovers(setup.action, capability)
    ? `Use “${setup.actionLabel}” below — PortOS does this for you.`
    : null;
}

/**
 * What a runtime's own model cache says while its server is down — the half of
 * the model check `GET /v1/models` cannot answer because nothing is listening.
 *
 * `capability` is one of `installs` / `starts` / `provisions` — the axes
 * `localRuntimeSetup.js` declares for every action.
 *
 * `null` when there is nothing to add, so the caller keeps its own prose.
 */
function weightsDetail(runtime, weights) {
  // A runtime whose local setup is not "a model cache" describes its own states
  // — vLLM's is a compose project that was never cloned or prepared, and
  // "no model weights cached" would send the operator after a download that was
  // never the first step. Keyed by STATE rather than one override per divergence,
  // so the next runtime supplies whichever states read wrong for it.
  const override = runtime.setupStateDetail?.[weights];
  if (override) return override;
  if (weights === 'empty') return `${runtime.label} has no model weights cached, so its server exits before it binds a port.`;
  if (weights === 'partial') return `${runtime.label}'s cache holds only an unfinished download — no complete checkpoint to serve.`;
  if (weights === 'ready') return 'Weights are cached locally; this can be confirmed once the server is running.';
  return null;
}

/**
 * The page that manages this runtime, named the way the sidebar names it.
 *
 * Every hint below used to say "Models → LLMs" outright, which was true while
 * that one page owned every local server. #7414 moved server lifecycle to
 * Models → Runtimes and left the weights catalog on LLMs, so a fixed breadcrumb
 * now sends half these runtimes to the wrong sibling. Resolving it from the
 * route each runtime ALREADY links to keeps the prose and the link agreeing,
 * per runtime, without this module knowing which pages exist.
 *
 * `null` for a runtime with no page (vLLM, SGLang) — callers drop the clause
 * rather than inventing a destination.
 */
const managePage = (runtime) => getNavPageForPath(runtime.manageUrl)?.breadcrumb ?? null;

/** The `runtime` check — is the daemon's software here at all? */
function runtimeCheck(runtime, { onPath, appInstalled, installed, reachable, setup }) {
  const detail = onPath ? `\`${runtime.command}\` is on PortOS's PATH.`
    : appInstalled ? `${runtime.label} is installed as an app.`
      : reachable ? `Something is already serving ${runtime.endpoint}.`
        : `\`${runtime.command}\` was not found on PortOS's PATH.`;
  const page = managePage(runtime);
  const fixHint = installed ? null
    : setupHint(setup, 'installs')
      || (page ? `Install ${runtime.label} from ${page}.`
        : `Use the setup button below to install ${runtime.label}.`);
  return { id: 'runtime', label: `${runtime.label} installed`, ok: installed, detail, fixHint };
}

/** The `server` check — is it running where THIS provider points? */
function serverCheck(runtime, { installed, result, setup, weights = 'unknown' }) {
  if (result.reachable) {
    return {
      id: 'server',
      label: `${runtime.label} server responding`,
      ok: true,
      detail: `${runtime.endpoint} answered.`,
      fixHint: null,
    };
  }
  const page = managePage(runtime);
  const start = `Start ${runtime.label}${page ? ` from ${page}` : ''}.`;
  const modelsHint = expandPageToken(runtime.modelsHint, runtime);
  const fallback = installed
    ? `${start} ${modelsHint}`
    : `Install ${runtime.label} first, then start it. ${modelsHint}`;
  // Name the blocker in the SAME line that says nothing answered. Otherwise the
  // checklist reads "installed ✓ / not responding — just press Start", and
  // Start is the thing that cannot work until the weights land.
  const blocked = weightsBlockStart(weights) ? weightsDetail(runtime, weights) : null;
  return {
    id: 'server',
    label: `${runtime.label} server responding`,
    ok: false,
    detail: `Nothing answered at ${runtime.endpoint}${result.error ? ` (${result.error})` : ''}.${blocked ? ` ${blocked}` : ''}`,
    fixHint: setupHint(setup, 'starts') || fallback,
  };
}

/**
 * The `detail`/`fixHint` a model-facing check reports when the endpoint never
 * answered, or answered unreadably. Shared by both such checks so a listing
 * PortOS could not read is reported as unknown, never as missing — the user
 * chases the check that IS actionable.
 */
function unreadableListing(runtime, probeError, weights, setup) {
  // A server that answered 401 IS responding — "until the server responds"
  // would send the user to start a container that is already up, when the
  // actual fix is to paste its key onto this provider.
  const detail = probeError === 'authentication required'
    ? `${runtime.label} refused the model listing without an API key — paste the server's key on this provider.`
    : weightsDetail(runtime, weights) || 'Cannot be checked until the server responds.';
  // An unservable cache is the ONE unknown here that has a fix: the download
  // that makes a start possible at all.
  const fixHint = weightsBlockStart(weights) ? setupHint(setup, 'provisions') : null;
  return { detail, fixHint };
}

/**
 * The `model` check — does the running daemon serve what this provider asks
 * for? This is the alias mismatch: `llama-server --alias dflash` against a
 * provider pinned to `dspark` fails HERE rather than inside a dead agent run.
 */
function modelCheck(runtime, wanted, served, probeError = null, { weights = 'unknown', setup = null } = {}) {
  const label = `Model \`${wanted}\` available`;
  if (!Array.isArray(served)) {
    return { id: 'model', label, ok: null, ...unreadableListing(runtime, probeError, weights, setup) };
  }
  if (served.includes(wanted)) {
    return { id: 'model', label, ok: true, detail: `${runtime.label} is serving \`${wanted}\`.`, fixHint: null };
  }
  const listed = served.slice(0, 3).map((id) => `\`${id}\``).join(', ');
  // WHY only one id is listed, in the same line that lists it. A daemon that
  // serves one model per process is not "missing" the others — nothing was ever
  // started under those names — and without saying so the checklist reads as a
  // missing multi-gigabyte download, which is the wrong fix and an expensive one
  // to chase.
  const oneModel = runtime.servesOneModel
    ? ` ${runtime.label} serves one model per process${runtime.aliasFlag ? `, answering under the id its \`${runtime.aliasFlag}\` set rather than the weights' own name` : ''}.`
    : '';
  const detail = served.length === 0
    ? `${runtime.label} is running but has no model loaded.`
    : `${runtime.label} is serving ${listed}${served.length > 3 ? ` +${served.length - 3} more` : ''}.${oneModel}`;
  // A runtime whose served id is a launch-line LABEL can be renamed onto the
  // weights it is already running — no download, no model swap. That makes the
  // mismatch fixable from EITHER end, so the hint names both rather than
  // implying the provider is the only thing that may move.
  const renameTo = served.length > 0 && runtime.aliasFlag ? wanted : null;
  const page = managePage(runtime);
  const fixHint = served.length === 0
    ? (page
      ? `No model is loaded. Start a preset from ${page}.`
      : 'No model is loaded. Use the setup controls on this card to load one.')
    : renameTo
      ? `Same server, two names for it — nothing needs downloading. Use the button below to point this provider at ${listed}, or “Serve as \`${wanted}\`” to relaunch ${runtime.label} on the weights it already has under that id.`
      : `This provider will send \`${wanted}\`, but the running server only accepts ${listed}. Use the button below to match them${page ? ` (or change the loaded weights on ${page})` : ''}.`;
  return {
    id: 'model',
    label,
    ok: false,
    detail,
    fixHint,
    servedModels: served,
    // The id a one-click relaunch would put on the launch line, or null when
    // this runtime has no label of its own to change.
    renameTo,
  };
}

/**
 * The `catalog` check — can this provider dispatch the models it OFFERS?
 *
 * `modelCheck` above validates the pin. This validates everything else in the
 * provider's `models` list, which is what a task or a pipeline stage actually
 * picks from. Without it a provider that pins nothing (every shipped OpenCode
 * local wrapper) passed readiness while its whole catalog named ids the daemon
 * stopped serving — a stage pinned to one spawned, produced no output, and died
 * at spawn time rather than failing resolution up front (#6125).
 *
 * `servesOneModel` runtimes are graded leniently on purpose: llama.cpp, MTPLX,
 * vLLM, SGLang and Slotstream run ONE checkpoint per process, so a catalog
 * listing the other presets is normal configuration rather than rot. One
 * servable id is all such a provider ever needs. A multi-model daemon (Ollama,
 * LM Studio) serves everything it holds at once, so there any unservable id is
 * a real pin waiting to fail.
 */
function catalogCheck(runtime, offered, served, probeError = null, { weights = 'unknown', setup = null, pinned = null } = {}) {
  const label = `Offered models available (${offered.length})`;
  if (!Array.isArray(served)) {
    return { id: 'catalog', label, ok: null, ...unreadableListing(runtime, probeError, weights, setup) };
  }
  const unserved = offered.filter((id) => !served.includes(id));
  const ok = runtime.servesOneModel ? unserved.length < offered.length : unserved.length === 0;
  if (ok) {
    const detail = unserved.length === 0
      ? `${runtime.label} is serving every model this provider offers.`
      : `${runtime.label} is serving ${offered.length - unserved.length} of them; it runs one model per process, so the rest are presets nothing was started under.`;
    return { id: 'catalog', label, ok: true, detail, fixHint: null };
  }
  const listed = unserved.slice(0, 3).map((id) => `\`${id}\``).join(', ');
  const more = unserved.length > 3 ? ` +${unserved.length - 3} more` : '';
  // "every run fails" is only true when the pin cannot carry the provider
  // either — otherwise the provider still runs, and only a stage that pinned one
  // of these ids dies.
  const nothingDispatchable = unserved.length === offered.length && !(pinned && served.includes(pinned));
  const page = managePage(runtime);
  const detail = nothingDispatchable
    ? `${runtime.label} serves none of them (${listed}${more}), so every run dispatched onto this provider fails before producing output.`
    : `${runtime.label} does not serve ${listed}${more}, so a task or stage pinned to one of those fails at spawn.`;
  return {
    id: 'catalog',
    label,
    ok: false,
    detail,
    fixHint: `Refresh this provider's models so it only offers ids the server serves${page ? `, or pull the missing ones from ${page}.` : `. ${expandPageToken(runtime.modelsHint, runtime)}`}`,
    unservedModels: unserved,
  };
}

/**
 * The requirements checklist for one provider, or `null` when the provider does
 * not depend on a local daemon.
 *
 * Deps are injectable so the suite can drive every combination without a daemon
 * or a real PATH on the host.
 *
 * @param {object} provider - the RAW provider record (endpoint + envVars intact;
 *   a sanitized copy has its secret env values redacted, which would hide a
 *   custom base URL)
 * @param {{findCommand?:Function, probe?:Function}} [deps]
 */
export async function getProviderReadiness(provider, deps = {}) {
  const runtime = localRuntimeForProvider(provider);
  if (!runtime) return null;

  const findCommand = deps.findCommand || findCommandCached;
  const probe = deps.probe || probeOpenAiModelsCached;

  // The wrapper's own key: a vLLM container is started behind `VLLM_API_KEY`
  // and 401s an unauthenticated `/v1/models`, which would leave the model check
  // permanently unknown on an otherwise-healthy stack.
  const result = await probe(runtime.endpoint, provider?.apiKey || '');
  // A daemon that answers is installed, whatever PATH says — Ollama's macOS app
  // and LM Studio both serve without putting a CLI on PortOS's PATH.
  const onPath = Boolean(runtime.command && findCommand(runtime.command));
  // LM Studio ships as a macOS app bundle whose `lms` shim the user opts into
  // separately, so PATH alone says "not installed" for a perfectly installed
  // copy. The Models → LLMs page already counts the bundle (`localLlm.getStatus`);
  // without the same signal here the card would render "LM Studio installed"
  // and "install LM Studio" two lines apart, and send the user after the wrong
  // fix — the real one is "start its server".
  const appInstalled = runtime.kind === 'lmstudio' && (deps.isAppInstalled || isLmStudioAppInstalled)();
  const installed = onPath || appInstalled || result.reachable;

  // What the runtime's own model cache holds. Only worth asking when the daemon
  // is installed and NOT answering: a running server's `/v1/models` is the
  // better answer, and an uninstalled runtime has no cache to read.
  const weights = installed && !result.reachable
    ? await (deps.readWeights || readWeightsStateCached)(runtime.kind)
    : 'unknown';

  // Resolved BEFORE the checks so each unmet one can point at the button that
  // fixes it rather than at a setup doc.
  const setup = describeRuntimeSetup(runtime.kind, { installed, running: result.reachable, weights });
  const standby = runtime.standbyWhenStopped === true
    && installed
    && !result.reachable
    && !weightsBlockStart(weights);

  const checks = [
    runtimeCheck(runtime, { onPath, appInstalled, installed, reachable: result.reachable, setup }),
    serverCheck(runtime, { installed, result, setup, weights }),
  ];
  const wanted = servedModelId(provider, runtime.kind);
  // The probe returns `models: null` on every unreachable path, so this
  // needs no second reachability test.
  if (wanted) checks.push(modelCheck(runtime, wanted, result.models, result.error, { weights, setup }));
  const offered = offeredModelIds(provider, runtime.kind);
  if (offered.length > 0) checks.push(catalogCheck(runtime, offered, result.models, result.error, { weights, setup, pinned: wanted }));

  return {
    kind: runtime.kind,
    label: runtime.label,
    endpoint: runtime.endpoint,
    manageUrl: runtime.manageUrl,
    // `ready` is strict: a check that could not be evaluated (`ok: null`) is not
    // a pass, so the card never claims a provider is good to go on unknowns.
    ready: checks.every((check) => check.ok === true),
    // Standby is deliberately separate from `ready`: no endpoint is currently
    // available, but an installed model-selecting runtime is not missing setup
    // merely because no model was chosen to occupy resources right now.
    standby,
    standbyDetail: standby ? expandPageToken(runtime.standbyDetail, runtime) : null,
    checks,
    // What this daemon is SERVING right now. It rides the readiness payload
    // BECAUSE it is observed runtime state and must never reach `providers.json`
    // (#7441) — this is the channel the provider card budgets its meter from, so
    // the card shows the window the dispatch gate would actually enforce.
    contextWindows: aliasServedContextWindows(provider, runtime, result.contextWindows),
    // The window PortOS holds this daemon at, resolved from BOTH rungs — the
    // provider's `numCtx` and the ambient `OLLAMA_CONTEXT_LENGTH` this install
    // launches Ollama with. It rides the same payload for the same reason the
    // served windows do: it is runtime state, not configuration, and the card's
    // meter must show the ceiling the dispatch gate enforces. The browser cannot
    // read server env, so without this the card stayed on the `numCtx`-only rung
    // and promised a window a run would then be refused for (#7472).
    runtimeContextWindow: managedOllamaContextLength(provider),
    // What a one-click "set this up for me" button can do about the unmet
    // checks, or `null` when nothing here is auto-fixable (see
    // `localRuntimeSetup.js`). Carried on the readiness payload so the card
    // offers the ACTION next to the failing check instead of sending the user
    // out of the app.
    setup,
  };
}

/**
 * Readiness for every ENABLED provider that needs a local daemon, keyed by
 * provider id. Providers with no local dependency are omitted entirely, so the
 * client can treat "absent" as "nothing to report".
 *
 * Disabled providers are skipped: a stock install ships most of the local
 * presets disabled, and probing daemons for providers the user switched off
 * spends most of every poll on cards that would show the checklist for a
 * provider they cannot run anyway.
 *
 * @param {object[]} providers - RAW provider records
 */
export async function getProviderReadinessMap(providers, deps = {}) {
  const list = (Array.isArray(providers) ? providers : []).filter((provider) => provider?.enabled !== false);
  // One PATH scan per distinct binary and one probe per distinct endpoint for
  // the whole batch — including when the caller injects its own probe, which
  // would otherwise bypass the module-level caches that normally collapse them.
  const findCommand = memoize(deps.findCommand || findCommandCached);
  const probe = memoize(deps.probe || probeOpenAiModelsCached);
  // Same reasoning as the two above: one `mtplx models` subprocess per batch,
  // not one per provider pointed at the same runtime.
  const readWeights = memoize(deps.readWeights || readWeightsStateCached);

  const entries = await Promise.all(list.map(async (provider) => {
    // Spread `deps` first so any other injected dep (isAppInstalled) survives,
    // then override the three the batch memoizes.
    const readiness = await getProviderReadiness(provider, { ...deps, findCommand, probe, readWeights });
    return readiness ? [provider.id, readiness] : null;
  }));
  return Object.fromEntries(entries.filter(Boolean));
}

/**
 * Per-batch memo (the returned promise is shared, not awaited). EVERY argument
 * is forwarded and folded into the key — a memo that keyed on the first argument
 * alone silently dropped the probe's API key, so the batch path (which is what
 * `GET /api/providers/readiness` uses) probed a key-gated container
 * unauthenticated while the single-provider path authenticated fine.
 */
function memoize(fn) {
  const seen = new Map();
  return (...args) => {
    const key = args.join('\n');
    if (!seen.has(key)) seen.set(key, fn(...args));
    return seen.get(key);
  };
}

/**
 * Drops the probe caches so the next read reflects a daemon that was just
 * started, stopped, or installed — called by the llama-server lifecycle routes,
 * which change exactly what these caches remember.
 */
export function resetProviderReadinessCache() {
  invalidateModelObservation('provider-readiness');
  // The endpoint listing cache lives in `lib/openAiModelsProbeCache.js` — the
  // dispatch path's context gate shares it, and a daemon relaunched at a
  // different context size must invalidate BOTH readers or a run is gated on
  // the old window.
  resetOpenAiModelsProbeCache();
  binaryCache.clear();
  weightsCache.clear();
}