Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

27.3 KB
Raw Blame
/**
 * Measure a batch of local models in one pass — the overnight sweep.
 *
 * Measuring one model takes minutes; measuring a machine's worth of them takes
 * hours. That is a job you start at the end of the day and read in the morning,
 * which rules out the shape the single-model path uses (one blocking POST held
 * open by the browser tab). This module runs the queue SERVER-SIDE: the request
 * that starts it returns immediately, the loop keeps going with nobody watching,
 * and the page reads its progress from a status endpoint plus the existing
 * `localLlm:progress` socket event.
 *
 * ## Two dimensions, one queue
 *
 * The queue walks a list of (model, tuning) pairs, and which list it walks is
 * the only difference between the two sweeps this module serves:
 *
 *   - **Model sweep** — every model a scope covers, each under the tuning it
 *     already carries. Answers "which of my models is worth using?".
 *   - **Tuning sweep** — one model across a grid of tunings
 *     (`lib/localModelTuning.js#tuningGridFor`). Answers "which flags suit this
 *     model here?", which is what `compareTunings` ranks.
 *
 * They deliberately share the queue rather than getting one each: cancellation,
 * progress, the accelerator claim, and the "one measurement at a time" rule are
 * identical, and a second queue could run concurrently with this one — which
 * would make every number either produced describe the contention.
 *
 * ## AI Provider Usage Policy (root AGENTS.md) — read before editing
 *
 * A sweep calls a provider once per queued measurement, so it is STRICTLY
 * user-triggered, the same as the single-model run:
 *
 *   - `startSweep()` is reachable only from `POST /api/local-llm/assessments/sweep`,
 *     behind a consent gate that names the exact measurement count (models, or
 *     tuning variants) and the total generation count.
 *   - `getSweepStatus()` reads module state only. Zero LLM calls, safe to reconcile.
 *
 * The prohibition the single-model service states — "no scheduler, no boot hook,
 * no background sweep" — is about work the user did not ask for. This IS the ask:
 * a button they pressed, having been told what it will run. What must never
 * appear is a cron entry, a boot hook, or an auto-start that fires this without a
 * click. Do not add one.
 *
 * `lib/sseUtils.js#createSseRunner` owns a similar lifecycle (runId, abort,
 * cancel, active-guard) but is SSE-transport-bound and keeps no cumulative record
 * of what a run has produced so far — which is the one thing a page reloaded at
 * 3am needs. Hence a small queue of its own rather than a transport it would have
 * to work around.
 *
 * ## Restart behavior
 *
 * The queue lives in module memory, so a server restart mid-sweep ends it. That
 * is deliberate rather than unfinished: every COMPLETED measurement is already
 * durable (`runAssessment` persists as it goes), so a restart costs the model in
 * flight and nothing else, and re-running the sweep afterwards picks up exactly
 * what is still unmeasured. Persisting the queue would buy a resumable pointer
 * into a job that is cheap to restart and must never auto-resume on boot.
 *
 * One consequence is worth naming: a restart during a MODEL or TUNING sweep
 * also loses the captured launch configuration, so llama-server stays on
 * whichever variant/model was in flight. That is visible and fixable on the
 * LLMs page, and the alternative — persisting it and relaunching a daemon at
 * boot to put it back — is exactly the boot-time work this module refuses to do.
 */

import { selectSweepTargets, SWEEP_SCOPES } from '../lib/localModelAssessment.js';
import { isEmbeddingModel } from '../lib/localModelHeuristics.js';
import { tuningGridFor } from '../lib/localModelTuning.js';
import {
  captureLaunchState,
  getAssessmentReport,
  isTuningSweepable,
  restoreLaunchState,
  runAssessment,
} from './localModelAssessments.js';
import { claimHeavyLocalJob } from '../lib/heavyJobClaim.js';

// One sweep at a time. This is a re-entrancy guard, not a concurrency control:
// PortOS serves one user, and the thing being prevented is a second click (or a
// second tab) queuing the same models against a machine that can only run one
// model at a time anyway.
let sweep = null;

// Held across `startSweep`'s `await getAssessmentReport()`. The `sweep` object
// itself cannot be published until the targets are known, so without this a
// second request arriving during that await passes the "already running" check,
// and both requests go on to assign `sweep` and launch a detached loop — two
// overnight queues measuring the same models against each other, with only the
// second one's status visible. The slot has to be reserved SYNCHRONOUSLY, before
// the first await, or the check and the claim are not atomic.
let startingSweep = false;

// A sweep is sequential BY DESIGN. Two models measured at once contend for the
// same memory and the same GPU, and every number either one produces would
// describe that contention rather than the model — which is the one thing this
// whole feature exists to measure accurately.

// How long one model waits for the machine-wide accelerator claim before giving
// up on its turn. An interactive Measure click refuses immediately; a queue
// running unattended should ride out a short image render rather than throwing
// away a model's slot over it — but not wait all night behind a LoRA training
// run, which would leave the sweep looking hung with nothing to show.
const CLAIM_WAIT_MS = 10 * 60 * 1000;

const nowIso = () => new Date().toISOString();

/** Public snapshot. Never leaks the AbortController or the emit hook. */
function snapshot() {
  if (!sweep) {
    // `status: 'idle'` with a null run is the honest "nothing has been started",
    // distinct from a finished sweep whose results are still worth showing.
    return { status: 'idle', mode: null, scope: null, target: null, settled: true, startedAt: null, finishedAt: null, total: 0, completed: 0, current: null, results: [], cancelRequested: false, restoreError: null };
  }
  return {
    status: sweep.status,
    // 'models' = every model the scope covers, each under one tuning.
    // 'tunings' = ONE model across a grid of tunings. The page renders a
    // different sentence for each, so it cannot be inferred from the counts.
    mode: sweep.mode,
    scope: sweep.scope,
    // The single model a tuning sweep is measuring; null for a model sweep.
    target: sweep.target,
    // Whether the sweep has let go of the machine, which outlasts `status`: a
    // cancelled sweep is still aborting its last measurement and, for a tuning
    // sweep, still has a launch configuration to put back — a relaunch. The page
    // keeps its per-model actions disabled while this is false.
    settled: sweep.settled === true,
    startedAt: sweep.startedAt,
    finishedAt: sweep.finishedAt,
    total: sweep.total,
    completed: sweep.results.length,
    // Which model is being measured right now — `null` between models and once
    // the sweep ends.
    current: sweep.current ? { ...sweep.current } : null,
    results: sweep.results.map((result) => ({ ...result })),
    cancelRequested: sweep.cancelRequested,
    error: sweep.error || null,
    // A sweep whose MEASUREMENTS all landed but whose daemon could not be put
    // back is not a failed sweep — but the user has to be told, because their
    // runtime is now serving the last variant's launch line and nothing else on
    // the page would say so. Separate from `error` so neither one lies.
    restoreError: sweep.restoreError || null,
    // `launchStates` is deliberately NOT exposed: it holds on-disk model paths,
    // and the page has no use for the configuration being put back.
  };
}

export function getSweepStatus() {
  return snapshot();
}

/**
 * Put the captured launch configuration back, holding the machine-wide claim
 * while it happens.
 *
 * A restore STOPS and STARTS the daemon, which takes seconds to minutes for a
 * large GGUF — the same disruption the claim exists to keep away from a
 * measurement. `runAssessment` released it after the last variant, and a Stop
 * click re-enables the per-model Measure button immediately, so without a claim
 * of its own the restore can bounce llama-server in the middle of a reading
 * somebody started meanwhile (from the page, ⌘K, or curl) and that reading would
 * record the restart.
 *
 * A claim that cannot be had is NOT a reason to skip: the daemon would be left
 * on the sweep's last variant, which is the thing this exists to prevent. It
 * waits, bounded, then restores anyway and lets the reason be reported.
 */
async function restoreUnderClaim(backend, launchState) {
  const claim = await claimHeavyLocalJob({
    kind: 'local-model assessment',
    id: `${backend} launch restore`,
    timeoutMs: CLAIM_WAIT_MS,
  });
  if (!claim.ok) {
    console.error(`⚠️ Local LLM: restoring the launch configuration without the accelerator claim — ${claim.message}`);
  }
  try {
    return await restoreLaunchState(backend, launchState);
  } finally {
    if (claim.ok) await claim.release();
  }
}

/**
 * Restore every backend touched by a model or tuning sweep, one at a time.
 *
 * Only CAPTURED backends are in the map, so every entry is a real restore —
 * there is no "nothing to restore" here to mistake for a failure. Each is caught
 * on its own: one runtime that will not come back must not leave the next one
 * serving the sweep's launch line.
 */
async function restoreAllUnderClaim(run) {
  const failures = [];
  for (const [backend, launchState] of Object.entries(run.launchStates)) {
    const result = await restoreUnderClaim(backend, launchState)
      .catch((err) => ({ restored: false, reason: err?.message || 'restore failed' }));
    if (result?.restored === false) {
      failures.push(`${backend}: ${result.reason || 'the launch configuration could not be put back'}`);
    }
  }
  return failures;
}

/**
 * Ask the running sweep to stop.
 *
 * Aborts the model in flight (which `runAssessment` treats as a cancel and does
 * NOT record) and drops the rest of the queue. Everything already measured stays
 * on disk — an interrupted overnight run is still worth what it finished.
 */
export function cancelSweep() {
  if (!sweep || sweep.status !== 'running') return snapshot();
  sweep.cancelRequested = true;
  sweep.controller.abort();
  // Flip the status here rather than waiting for the loop's `finally`: the
  // caller is an HTTP handler returning this snapshot NOW, and reporting
  // `running` to a client that just stopped it would leave a Stop button on
  // screen for a queue that is already winding down. The `finally` re-affirms
  // `cancelled` (it reads the same flag), so the two cannot disagree.
  sweep.status = 'cancelled';
  sweep.notifyChange();
  console.log(`🛑 Local LLM: assessment sweep cancelled after ${sweep.results.length}/${sweep.total}`);
  return snapshot();
}

/** Test seam: forget any sweep state between suites. */
export function __resetSweep() {
  sweep?.controller?.abort();
  sweep = null;
  startingSweep = false;
}

// The sweep loop runs OUTSIDE the request lifecycle — there is no `next(err)` to
// bubble to, so an uncaught throw here would take the process down (root
// AGENTS.md). Every model is individually caught, and the loop itself is wrapped.
//
// It mutates the `run` it was handed, NOT the module-level `sweep`: a cancelled
// queue is replaceable the moment it is cancelled, and a loop still winding down
// its last model must not write its results into the sweep that succeeded it.
async function runSweepLoop(run, targets, contextTokens, emit) {
  // A tuning sweep asks each variant to be the COMPLETE tuning, while a model
  // sweep applies each model's recorded tuning. In both cases the launch line
  // is made explicit before the measurement, and `beginSweep` captured what
  // was running first so the final restore can put the user's configuration
  // back.
  for (const target of targets) {
    if (run.cancelRequested) break;
    run.current = { backend: target.backend, modelId: target.modelId, tuningLabel: target.tuningLabel, startedAt: nowIso() };
    run.notifyChange();
    emit({
      scope: 'assessment-sweep',
      event: 'model-start',
      backend: target.backend,
      modelId: target.modelId,
      // The label is what distinguishes one step of a TUNING sweep from the
      // next — every step names the same model, so a message without it reads
      // as one measurement repeating.
      tuningLabel: target.tuningLabel || null,
      completed: run.results.length,
      total: run.total,
      message: `Sweep ${run.results.length + 1}/${run.total}: measuring ${target.modelId}${target.tuningLabel ? ` (${target.tuningLabel})` : ''}…`,
    });

    // One model failing is a RESULT, not a reason to abandon the queue — the
    // whole point of an overnight run is that it gets through the list.
    // A llama model sweep must reset the launch line before applying the stored
    // tuning for each model. Without this, model B inherited model A's batch /
    // KV / speculative flags while its record named only B's tuning (#4774).
    // Other endpoint runtimes have no sweep-safe reset/restore path, so they
    // remain ordinary model measurements.
    //
    // Gated on a CAPTURED configuration rather than on `isTuningSweepable`
    // alone: a reset renders the cleared launch line, which wipes knobs the USER
    // may have set on the LLMs page, and only a caller holding what was running
    // there is entitled to do that. `beginSweep` captured every sweepable
    // backend it could, so a runtime that could not be captured — llama-server
    // stopped, or started outside PortOS — measures under what is running
    // instead of losing flags nothing can put back.
    const resetTuning = Boolean(run.launchStates[target.backend]);
    const result = await runAssessment({
      backend: target.backend,
      modelId: target.modelId,
      contextTokens,
      tuning: target.tuning,
      resetTuning,
      signal: run.controller.signal,
      onProgress: emit,
      claimTimeoutMs: CLAIM_WAIT_MS,
    }).catch((err) => ({ error: err?.message || 'assessment failed' }));

    run.current = null;
    // A cancelled run recorded nothing, so it is not a result — recording it as
    // one would make a stopped sweep look like it measured what it abandoned.
    if (result?.cancelled) {
      run.notifyChange();
      break;
    }
    run.results.push({
      backend: target.backend,
      modelId: target.modelId,
      tuningLabel: target.tuningLabel,
      finishedAt: nowIso(),
      // `null` verdict + an error means the run threw before producing evidence.
      verdict: result?.error ? null : (result?.verdict || 'unknown'),
      error: result?.error || null,
      meanTokensPerSecond: Number.isFinite(result?.performance?.meanTokensPerSecond) ? result.performance.meanTokensPerSecond : null,
      meanCharsPerSecond: Number.isFinite(result?.performance?.meanCharsPerSecond) ? result.performance.meanCharsPerSecond : null,
      // Travels WITH the rate it qualifies. Dropping it here would render a
      // frame-counted estimate as a tokenizer measurement in the results list.
      tokensEstimated: typeof result?.performance?.tokensEstimated === 'boolean' ? result.performance.tokensEstimated : null,
      // `false` means the knobs never reached the daemon, so these numbers
      // describe some OTHER configuration. The report already refuses to rank
      // such a record; without these the sweep's own list would still present it
      // as a clean reading for the tuning it names — which is exactly the row a
      // tuning sweep is read for.
      tuningApplied: typeof result?.tuningApplied === 'boolean' ? result.tuningApplied : null,
      tuningNotApplied: result?.tuningNotApplied || null,
    });
    run.notifyChange();
  }
}

/**
 * Start measuring every model the scope covers. **Calls a provider, once per
 * model** — see the policy note at the top.
 *
 * Returns as soon as the queue is built: the loop runs detached so the HTTP
 * request that started it can return, and so closing the browser does not stop
 * an overnight run.
 *
 * Set `tunings` and it measures ONE named model across the grid of launch
 * configurations its runtime declares — the tuning sweep. The grid is chosen
 * HERE, from `tuningGridFor`, rather than accepted from the caller: what a
 * "tuning sweep of this model" means is a domain decision, and deriving it in
 * one place is what keeps the count a consent gate names equal to the count that
 * runs.
 *
 * @param {object} options
 * @param {'unmeasured'|'stale'|'all'} [options.scope] which models, when no
 *   explicit model is named
 * @param {string} [options.backend] measure only this model (with `modelId`)
 * @param {string} [options.modelId]
 * @param {boolean} [options.tunings] sweep the named model's tuning grid instead
 *   of measuring it under the tuning it already carries
 * @param {number[]} [options.contextTokens] passed through to each measurement
 * @param {(frame: object) => void} [options.onProgress] forwarded to the socket
 * @param {(state: object) => void} [options.onChange] public queue snapshots on state changes
 * @returns {Promise<object>} the initial snapshot, or `{ rejected }` when a
 *   sweep is already running or the request covers nothing
 */
export async function startSweep({ scope = 'unmeasured', backend, modelId, tunings = false, contextTokens, onProgress, onChange } = {}) {
  if (sweep?.status === 'running' || startingSweep) return { ...snapshot(), rejected: 'a sweep is already running' };
  // A CANCELLED sweep is not finished with the machine: its last measurement is
  // still aborting and it may have a launch configuration to put back. Starting
  // the next one on top of that would have the old sweep relaunch the daemon
  // partway through the new one's first measurement — and every number the new
  // sweep produced up to then would describe that relaunch.
  if (sweep && !sweep.settled) return { ...snapshot(), rejected: 'the previous sweep is still winding down' };
  // Reserved here, synchronously, so the check above and this claim cannot be
  // split by the await inside. Released in the `finally` — by which point either
  // `sweep` is published (and the `running` check covers the slot) or the start
  // was refused and the slot is free again.
  startingSweep = true;
  try {
    return await beginSweep({ scope, backend, modelId, tunings, contextTokens, onProgress, onChange });
  } finally {
    startingSweep = false;
  }
}

async function beginSweep({ scope, backend, modelId, tunings, contextTokens, onProgress, onChange }) {
  const resolvedScope = SWEEP_SCOPES.includes(scope) ? scope : 'unmeasured';
  const named = backend && modelId ? { backend, modelId } : null;

  // A tuning sweep varies the configuration of ONE model, so without a model
  // there is nothing to hold constant and the request is meaningless.
  if (tunings && !named) return { ...snapshot(), rejected: 'a tuning sweep needs a model to sweep' };
  // A runtime PortOS cannot reset and put back must not be swept: its baseline
  // would measure the previous variant's configuration, and the sweep would
  // leave its knobs set for good. See `LAUNCH_APPLIERS` and #4763.
  if (tunings && !isTuningSweepable(named.backend)) {
    return { ...snapshot(), rejected: `PortOS cannot sweep ${named.backend} tunings yet — it has no way to put the runtime back afterwards` };
  }
  // A named embedding model would also produce an empty list (the selector drops
  // it), and "not installed" would be a lie — it is installed, it just has no
  // generation to measure. Say which it is.
  if (named && isEmbeddingModel(named.modelId)) {
    return { ...snapshot(), rejected: `${named.modelId} is an embedding model — it has no chat/generation to measure` };
  }
  const grid = named && tunings ? tuningGridFor(named.backend) : null;
  // A runtime PortOS cannot pass flags to yields the baseline alone — one
  // measurement with nothing to compare it to. Refuse rather than spend minutes
  // of GPU on a table `compareTunings` will not render.
  if (grid && grid.length < 2) {
    return { ...snapshot(), rejected: `${named.backend} has no tuning knobs PortOS can sweep` };
  }
  const mode = grid ? 'tunings' : 'models';

  // Read the report ONCE, here, rather than per model: it lists installed models
  // across every runtime and annotates staleness, which is exactly the input the
  // target selection needs. It is disk-plus-listing only — no LLM call.
  const report = await getAssessmentReport({ intent: 'balanced' });
  const uninstalledKeys = new Set(report.uninstalled.map((u) => `${u.backend}:${u.modelId}`));
  const installed = report.assessments.filter((a) => !uninstalledKeys.has(`${a?.backend}:${a?.modelId}`));

  const targets = selectSweepTargets({
    // Records for models that are no longer installed cannot be re-measured, so
    // they never enter the queue.
    assessments: installed,
    unassessed: report.unassessed,
    scope: resolvedScope,
    only: named,
    tunings: grid,
  });

  // An empty list for a NAMED model means it is not installed — the selector
  // found neither a record nor a listing for it. Saying so beats queuing one
  // doomed measurement per variant and reporting the sweep as failed.
  if (!targets.length) {
    return {
      ...snapshot(),
      rejected: named ? `${named.modelId} is not installed on ${named.backend}` : 'nothing to measure for that scope',
    };
  }

  const emit = (frame) => {
    if (typeof onProgress !== 'function') return;
    // A broken listener (a closed socket) must never abort a job the user is
    // paying hours of compute for.
    try { onProgress(frame); }
    catch (err) { console.error(`❌ Local LLM: sweep progress listener failed: ${err.message}`); }
  };

  // Capture BEFORE the first model/variant relaunches anything. A model sweep
  // can cross several llama targets, each with a different stored tuning; the
  // llama daemon is machine-wide, so the one launch state has to be restored
  // after the whole queue, not after each model. Endpoint runtimes that cannot
  // reset safely are intentionally absent from this map.
  //
  // A backend whose capture comes back `null` is LEFT OUT rather than stored as
  // null: the daemon is stopped or somebody else started it, so there is nothing
  // to put back — and the loop reads presence here as its permission to reset.
  const restoreBackends = [...new Set(targets.map((target) => target.backend).filter(isTuningSweepable))];
  const launchStates = {};
  for (const restoreBackend of restoreBackends) {
    const state = await captureLaunchState(restoreBackend);
    if (state) launchStates[restoreBackend] = state;
  }
  // Nothing to capture means the daemon is stopped, or someone else started it —
  // either way PortOS cannot put a tuning on its launch line, so every variant
  // would record `tuningApplied: false` and the comparison would rank a set of
  // configurations that never ran. Refusing costs a click; running costs the
  // whole sweep.
  //
  // Only a TUNING sweep refuses. Varying the launch line is its entire job, so
  // an uncapturable runtime leaves it nothing to do — but a MODEL sweep is a
  // queue of real measurements across every runtime the scope covers, and one
  // llama-server somebody else started must not cancel the Ollama and LM Studio
  // models alongside it. It measures that runtime under whatever is running, as
  // it did before the reset existed.
  if (grid && !launchStates[named.backend]) {
    return {
      ...snapshot(),
      rejected: `PortOS is not running ${named.backend}, so it cannot vary the launch configuration — start it from the LLMs page first`,
    };
  }

  const run = {
    status: 'running',
    mode,
    // A tuning sweep of ONE named model did not consult the scope, so reporting
    // one would tell the page it measured a set it never looked at.
    scope: named ? null : resolvedScope,
    target: named,
    startedAt: nowIso(),
    finishedAt: null,
    total: targets.length,
    current: null,
    results: [],
    cancelRequested: false,
    error: null,
    restoreError: null,
    // Flipped in the loop's `finally`, after the restore. `startSweep` refuses
    // while it is false, so a cancelled sweep cannot relaunch the daemon
    // underneath the sweep that replaced it.
    settled: false,
    launchStates,
    controller: new AbortController(),
    notifyChange: () => {
      if (sweep !== run || typeof onChange !== 'function') return;
      // Socket delivery must not abort the user's running measurements.
      try { onChange(snapshot()); }
      catch (err) { console.error(`❌ Local LLM: sweep state listener failed: ${err.message}`); }
    },
  };
  sweep = run;
  run.notifyChange();

  // A tuning sweep measures ONE thing many ways, so counting "models" would be
  // wrong in exactly the place the user is reading for reassurance.
  const plural = targets.length === 1 ? '' : 's';
  const what = grid
    ? `${named.modelId} across ${targets.length} tuning${plural}`
    : `${targets.length} model${plural}`;
  console.log(`📏 Local LLM: assessment sweep started — ${targets.length} measurement${plural} (${named ? named.modelId : resolvedScope})`);
  emit({ scope: 'assessment-sweep', event: 'start', total: targets.length, completed: 0, message: `Measuring ${what}…` });

  // Detached on purpose: the caller is an HTTP handler that must return now.
  runSweepLoop(run, targets, contextTokens, emit)
    .catch((err) => {
      run.error = err?.message || 'sweep failed';
      console.error(`❌ Local LLM: assessment sweep failed: ${run.error}`);
    })
    // Not in the `finally`: putting the daemon back is real work that can fail,
    // and it has to finish before the page is told the sweep is over — a user
    // reading "complete" must not still be racing a relaunch.
    .then(() => restoreAllUnderClaim(run)
      .then((failures) => {
        // A captured state that did not restore means that backend is still on
        // the last variant. Recording the reason is the only way the user finds
        // out; swallowing it leaves them running a configuration they did not
        // choose and cannot see.
        if (failures.length) {
          run.restoreError = failures.join('; ');
          console.error(`❌ Local LLM: sweep finished but the launch configuration was not restored — ${run.restoreError}`);
        }
      })
      .catch((err) => {
        run.restoreError = err?.message || 'the launch configuration could not be put back';
        console.error(`❌ Local LLM: could not restore the launch configuration after the sweep: ${run.restoreError}`);
      }))
    .finally(() => {
      run.status = run.cancelRequested ? 'cancelled' : (run.error ? 'failed' : 'complete');
      run.finishedAt = nowIso();
      run.current = null;
      // Set LAST, once the queue and the restore are both done — this is what
      // lets the next sweep start.
      run.settled = true;
      console.log(`📏 Local LLM: assessment sweep ${run.status} — ${run.results.length}/${run.total} measured`);
      // A queue the user already replaced has nothing to report — emitting its
      // terminal frame would tell the page the CURRENT sweep just finished.
      if (sweep !== run) return;
      run.notifyChange();
      emit({
        scope: 'assessment-sweep',
        event: 'complete',
        status: run.status,
        completed: run.results.length,
        total: run.total,
        message: `Sweep ${run.status}: ${run.results.length}/${run.total} measured`,
      });
    });

  return snapshot();
}