Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

21.6 KB
Raw Blame
/**
 * Video render speed profiles (issue #4875).
 *
 * A *speed profile* is a named, pre-validated sampler configuration a user can
 * pick instead of hand-tuning steps and CFG: it swaps the whole schedule at
 * once (stage-1 / stage-2 step counts, guidance, and runtime acceleration
 * levers like TeaCache) for a documented wall-time saving.
 *
 * The relationship between a model and its profiles is DECLARED here — not
 * inferred from a runtime name — and attached to registry entries as
 * `speedProfiles` by `applyVideoSpeedProfiles`, the same load-time backfill
 * pattern `applyVideoDisclosures` (lib/videoDisclosure.js) and
 * `applyVideoFinishProfiles` (lib/videoFinishProfiles.js) use. Migration 295
 * makes it durable on disk for installs that already persisted
 * `data/media-models.json`.
 *
 * ## Why the DEFAULT profile is a first-class entry rather than "absence"
 *
 * `SPEED_PROFILE_DEFAULT_ID` ('quality') is the shipped contract: the exact
 * sampler the model has always rendered with. It exists as a named option so
 * the picker has something to show as selected, but it is deliberately a
 * NO-OP — `resolveVideoSpeedProfile` returns `null` for it, so a quality render
 * builds byte-identical spawn args to a render from before this feature
 * existed, and stamps no extra history fields. Absence and `'quality'` are the
 * same request (`isDefaultSpeedProfile`), exactly as `'stock'` and absence are
 * the same request in lib/videoTextEncoders.js.
 *
 * ## Why compatibility is checked rather than assumed
 *
 * A profile's numbers are validated against ONE pipeline on ONE set of weights.
 * Applying them anywhere else would be a speed claim we cannot back:
 *
 *   - **Weights.** `shippedRepo` + `shippedRevision` must both still match the
 *     entry. A user who re-pointed `repo` at a fork, or moved `revision` off
 *     the pin the schedule was measured against, keeps no profile rather than
 *     inheriting a claim about weights we can no longer vouch for. (The
 *     finish-profile precedent guards `repo` only; a sampler schedule is
 *     revision-sensitive in a way a draft→delivery edge is not, so this guards
 *     both.)
 *   - **Mode.** Only the modes in `profile.modes` route through the pipeline
 *     the schedule was measured on. `fflf`/`extend`/`a2v`/`ic` run entirely
 *     different pipelines (KeyframeInterpolation, Extend, A2Vid, IC-LoRA), so
 *     a profile is DECLINED for them with an explicit reason instead of
 *     silently applying numbers that mean nothing there.
 *   - **Sampler lock.** A `samplerLocked` entry pins its own validated
 *     schedule; a speed profile would be a second authority over the same
 *     dials, so the two are mutually exclusive by construction (asserted by
 *     `validateSpeedProfileTable`).
 *
 * Everything the RUNNER cannot promise ahead of time — is the pinned
 * `ltx_pipelines_mlx` new enough to accept `enable_teacache`? is the distilled
 * adapter actually in the model pack? — is probed by the Python helper at
 * render time, which emits a `SPEEDPROFILE:` line naming what it actually
 * applied. That is what keeps an unavailable lever from becoming a misleading
 * speed claim: the profile still renders (at its step schedule), the missing
 * lever is reported as `degraded`, and the history record says so.
 *
 * Pure module: no I/O, and no imports out to services (the one import is a
 * sibling lib constant).
 */

import { LTX2_FAMILY_RUNTIMES } from './runners.js';

/**
 * The implicit "unchanged" profile. Never resolves to an override.
 */
export const SPEED_PROFILE_DEFAULT_ID = 'quality';

/**
 * Modes that route through the LTX two-stage pipeline (`run_two_stage` in
 * scripts/generate_ltx2.py), which is the only pipeline the shipped schedules
 * below were measured on. Declared once so a new profile can't quietly widen
 * itself past what was validated.
 */
export const TWO_STAGE_SPEED_PROFILE_MODES = Object.freeze(['text', 'image']);

/**
 * Shipped profiles, keyed by registry entry id.
 *
 * `shippedRepo` / `shippedRevision` are the pin guards described above.
 * `profiles` is the list attached to the entry, in picker order. Each profile:
 *
 *   id              stable key — persisted in history and submitted by the UI
 *   name            picker label
 *   description     one line under the picker; states the trade honestly
 *   speedupLabel    the measured claim, or null when there is nothing to claim
 *   steps           stage-1 step count (overrides the entry's `steps`)
 *   stage2Steps     explicit stage-2 step count (null → pipeline default)
 *   guidance        CFG scale (1.0 drops the negative branch entirely, which is
 *                   where most of the saving comes from on a two-stage render)
 *   teacache        request stage-1 TeaCache from the runner
 *   teacacheThresh  rel_l1_thresh override, or null for the pin's calibrated
 *                   default (0.5 at the current LTX-2 pin)
 *   requiresAdapter distilled-adapter filename the schedule was measured with,
 *                   or null when the profile doesn't depend on one
 *   modes           modes the profile may resolve for
 */
export const VIDEO_SPEED_PROFILES = Object.freeze({
  // LTX-2.5 Q8. The step schedule here is NOT invented: it is the same
  // 8 stage-1 / 3 stage-2 / CFG 1.0 configuration the env-gated
  // PORTOS_T2V_TWO_STAGE experiment measured at ~30-35% off wall time
  // (resolveT2vTwoStageOverride, server/services/videoGen/local.js), promoted
  // from a hidden env knob to a user-facing, capability-gated choice.
  //
  // The 2.5 pack adds the lever 2.3 did not have: its own distilled adapter
  // (ltx-2.5-22b-distilled-lora-450) plus a pinned runtime whose two-stage
  // `generate_and_save` accepts `enable_teacache`. TeaCache caches stage-1
  // residuals between steps against upstream's calibrated coefficients; PortOS
  // has always left it off on this path because it never passed the kwarg.
  ltx25_mlx_q8: Object.freeze({
    shippedRepo: 'MrMofer/ltx-2.5-mlx-q8',
    shippedRevision: 'f1b56e7dc89f71a9af2cddac787b89ed22a8b7fc',
    profiles: Object.freeze([
      Object.freeze({
        id: 'fast',
        name: 'Fast',
        description: 'Validated 8+3-step two-stage schedule at CFG 1.0 with stage-1 TeaCache. Softer detail and weaker prompt adherence than Quality.',
        // Deliberately conservative, and traceable to the two numbers this repo
        // can actually cite: the step schedule measured ~30-35% off wall time
        // (resolveT2vTwoStageOverride, i.e. ~1.4-1.5x), and the pin's own
        // --teacache-thresh help documents ~1.2x at the calibrated default —
        // ~1.7-1.85x combined, so ~35-45% off. NOT "2x": that would overstate
        // the evidence, and on a pin without the TeaCache kwarg the cache half
        // degrades away entirely (leaving only the ~30-35%). For a feature
        // whose whole contract is an honest speed claim, the label is the last
        // place to round up. Raise this only against an A/B measured on 2.5.
        speedupLabel: '~35-45% faster',
        steps: 8,
        stage2Steps: 3,
        guidance: 1.0,
        teacache: true,
        teacacheThresh: null,
        requiresAdapter: 'ltx-2.5-22b-distilled-lora-450.safetensors',
        modes: TWO_STAGE_SPEED_PROFILE_MODES,
      }),
    ]),
  }),
});

const isEntry = (entry) => !!entry && typeof entry === 'object' && typeof entry.id === 'string';

/**
 * Attach `speedProfiles` to shipped entries that don't already carry the key.
 * Pure; returns a new array and never mutates the input entries.
 *
 * Preservation contract (mirrors migration 295, and the two sibling
 * decorators):
 *   - `'speedProfiles' in entry` → user/existing value wins (including `null`
 *     or `[]`, the explicit "no speed profiles on this model" override)
 *   - entry id not shipped → custom model, left as-is
 *   - `repo` or `revision` differs from the pin → left as-is (see the pin
 *     guard rationale in the module docblock)
 */
export const applyVideoSpeedProfiles = (list) => {
  if (!Array.isArray(list)) return list;
  return list.map((entry) => {
    if (!isEntry(entry)) return entry;
    if ('speedProfiles' in entry) return entry;
    const spec = VIDEO_SPEED_PROFILES[entry.id];
    if (!spec) return entry;
    if (spec.shippedRepo !== null && entry.repo !== spec.shippedRepo) return entry;
    if (spec.shippedRevision !== null && entry.revision !== spec.shippedRevision) return entry;
    return { ...entry, speedProfiles: spec.profiles.map((p) => ({ ...p })) };
  });
};

/**
 * Absence and the default id are the same request — so a client that always
 * submits its picker value doesn't create a second spelling of "unchanged".
 */
export const isDefaultSpeedProfile = (id) => id == null || id === '' || id === SPEED_PROFILE_DEFAULT_ID;

/**
 * The profiles a model declares, or `[]`. Deliberately does NOT prepend the
 * implicit default entry: that is a PRESENTATION rule (the picker owns it, the
 * same way `TextEncoderPicker` owns hide-when-there-is-no-real-choice), and
 * folding it in here would make every model look like it has a profile.
 */
export const videoSpeedProfiles = (model) => (
  Array.isArray(model?.speedProfiles) ? model.speedProfiles.filter((p) => typeof p?.id === 'string') : []
);

/**
 * Does this model offer any speed profile at all?
 */
export const supportsSpeedProfiles = (model) => videoSpeedProfiles(model).length > 0;

/**
 * Look up one declared profile by id. `null` for the default id, an unknown
 * id, or a model with none.
 */
export const findVideoSpeedProfile = (model, id) => (
  isDefaultSpeedProfile(id) ? null : videoSpeedProfiles(model).find((p) => p.id === id) || null
);

/**
 * Why a requested profile could not be applied — a machine-readable reason plus
 * a sentence the UI/log can show verbatim. `null` when the profile applies.
 *
 * Returned rather than thrown: the render still proceeds at the model's own
 * default sampler. The point of the feature is an honest fallback, not a 400
 * on a knob that only ever makes a render faster.
 */
export const speedProfileDeclineReason = ({ model, profileId, mode }) => {
  if (isDefaultSpeedProfile(profileId)) return null;
  if (model?.samplerLocked === true) {
    return {
      code: 'SPEED_PROFILE_SAMPLER_LOCKED',
      message: `${model?.name || model?.id || 'This model'} locks its own validated sampler, so speed profiles don't apply.`,
    };
  }
  const profile = findVideoSpeedProfile(model, profileId);
  if (!profile) {
    return {
      code: 'SPEED_PROFILE_UNSUPPORTED',
      message: `Speed profile "${profileId}" isn't offered by ${model?.name || model?.id || 'this model'} — rendering at its default sampler.`,
    };
  }
  const modes = Array.isArray(profile.modes) ? profile.modes : TWO_STAGE_SPEED_PROFILE_MODES;
  // Absence is read as the plain text render — which is only SAFE because the
  // caller resolves the runner's own inference first (see `inferEffectiveMode`
  // below, used by generateVideo). A direct caller that hands us a raw absent
  // `mode` alongside keyframes would otherwise be told the profile applies,
  // while buildLtx2Args infers `fflf` and runs an entirely different pipeline.
  const effectiveMode = mode == null || mode === '' ? 'text' : mode;
  if (!modes.includes(effectiveMode)) {
    return {
      code: 'SPEED_PROFILE_MODE_UNSUPPORTED',
      message: `The ${profile.name} profile is validated for ${modes.join('/')} renders only — ${effectiveMode} renders use a different pipeline and keep the default sampler.`,
    };
  }
  return null;
};

/**
 * Resolve the sampler override a requested speed profile implies.
 *
 * Returns `null` when nothing should change (default profile, unsupported
 * model, incompatible mode) — the caller then leaves every dial exactly where
 * it was. Otherwise returns the concrete override the render should use.
 *
 * @returns {{ id, steps, stage2Steps, guidance, teacache, teacacheThresh, requiresAdapter } | null}
 */
export const resolveVideoSpeedProfile = ({ model, profileId, mode }) => {
  if (speedProfileDeclineReason({ model, profileId, mode })) return null;
  const profile = findVideoSpeedProfile(model, profileId);
  if (!profile) return null;
  return {
    id: profile.id,
    steps: profile.steps,
    stage2Steps: profile.stage2Steps ?? null,
    guidance: profile.guidance,
    teacache: profile.teacache === true,
    teacacheThresh: profile.teacacheThresh ?? null,
    requiresAdapter: profile.requiresAdapter ?? null,
  };
};

/**
 * The mode a render will EFFECTIVELY run in when the caller omitted one.
 *
 * `buildLtx2Args` infers the helper mode from the conditioning it was handed —
 * `fflf` from multi-keyframes, `image` from a source image — so a gate that
 * read a raw absent `mode` as "text" would green-light a two-stage schedule for
 * a KeyframeInterpolation render the profile was never validated on. This is
 * that same inference, narrowed to what the gate needs, so the two agree by
 * construction rather than by comment.
 */
export const inferEffectiveVideoMode = ({ mode, keyframes, sourceImagePath, extendFromVideoPath, audioFilePath }) => {
  if (mode != null && mode !== '') return mode;
  if (Array.isArray(keyframes) && keyframes.length >= 2) return 'fflf';
  if (audioFilePath) return 'a2v';
  if (extendFromVideoPath) return 'extend';
  if (sourceImagePath) return 'image';
  return 'text';
};

/**
 * Resolve a profile for a MULTI-RENDER request (a chained render), where the
 * chunks do not all run in the same mode.
 *
 * A chained render is one clip. Applying the profile to the chunks whose mode
 * supports it and not to the rest would stitch a fast chunk (CFG 1.0, fewer
 * stage-2 steps, TeaCache) onto quality chunks — a visible seam mid-clip, and
 * a chain-level ETA that assumes a speed-up three quarters of the render never
 * takes. So the profile applies to the whole chain or to none of it.
 *
 * This is the same "degrade rather than half-apply" rule `resolveVideoSampler`
 * enforces for one render, lifted to the unit the user actually receives.
 *
 * @param {string[]} modes - every mode the chunks will run in
 * @returns the override when EVERY mode accepts it, otherwise `null`
 */
export const resolveVideoSpeedProfileForModes = ({ model, profileId, modes }) => {
  if (isDefaultSpeedProfile(profileId)) return null;
  const list = Array.isArray(modes) && modes.length > 0 ? modes : [null];
  if (list.some((mode) => speedProfileDeclineReason({ model, profileId, mode }))) return null;
  return resolveVideoSpeedProfile({ model, profileId, mode: list[0] });
};

/**
 * The first reason any of `modes` declines the profile, for logging. `null`
 * when every mode accepts it (or the request is the default profile).
 */
export const speedProfileDeclineReasonForModes = ({ model, profileId, modes }) => {
  if (isDefaultSpeedProfile(profileId)) return null;
  const list = Array.isArray(modes) && modes.length > 0 ? modes : [null];
  for (const mode of list) {
    const reason = speedProfileDeclineReason({ model, profileId, mode });
    if (reason) return reason;
  }
  return null;
};

/**
 * The effective sampler for one render — the SINGLE place the precedence
 * between a locked sampler, a speed profile, an explicit user value and the
 * model default is decided.
 *
 * Precedence, highest first:
 *   1. `model.samplerLocked` — the model's own validated schedule wins over
 *      everything, including a user-typed value (pre-existing contract).
 *   2. An applicable speed profile — the user picked a named schedule, so it
 *      drives steps AND guidance together. Same shape as a locked sampler:
 *      the UI disables the two inputs while a profile is active, and a direct
 *      API caller that sends both gets the profile it asked for rather than a
 *      half-applied hybrid whose speed claim would be false.
 *   3. An explicit `steps` / `guidanceScale` from the request.
 *   4. The model's registry defaults.
 *
 * `stage2Steps` is `null` unless a profile asked for an explicit one, so the
 * pipeline keeps its own default on every other path.
 *
 * Exported and pure so the render path and the chained-render ETA agree by
 * construction instead of by two copies of the same rule.
 */
export const resolveVideoSampler = ({ model, steps, guidanceScale, speedProfile = null }) => {
  if (model?.samplerLocked === true) {
    return { steps: model.steps, guidance: model.guidance, stage2Steps: null };
  }
  if (speedProfile) {
    return {
      steps: speedProfile.steps,
      guidance: speedProfile.guidance,
      stage2Steps: speedProfile.stage2Steps ?? null,
    };
  }
  return {
    steps: steps ? Number(steps) : model?.steps,
    guidance: guidanceScale != null && guidanceScale !== '' ? Number(guidanceScale) : model?.guidance,
    stage2Steps: null,
  };
};

/**
 * Validate one platform's video list for speed-profile problems. Pure — returns
 * `{ id, profileId, reason }` rows (empty when sound). Callers decide whether
 * to warn-and-strip (load path) or fail (the test that pins the shipped
 * registry).
 *
 * Checked, in the order a hand-edit is most likely to break:
 *   - `speedProfiles` is an array (or absent)
 *   - every profile has a non-empty string id, unique within the entry
 *   - no profile reuses the reserved default id
 *   - `steps` is a positive integer and `guidance` a finite non-negative number
 *     — a profile that resolved to NaN steps would spawn a broken render
 *   - `stage2Steps`, when present, is a positive integer
 *   - `modes` is a non-empty array of strings
 *   - the entry is not also `samplerLocked` (two authorities over one dial)
 */
export const validateSpeedProfileTable = (list) => {
  if (!Array.isArray(list)) return [];
  const problems = [];
  const fail = (id, profileId, reason) => problems.push({ id, profileId, reason });
  for (const entry of list) {
    if (!isEntry(entry)) continue;
    if (!('speedProfiles' in entry) || entry.speedProfiles == null) continue;
    if (!Array.isArray(entry.speedProfiles)) {
      fail(entry.id, null, 'speedProfiles must be an array');
      continue;
    }
    if (entry.speedProfiles.length > 0 && entry.samplerLocked === true) {
      fail(entry.id, null, 'a samplerLocked model cannot also declare speedProfiles');
      continue;
    }
    // Only buildLtx2Args emits the profile's runner flags. On any other runtime
    // the step schedule would still apply (resolveVideoSampler is runtime-blind)
    // while TeaCache and the adapter silently would not — and no SPEEDPROFILE:
    // report would come back, so history would read as a FULL speed-up that
    // never happened. Refuse the declaration rather than ship that gap.
    if (entry.speedProfiles.length > 0 && !LTX2_FAMILY_RUNTIMES.includes(entry.runtime)) {
      fail(entry.id, null, `speedProfiles need an LTX-2-family runtime (got "${entry.runtime || 'none'}") — no other builder emits their flags`);
      continue;
    }
    const seen = new Set();
    for (const profile of entry.speedProfiles) {
      const pid = profile?.id;
      if (typeof pid !== 'string' || pid.length === 0) {
        fail(entry.id, pid ?? null, 'profile id must be a non-empty string');
        continue;
      }
      if (pid === SPEED_PROFILE_DEFAULT_ID) {
        fail(entry.id, pid, `"${SPEED_PROFILE_DEFAULT_ID}" is the reserved default profile id`);
        continue;
      }
      if (seen.has(pid)) {
        fail(entry.id, pid, `duplicate profile id "${pid}"`);
        continue;
      }
      seen.add(pid);
      if (!Number.isInteger(profile.steps) || profile.steps <= 0) {
        fail(entry.id, pid, 'steps must be a positive integer');
        continue;
      }
      if (!Number.isFinite(profile.guidance) || profile.guidance < 0) {
        fail(entry.id, pid, 'guidance must be a finite number >= 0');
        continue;
      }
      if (profile.stage2Steps != null
        && (!Number.isInteger(profile.stage2Steps) || profile.stage2Steps <= 0)) {
        fail(entry.id, pid, 'stage2Steps must be a positive integer when present');
        continue;
      }
      if (!Array.isArray(profile.modes) || profile.modes.length === 0
        || profile.modes.some((m) => typeof m !== 'string' || m.length === 0)) {
        fail(entry.id, pid, 'modes must be a non-empty array of strings');
        continue;
      }
      // Both of these reach the helper's argv. A non-numeric threshold exits the
      // child at argparse (`type=float`) mid-job, and a non-string adapter name
      // throws out of spawn — so they belong in the same fail-closed check as
      // the sampler numbers rather than being trusted from a hand-edited file.
      if (profile.teacacheThresh != null
        && (!Number.isFinite(profile.teacacheThresh) || profile.teacacheThresh <= 0)) {
        fail(entry.id, pid, 'teacacheThresh must be a positive number when present');
        continue;
      }
      if (profile.requiresAdapter != null
        && (typeof profile.requiresAdapter !== 'string' || profile.requiresAdapter.length === 0)) {
        fail(entry.id, pid, 'requiresAdapter must be a non-empty string when present');
      }
    }
  }
  return problems;
};

/**
 * Load-time guard: strip every invalid `speedProfiles` list, logging each
 * problem. A user-edited (or migration-stale) registry must not be able to
 * offer a profile that would spawn a broken render, and must not crash boot
 * (`loadMediaModels` runs at import time). Returns the input array unchanged
 * when the table is sound, so the common path allocates nothing.
 */
export const sanitizeSpeedProfiles = (list) => {
  const problems = validateSpeedProfileTable(list);
  if (problems.length === 0) return list;
  const bad = new Set(problems.map((p) => p.id));
  for (const p of problems) {
    console.log(`⚠️ media-models: dropping speedProfiles on "${p.id}" — ${p.reason}`);
  }
  return list.map((entry) => {
    if (!isEntry(entry) || !bad.has(entry.id)) return entry;
    const { speedProfiles: _dropped, ...rest } = entry;
    return rest;
  });
};