Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

64.1 KB
Raw Blame
/**
 * Shared scaffolding for migrations. Several families live here:
 *
 *   1. Hash-driven prompt-replace migrations — every one from 003 onward uses
 *      `makePromptReplaceMigration` to collapse onto ~50 lines (hash table +
 *      label + customized-skip hint).
 *   2. Dashboard-layout seeding migrations — `readLayoutsDoc` /
 *      `writeLayoutsDoc` collapse the read → JSON.parse → `Array.isArray`
 *      guard → write shell shared by every migration that mutates built-in
 *      layouts in `data/dashboard-layouts.json` (029, 030, 033, …).
 *   3. Monolithic → per-record split migrations — `makeSplitMigration`.
 *   4. Brain seed-record migrations — `makeBrainSeedMigration`.
 *   5. Provider-seed migrations — `makeProviderSeedMigration` collapses the
 *      read → guard → add-missing-ids → write shell every migration that ships
 *      a new `data/providers.json` entry repeats (149, 152, 185, 195, 201, 231).
 *   6. Media-model-registry migrations — `readMediaRegistry` /
 *      `writeMediaRegistry` collapse the strict-read → absent/unreadable skip →
 *      bucket-array guard shell shared by every migration that patches
 *      `data/media-models.json` (244, 247, …).
 *   7. Seeded-provider-tier bumps — `makeSeededProviderTierMigration` collapses
 *      the exact-match → rewrite-models → swap-retired-pointers shell that
 *      032 / 058 / 153 / 206 each hand-copied. Those four stay frozen; the
 *      factory is for the next RETIREMENT bump. It expresses id→id retirement
 *      only: `idMap` maps 1:1 over `oldModels`, and `swapTierPointers` moves
 *      EVERY pointer off a retired id. This is a **pristine-list rewrite** —
 *      only an EXACT match of the prior seeded `models` array qualifies, so a
 *      curated list is left alone rather than reset.
 *   7b. Additive provider-model inserts — `makeAdditiveProviderInsertMigration`
 *       (337 is the first and, so far, only member) is the opposite policy for
 *       the SAME `data/providers.json` targets: it runs against a list the user
 *       may have curated, so it never rewrites or drops anything — it only
 *       splices `current` in immediately after `retired` when `retired` is
 *       listed and `current` is not, regardless of what else the list holds.
 *       Tier pointers are left alone (a curated list is exactly where
 *       re-pointing would override a deliberate choice). Reach for family 7
 *       when retiring an id from the shipped seed; reach for 7b when a later
 *       tier needs to be OFFERED without disturbing whatever the user already
 *       has listed.
 *   8. CoS config reads — `readCosConfig` resolves the durable CoS settings a
 *      migration needs to honour an install-specific path, across both the
 *      pre- and post-339 layouts (`data/cos/state.json`'s `config` slice and
 *      `data/cos/config.json`).
 *   9. Plain document reads — `readJsonOrNull` is the "absent or unparseable
 *      → null, anything else throws" reader that thirty-odd applied migrations
 *      each declared as a local `readJson`. Those stay frozen;
 *      `migrationConventions.guard.test.js` stops a new migration from
 *      declaring its own copy of any helper exported here.
 *
 * Families 5, 7, and 7b all target `data/providers.json` and share its
 * read → parse → shape-guard preamble via `readProvidersDoc`; each still owns
 * its own log copy and result shape.
 *
 * The runner (`scripts/run-migrations.js`) explicitly skips `_`-prefixed
 * files so this module is never imported as a migration.
 */

import { readFile, writeFile, unlink, readdir, rename, mkdir, stat } from 'fs/promises';
import { join } from 'path';
import { pathToFileURL } from 'url';
import { createHash } from 'crypto';
import { atomicWrite, readJSONFileStrict } from '../../server/lib/fileUtils.js';
import { VIDEO_BUCKET_MLX, resolveVideoBucketKey } from '../../server/lib/mediaModelBuckets.js';

// ---- monolithic → per-record split-migration family ----
//
// Migrations 034 / 035 / 036 / 059 all split a single `data/<legacy>.json`
// (`{ [recordsKey]: [...] }`) into per-record `data/<typeDir>/<id>/index.json`
// files plus a type-level `data/<typeDir>/index.json` that stamps the storage
// `schemaVersion`. They share the same gate1 (already-applied) / gate2
// (fresh-install) / recovery / split / stamp / backup skeleton and differ only
// in a handful of config values. `makeSplitMigration` collapses that skeleton;
// see `server/lib/collectionStore.js` for the on-disk layout it targets.
//
// Behavioral divergences across the four are PRESERVED via flags, not
// homogenized — they were deliberate (a split that silently changed what an
// applied migration did to existing data would be a corruption bug):
//
//   - `onUnreadable: 'return' | 'throw'` — 034/035/036 return
//     `{ ok:false, reason:'unreadable' }` (the runner marks them applied; a
//     repaired file is NOT re-split). 059 THROWS so the runner leaves it
//     pending and a repaired file re-splits on the next boot.
//   - `dedupe` — 059 claims each id as it writes so a duplicate id later in the
//     legacy array is skipped (first-wins, mirroring the old monolithic
//     `listCollections` dedup). 034/035/036 never had duplicate-id concerns.
//   - `extraValid(record)` — 059 additionally rejects a blank/missing `name`
//     (mirroring `sanitizeCollection`'s read-time drop) so an unsanitizable
//     leading row can't shadow a later valid duplicate. The others validate id
//     only.
//   - `buildConfig(doc)` — 034 moves the legacy cross-record `runs[]` into
//     `config.runs`; the others stamp `config: {}`.
//   - `idPattern` / `invalidWarn` / `recordNoun` — per-kind id shape + log copy.
//   - `recordsShape: 'array' | 'map'` — the legacy container shape. 034/035/036/
//     059 store `{ [recordsKey]: [ {id, …}, … ] }` (an ARRAY; id comes from
//     `record.id`) — the default `'array'`. Brain entity stores (migration 200)
//     store `{ [recordsKey]: { <id>: record } }` (an object MAP; id is the KEY,
//     the record value carries no id field). `'map'` iterates `Object.entries`,
//     derives the id from the key, and — because the value is written verbatim —
//     PRESERVES in-place tombstones (`{ _deleted: true, updatedAt, … }`) so a
//     split can't drop the last-writer-wins markers federation depends on. Map
//     mode also skips the reserved keys `__proto__` / `constructor` / `prototype`
//     (a legacy JSON map can carry them as own keys; the collectionStore id
//     allowlist would accept them, so guard here) — left in the backup.

const splitFileExists = (path) => stat(path).then(() => true, (err) => {
  if (err.code === 'ENOENT') return false;
  throw err;
});

// Two read variants so we distinguish "missing file" from "present but
// unparseable" — the latter is a recovery-required state reported through the
// migration's return value (or a throw) rather than crashing the boot.
const splitReadJsonStrict = async (path) => {
  const raw = await readFile(path, 'utf-8').catch((err) => {
    if (err.code === 'ENOENT') return null;
    throw err;
  });
  if (raw == null) return null;
  return JSON.parse(raw);
};

const splitReadJsonTolerant = async (path) => {
  const raw = await readFile(path, 'utf-8').catch((err) => {
    if (err.code === 'ENOENT') return null;
    throw err;
  });
  if (raw == null) return null;
  try { return JSON.parse(raw); } catch { return { __unreadable: true }; }
};

// Atomic write (temp + rename) so a crash mid-write can't leave a truncated
// `<id>/index.json`. Without it, a retry's `splitExistingRecordIds` scan would
// see the partial file, trust it as already-split, skip re-splitting from the
// authoritative legacy value, then stamp + rename the legacy store — and the
// truncated record would forever load as null. rename() is atomic within one
// filesystem, and tmp sits in the same dir as its target, so the swap is safe.
// Also used by the provider-seed family, where a truncated `data/providers.json`
// would cost the user every stored apiKey and provider customization, and by
// the prompt-replace family: a half-written stage prompt hashes as neither the
// accepted-old nor the shipped body, so the next pass would classify it as
// user-customized and leave the truncated prompt in place forever.
export const writeTextAtomic = async (path, content) => {
  const tmp = `${path}.tmp-${process.pid}`;
  await writeFile(tmp, content);
  await rename(tmp, path);
};

export const writeJsonAtomic = (path, value) =>
  writeTextAtomic(path, JSON.stringify(value, null, 2) + '\n');

/**
 * Read a JSON document, or null when it is absent (ENOENT) or not valid JSON.
 * Any other read error (EACCES, EISDIR, …) propagates: a store that exists
 * but cannot be read is not "nothing to migrate", and treating it so would
 * let a later write replace it. A migration that must tell "absent" from
 * "unreadable" apart (to skip one and warn on the other) reads through
 * `readJSONFileStrict` from `server/lib/fileUtils.js` instead.
 */
export const readJsonOrNull = async (path) => {
  const raw = await readFile(path, 'utf-8').catch((err) => {
    if (err.code === 'ENOENT') return null;
    throw err;
  });
  if (raw == null) return null;
  try { return JSON.parse(raw); } catch { return null; }
};

// Reserved keys a legacy `map`-shape doc can carry as own properties (JSON.parse
// surfaces a literal `"__proto__"` key as an own property). They pass a typical
// `idPattern`, so a map split must drop them explicitly rather than write a
// `__proto__/index.json` record dir — left in the backup for manual recovery.
// Shared with the provider-seed family below, which faces the mirror-image
// hazard: these keys are INHERITED on every plain object, so a `map[id]`
// presence probe reads truthy for them even on an empty map.
const RESERVED_MAP_KEYS = new Set(['__proto__', 'constructor', 'prototype']);

// Scan the type dir for records already split in a prior partial run. Uses
// `withFileTypes` so stray non-directory entries (user `.bak` files, editor
// swap files) are skipped without statting INTO them — `stat('foo.bak/index.json')`
// would raise ENOTDIR and crash the migration.
async function splitExistingRecordIds(typeDir) {
  const ids = new Set();
  if (!await splitFileExists(typeDir)) return ids;
  // No catch-to-empty: a failed listing is not proof that no records exist, and
  // an empty set would let the legacy values overwrite fresher split records.
  const entries = await readdir(typeDir, { withFileTypes: true });
  for (const entry of entries) {
    if (entry.name === 'index.json' || entry.name.startsWith('.') || !entry.isDirectory()) continue;
    if (await splitFileExists(join(typeDir, entry.name, 'index.json'))) ids.add(entry.name);
  }
  return ids;
}

/**
 * Build a monolithic→per-record split migration's `up()`. Returns `{ up }`.
 *
 * Required config:
 *   - `migrationLabel`   — log tag, e.g. `'migration 034'`
 *   - `typeDirName`      — `data/<typeDirName>/` (e.g. `'universes'`)
 *   - `legacyFilename`   — `data/<legacyFilename>` (e.g. `'universe-builder.json'`)
 *   - `backupSuffix`     — appended to the legacy path on backup (e.g. `'.bak-034'`)
 *   - `typeSchemaVersion`— the type-level layout version this migration stamps
 *   - `typeLabel`        — the `type` field written into the type index
 *   - `recordsKey`       — array key in the legacy doc (e.g. `'universes'`)
 *   - `idPattern`        — RegExp a record id must match to be split
 *   - `recordNoun`       — singular noun for the split-count log (e.g. `'universe'`)
 *
 * Optional:
 *   - `buildConfig(doc)` — returns the type index `config`; default `() => ({})`
 *   - `extraValid(record)`— extra per-record validity gate beyond id (059's name check)
 *   - `dedupe`           — claim ids as written so later duplicates skip (059); default false
 *   - `onUnreadable`     — `'return'` (default) or `'throw'`
 *   - `recordsShape`     — `'array'` (default; id from `record.id`) or `'map'`
 *                          (legacy `{ [recordsKey]: { <id>: record } }`; id is the
 *                          object key, tombstones preserved, reserved keys skipped)
 */
export function makeSplitMigration({
  migrationLabel,
  typeDirName,
  legacyFilename,
  backupSuffix,
  typeSchemaVersion,
  typeLabel,
  recordsKey,
  idPattern,
  recordNoun,
  buildConfig = () => ({}),
  extraValid = null,
  dedupe = false,
  onUnreadable = 'return',
  recordsShape = 'array',
}) {
  const up = async ({ rootDir }) => {
    const dataDir = join(rootDir, 'data');
    const typeDir = join(dataDir, typeDirName);
    const typeIndexPath = join(typeDir, 'index.json');
    const legacyPath = join(dataDir, legacyFilename);
    const backupPath = legacyPath + backupSuffix;

    // Gate 1: type index already at/above target → no-op (a re-run after full
    // success lands here). Strict read — a corrupted index.json should throw.
    const typeIndex = await splitReadJsonStrict(typeIndexPath);
    if (typeIndex && typeIndex.schemaVersion >= typeSchemaVersion) {
      console.log(`📦 ${migrationLabel}: ${typeLabel} already at schemaVersion=${typeIndex.schemaVersion} — no-op`);
      return { ok: true, reason: 'already-applied' };
    }

    const legacyExists = await splitFileExists(legacyPath);
    const backupExists = await splitFileExists(backupPath);

    // Gate 2: fresh install — no legacy, no backup. Stamp the type index so the
    // boot-time verifyCollectionVersions doesn't flag it missing.
    if (!legacyExists && !backupExists) {
      await mkdir(typeDir, { recursive: true });
      await writeJsonAtomic(typeIndexPath, {
        schemaVersion: typeSchemaVersion,
        type: typeLabel,
        updatedAt: new Date().toISOString(),
        config: buildConfig(null),
      });
      console.log(`📦 ${migrationLabel}: fresh install — stamped data/${typeDirName}/index.json @ v${typeSchemaVersion}`);
      return { ok: true, reason: 'fresh-install' };
    }

    // Recovery gate: a prior run split records but didn't finish renaming the
    // legacy file. Use whichever file is present — prefer the live file if both
    // somehow exist (the split must not have happened).
    const sourcePath = legacyExists ? legacyPath : backupPath;
    const doc = await splitReadJsonTolerant(sourcePath);
    if (!doc || typeof doc !== 'object' || doc.__unreadable) {
      if (onUnreadable === 'throw') {
        // THROW (don't return) so the runner does NOT mark this migration applied
        // (run-migrations.js records any migration whose up() resolves) — keeps it
        // pending so a repaired file re-splits on the next boot. Server boot itself
        // survives via the runMigrations().catch() in server/index.js.
        throw new Error(`${migrationLabel}: ${sourcePath} is unreadable — repair or remove it, then reboot to retry the split`);
      }
      console.warn(`⚠️ ${migrationLabel}: ${sourcePath} unreadable — skipping. Resolve manually before next boot.`);
      return { ok: false, reason: 'unreadable' };
    }

    // Normalize both container shapes into `[id, record]` pairs so the split
    // loop below is shape-agnostic. Array: id from `record.id` (034/035/036/059).
    // Map: id from the object KEY, value written verbatim (migration 200 — brain
    // `{ records: { <id>: record } }`, tombstones and all). Both feed the same
    // downstream object/id/idPattern validation, so a null/non-string id or a
    // non-object record is rejected there, not here.
    let pairs;
    if (recordsShape === 'map') {
      const container = doc[recordsKey];
      // Distinguish "records key absent" (a legitimately empty store → 0 records)
      // from "records present but NOT an object map" (corruption). Silently
      // emptying the latter would stamp v1, rename the legacy file to backup, and
      // mark the migration applied while the real data sits stranded and unread in
      // the backup — so honor `onUnreadable` for a malformed container exactly as
      // for unparseable JSON.
      if (container != null && !(typeof container === 'object' && !Array.isArray(container))) {
        if (onUnreadable === 'throw') {
          throw new Error(`${migrationLabel}: ${sourcePath} has a malformed "${recordsKey}" container (expected an object map) — repair or remove it, then reboot to retry the split`);
        }
        console.warn(`⚠️ ${migrationLabel}: ${sourcePath} has a malformed "${recordsKey}" container — skipping. Resolve manually before next boot.`);
        return { ok: false, reason: 'unreadable' };
      }
      pairs = Object.entries(container ?? {}).filter(([key]) => !RESERVED_MAP_KEYS.has(key));
    } else {
      const records = Array.isArray(doc[recordsKey]) ? doc[recordsKey] : [];
      pairs = records.map((record) => [record?.id, record]);
    }
    const existingIds = await splitExistingRecordIds(typeDir);
    await mkdir(typeDir, { recursive: true });

    let written = 0;
    let skipped = 0;
    let invalid = 0;
    for (const [rawId, record] of pairs) {
      if (!record || typeof record !== 'object') {
        invalid += 1;
        console.warn(`⚠️ ${migrationLabel}: skipping non-object ${recordNoun} record (left in backup for manual recovery)`);
        continue;
      }
      const id = typeof rawId === 'string' ? rawId : null;
      if (!id || !idPattern.test(id)) {
        invalid += 1;
        console.warn(`⚠️ ${migrationLabel}: skipping ${recordNoun} with invalid id "${id}" (left in backup for manual recovery)`);
        continue;
      }
      if (extraValid && !extraValid(record)) {
        invalid += 1;
        console.warn(`⚠️ ${migrationLabel}: skipping ${recordNoun} id "${id}" — failed validity check (left in backup for manual recovery)`);
        continue;
      }
      if (existingIds.has(id)) {
        // Already split — in a prior partial run (trust the on-disk per-record
        // file, which may hold fresher post-crash state) OR, when `dedupe` is
        // on, earlier in THIS loop (a duplicate id within the legacy array →
        // first-wins, matching the old monolithic reader's dedup).
        skipped += 1;
        continue;
      }
      const recordDir = join(typeDir, id);
      await mkdir(recordDir, { recursive: true });
      await writeJsonAtomic(join(recordDir, 'index.json'), record);
      if (dedupe) existingIds.add(id); // first-wins: later duplicates skip above
      written += 1;
    }

    // Stamp the type index AFTER all records land so a crash mid-split leaves
    // it missing — the next boot's gate 1 won't trip and gate 2/recovery re-runs.
    await writeJsonAtomic(typeIndexPath, {
      schemaVersion: typeSchemaVersion,
      type: typeLabel,
      updatedAt: new Date().toISOString(),
      config: buildConfig(doc),
    });

    // Backup the legacy file (skip when the recovery path was driven from the
    // backup). Renaming preserves data; manual restore is
    // `mv <legacy>${backupSuffix} <legacy>`.
    if (legacyExists) await rename(legacyPath, backupPath);

    console.log(
      `📦 ${migrationLabel}: split ${written} ${recordNoun}(s) into data/${typeDirName}/<id>/index.json ` +
      `(${skipped} already split, ${invalid} invalid); stamped index.json @ v${typeSchemaVersion}; ` +
      `legacy file backed up as ${legacyFilename}${backupSuffix}`,
    );

    return { ok: true, reason: 'split', written, skipped, invalid };
  };

  return { up };
}

/**
 * Newline-normalized MD5. Both `\r\n` and bare `\r` collapse to `\n` before
 * hashing so Windows checkouts of the same template hash identically.
 */
export const md5 = (str) => {
  const normalized = str.replace(/\r\n/g, '\n').replace(/\r/g, '\n');
  return createHash('md5').update(normalized).digest('hex');
};

/**
 * Core scan — exposed so tests can pass synthetic `accepted` / `current`
 * tables to exercise the OLD→NEW branch without pinning to a real shipped
 * hash.
 *
 * Per-migration opt-ins (both default `false`):
 *
 * - `createIfMissing` — when the data-side file is absent, copy the sample
 *   file in. Used by migration 005, whose `pipeline-arc-resolve.md` may not
 *   have shipped in `data.reference/` yet at the time it was authored.
 *
 * - `retireOnSampleMissing` — when the sample-side file is absent (the prompt
 *   was renamed or retired by a later commit), treat it as a soft delete:
 *   unlink the data-side file when it still matches an accepted-old hash,
 *   and warn (counting as `skipped`) when it's been customized. Without this
 *   flag, a missing sample raises an ENOENT at read time. Used by migration
 *   003 to handle the `pipeline-tv-script.md` → `pipeline-teleplay.md`
 *   rename.
 *
 * `subdirs` mirrors a migration's `DRIFT_SUBDIRS` export: a per-filename
 * override of the `prompts/<subdir>/` directory both sides resolve under.
 * Stage prompts (the default, `'stages'`) need no entry; prompt fragments in
 * `prompts/_partials/` do. Passing the migration's own `DRIFT_SUBDIRS` object
 * keeps the replace pass and the setup-data.js drift sweep reading one table.
 */
export async function applyPromptReplaceMigration({
  rootDir,
  accepted,
  current,
  label,
  customizedHint,
  createIfMissing = false,
  retireOnSampleMissing = false,
  subdirs = {},
}) {
  const subdirFor = (filename) => subdirs[filename] || 'stages';
  const dataPathFor = (filename) => join(rootDir, 'data', 'prompts', subdirFor(filename), filename);
  const samplePathFor = (filename) => join(rootDir, 'data.reference', 'prompts', subdirFor(filename), filename);
  const filenames = Object.keys(accepted);

  // Two phases so parallelism never changes the migration's failure semantics:
  //
  //   1. PLAN (parallel) — every file's reads + hash compares run concurrently
  //      (that's the dominant cost, and they share no state). Each yields a
  //      side-effect-free descriptor: which write/unlink to perform, which
  //      counter to bump, and any deferred log line. A read rejection here
  //      rejects the whole migration BEFORE any file has been mutated — so a
  //      fail-stop on a missing/unreadable file leaves data/ untouched (a
  //      stricter guarantee than the old serial loop, which could leave an
  //      already-processed prefix mutated).
  //   2. APPLY (ordered) — perform the planned writes/unlinks in Object.keys
  //      order so log output and the on-disk result are deterministic, exactly
  //      as the serial loop produced them.
  const plans = await Promise.all(filenames.map(async (filename) => {
    const dataPath = dataPathFor(filename);
    const samplePath = samplePathFor(filename);

    const existing = await readFile(dataPath, 'utf-8').catch((err) => {
      if (err.code !== 'ENOENT') throw err;
      return null;
    });

    if (existing === null) {
      if (createIfMissing) {
        const sampleContent = await readFile(samplePath, 'utf-8').catch(() => null);
        if (sampleContent != null) {
          return { write: { path: dataPath, content: sampleContent }, counter: 'created', log: `📄 created ${label}: ${filename}` };
        }
      }
      return { log: `📄 ${label} ${filename}: not present in data/, will be created by setup-data.js` };
    }

    const existingMd5 = md5(existing);
    const acceptedOld = accepted[filename];
    const matchesAcceptedOld = acceptedOld.includes(existingMd5);
    const matchesCurrent = existingMd5 === current[filename];

    if (retireOnSampleMissing) {
      // Peek at the sample before any other branch: a missing sample means
      // the prompt was renamed or retired upstream, so the on-disk file is
      // obsolete regardless of which shipped version it matches. Unmodified
      // copies (either accepted-old or current hash) are unlinked; customized
      // copies warn and skip.
      const sampleExists = await readFile(samplePath, 'utf-8').then(() => true, (err) => {
        if (err.code === 'ENOENT') return false;
        throw err;
      });
      if (!sampleExists) {
        if (matchesAcceptedOld || matchesCurrent) {
          return { unlink: dataPath, counter: 'retired', log: `🗑️  ${label} ${filename} was renamed/retired upstream — removed unmodified copy from data/` };
        }
        return {
          counter: 'skipped',
          warn: `⚠️  ${label} ${filename} was renamed/retired upstream but your local copy has been customized.\n` +
            `   Check data.reference/prompts/${subdirFor(filename)}/ for the replacement file and merge any custom edits manually.`,
        };
      }
    }

    if (matchesCurrent) {
      return { counter: 'alreadyCurrent' };
    }

    if (!matchesAcceptedOld) {
      return {
        counter: 'skipped',
        warn: `⚠️  ${label} ${filename} has been customized — skipping auto-update.\n` + customizedHint(filename),
      };
    }

    // Sample read stays in the plan phase: a missing sample for an accepted-old
    // file is the migration-author error the old loop surfaced by throwing, so
    // it must still abort before any write lands.
    const sampleContent = await readFile(samplePath, 'utf-8');
    return { write: { path: dataPath, content: sampleContent }, counter: 'updated', log: `✅ updated ${label}: ${filename}` };
  }));

  const counts = { updated: 0, alreadyCurrent: 0, skipped: 0, created: 0, retired: 0 };
  for (const plan of plans) {
    if (plan.unlink) await unlink(plan.unlink);
    if (plan.write) await writeTextAtomic(plan.write.path, plan.write.content);
    if (plan.warn) console.warn(plan.warn);
    if (plan.log) console.log(plan.log);
    if (plan.counter) counts[plan.counter]++;
  }

  return counts;
}

/**
 * Factory: returns `{ applyMigration, up }`. `skipFooter(count)` is optional
 * — when provided, the wrapped `up()` logs it after the per-file pass if any
 * file was skipped (user-facing guidance about what the customized files miss).
 */
export function makePromptReplaceMigration({
  accepted,
  current,
  label,
  customizedHint,
  skipFooter,
  createIfMissing = false,
  retireOnSampleMissing = false,
  subdirs = {},
}) {
  const applyMigration = (opts = {}) =>
    applyPromptReplaceMigration({
      accepted,
      current,
      label,
      customizedHint,
      createIfMissing,
      retireOnSampleMissing,
      subdirs,
      ...opts,
    });

  const up = async ({ rootDir }) => {
    const { updated, alreadyCurrent, skipped, created, retired } = await applyMigration({ rootDir });

    if (updated > 0 || created > 0 || retired > 0) {
      console.log(`📝 ${label} migration: ${updated} updated, ${created} created, ${retired} retired, ${alreadyCurrent} already current, ${skipped} skipped (customized)`);
    } else if (skipped > 0) {
      console.log(`📝 ${label} migration: all files either current or customized (${skipped} skipped)`);
    } else {
      console.log(`📝 ${label} migration: all files already up to date`);
    }

    if (skipped > 0 && skipFooter) {
      console.warn('\n' + skipFooter(skipped));
    }
  };

  return { applyMigration, up };
}

/**
 * Read + parse + guard `data/dashboard-layouts.json` for a layout-seeding
 * migration. Collapses the preamble every such migration repeats: resolve the
 * path, read the file (absent → fresh install, nothing to do), JSON-parse it
 * (unreadable → skip), and verify `doc.layouts` is an array.
 *
 * Returns a discriminated result:
 * - `{ ok: false, reason: 'no-state' | 'unreadable' | 'no-layouts-array', path }`
 *   — the caller short-circuits with `return { updated: 0, reason: result.reason }`.
 * - `{ ok: true, doc, path }` — mutate `doc.layouts` in place, then persist
 *   with `writeLayoutsDoc(path, doc)`.
 *
 * `label` is the migration's human tag (e.g. `'migration 029'`); it keeps the
 * no-state / unreadable log lines per-migration identifiable.
 */
export async function readLayoutsDoc({ rootDir, label }) {
  const path = join(rootDir, 'data', 'dashboard-layouts.json');
  const raw = await readFile(path, 'utf-8').catch((err) => {
    if (err.code === 'ENOENT') return null;
    throw err;
  });
  if (raw == null) {
    console.log(`📦 ${label}: no dashboard-layouts.json yet — fresh install will seed from defaults.`);
    return { ok: false, reason: 'no-state', path };
  }
  let doc;
  try {
    doc = JSON.parse(raw);
  } catch {
    console.log(`📦 ${label}: dashboard-layouts.json unreadable — skipping.`);
    return { ok: false, reason: 'unreadable', path };
  }
  if (!doc || !Array.isArray(doc.layouts)) {
    return { ok: false, reason: 'no-layouts-array', path };
  }
  return { ok: true, doc, path };
}

/** Persist a layouts doc with the canonical 2-space indentation (no trailing newline, matching the store). */
export async function writeLayoutsDoc(path, doc) {
  await writeTextAtomic(path, JSON.stringify(doc, null, 2));
}

// ---- dashboard-grid helpers ----
//
// Two grid shapes exist in `data/dashboard-layouts.json` history. Before
// migration 269 every grid entry was a rectangle (`{ id, x, y, w, h }`) and a
// seed had to find a free slot; 269 converted grids to a packing sequence
// (`{ id, x, w, h, order }`) where a seed appends at `max(order) + 1`. The
// rectangle helpers are kept for a migration that has to read or repair a
// legacy rectangle grid; a new widget seed uses `applyToLayout` (or the
// `makeWidgetSeedMigration` factory around it).

/** Axis-aligned overlap of two `{ x, y, w, h }` rectangles. */
export function rectsOverlap(a, b) {
  return a.x < b.x + b.w && b.x < a.x + a.w && a.y < b.y + b.h && b.y < a.y + a.h;
}

/** Whether `candidate` overlaps any entry of a rectangle grid. */
export function collidesWith(grid, candidate) {
  return grid.some((item) => rectsOverlap(item, candidate));
}

/**
 * Place `widgetId` at the preferred rectangle `slot` when it is free, else on
 * a clean row below everything in the grid (a user rearranged the layout but
 * kept the built-in id).
 */
export function pickGridEntry(grid, widgetId, slot) {
  const candidate = { id: widgetId, ...slot };
  if (!collidesWith(grid, candidate)) return candidate;
  const bottom = grid.reduce((max, it) => Math.max(max, (it.y ?? 0) + (it.h ?? 0)), 0);
  return { id: widgetId, x: 0, y: bottom, w: slot.w, h: slot.h };
}

/**
 * Seed `widgetId` into one layout of the post-269 (`order`-packed) shape:
 * append it to `widgets` when absent and to the end of the packing sequence
 * when it has no grid entry. Returns whether the layout changed. `targets`
 * (a Set of layout ids) limits the seed to built-in layouts; custom/user
 * layouts are the layout editor's to manage. Idempotent — a rerun heals
 * whichever of the two arrays is missing the widget.
 */
export function applyToLayout(layout, { widgetId, cell, targets }) {
  if (!layout || !targets.has(layout.id) || !Array.isArray(layout.widgets)) return false;
  const hasWidget = layout.widgets.includes(widgetId);
  const hasGridEntry = Array.isArray(layout.grid) && layout.grid.some((item) => item?.id === widgetId);
  if (hasWidget && hasGridEntry) return false;

  if (!hasWidget) layout.widgets = [...layout.widgets, widgetId];
  if (!hasGridEntry) {
    const grid = Array.isArray(layout.grid) ? layout.grid : [];
    const maxOrder = grid.reduce((max, item) => Math.max(max, Number.isFinite(item?.order) ? item.order : -1), -1);
    layout.grid = [...grid, { id: widgetId, x: 0, w: cell.w, order: maxOrder + 1, h: cell.h }];
  }
  return true;
}

/**
 * Factory for the "seed a new dashboard widget into built-in layouts"
 * migration family (327 seeded today-agenda by hand; 329 onward use this).
 *
 * Appends `widgetId` to each targeted built-in layout's `widgets` list and to
 * the end of its `grid` packing sequence (`{ id, x: 0, w, order: max+1, h }`)
 * — a gated widget that gates off leaves only harmless trailing space, so
 * appending is always safe. Custom/user layouts are intentionally untouched:
 * the layout editor is the user's source of truth for those. Idempotent —
 * re-running heals whichever of the two arrays is missing the widget and
 * reports `already-applied` when both carry it.
 *
 * @param {object} opts
 * @param {string} opts.label     Human tag, e.g. `'migration 329'`
 * @param {string} opts.widgetId  Registry id being seeded
 * @param {string[]} [opts.layoutIds] Built-in layout ids to seed (defaults to
 *   the Everything + Morning Review pair every widget seed so far targets)
 * @param {{ w: number, h: number }} opts.cell Grid cell size for the append
 * @param {string} opts.logLine   Success log line (count is appended)
 */
export function makeWidgetSeedMigration({ label, widgetId, layoutIds = ['default', 'morning-review'], cell, logLine }) {
  const targets = new Set(layoutIds);

  return {
    async up({ rootDir }) {
      const result = await readLayoutsDoc({ rootDir, label });
      if (!result.ok) return { updated: 0, reason: result.reason };
      const { doc, path } = result;
      let updated = 0;
      for (const layout of doc.layouts) {
        if (applyToLayout(layout, { widgetId, cell, targets })) updated += 1;
      }
      if (updated === 0) return { updated: 0, reason: 'already-applied' };
      await writeLayoutsDoc(path, doc);
      console.log(`${logLine} ${updated} built-in dashboard layout(s).`);
      return { updated };
    },
  };
}

// ---- media-model-registry migration family ----

const MEDIA_MODELS_REL_PATH = 'data/media-models.json';

/**
 * Read `data/media-models.json` whole, for a migration that patches something
 * other than one bucket's entry array (270 renames the bucket keys themselves).
 *
 * `readJSONFileStrict`'s `ok` flag is what separates "never written" (fresh
 * install — the seed in `data.reference/` already carries the change) from
 * "unreadable or corrupt", where rewriting the file would destroy a registry we
 * couldn't read. Both are skips, but only one of them is a warning.
 */
export async function readMediaRegistryConfig({ rootDir } = {}) {
  const path = join(rootDir, MEDIA_MODELS_REL_PATH);
  const { ok, value: config } = await readJSONFileStrict(path, null);
  if (!ok) {
    console.log(`⚠️ ${MEDIA_MODELS_REL_PATH}: unreadable or invalid JSON, skipping`);
    return { ok: false };
  }
  if (config == null) {
    console.log(`📄 ${MEDIA_MODELS_REL_PATH} not present — skipping (fresh install seeds from data.reference)`);
    return { ok: false };
  }
  return { ok: true, config, path };
}

/**
 * Read `data/media-models.json` for a registry-patching migration. Collapses
 * the preamble every such migration repeats (244, 247, …):
 * `readMediaRegistryConfig` above, plus pulling out one bucket's entry array.
 *
 * Returns `{ ok: false }` when the caller should do nothing, or
 * `{ ok: true, config, entries, bucketKey, path }` — mutate `config` in place,
 * then persist with `writeMediaRegistry(path, config)`. `bucket` selects which
 * video bucket lands in `entries`; `bucketKey` is the key it was actually found
 * under, which a caller that REPLACES the whole array must write back to.
 *
 * The bucket key is resolved canonical-first with a fallback to the pre-#4142
 * `macos` / `windows` spelling (see server/lib/mediaModelBuckets.js). Every
 * member of this family predates the rename and can therefore meet either
 * shape: an install upgrading from an older release still has the legacy keys
 * when they run, while a fresh install seeds `data/media-models.json` from the
 * canonical-keyed `data.reference/` copy before any migration runs.
 */
export async function readMediaRegistry({ rootDir, bucket = VIDEO_BUCKET_MLX } = {}) {
  const { ok, config, path } = await readMediaRegistryConfig({ rootDir });
  if (!ok) return { ok: false };
  const bucketKey = resolveVideoBucketKey(config?.video, bucket);
  const entries = bucketKey === null ? undefined : config.video[bucketKey];
  if (!Array.isArray(entries)) {
    console.log(`⚠️ ${MEDIA_MODELS_REL_PATH}: no video.${bucket}[] array — skipping`);
    return { ok: false };
  }
  return { ok: true, config, entries, bucketKey, path };
}

/** Persist a media-model registry with the canonical trailing-newline shape. */
export async function writeMediaRegistry(path, config) {
  await atomicWrite(path, `${JSON.stringify(config, null, 2)}\n`);
}

// ---- CoS config readers ----

/**
 * Read the durable CoS user configuration a migration needs to resolve an
 * install-specific path (`userTasksFile` / `cosTasksFile` are the ones that
 * matter today). Returns `{}` when there is nothing usable — every caller
 * already falls back to the shipped defaults.
 *
 * Config moved out of `data/cos/state.json` into `data/cos/config.json` in
 * migration 339. Both are consulted, config file first, so this resolves
 * correctly whether the reading migration runs before or after that split (and
 * on an install restored from a pre-split backup).
 *
 * Deliberately tolerant: a truncated/corrupt file left by a prior crash must
 * not abort the reading migration and every one after it — the state loader
 * owns quarantining and repairing those files.
 */
export async function readCosConfig({ rootDir, label = 'migration' }) {
  for (const relPath of ['data/cos/config.json', 'data/cos/state.json']) {
    const raw = await readFile(join(rootDir, relPath), 'utf-8').catch(() => null);
    if (raw === null) continue;
    let parsed;
    try {
      parsed = JSON.parse(raw);
    } catch {
      // Not necessarily the final answer — the next source in the list may
              // still carry a usable config, so this says only what was skipped.
              console.warn(`⚠️ ${label}: invalid JSON in ${relPath}; ignoring it`);
      continue;
    }
    // config.json IS the config; state.json nests it under `config`.
    const config = relPath.endsWith('config.json') ? parsed : parsed?.config;
    if (config && typeof config === 'object' && !Array.isArray(config)) return config;
  }
  return {};
}

// ---- brain seed-record migration family ----
//
// `scripts/setup-data.js` copies `data.reference/` wholesale, so a FRESH install
// picks up every seed record in `data.reference/brain/<type>.json` for free. But
// setup-data only copies MISSING files — an install whose `data/brain/<type>.json`
// already exists (even empty, from the feature's first boot) never receives a
// seed added later. Each such addition therefore ships a migration that merges
// its own ids in; 209 (the drum-format worked example) and 213 (the House of the
// Rising Sun drum arrangement) are the current members.
//
// They all want the same skeleton, so it lives here once:
//   - Since migration 200, brain stores live per-record at
//     `data/brain/<type>/<id>/index.json` (collectionStore layout) — that's the
//     primary write target.
//   - The legacy monolithic `data/brain/<type>.json` is ALSO topped up when it's
//     still present, so an install whose 200 split hasn't run yet still picks the
//     seed up when it finally does. It is never CREATED — doing so on a split
//     install would resurrect a shape nothing reads.
//   - An id already present — a user-edited copy, a peer-synced copy, or a
//     tombstone from a deliberate delete — is NEVER overwritten, so a deleted
//     seed stays deleted, and a second run is a no-op.
//   - Nothing is written when the existing record (or the legacy file) is
//     unreadable: possibly-recoverable user data beats a cosmetic starter record.
//
// Each migration owns an explicit `seedIds` list rather than "everything in the
// reference file", so a later seed addition gets its own migration instead of
// silently riding along on a re-run of an older one.

// Tagged read: 'missing' (ENOENT — nothing there, safe to create) is NOT the
// same as 'invalid' (exists but won't parse — user data a write would destroy).
// The migration runner executes before the service layer is wired, so this can't
// reach for server/lib helpers.
async function seedReadJsonTagged(path) {
  let raw;
  try {
    raw = await readFile(path, 'utf-8');
  } catch (err) {
    return err?.code === 'ENOENT' ? { state: 'missing' } : { state: 'invalid', error: err?.message };
  }
  try {
    return { state: 'ok', doc: JSON.parse(raw) };
  } catch (err) {
    return { state: 'invalid', error: err?.message };
  }
}

/**
 * Build a brain seed-record migration's `up()`. Returns `{ up }`.
 *
 *   - `logTag`     — emoji-prefixed log tag, e.g. `'🥁 drum-seed'`
 *   - `entityType` — brain entity type, e.g. `'songs'` (names all three paths)
 *   - `seedIds`    — the record ids THIS migration owns
 *   - `seedLabel`  — human phrase for the log lines ("the drum example groove")
 *   - `storeLabel` — where it lands, for the success line ("the SongBook")
 *
 * Resolves to `{ ok: true, reason: 'no-seeds' | 'already-present' | 'seeded',
 * added, legacyAdded, skipped }`.
 */
export function makeBrainSeedMigration({ logTag, entityType, seedIds, seedLabel, storeLabel }) {
  async function up({ rootDir }) {
    const seedPath = join(rootDir, 'data.reference', 'brain', `${entityType}.json`);
    const perRecordDir = join(rootDir, 'data', 'brain', entityType);
    const legacyPath = join(rootDir, 'data', 'brain', `${entityType}.json`);

    const seedRead = await seedReadJsonTagged(seedPath);
    const seedRecords = seedRead.state === 'ok' && seedRead.doc?.records && typeof seedRead.doc.records === 'object'
      ? seedRead.doc.records
      : {};
    const present = seedIds.filter((id) => seedRecords[id] !== undefined);
    if (present.length === 0) {
      console.log(`${logTag}: no ${seedLabel} in data.reference — no-op.`);
      return { ok: true, reason: 'no-seeds' };
    }

    // --- Per-record store (the post-migration-200 layout) -------------------
    let added = 0;
    let skipped = 0;
    for (const id of present) {
      const recordPath = join(perRecordDir, id, 'index.json');
      const read = await seedReadJsonTagged(recordPath);
      if (read.state !== 'missing') {
        // Present (a user-edited copy / peer copy / tombstone) or unreadable —
        // either way, leave it alone.
        if (read.state === 'invalid') {
          console.error(`❌ ${logTag}: ${id}/index.json is unreadable (${read.error}) — leaving it untouched.`);
        }
        skipped += 1;
        continue;
      }
      await mkdir(join(perRecordDir, id), { recursive: true });
      await writeJsonAtomic(recordPath, seedRecords[id]);
      added += 1;
    }

    // --- Legacy monolithic file (only when it still exists) -----------------
    // The tagged read distinguishes all three cases in one syscall: 'missing'
    // means the install is already split, so leave it alone (creating the file
    // would resurrect a shape nothing reads); 'invalid' means user data a write
    // would destroy.
    let legacyAdded = 0;
    const legacyRead = await seedReadJsonTagged(legacyPath);
    if (legacyRead.state === 'invalid') {
      console.error(`❌ ${logTag}: data/brain/${entityType}.json exists but is unreadable (${legacyRead.error}) — leaving it untouched.`);
    } else if (legacyRead.state === 'ok') {
      const live = legacyRead.doc && typeof legacyRead.doc === 'object' ? legacyRead.doc : {};
      if (!live.records || typeof live.records !== 'object') live.records = {};
      for (const id of present) {
        if (live.records[id] !== undefined) continue;
        live.records[id] = seedRecords[id];
        legacyAdded += 1;
      }
      if (legacyAdded > 0) await writeJsonAtomic(legacyPath, live);
    }

    if (added === 0 && legacyAdded === 0) {
      console.log(`${logTag}: ${seedLabel} already present — no-op.`);
      return { ok: true, reason: 'already-present', added: 0, legacyAdded: 0, skipped };
    }

    console.log(`${logTag}: added ${seedLabel} (${added} per-record, ${legacyAdded} legacy) to ${storeLabel}.`);
    return { ok: true, reason: 'seeded', added, legacyAdded, skipped };
  }

  return { up };
}

// ---- provider-seed migration family ----
//
// `scripts/setup-data.js` merges *missing* `data/providers.json` entries from
// data.reference, but only when an install actually re-runs setup. Every
// migration that ships a new provider therefore delivers it on a plain server
// restart too, and each one repeats the identical shell: read
// `data/providers.json` → ENOENT skip (a fresh install seeds from
// data.reference) → JSON.parse guard → shape guard → add each missing id →
// write only when something changed. 149 / 152 / 185 / 195 / 201 / 231 are the
// current members; `makeProviderSeedMigration` is that shell.
//
// What each migration still owns is its `defs` — a FROZEN literal in its own
// file, never read from `data.reference/providers.json` at migration time. A
// migration is the historical record of what it installed; later default
// changes ride their own migrations (precedent: 058/153/206 bumping the Claude
// defaults). Reading the live reference would hand an upgrading install a
// different payload than the migration's own tests assert, and a different one
// than earlier users received.

const PROVIDERS_REL_PATH = 'data/providers.json';

/**
 * Read + parse + shape-guard `data/providers.json` for the two provider
 * migration families below. Returns a discriminated result:
 *
 * - `{ ok: false, reason: 'no-file' | 'unreadable' | 'bad-shape', path }` —
 *   absent (a fresh install seeds from data.reference), unparseable, or missing
 *   its `providers` map. In every case the caller leaves the file untouched: a
 *   migration must never clobber a user's stored apiKeys to "fix" a shape.
 * - `{ ok: true, config, providers, path }` — mutate `providers` in place, then
 *   persist the whole `config` with `writeJsonAtomic(path, config)`.
 *
 * Deliberately silent: each family owns its own log copy (they say different
 * things about what the skip costs the user), so the wording stays per-family
 * while the read shell is shared. `err` carries the parse failure for the
 * `'unreadable'` message.
 */
export async function readProvidersDoc({ rootDir }) {
  const path = join(rootDir, PROVIDERS_REL_PATH);
  const raw = await readFile(path, 'utf-8').catch((err) => {
    if (err.code === 'ENOENT') return null;
    throw err;
  });
  if (raw == null) return { ok: false, reason: 'no-file', path };

  let config;
  try {
    config = JSON.parse(raw);
  } catch (err) {
    return { ok: false, reason: 'unreadable', path, err };
  }

  const providers = config?.providers;
  if (!providers || typeof providers !== 'object') return { ok: false, reason: 'bad-shape', path };

  return { ok: true, config, providers, path };
}

/**
 * Build a provider-seed migration's `up()`. Returns `{ up }`, so a migration is
 * `export default makeProviderSeedMigration({ label, defs })`.
 *
 *   - `label` — human name of the provider family for the log lines
 *     ("Grok", "Cerebras", "OpenCode Ollama TUI").
 *   - `defs`  — the frozen provider objects this migration installs, each with
 *     an `id`. Added only when the id is missing, so an existing entry (user
 *     edits, a stored apiKey, a refreshed model list) is never clobbered and a
 *     second run is a no-op.
 *
 * Each def is installed via `structuredClone`, which fully detaches nested
 * arrays/objects — necessary because sibling defs commonly share one constant
 * (231's two entries both reference `CURSOR_MODELS`), so a shallow spread would
 * leak a later in-memory mutation across both entries and back into the frozen
 * module-level default.
 *
 * Resolves to `{ ok, reason: 'no-file' | 'unreadable' | 'bad-shape' |
 * 'already-present' | 'seeded', added }`.
 */
export function makeProviderSeedMigration({ label, defs }) {
  const noun = defs.length === 1 ? 'provider' : 'providers';

  async function up({ rootDir }) {
    const doc = await readProvidersDoc({ rootDir });
    if (!doc.ok) {
      if (doc.reason === 'no-file') console.log(`📄 ${PROVIDERS_REL_PATH} not present — skipping (fresh install seeds ${label} from data.reference)`);
      else if (doc.reason === 'unreadable') console.log(`⚠️ ${PROVIDERS_REL_PATH}: invalid JSON, skipping (${doc.err.message})`);
      else console.log(`⚠️ ${PROVIDERS_REL_PATH}: unexpected shape, skipping`);
      return { ok: false, reason: doc.reason, added: 0 };
    }

    const { config, providers, path: providersPath } = doc;
    let added = 0;

    for (const def of defs) {
      // Same guard the split family applies to legacy map keys: `__proto__` /
      // `constructor` / `prototype` are inherited on any plain object, so the
      // `providers[def.id]` presence probe below reads TRUTHY for them even on
      // an empty map — the id would be silently skipped forever — and writing
      // one would mutate the prototype rather than add a provider. A provider
      // id is never one of these; refuse loudly if a def ever carries one.
      if (RESERVED_MAP_KEYS.has(def.id)) {
        console.log(`⚠️ ${PROVIDERS_REL_PATH}: refusing reserved provider id ${def.id}`);
        continue;
      }
      if (!providers[def.id]) {
        providers[def.id] = structuredClone(def);
        added++;
        console.log(`📝 ${PROVIDERS_REL_PATH}: added ${def.id} provider`);
      }
    }

    if (added === 0) {
      console.log(`✅ ${PROVIDERS_REL_PATH}: ${label} ${noun} already present — no change`);
      return { ok: true, reason: 'already-present', added: 0 };
    }

    await writeJsonAtomic(providersPath, config);
    return { ok: true, reason: 'seeded', added };
  }

  return { up };
}

// ---- seeded-provider-tier bump migration family ----
//
// Migrations 032 / 058 / 153 / 206 each bump ONE model tier of the seeded
// Claude provider entries (`claude-code`, `claude-code-tui`, and their
// `-bedrock` twins) from a retired model id to its replacement. `setup-data.js`
// merges *missing* provider entries but never updates existing ones, so an
// existing install only picks a new default up when a migration rewrites its
// `data/providers.json`.
//
// All four are the same shell with different data, and it is a deliberately
// conservative shell:
//   - `models` is rewritten only when it matches the prior seeded list EXACTLY
//     (order-sensitive). A curated list — reordered, trimmed, extended — is left
//     alone rather than silently reset to the shipped default.
//   - On a rewrite, every retired id is swapped to its mapped replacement
//     wherever it appears (the `models` array and any tier pointer). Pointers
//     parked on still-current models are preserved.
//   - Bedrock ids map like-for-like, so a long-context `…[1m]` pin lands on the
//     new `…[1m]` id instead of silently dropping to the standard-context id.
//   - The "already-new models but stale pointer" case is repaired: an install
//     freshly seeded from the new data.reference can still carry a tier pointer
//     at a now-absent id, which would leave it requesting a model it no longer
//     lists.
//
// The four shipped copies stay FROZEN and do not consume this factory. A
// migration is the historical record of what it did to an install; rewriting an
// applied one to route through shared code would change that record and risk
// changing its behavior for anyone who has not run it yet. This factory is the
// shell the NEXT tier bump uses, and `_testHelpers.js#runSeededProviderTierMigrationTests`
// is its companion test runner.

// The four tier pointers a provider entry can park on a model id. A bump must
// consider all of them, not just `defaultModel` — an install that pinned
// `heavyModel` to the retired id would otherwise be left pointing at a model
// that is no longer in its `models` list.
const TIER_POINTER_KEYS = ['defaultModel', 'lightModel', 'mediumModel', 'heavyModel'];

/**
 * The post-bump `models` array for one target, derived from its prior seeded
 * list plus its id map — so the two can never drift apart in a caller's data
 * table. Exported for the shared test runner (and for a migration that wants to
 * assert the shape it is about to ship).
 */
export const seededProviderTierModels = ({ oldModels, idMap }) =>
  oldModels.map((id) => (Object.hasOwn(idMap, id) ? idMap[id] : id));

// Order-sensitive equality. Reordering the seeded list counts as customization
// (skipped) — that is the "left alone" promise 032/058/153/206 all made.
const sameModelList = (a, b) =>
  Array.isArray(a) && Array.isArray(b) && a.length === b.length && a.every((v, i) => v === b[i]);

// Swap any tier pointer still referencing a retired id to its replacement.
// `Object.hasOwn` before the lookup: a bare `idMap[provider[key]]` would inherit
// an Object.prototype member for a pointer literally named `constructor` /
// `toString`. Unreachable via the UI or any seed, but the guard costs nothing.
// Mutates in place; returns true if any pointer changed.
const swapTierPointers = (provider, idMap) => {
  let changed = false;
  for (const key of TIER_POINTER_KEYS) {
    const mapped = Object.hasOwn(idMap, provider[key]) ? idMap[provider[key]] : null;
    if (mapped) {
      provider[key] = mapped;
      changed = true;
    }
  }
  return changed;
};

/**
 * Build a seeded-provider-tier bump migration's `up()`. Returns `{ up }`, so a
 * migration collapses to `export default makeSeededProviderTierMigration({…})`
 * over a small data table.
 *
 *   - `targets`   — `{ [providerId]: { oldModels, idMap } }`.
 *       - `oldModels` — the EXACT prior seeded `models` array, in order. Only an
 *         exact match is rewritten.
 *       - `idMap` — retired id → replacement id, for every id this bump
 *         retires. Bedrock targets list the plain and `[1m]` ids separately so
 *         each maps like-for-like. Ids absent from the map are still-current
 *         tiers and are carried through untouched.
 *     Sibling providers that ship identical lists (the CLI/TUI pair, the two
 *     Bedrock entries) should share one spec object.
 *   - `tierLabel` — the human phrase for the tier being bumped, used in the log
 *     lines (e.g. `'opus tier claude-opus-5'`).
 *
 * The summary log reports each touched provider's resulting `defaultModel` —
 * the "what will this install actually run now" value — regardless of which
 * tier was bumped.
 *
 * Resolves to `{ ok, reason: 'no-file' | 'unreadable' | 'bad-shape' |
 * 'no-change' | 'bumped', touched, alreadyCurrent, customized }`, where the
 * three arrays hold provider ids.
 */
export function makeSeededProviderTierMigration({ targets, tierLabel }) {
  // Derive each target's post-bump list once, at build time.
  const plans = Object.entries(targets).map(([id, target]) => ({
    id,
    idMap: target.idMap,
    oldModels: target.oldModels,
    newModels: seededProviderTierModels(target),
  }));

  async function up({ rootDir }) {
    const doc = await readProvidersDoc({ rootDir });
    if (!doc.ok) {
      if (doc.reason === 'no-file') console.log(`📄 ${PROVIDERS_REL_PATH} not present — skipping (fresh install seeds from data.reference with the new defaults)`);
      else if (doc.reason === 'unreadable') console.log(`⚠️ ${PROVIDERS_REL_PATH}: invalid JSON, skipping (${doc.err.message})`);
      else console.log(`⚠️ ${PROVIDERS_REL_PATH}: no providers map — skipping`);
      return { ok: false, reason: doc.reason, touched: [], alreadyCurrent: [], customized: [] };
    }

    const { config, providers, path: providersPath } = doc;

    const touched = [];
    const alreadyCurrent = [];
    const customized = [];

    for (const plan of plans) {
      // `Object.hasOwn` rather than a bare `providers[plan.id]` probe: every
      // plain object inherits `constructor` / `toString`, so a target id
      // colliding with one would read as present and get "bumped" on the
      // prototype. No seeded provider id is one of those, but the probe stays
      // honest for free.
      if (!Object.hasOwn(providers, plan.id)) continue;
      const provider = providers[plan.id];
      if (!provider || typeof provider !== 'object') continue;

      if (sameModelList(provider.models, plan.oldModels)) {
        // Prior seeded list → rewrite models + swap retired pointers.
        provider.models = [...plan.newModels];
        swapTierPointers(provider, plan.idMap);
        touched.push(plan.id);
        continue;
      }

      if (sameModelList(provider.models, plan.newModels)) {
        // Models already current — only act if a tier pointer is still orphaned
        // at a now-absent retired id.
        if (swapTierPointers(provider, plan.idMap)) touched.push(plan.id);
        else alreadyCurrent.push(plan.id);
        continue;
      }

      customized.push(plan.id);
    }

    if (touched.length === 0) {
      const notes = [];
      if (alreadyCurrent.length > 0) notes.push(`already current: ${alreadyCurrent.join(', ')}`);
      if (customized.length > 0) notes.push(`customized: ${customized.join(', ')}`);
      console.log(`✅ ${PROVIDERS_REL_PATH}: nothing to bump for ${tierLabel}${notes.length ? ` (${notes.join('; ')})` : ''}`);
      return { ok: true, reason: 'no-change', touched, alreadyCurrent, customized };
    }

    await writeJsonAtomic(providersPath, config);
    const summary = touched.map((id) => `${id} (default: ${providers[id].defaultModel})`).join(', ');
    console.log(`📝 ${PROVIDERS_REL_PATH}: updated ${summary} → ${tierLabel}`);
    return { ok: true, reason: 'bumped', touched, alreadyCurrent, customized };
  }

  return { up };
}

// ---- additive provider-model insert migration family (7b) ----
//
// The opposite policy from family 7, for the same `data/providers.json`
// targets: family 7 rewrites a PRISTINE seeded list wholesale; this one splices
// a new id into a list the user may have curated, touching nothing else.
// 337 is the first member and the shell every future "offer a model without
// disturbing what's already listed" migration should reach for instead of
// hand-copying it a fourth time.

/**
 * Build an additive provider-model insert migration's `up()`. Returns `{ up }`,
 * so a migration collapses to
 * `export default makeAdditiveProviderInsertMigration({ targets, label })`
 * over a small data table.
 *
 *   - `targets` — `[{ id, retired, current }, …]`, one entry per provider
 *     record this migration can touch. `retired` and `current` are per-target
 *     (not shared across the table) so sibling records that spell the same
 *     tier differently — e.g. a Bedrock record's region-qualified id versus the
 *     bare CLI id — each get their own exact strings; inserting one target's
 *     `current` onto another target's record would offer an id that record's
 *     environment cannot resolve.
 *   - `label` — the human name of the tier being offered, used in the log
 *     copy (`"offered ${label} on …"`, `"${label} already current"`).
 *
 * For each target: if `retired` is listed in the record's `models` and
 * `current` is not, `current` is spliced in immediately after `retired` —
 * `retired` itself, every other entry, and every tier pointer are left
 * exactly as found. A record already listing `current`, or never listing
 * `retired`, is untouched — which makes a second run, and a fresh install
 * already seeded with `current`, both no-ops.
 *
 * Resolves to `{ ok, reason: 'no-file' | 'unreadable' | 'bad-shape' |
 * 'already-current' | 'updated', updated }`.
 */
export function makeAdditiveProviderInsertMigration({ targets, label }) {
  async function up({ rootDir }) {
    const doc = await readProvidersDoc({ rootDir });
    if (!doc.ok) {
      if (doc.reason === 'no-file') console.log(`📄 ${PROVIDERS_REL_PATH} not present — skipping (fresh install seeds ${label} from data.reference)`);
      else if (doc.reason === 'unreadable') console.log(`⚠️ ${PROVIDERS_REL_PATH}: invalid JSON, skipping (${doc.err.message})`);
      else console.log(`⚠️ ${PROVIDERS_REL_PATH}: unexpected shape, skipping`);
      return { ok: false, reason: doc.reason, updated: 0 };
    }

    const { config, providers, path: providersPath } = doc;
    const touched = [];

    for (const { id, retired, current } of targets) {
      const provider = providers[id];
      if (!provider || !Array.isArray(provider.models)) continue;
      const at = provider.models.indexOf(retired);
      // Nothing to repair unless the retired id is listed AND the current one
      // isn't: an already-current record (seeded, or bumped by a prior run) is
      // a no-op, and a record that never listed the retired tier is not this
      // migration's concern.
      if (at === -1 || provider.models.includes(current)) continue;
      provider.models = [
        ...provider.models.slice(0, at + 1),
        current,
        ...provider.models.slice(at + 1),
      ];
      touched.push(id);
    }

    if (touched.length === 0) {
      console.log(`✅ ${PROVIDERS_REL_PATH}: ${label} already current — no change`);
      return { ok: true, reason: 'already-current', updated: 0 };
    }

    await writeJsonAtomic(providersPath, config);
    console.log(`📝 ${PROVIDERS_REL_PATH}: offered ${label} on ${touched.join(', ')}`);
    return { ok: true, reason: 'updated', updated: touched.length };
  }

  return { up };
}

/**
 * Build the per-subdir prompt-drift tables that `scripts/setup-data.js` uses
 * for its "pending migration" warning by sweeping every numbered migration's
 * exported drift constants — instead of hand-mirroring them in setup-data.js
 * (the spot most likely to drift out of sync with the migrations).
 *
 * Each prompt-touching migration exports:
 *   - `ACCEPTED_OLD_MD5` — `{ 'file.md': hash | [hash, …] }` (the shipped
 *     hashes a still-unmodified installed copy may carry before this migration)
 *   - `NEW_SHIPPED_MD5`  — `{ 'file.md': hash }` (the hash this migration ships)
 *   - `DRIFT_SUBDIRS`    — (optional) `{ 'file.md': '_partials' }` for prompt
 *     fragments under `data/prompts/_partials/` rather than the default
 *     `data/prompts/stages/`.
 *
 * Merge rules across a file's migration lineage. Migrations sort numerically by
 * filename, so the highest-numbered one that ships a file defines its current
 * shape:
 *   - current/new hash  = the LAST `NEW_SHIPPED_MD5` entry for the file.
 *   - accepted-old set  = union of every `ACCEPTED_OLD_MD5` entry PLUS every
 *     intermediate `NEW_SHIPPED_MD5` (each earlier shipped hash is itself
 *     auto-updatable to the latest), minus the current hash.
 *
 * Only migration files whose source text mentions the export names are
 * imported — this skips the heavier split/seed migrations (which pull in
 * server-side modules) and keeps the sweep a cheap, side-effect-free read.
 * `_`-prefixed files (this module) are excluded by the numeric-prefix filter.
 * Specialist prompt lineages that manage their own drift inline without
 * exporting these constants (e.g. the importer-stage migrations 015/016/020)
 * are intentionally not swept — they were never in setup-data.js's tables.
 *
 * Returns `{ [subdir]: { oldMap, newMap, files } }` keyed by `'stages'` /
 * `'_partials'`, matching the shape `collectDrift` in setup-data.js consumes.
 */
export async function buildPromptDriftTables(migrationsDir) {
  // Sort by the leading migration number, not lexicographically — so the
  // `current` (latest) hash selection holds even if a future migration name
  // isn't zero-padded (e.g. `7-foo.js` must order before `60-foo.js`).
  const candidates = (await readdir(migrationsDir))
    .filter((f) => /^\d.*\.js$/.test(f) && !f.endsWith('.test.js'))
    .sort((a, b) => (parseInt(a, 10) - parseInt(b, 10)) || a.localeCompare(b));

  // key = `${subdir}/${filename}` so stages + _partials never collide.
  const merged = new Map();
  for (const file of candidates) {
    const filePath = join(migrationsDir, file);
    const source = await readFile(filePath, 'utf-8');
    if (!source.includes('ACCEPTED_OLD_MD5') && !source.includes('NEW_SHIPPED_MD5')) continue;
    const mod = await import(pathToFileURL(filePath).href);
    const oldMap = mod.ACCEPTED_OLD_MD5 || {};
    const newMap = mod.NEW_SHIPPED_MD5 || {};
    const subdirs = mod.DRIFT_SUBDIRS || {};
    const names = new Set([...Object.keys(oldMap), ...Object.keys(newMap)]);
    for (const name of names) {
      const subdir = subdirs[name] || 'stages';
      const key = `${subdir}/${name}`;
      const entry = merged.get(key) || { subdir, name, old: new Set(), newSeq: [] };
      const ov = oldMap[name];
      if (ov != null) (Array.isArray(ov) ? ov : [ov]).forEach((h) => entry.old.add(h));
      if (newMap[name]) entry.newSeq.push(newMap[name]);
      merged.set(key, entry);
    }
  }

  const tables = {};
  for (const { subdir, name, old, newSeq } of merged.values()) {
    // A file with only accepted-old hashes and no shipped current has nothing
    // to update *to* — skip it rather than emit a half table.
    if (newSeq.length === 0) continue;
    const current = newSeq[newSeq.length - 1];
    const olds = new Set([...old, ...newSeq.slice(0, -1)]);
    olds.delete(current);
    const table = (tables[subdir] ||= { oldMap: {}, newMap: {}, files: [] });
    table.oldMap[name] = [...olds];
    table.newMap[name] = current;
    table.files.push(name);
  }
  return tables;
}