Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Migration 220 — backfill `source: 'auto'` on media collections that a machine
* flow created before the field existed (#3311).
*
* Background:
* Every Creative Director project, Writers Room work, and universe/series
* render bucket auto-files itself as a media collection. Until #3311 nothing
* recorded that fact, so the grid reverse-engineered it on the CLIENT from
* four independent markers — a name prefix, an `Auto-…` description, a
* `uc-`/`sc-` id, or a universe/series link. Every new auto-creator silently
* regressed the ordering until someone remembered to extend that list.
*
* `server/services/mediaCollections.js` now stamps `source` at mint time.
* This migration performs the one-time classification of records already on
* disk, using the SAME four markers, so the client heuristic stops being
* load-bearing for anything this install already had.
*
* What it writes:
* `source: 'auto'` on each live record matching a non-forgeable marker and
* carrying no `source` yet (see the marker note below — a bare name prefix is
* deliberately NOT enough, since the create route lets a user pick one).
* Nothing else — in particular `updatedAt` is deliberately NOT
* bumped: this is a derived classification, not a user edit, and advancing
* the LWW clock would make every collection look freshly edited to peers and
* out-race real remote edits. `source` is not part of the conflict journal's
* scalar projection either, so no base hashes churn.
*
* Deliberately NOT written:
* - `source: 'user'` on non-matching records. The client treats an ABSENT
* stamp as "fall back to the marker heuristic" — which is exactly the
* classification those records get today — so stamping them would add
* writes without changing behavior, while freezing an answer the heuristic
* would otherwise re-evaluate (a name-only match is the case that matters:
* it stays auto until the user renames the collection, and then correctly
* stops being auto).
* - Tombstones (`deleted: true`). `deleteCollection` clears the owner links
* and items; nothing renders a tombstone, and rewriting one only adds sync
* noise.
*
* Cross-install: `source` is LOCAL-ONLY — `sanitizeRecordForWire` strips it, so
* no `schemaVersions` bump is needed and no peer's checksum moves. Each peer
* upgrades and runs this independently over ITS OWN copy of the shared
* collections; because the markers are deterministic, they classify the same
* record identically.
*
* The legacy monolithic `data/media-collections.json` is not read — migration
* 059 split it into `data/media-collections/<id>/index.json` and runs first.
*
* Idempotent: a record that already carries a `source` is skipped, so a second
* pass writes nothing.
*/
import { readdir, readFile, writeFile, rename } from 'fs/promises';
import { join } from 'path';
// Frozen snapshot of the markers as of #3311. Kept INLINE rather than imported
// from `client/src/lib/mediaCollectionList.js` / `services/mediaCollections.js`
// on purpose: a migration is a point-in-time transform and must not shift if
// those lists later evolve.
//
// The client heuristic has a FOURTH marker this migration deliberately does not
// use: the `Creative Director: ` / `Writers Room: ` / `Universe: ` / `Series: `
// NAME prefix. Nothing reserves those prefixes — the create route takes a free
// name — so a user-made collection called `Universe: Notes` matches it, and a
// stamped record is skipped on every later pass rather than re-evaluated.
// Leaving name-only matches unstamped keeps them on the client fallback, which
// classifies them exactly as it does today and still re-evaluates if the user
// renames them. Nothing is lost: every real auto-creator also carries one of
// the markers below — Creative Director and Writers Room stamp an `Auto-…`
// description, and universe/series buckets carry the deterministic id and/or
// the owner link.
//
// The description prefixes are far weaker evidence than the id/link markers
// (the POST/PATCH routes do accept a free description), but they are the ONLY
// markers a legacy Creative Director / Writers Room bucket carries — those get
// a random uuid and no owner link. `PATCH /api/media/collections/:id` accepts
// `source`, so the rare user collection described as `Auto-created for project
// …` can be corrected rather than being stuck.
const AUTO_DESCRIPTION_PREFIXES = ['Auto-created for project ', 'Auto-generated images for '];
const AUTO_ID_PREFIXES = ['uc-', 'sc-'];
const isNonEmptyString = (v) => typeof v === 'string' && v !== '';
/**
* True when a record carries a marker of machine creation that a user could not
* have produced through the create form. A subset of `isAutoCollection` in
* `client/src/lib/mediaCollectionList.js` — see the note above on why the name
* prefix is excluded. ANY one marker suffices, because an install can hold
* records from before a given marker existed.
* @param {object} record
* @returns {boolean}
*/
export function isAutoCreated(record) {
if (!record || typeof record !== 'object') return false;
if (typeof record.description === 'string' && AUTO_DESCRIPTION_PREFIXES.some((p) => record.description.startsWith(p))) return true;
if (typeof record.id === 'string' && AUTO_ID_PREFIXES.some((p) => record.id.startsWith(p))) return true;
// Owner link must be a non-empty STRING, matching `sanitizeCollection` — this
// reads raw disk, where the client only ever sees sanitized records. A
// hand-edited/corrupt `universeId: true` is dropped to null on the next read,
// so treating it as a machine-owned marker would stamp a record that has none.
return isNonEmptyString(record.universeId) || isNonEmptyString(record.seriesId);
}
/**
* Decide the new record for one on-disk collection. Returns `null` when the
* record should be left exactly as it is (already stamped, tombstoned, or no
* marker matched).
* @param {object} record
* @returns {object|null}
*/
export function stampAutoSource(record) {
if (!record || typeof record !== 'object') return null;
if (record.source === 'auto' || record.source === 'user') return null;
if (record.deleted === true) return null;
if (!isAutoCreated(record)) return null;
return { ...record, source: 'auto' };
}
const readJson = async (abs) => {
const raw = await readFile(abs, 'utf-8').catch((err) => { if (err.code === 'ENOENT') return null; throw err; });
if (raw == null) return null;
try { return JSON.parse(raw); } catch { return null; }
};
const writeJsonAtomic = async (abs, value) => {
const tmp = `${abs}.tmp-220`;
await writeFile(tmp, JSON.stringify(value, null, 2) + '\n');
await rename(tmp, abs);
};
export default {
async up({ rootDir }) {
const typeDir = join(rootDir, 'data', 'media-collections');
const entries = await readdir(typeDir, { withFileTypes: true })
.catch((err) => { if (err.code === 'ENOENT') return null; throw err; });
if (entries === null) {
console.log('📦 migration 220: no data/media-collections dir — fresh install, no-op');
return { ok: true, reason: 'no-collections' };
}
let stamped = 0;
let scanned = 0;
for (const entry of entries) {
if (!entry.isDirectory()) continue; // skips the type-level index.json
const recordPath = join(typeDir, entry.name, 'index.json');
const record = await readJson(recordPath);
if (!record) continue; // missing or unparseable — leave it for the store to drop
scanned += 1;
const next = stampAutoSource(record);
if (!next) continue;
await writeJsonAtomic(recordPath, next);
stamped += 1;
}
console.log(`📦 migration 220: stamped source:auto on ${stamped} of ${scanned} media collection(s)`);
return { ok: true, reason: stamped ? 'migrated' : 'nothing-to-stamp', stamped, scanned };
},
};