Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
/**
* Non-blocking conflict journal.
*
* All cross-install merges are last-write-wins by `updatedAt`. When LWW would
* OVERWRITE a local record but BOTH sides edited it independently since the
* last time they were in sync (a true 3-way divergence), the losing local
* version is archived here BEFORE the overwrite — so convergence is preserved
* (peers still agree on the LWW winner, sync never wedges) AND no edit is
* silently lost (the user can review and restore from the Sharing → Conflicts
* tab).
*
* Divergence signal — per-record `syncBaseHash` (the content hash as of the
* last time this instance and a peer demonstrably held the same content):
*
* conflict := base != null
* && hLocal !== base // local diverged from the common base
* && hRemote !== base // remote also diverged from it
* && hLocal !== hRemote // and they actually differ
*
* `base == null` ⇒ treat as a clean update (conservative: we only ever MISS
* journaling the first divergence after this feature ships for a given record,
* never wedge and never false-positive on routine sequential updates) — UNLESS
* the base-hash store itself is untrustworthy: a present-but-unreadable
* `sync_base_hashes.json` latches `_persistBlocked` (#7260), under which every
* overwrite journals fail-closed (an unknown baseline is a possible
* divergence) and nothing ever writes back over the unreadable file.
*
* The base hash advances to the remote's hash on EVERY accepted overwrite
* (clean OR conflicting) — without that, the 60s snapshot loop would re-journal
* the same unresolved divergence every cycle.
*
* This store + the base-hash side store are LOCAL-ONLY: they are never enrolled
* in PEER_SUBSCRIBABLE_KINDS / snapshot categories / sanitizeStateForWire, so
* they can never cross the wire (no schema-version bump needed).
*/
import { join } from 'path';
import { randomUUID, createHash } from 'crypto';
import { PATHS, atomicWrite, readJSONFileStrict, ensureDir } from './fileUtils.js';
import { createCollectionStore } from './collectionStore.js';
import { canonicalStringify, isPlainObject } from './objects.js';
import { sanitizeRecordForWire, stripMusicVideoLocalRenderPins } from './syncWire.js';
const JOURNAL_TYPE_SCHEMA_VERSION = 1;
const BASE_HASH_FILE = () => join(PATHS.data, 'sharing', 'sync_base_hashes.json');
// ---- conflict-journal collection store ----
let _store = null;
const store = () => {
if (_store && _store.dir === join(PATHS.data, 'conflict-journal')) return _store;
_store = createCollectionStore({
dir: join(PATHS.data, 'conflict-journal'),
type: 'conflictJournal',
schemaVersion: JOURNAL_TYPE_SCHEMA_VERSION,
});
return _store;
};
export const conflictJournalStore = () => store();
// ---- content hashing (matches the sender/receiver wire convention) ----
// mediaCollection divergence is detected over a SCALAR SUBSET, not the whole
// wire record. The merge (`mergeMediaCollectionsFromSync`) union-merges `items`
// — neither side ever loses a render it knows about — and bumps `updatedAt` on
// every `addItem`. So hashing the full record would false-positive whenever two
// peers independently added different items to the same collection (items +
// updatedAt differ, yet NOTHING was overwritten). The journal only cares about
// the LWW-overwritten scalars; we hash exactly those so an item-only divergence
// is invisible to detection while a real name/description/cover/link overwrite
// still trips it. `updatedAt` is the LWW *key*, not content, so it's excluded
// too. Soft-delete fields stay in (a remote delete that erases a diverged local
// edit is a real conflict — mirrors the universe/series delete-vs-edit case).
const MEDIA_COLLECTION_SCALAR_FIELDS = Object.freeze(['name', 'description', 'coverKey', 'universeId', 'seriesId']);
// Fields dropped from the conflict-detection hash, per kind. These are
// server-managed values that mutate WITHOUT a user edit (and without bumping
// `updatedAt`), so leaving them in the hash would read as a false divergence
// and journal a spurious conflict. Issue `number` is renumber-managed:
// `applyVolumeOrderedNumbers` shifts it in place when a *sibling* issue is
// added/removed, so a local sibling-delete would otherwise make this issue's
// `localHash !== base` even though no restorable content changed (the resulting
// conflict card would have an empty diffSummary, since `number` isn't
// restorable). Excluding it keeps the hash aligned with the restorable-field
// set the UI actually shows. (mediaCollection narrows via its own scalar
// projection below; universe/series exclude nothing, so their already-seeded
// base hashes stay valid across upgrade.)
// Fields dropped from the CONTENT HASH (not the wire payload — they still
// replicate). `issue.number` is renumber-managed. `track.renders` is an additive
// field synthesized (backfilled) onto every existing track on first read: hashing
// it would invalidate every pre-upgrade `sync_base_hash` and journal a one-time
// FALSE 3-way conflict on the first cross-peer edit after both peers upgrade
// (local and remote both differ from a base computed without renders). Excluding
// it from the hash keeps base-hash compatibility; render-only divergences are
// LWW-ordered by `updatedAt` regardless, and renders stays restorable (below).
// This is for fields that are GENUINELY server-managed (mutate without a user
// edit) and must NEVER participate in conflict detection — contrast with
// HASH_FIELDS below, for user-authored additive fields that SHOULD eventually
// participate once it's safe to.
const HASH_EXCLUDED_FIELDS = Object.freeze({ issue: ['number'], track: ['renders'] });
// User-authored fields that were added to a kind's wire shape at a later
// point, keyed by a LOCAL version number (this mechanism's own — deliberately
// independent of PORTOS_SCHEMA_VERSIONS in schemaVersions.js, which gates
// cross-peer WIRE compatibility, a different concern from "which fields does
// the local base-hash store already know about"). Unlike HASH_EXCLUDED_FIELDS
// these DO belong in conflict detection — the problem #2912 solves isn't
// "never hash this," it's "don't let adding it retroactively invalidate every
// base hash stamped before it existed." `contentHashForRecord`'s `maxVersion`
// option drops any field introduced after that version; `setSyncBaseHash`
// always stamps at the CURRENT (highest defined) version for the kind, and
// `detectConflict` recomputes local/remote at the STORED base's version so
// the comparison is apples-to-apples. A base hash predating the field will
// therefore MISS a divergence limited to that field until the next successful
// sync re-stamps it at the current version — the same one-time-miss tradeoff
// this file already accepts for `baseHash == null` (see the module doc
// comment above).
const HASH_FIELDS = Object.freeze({
// v3 (#2911): chiptuneScore/chiptunePrompt. Previously permanently excluded
// via HASH_EXCLUDED_FIELDS; migrated here by #2912 so two peers that both
// understand the field (base stamped at v3+) get real conflict detection on
// it instead of silent LWW. The version number "3" is chosen to match
// PORTOS_SCHEMA_VERSIONS.tracks for a human reading both files side by side
// — nothing enforces the two staying in lockstep; it's fine for them to
// diverge (a wire-compat bump doesn't always change what the local hash
// cares about, and vice versa).
// v4: concept, the user-authored creative brief used by resumable music
// designer drafts.
// v7 (#8376): waveSketch/waveSketchPrompt, the Music Designer's drawn wave
// sketch — hashed only for bases stamped at v7+.
track: Object.freeze({
3: Object.freeze(['chiptuneScore', 'chiptunePrompt']),
4: Object.freeze(['concept']),
7: Object.freeze(['waveSketch', 'waveSketchPrompt']),
}),
});
// Precomputed field→version lookup per kind, inverted from HASH_FIELDS once
// at module load (HASH_FIELDS is static, so rebuilding this per hash call —
// on every record write in the system — would be pure waste).
const HASH_FIELD_VERSION_BY_KIND = Object.freeze(Object.fromEntries(
Object.entries(HASH_FIELDS).map(([kind, versioned]) => {
const byField = new Map();
for (const [v, fields] of Object.entries(versioned)) {
for (const f of fields) byField.set(f, Number(v));
}
return [kind, byField];
}),
));
// The highest HASH_FIELDS version defined for `kind` — the version
// `setSyncBaseHash` stamps new base hashes at. Kinds with no HASH_FIELDS
// entry have nothing to gate; the fallback of 1 is inert for them (no field
// is ever excluded by a maxVersion check that never finds a match).
function currentHashVersionForKind(kind) {
const versioned = HASH_FIELDS[kind];
if (!versioned) return 1;
return Math.max(1, ...Object.keys(versioned).map(Number));
}
/**
* sha256 of the canonical content projection. Reuses sanitizeRecordForWire +
* canonicalStringify so the sender hashing what it pushes and the receiver
* hashing its local copy agree byte-for-byte. For mediaCollection the wire form
* is further narrowed to its overwritable scalars (see
* MEDIA_COLLECTION_SCALAR_FIELDS); for issue the renumber-managed `number` is
* dropped (see HASH_EXCLUDED_FIELDS) — both sides apply the same narrowing, so
* the base hash stays consistent across peers. Returns null when the record has
* no wire form (ephemeral non-tombstone, or invalid) — callers treat a null
* hash as "cannot compare".
*
* `maxVersion` (see HASH_FIELDS above) restricts the hash to fields introduced
* at or below that version — pass the STORED base hash's version when
* recomputing for comparison so an old base isn't compared against a
* differently-shaped hash. Omitted (the default, used when stamping a NEW base
* hash) means "include every field at the current version."
*/
export function contentHashForRecord(kind, record, { maxVersion } = {}) {
const wire = sanitizeRecordForWire(kind, record);
if (!wire) return null;
let hashInput = wire;
if (kind === 'mediaCollection') {
hashInput = projectCollectionScalars(wire);
} else if (kind === 'musicVideoProject') {
hashInput = stripMusicVideoLocalRenderPins(wire);
} else if (HASH_EXCLUDED_FIELDS[kind] || HASH_FIELD_VERSION_BY_KIND[kind]) {
const excluded = HASH_EXCLUDED_FIELDS[kind] || [];
const introducedAt = HASH_FIELD_VERSION_BY_KIND[kind];
hashInput = Object.fromEntries(Object.entries(wire).filter(([k]) => {
if (excluded.includes(k)) return false;
const fieldVersion = introducedAt?.get(k);
if (fieldVersion == null) return true; // not a versioned field — always included
return maxVersion == null || fieldVersion <= maxVersion;
}));
}
return createHash('sha256').update(canonicalStringify(hashInput)).digest('hex');
}
// Narrow a wire-form collection to the scalars whose overwrite the journal
// tracks, plus the (already-normalized) soft-delete pair. `items`/`updatedAt`/
// `id` and all other fields are dropped. canonicalStringify sorts keys, so the
// insertion order here is irrelevant to the resulting hash.
function projectCollectionScalars(wire) {
const out = {};
for (const f of MEDIA_COLLECTION_SCALAR_FIELDS) if (f in wire) out[f] = wire[f];
out.deleted = wire.deleted === true;
out.deletedAt = out.deleted ? (wire.deletedAt ?? null) : null;
return out;
}
// ---- base-hash side store (in-memory cache, batched write-through) ----
// Map<`${kind}:${id}`, sha256 | { h: sha256, v: number }>. A bare string is a
// legacy entry written before #2912 (or a v1 entry — the two are equivalent,
// see readBaseEntry's "absent = v1" fallback); new entries are always written
// as `{ h, v }` so the hash-fields version travels with the hash it was
// computed under.
let _baseHashes = null;
let _loadPromise = null;
let _baseDirty = false;
let _flushTail = Promise.resolve();
// >0 while a `withBaseHashFlushBatch` scope is active. Defers the disk write so
// an await-separated multi-record push loop (peer:online convergence) collapses
// N `sync_base_hashes.json` rewrites into one at the batch's terminal flush.
let _flushBatchDepth = 0;
// Latched when the base-hash file was present-but-unreadable at load (#7260 —
// the `persistBlocked` shape from services/mediaJobQueue). flushBaseHashes()
// whole-file-REPLACES sync_base_hashes.json with the in-memory map, so
// persisting on top of a baseline we failed to read would discard every entry
// the process didn't happen to stamp — and unlike a rebuildable cache, the map
// records what the two machines last agreed on, which no longer exists
// anywhere once it is gone. While latched: nothing flushes (the file survives
// for the user to repair), and maybeJournalBeforeOverwrite fails CLOSED —
// "baseline unknown" is treated as "possible conflict" and journaled. The
// latch is process-lifetime; nothing in production calls __reset.
let _persistBlocked = false;
const baseKey = (kind, id) => `${kind}:${id}`;
async function ensureBaseLoaded() {
if (_baseHashes) return _baseHashes;
if (!_loadPromise) {
_loadPromise = (async () => {
// Strict (#7260): a present-but-unreadable file (EACCES, EIO,
// truncated/partial bytes, a cloud-sync conflict copy) must NOT read as
// a genuine empty — that silently disabled conflict detection and let
// the next flush destroy every stored baseline. ENOENT is the only
// trustworthy empty (first sync). `allowArray: false` makes an
// array-shaped file unreadable; a non-object scalar parses `ok` but
// still isn't a baseline map, so it latches too.
const { ok, value } = await readJSONFileStrict(BASE_HASH_FILE(), {}, { allowArray: false });
if (ok && isPlainObject(value)) {
_baseHashes = new Map(Object.entries(value));
} else {
_persistBlocked = true;
_baseHashes = new Map();
// Name the recovery step: the latch is process-lifetime, so without
// this the user has conflict detection that silently stops persisting
// and no indication that repairing or deleting one file restores it.
console.error(`❌ conflictJournal: ${BASE_HASH_FILE()} is present but unreadable — base hashes are preserved on disk (persistence latched off) and overwrites journal fail-closed; repair or delete it and restart to re-enable persistence`);
}
return _baseHashes;
})();
}
return _loadPromise;
}
// Normalize a raw map value (legacy bare string, or `{ h, v }`) to
// `{ hash, version }`. Absent version defaults to 1 — the pre-#2912 hash
// never included any HASH_FIELDS-versioned field, so it's the correct floor.
function readBaseEntry(raw) {
if (raw == null) return null;
if (typeof raw === 'string') return { hash: raw, version: 1 };
const version = Number.isInteger(raw.v) ? raw.v : 1;
return { hash: raw.h ?? null, version };
}
/** Shared lookup behind getSyncBaseHash and detectConflict — one place that
* loads the map and normalizes the stored entry. */
async function readStoredBaseEntry(kind, id) {
const map = await ensureBaseLoaded();
return readBaseEntry(map.get(baseKey(kind, id)));
}
/** The stored base hash for (kind, id), or null. String-only for backward
* compatibility with every existing caller/test — use detectConflict (which
* reads the paired version internally) when the version matters. */
export async function getSyncBaseHash(kind, id) {
return (await readStoredBaseEntry(kind, id))?.hash ?? null;
}
/**
* Set the base hash in memory (write-through deferred to flushBaseHashes).
* Stamps the CURRENT hash-fields version for `kind` (see
* currentHashVersionForKind) unless `version` is explicitly given — tests use
* the override to plant a hash computed at an older version, simulating a
* base stamped before an additive HASH_FIELDS entry existed.
*/
export async function setSyncBaseHash(kind, id, hash, { version } = {}) {
const map = await ensureBaseLoaded();
if (!hash) return;
const v = Number.isInteger(version) ? version : currentHashVersionForKind(kind);
const key = baseKey(kind, id);
const existing = readBaseEntry(map.get(key));
if (existing?.hash === hash && existing?.version === v) return;
map.set(key, { h: hash, v });
_baseDirty = true;
}
/**
* Remove the base hash for a pruned record (idempotent — no-op if already
* absent). Call when a tombstone is force-pruned so the entry doesn't grow
* without bound on long-lived federated installs.
*/
export async function deleteSyncBaseHash(kind, id) {
const map = await ensureBaseLoaded();
const key = baseKey(kind, id);
if (!map.has(key)) return;
map.delete(key);
_baseDirty = true;
await flushBaseHashes();
}
/**
* Sweep base-hash entries whose record no longer resolves to a live record.
*
* The per-record prune paths that own base hashes already evict them when they
* hard-delete tombstones. This is the BACKSTOP for keys that escaped those
* paths: a record hand-deleted on disk, a key left behind by an older version
* before the prune paths evicted, or any future kind that seeds a base hash
* without a matching eviction. Without it the side store accumulates dead keys
* forever on a long-lived federated install.
*
* `resolves` is an async predicate `(kind, id) => boolean` — return true if the
* key still maps to a record we want to keep a base hash for. Keys whose kind
* the caller doesn't recognize are LEFT ALONE (the predicate should return true
* for unknown kinds) so a partial resolver can never strip live entries it
* simply didn't know how to check. Coalesces into one disk write via the dirty
* flag + a single trailing flush.
*
* Returns `{ pruned }`.
*/
export async function pruneOrphanedBaseHashes(resolves) {
if (typeof resolves !== 'function') return { pruned: 0 };
const map = await ensureBaseLoaded();
let pruned = 0;
for (const key of [...map.keys()]) {
const sep = key.indexOf(':');
if (sep < 0) continue; // malformed key — leave it, never guess
const kind = key.slice(0, sep);
const id = key.slice(sep + 1);
const keep = await resolves(kind, id).catch(() => true); // resolver error → conservative keep
if (keep) continue;
map.delete(key);
_baseDirty = true;
pruned += 1;
}
if (pruned > 0) await flushBaseHashes();
return { pruned };
}
/** Persist the base-hash map if dirty. Serialized so concurrent flushes can't
* interleave file writes. Call once after a merge's write loop.
*
* Inside a `withBaseHashFlushBatch` scope the write is deferred: the in-memory
* stamps stay coalesced (`_baseDirty`) and the single disk write fires when the
* outermost batch closes. Callers still get a thenable (`_flushTail`) so any
* `await flushBaseHashes()` resolves immediately rather than blocking inside
* the batch.
*
* No-ops while `_persistBlocked` is latched (#7260): the read that seeded the
* map was untrustworthy, so persisting it back would replace a file that may
* hold the real baselines with whatever this process stamped since boot.
* `deleteSyncBaseHash` / `pruneOrphanedBaseHashes` still evict in memory (the
* session map stays coherent) but their flush is refused for the same reason. */
export function flushBaseHashes() {
// Defer while inside a batch, refuse while the store is untrustworthy, and
// no-op when nothing is dirty — each way the outstanding `_flushTail`
// thenable is the right thing to await.
if (_persistBlocked || _flushBatchDepth > 0 || !_baseDirty) return _flushTail;
_flushTail = _flushTail.then(async () => {
if (!_baseDirty) return;
_baseDirty = false;
const map = await ensureBaseLoaded();
await ensureDir(join(PATHS.data, 'sharing'));
await atomicWrite(BASE_HASH_FILE(), Object.fromEntries(map));
}).catch((err) => {
_baseDirty = true; // retry on next flush
console.error(`❌ conflictJournal: base-hash flush failed: ${err?.message || err}`);
});
return _flushTail;
}
/**
* Run `fn` with base-hash disk writes coalesced into ONE flush at the end.
*
* The base-hash map already coalesces SYNCHRONOUS bursts (the `_baseDirty` flag
* means a run of `setSyncBaseHash` calls before a single `flushBaseHashes` pays
* one write). What it can't collapse on its own is an AWAIT-SEPARATED loop that
* stamps + flushes per iteration: each flush lands on the previous one's settled
* `_flushTail`, so an N-iteration loop rewrites `sync_base_hashes.json` N times.
* Wrapping the loop in this scope defers every interior flush; the single
* terminal write captures all N stamps.
*
* Two caller shapes use it: the `peer:online` convergence walk
* `retryPendingPushesForPeer` (pushes — and stamps — every subscribed record in
* sequence), and every base-hash-evicting `pruneTombstoned*` hard-prune loop.
* Eviction only marks `_baseDirty` for a key the map actually held, so a prune
* that removed nothing still costs zero writes.
*
* Re-entrant (depth-counted) so nested scopes collapse to the outermost batch,
* and the terminal flush runs in `finally` so a thrown/rejected `fn` still
* persists whatever stamps landed before it failed. Unlike a `setTimeout`
* debounce (tried and reverted — the unref'd timer leaked into `peerSync.test.js`'s
* settle-waits and made the suite flaky), this is deterministic: the flush is
* tied to the scope boundary, not a wall-clock timer, so tests just `await` it.
*
* Contract note: stamps performed inside the scope must have settled (the
* in-memory `setSyncBaseHash` awaited) before `fn` returns, or the terminal
* flush won't see `_baseDirty` for them. Every caller awaits its stamps/evictions.
*
* Depth is module-level, not per-async-context, so CONCURRENT batches merge
* rather than nest. The tombstone GC fans its `pruneTombstoned*` calls out
* through one `Promise.all`, so their scopes overlap: the last scope to close
* flushes every eviction they accumulated. An unbatched concurrent flush that
* lands while the depth is still positive rides along in that same write.
* Nothing is dropped — a deferred flush leaves `_baseDirty` set.
*/
export async function withBaseHashFlushBatch(fn) {
_flushBatchDepth += 1;
try {
return await fn();
} finally {
_flushBatchDepth -= 1;
if (_flushBatchDepth === 0) await flushBaseHashes();
}
}
// ---- conflict detection + journaling ----
/**
* Detect a true 3-way divergence for a record about to be overwritten.
* Returns `{ isConflict, baseHash, localHash, remoteHash }`.
*
* local/remote are recomputed at the STORED base's hash-fields version (see
* HASH_FIELDS), not the current one — so a base stamped before an additive
* field existed is compared apples-to-apples against the same field set it
* was originally built from, instead of unconditionally mismatching the
* moment the field starts appearing in the wire shape.
*/
export async function detectConflict({ kind, id, local, remote }) {
const base = await readStoredBaseEntry(kind, id);
const baseHash = base?.hash ?? null;
const maxVersion = base ? base.version : undefined;
const localHash = contentHashForRecord(kind, local, { maxVersion });
const remoteHash = contentHashForRecord(kind, remote, { maxVersion });
const isConflict = baseHash != null && localHash != null && remoteHash != null
&& localHash !== baseHash && remoteHash !== baseHash && localHash !== remoteHash;
// `baseUnavailable` (#7260): the stored baseline file could not be read, so
// `isConflict === false` here means "no base to compare against," NOT
// "proven clean". Callers that journal must branch on this flag — the merge
// path treats it as fail-closed (see maybeJournalBeforeOverwrite).
return { isConflict, baseHash, localHash, remoteHash, baseUnavailable: _persistBlocked };
}
// Restorable fields whose ABSENCE from a wire snapshot is meaningful state
// rather than "not captured". `sanitizeTemplate` persists universe `factual`
// only when true, so a fiction snapshot simply omits the key — and a restore
// that picks fields with a plain `in` check would skip it, silently turning
// "restore this universe to fiction" into a no-op while the true-direction
// restore worked. The resolver fills in the explicit `false`, which
// `sanitizeTemplate` then drops back to absent.
export const ABSENT_MEANS_FALSE_FIELDS = Object.freeze({
universe: Object.freeze(['factual']),
});
// User-authored content fields a restore/merge may write back, per kind. This
// is the SINGLE SOURCE OF TRUTH — the resolver imports it for its merge-fields
// allowlist, and diffSummary filters to it so the Conflicts UI only ever offers
// fields the resolver will accept (server-owned fields like id/createdAt/
// schemaVersion/locked/origin/deleted are neither restorable nor shown).
export const RESTORABLE_FIELDS = Object.freeze({
// `factual` (#7616) is here because it is FEDERATED and therefore hashed by
// contentHashForRecord — a peer flipping the factual/fiction axis is a real
// 3-way divergence. Omitting it would journal that conflict with a diff that
// cannot mention the field (the phantom-divergence trap `styleImageRefs`
// avoids by being stripped from the wire instead) and leave the user no way
// to restore it. See ABSENT_MEANS_FALSE_FIELDS for the clear direction.
universe: ['name', 'starterPrompt', 'logline', 'premise', 'styleNotes', 'categories', 'compositeSheets', 'influences', 'characters', 'places', 'objects', 'factual'],
series: ['name', 'logline', 'premise', 'styleNotes', 'styleGuide', 'titleLogo', 'author', 'stylePromptOverride', 'stylePromptOverrideMode', 'targetFormat', 'issueCountTarget', 'arc', 'seasons'],
// mediaCollection restores only the user-authored content scalars. `items`
// are union-merged (never lost, nothing to restore); `universeId`/`seriesId`
// are structural links managed by the link/unlink helpers, not `updateCollection`
// patches — so they're hashed for DETECTION but not offered for restore.
mediaCollection: ['name', 'description', 'coverKey'],
// Author personas: all user-authored text scalars are restorable. `id`, the
// LWW/tombstone trio, and `createdAt` are server-owned/structural. Mirrors the
// PATCHABLE field set in services/authors/logic.js. The author wire form is
// LWW-overwritten whole (no field-union like mediaCollection's items), so it
// is hashed in full by contentHashForRecord — no scalar-narrowing branch.
author: ['name', 'writingStyle', 'bio', 'physicalDescription', 'headshotStyle', 'headshotImageUrl'],
artist: ['name', 'genre', 'bio', 'musicalStyle', 'physicalDescription', 'portraitStyle', 'portraitImageUrl'],
album: ['title', 'artistId', 'artist', 'description', 'genre', 'releaseYear', 'coverImageUrl', 'trackIds'],
// `renders` IS restorable (a conflict restore must bring back the local render
// history) but is EXCLUDED FROM THE CONTENT HASH — see HASH_EXCLUDED_FIELDS for
// why (base-hash compatibility for this additive backfilled field).
// `chiptuneScore`/`chiptunePrompt` (#2911) are also restorable and, as of
// #2912, DO participate in the content hash via the version-gated HASH_FIELDS
// mechanism (see above) — a base hash predating the field just compares at
// its own stored version until the next sync re-stamps it. Same for
// `waveSketch`/`waveSketchPrompt` (#8376, hashed from v7).
track: ['title', 'albumId', 'artistId', 'artist', 'concept', 'lyrics', 'prompt', 'engine', 'modelId', 'durationSec', 'audioFilename', 'renders', 'chiptuneScore', 'chiptunePrompt', 'waveSketch', 'waveSketchPrompt'],
// Issue: the user-authored content the merge can restore. `stages` carries
// the bulk of the work (prose, comic pages, render metadata). Server-owned /
// structural fields are excluded deliberately — `number` is renumber-managed,
// `seriesId` is the immutable parent link, `origin` is share provenance —
// and `mergeIssuePatch` accepts every field listed here.
issue: ['title', 'status', 'seasonId', 'arcPosition', 'arcRole', 'lengthProfile', 'pageTarget', 'minutesTarget', 'stages'],
// Creative Director projects (#1564): the user-authored config + creative
// content the merge can restore. `treatment` (logline/synopsis/scenes) carries
// the bulk of the work. Server-owned / machine-managed fields are excluded —
// `id`/`createdAt`/the LWW-tombstone trip are structural, `collectionId`/
// `timelineProjectId`/`finalVideoId` are derived output pointers, and `runs[]`
// is render history (regenerated, not hand-authored). The whole record is
// hashed for DETECTION by contentHashForRecord (no scalar narrowing); this set
// is what the Conflicts UI offers for restore. `creativeDirectorProject`
// matches the record kind and mergeProjectRecord accepts every field here.
creativeDirectorProject: ['name', 'status', 'styleSpec', 'userStory', 'treatment', 'aspectRatio', 'quality', 'modelId', 'targetDurationSeconds', 'disableAudio', 'autoAcceptScenes', 'startingImageFile'],
// Mood boards (#1564): the user-authored fields the merge can restore. `items[]`
// (the pinned image/text references) carries the bulk of the work, alongside
// name/description. `style` is the composite board prompt and `posterImageRef`
// is the canonical poster filename — both user-triggered, so a restore must
// be able to bring them back. Server-owned / structural fields are excluded —
// `id`/`createdAt`/the LWW-tombstone trio. The whole record is hashed for
// DETECTION by contentHashForRecord (no scalar narrowing); this set is what
// the Conflicts UI offers for restore. `moodBoard` matches the record kind,
// and the resolver routes restore through restoreBoard → applyBoardRestore
// (which accepts items, unlike the route's applyBoardPatch).
moodBoard: ['name', 'description', 'items', 'style', 'posterImageRef', 'collageImageRef'],
// Writers Room works (#1565): the user-authored manifest fields the merge can
// restore through `updateWork` (which accepts exactly this set + the liveMode
// partial-merge). Server-owned / structural fields are excluded — `id`/
// `createdAt`/the LWW-tombstone trio, the draft-version structure
// (`drafts`/`activeDraftVersionId`, file-primary + lineage-managed), and the
// bridge links (`pipelineSeriesId`/`cdProjectId`/`mediaCollectionId`, set by
// their own helpers). The whole manifest is hashed for DETECTION by
// contentHashForRecord (no scalar narrowing); this set is what the Conflicts UI
// offers for restore. `writersRoomWork` matches the record kind.
writersRoomCharacters: ['characters'],
writersRoomPlaces: ['places'],
writersRoomObjects: ['objects'],
writersRoomWork: ['title', 'kind', 'status', 'folderId', 'imageStyle', 'liveMode'],
// Writers Room folders (#1645): the user-authored structural fields the merge
// can restore through `restoreFolder`. Server-owned / structural fields are
// excluded — `id`/`createdAt`/the LWW-tombstone trio. The whole folder is
// hashed for DETECTION by contentHashForRecord (no scalar narrowing); this set
// is what the Conflicts UI offers for restore. `writersRoomFolder` matches the
// record kind.
writersRoomFolder: ['name', 'parentId', 'sortOrder'],
// A commissionFeedback (#2686) is one taste reaction — the user-authored fields
// the Conflicts UI may restore are the rating + its note/tags. `commissionId`/
// `runId` are structural keys (they address the reaction) and are excluded.
commissionFeedback: ['rating', 'note', 'tags'],
// A creativeCommission (#2686) restores its federated BRIEF fields; the
// machine-local schedule/runs/assignment/enabled are never in a snapshot
// (stripped from the wire) so they're excluded.
creativeCommission: ['name', 'targetAbility', 'brief', 'generation', 'feedbackWindow'],
// A deck (#decks) restores its authored style guide + identity. `kind` is
// structural (the roster is minted from it and a CHECK constrains the
// column), `cards[]` is the roster itself (restored with the deck row by
// `restoreDeck`, not offered as an individually-overlayable field), and the
// install-capability pins never reach a snapshot (stripped from the wire).
deck: ['name', 'description', 'styleNotes', 'influences', 'layoutPrompt', 'cardOrientation', 'cardOrientationPrompt', 'samples', 'cardSize', 'universeId'],
// A storyboard restores its authored plan: identity fields, the panel-maker
// recipe, the plot, the temp track and the beats → panels → takes tree. `tempTrack`
// restores with `beats` so sanitizeStoryboard re-resolves each beat pin against
// the restored markers. `plotLlm` is a per-user provider pick and stays local;
// id and timestamps stay structural.
storyboard: ['name', 'logline', 'aspectRatio', 'artStyle', 'universeId', 'moodBoardId', 'recipe', 'plot', 'tempTrack', 'beats'],
// FableLoom restores the authored story graph, playback configuration, and
// story-level render preferences.
// Universe/series ids are structural links and stay on the live record.
fableLoom: ['name', 'logline', 'premise', 'styleNotes', 'format', 'playSettings', 'renderSettings', 'seriesPlan', 'productionStatus', 'protagonistCharacterId', 'protagonistWardrobeId', 'protagonistWardrobeLocked', 'participationMode', 'audienceCommunicationMedium', 'episodes'],
// Writers Room exercises (#1645): the user-authored sprint fields the merge can
// restore through `restoreExercise` — the appended prose + the sprint config.
// Server-owned / structural fields are excluded (`id`/`createdAt`/`startedAt`/
// the LWW-tombstone trio). `writersRoomExercise` matches the record kind.
writersRoomExercise: ['workId', 'prompt', 'durationSeconds', 'startingWords', 'endingWords', 'wordsAdded', 'appendedText', 'status', 'finishedAt'],
// Music Video projects (#1770): the user-authored config + creative content the
// merge can restore. `scenes[]` (the beat-aligned director board with prompts)
// carries the bulk of the work, alongside name/status/mode/trackId/
// uploadedAudioFilename/vocalStemFilename/concept. Server-owned / machine-managed fields are
// excluded — `id`/`createdAt`/the LWW-tombstone trio are structural,
// `renderHistoryId` is a derived output pointer, and `audioAnalysis` is the
// machine-derived beat cache (regenerated from the audio, not hand-authored).
// The whole record is hashed for DETECTION by contentHashForRecord (no scalar
// narrowing); this set is what the Conflicts UI offers for restore.
// `musicVideoProject` matches the record kind and updateProject (via
// applyProjectPatch's wholesale spread) accepts every field here.
musicVideoProject: ['name', 'status', 'mode', 'trackId', 'uploadedAudioFilename', 'vocalStemFilename', 'concept', 'scenes'],
});
const present = (v) => v !== undefined;
const changedFlag = (lv, rv) => (present(lv) && present(rv) ? 'both' : (present(rv) ? 'remote-only' : 'local-only'));
// First non-empty string among `fields`, in order — used both to pick a stable
// PAIR key for an array element and to pick a human display LABEL for it.
const firstStringField = (el, fields) => {
for (const k of fields) {
if (typeof el?.[k] === 'string' && el[k].trim()) return el[k];
}
return null;
};
// Stable key used to PAIR two array elements across the local/remote sides
// (prefer an explicit id so a reorder/insert doesn't cascade). Null ⇒ the
// element carries no identity and the array isn't deep-diffable by identity.
const entryMatchKey = (el) => firstStringField(el, ['id', 'key', 'slug', 'name']);
// Human-friendly LABEL for a changed sub-entry (a uuid id is a poor label, so a
// name/title is preferred for display even when the match used the id).
const entryLabel = (el, fallback) => firstStringField(el, ['name', 'title', 'label', 'key', 'slug', 'id']) ?? fallback;
// Diff two key→value maps: one part per key whose value differs. Each part
// carries a STABLE, unique `path` (the map key — used as the React render key)
// and a human `label(key, lv, rv)` for display; for an object map the two are
// the same, but for an identity-paired array the path is the (unique) match key
// while the label is a friendly name that may collide. Returns the parts array,
// or null when nothing differs.
const diffEntryMaps = (lMap, rMap, label) => {
const parts = [];
for (const key of new Set([...lMap.keys(), ...rMap.keys()])) {
const lv = lMap.get(key);
const rv = rMap.get(key);
if (canonicalStringify(lv) === canonicalStringify(rv)) continue;
parts.push({ path: key, label: label(key, lv, rv), localValue: lv, remoteValue: rv, changed: changedFlag(lv, rv) });
}
return parts.length ? parts : null;
};
// Build a Map keyed by each element's stable match key. Returns null — so the
// caller falls back to the whole-field diff — when ANY element is identity-less
// (null key) OR two elements share a key (a duplicate id, or two entries that
// both fall back to the same name): identity pairing isn't reliable, and a
// last-wins Map would silently drop the collided element from the diff.
const buildKeyedMap = (arr) => {
const map = new Map();
for (const el of arr) {
const key = entryMatchKey(el);
if (key === null || map.has(key)) return null;
map.set(key, el);
}
return map;
};
/**
* One-level structural diff of a single restorable field, used to render the
* Conflicts tab as "which sub-entry changed" instead of one giant JSON blob.
* Returns an array of changed sub-entries `[{ path, label, localValue,
* remoteValue, changed }]`, or `null` when the field isn't deepenable — a
* scalar, an array of scalars, an array of objects without unique stable
* identities, or a shape-mismatch (object vs array) — in which case the caller
* keeps the whole-field diff. Only entries that actually differ are emitted.
*
* - Object map / structured object (`categories`, `stages`, `arc`): one part
* per key whose value differs (path = label = key).
* - Array of identity-bearing objects (`characters`, `places`, `seasons`):
* paired by `entryMatchKey`; one part per identity that differs (a side
* missing ⇒ added/removed). The `path` is the unique match key; the `label`
* is a human name.
*/
export function deepFieldDiff(localVal, remoteVal) {
if (isPlainObject(localVal) && isPlainObject(remoteVal)) {
return diffEntryMaps(new Map(Object.entries(localVal)), new Map(Object.entries(remoteVal)), (key) => key);
}
if (Array.isArray(localVal) && Array.isArray(remoteVal)) {
if (localVal.length === 0 && remoteVal.length === 0) return null;
const lMap = buildKeyedMap(localVal);
const rMap = buildKeyedMap(remoteVal);
if (!lMap || !rMap) return null; // identity-less or duplicate keys → whole-field diff
return diffEntryMaps(lMap, rMap, (key, lv, rv) => entryLabel(lv ?? rv, key));
}
return null;
}
// Per-field diff over the kind's restorable content fields. A scalar field
// carries its whole-field `localValue`/`remoteValue` for InlineDiff; an
// object-map or array-of-objects field instead carries `parts` (the changed
// sub-entries) so the UI renders one focused diff per entry rather than a
// single giant JSON blob. The merge-fields selection stays at FIELD granularity
// (the resolver applies whole snapshot fields) — `parts` is display-only.
const diffSummary = (kind, local, remote) => {
const fields = RESTORABLE_FIELDS[kind] || [];
const out = [];
for (const field of fields) {
const lv = local?.[field];
const rv = remote?.[field];
if (canonicalStringify(lv) === canonicalStringify(rv)) continue;
const changed = changedFlag(lv, rv);
const parts = deepFieldDiff(lv, rv);
if (parts) out.push({ field, changed, parts });
else out.push({ field, localValue: lv, remoteValue: rv, changed });
}
return out;
};
/**
* Archive a losing local version. Writes a `pending` journal entry; returns its
* id. Snapshots are stored in sanitized wire form so a later restore re-applies
* a clean shape.
*/
export async function journalConflict({ kind, id, local, remote, source, hashes }) {
const localSnapshot = sanitizeRecordForWire(kind, local) ?? local;
const remoteSnapshot = sanitizeRecordForWire(kind, remote) ?? remote;
const entry = {
id: randomUUID(),
recordKind: kind,
recordId: id,
detectedAt: new Date().toISOString(),
source: source || { via: 'unknown', peerId: null, bucketId: null },
baseHash: hashes?.baseHash ?? null,
localHash: hashes?.localHash ?? null,
remoteHash: hashes?.remoteHash ?? null,
localSnapshot,
remoteSnapshot,
localUpdatedAt: local?.updatedAt ?? null,
remoteUpdatedAt: remote?.updatedAt ?? null,
diffSummary: diffSummary(kind, localSnapshot, remoteSnapshot),
status: 'pending',
resolvedAt: null,
resolution: null,
};
await store().saveOne(entry.id, entry);
console.log(`🪢 conflictJournal: archived ${kind} ${String(id).slice(0, 12)} (source=${entry.source.via})`);
return entry.id;
}
/**
* The one call every merge makes RIGHT BEFORE its overwrite. Detects a true
* conflict, archives the losing local version if so, and advances the base hash
* to the remote's (in memory — caller flushes after its loop). NEVER throws into
* the merge — a journal failure logs and the merge proceeds (convergence wins).
*
* Skips journaling (but still advances base) when there is no local content to
* lose: an absent/ephemeral local, or a local tombstone being overwritten.
*/
export async function maybeJournalBeforeOverwrite({ kind, id, local, remote, source }) {
try {
const { isConflict, baseHash, localHash, remoteHash, baseUnavailable } = await detectConflict({ kind, id, local, remote });
const localIsTombstone = local?.deleted === true;
// Fail CLOSED while the baseline store is untrustworthy (#7260): with the
// base hashes unreadable we cannot prove this merge ISN'T a divergence, so
// "baseline unknown" is treated as "possible conflict" and journaled. A
// spurious journal entry costs the user one dismissal; a missed one costs
// their edit. The nothing-to-lose guards still apply — an ephemeral local
// (no hashable wire form) or a local tombstone has no content to preserve.
if ((isConflict || baseUnavailable) && localHash != null && !localIsTombstone) {
await journalConflict({ kind, id, local, remote, source, hashes: { baseHash, localHash, remoteHash } });
}
// Advance the base to an UNRESTRICTED (current-version) hash of `remote`,
// never the version-restricted `remoteHash` used for the comparison above.
// setSyncBaseHash tags whatever it's given with the CURRENT version (see
// its own doc comment) — stamping the restricted hash would mislabel it:
// the tag would claim current-version field coverage the hash bytes don't
// actually have, so every later comparison trusts a version the stored
// hash never earned and can never match again once a HASH_FIELDS field is
// added (guaranteed permanent churn — the exact bug #2912 exists to fix).
const stampHash = contentHashForRecord(kind, remote);
if (stampHash != null) await setSyncBaseHash(kind, id, stampHash);
} catch (err) {
console.error(`❌ conflictJournal: maybeJournalBeforeOverwrite(${kind}:${id}) failed (proceeding): ${err?.message || err}`);
}
}
/**
* True while the `_persistBlocked` latch is held (#7260): the base-hash file
* was present-but-unreadable at load, so persistence is latched off (the file
* is preserved for repair) and overwrites journal fail-closed. Surfaced in
* the conflict-journal status payload so an empty Conflicts list can't be
* mistaken for "no conflicts" while detection is degraded.
*
* Awaits the memoized load so the flag reflects the file's actual state even
* when no merge has touched the store yet this process — a status poll must
* not report healthy just because nothing has read the file.
*/
export async function isBaseHashPersistBlocked() {
await ensureBaseLoaded();
return _persistBlocked;
}
// Test-only reset of the in-memory base-hash cache so suites that swap PATHS.data
// between tests don't bleed state across the module-level cache.
export function __resetBaseHashCacheForTests() {
_baseHashes = null;
_loadPromise = null;
_baseDirty = false;
_flushTail = Promise.resolve();
_flushBatchDepth = 0;
_persistBlocked = false;
_store = null;
}