Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

23.3 KB
Raw Blame
/**
 * Collection Store — per-type, per-record JSON storage with explicit
 * `schemaVersion` stamping.
 *
 * Layout on disk:
 *
 *     {dir}/
 *     ├── index.json          // { schemaVersion, type, updatedAt, config: {…} }
 *     ├── <id-1>/
 *     │   └── index.json      // the record itself
 *     └── <id-2>/
 *         └── index.json
 *
 * Replaces the legacy "one big JSON file per type" pattern (where every write
 * rewrites the whole file and every load parses every record) once a
 * collection has outgrown it. See `data/runs/` for the proven prior-art shape
 * this generalizes.
 *
 * The `config` slot — type-level shared state (convention):
 *   The type index's `config` object holds cross-record state that belongs to
 *   the collection as a whole rather than to any single record — the things
 *   that would otherwise need their own sidecar file. The convention is
 *   grounded in the first real consumer (universeBuilder, see migration 034),
 *   which keeps a capped `config.runs[]` history log alongside its per-universe
 *   records. The agreed shape (see the `TypeIndexConfig` typedef below):
 *
 *     config: {
 *       runs?: [],   // append-mostly cross-record history/audit log,
 *                    //   capped by the consumer (universeBuilder caps
 *                    //   at the last 200) — the established slot
 *       …            // consumers MAY add their own keys; document the
 *                    //   shape next to the consumer that owns it
 *     }
 *
 *   `runs` is the only slot with a shipped consumer today. Treat the slot as
 *   open-but-conventional: each key's value should be an object or array so
 *   the shallow-merge semantics below stay predictable.
 *
 *   Merge semantics: `saveTypeIndex({ config })` shallow-merges at the TOP
 *   level of `config` — `{ ...current.config, ...patch.config }`. A patch that
 *   sets `runs` REPLACES the whole `runs` array; it does NOT deep-merge into
 *   it. So a read-modify-write of an array (or nested-object) slot must load
 *   the current value, mutate a copy, and write the full replacement (see
 *   `recordRun` in universeBuilder). The merge is one level deep only.
 *
 *   Write-fence: `config`-slot writes all share ONE file (`index.json`) across
 *   every record in the type, so a read→modify→write cycle on a slot must run
 *   inside `queueTypeIndexWrite(fn)` to avoid clobbering a concurrent edit.
 *   `recordRun` is the canonical example: it loads the index, appends to
 *   `runs`, caps, and writes — all inside the type-index queue. Use
 *   `saveTypeIndex` for a simple patch (it queues internally); reach for
 *   `queueTypeIndexWrite` directly only when the read and the write must be
 *   fenced together. `defaultTypeIndexConfig` seeds the slot on a fresh type
 *   index (and on any read where `config` is missing or not a plain object);
 *   it is NOT re-applied on later writes.
 *
 * Two versioning concerns coexist:
 *   - The TYPE-LEVEL `schemaVersion` (this file): describes the on-disk
 *     storage layout. Bumped by a migration that changes layout (e.g. a split
 *     from a monolithic JSON to per-record dirs).
 *   - The RECORD-LEVEL `schemaVersion` (carried inside each record by some
 *     services, e.g. universeBuilder): describes the shape of one record's
 *     fields. Bumped by in-memory sanitizers when fields move/rename.
 *
 * These are distinct: a per-record sanitizer can mature without ever bumping
 * the type-level version, and a layout change can stamp a new type-level
 * version while record shape stays at v4.
 *
 * Concurrency model:
 *   - `queueRecordWrite(id, fn)` tail-chains per-id, so two concurrent
 *     read-modify-write cycles on the SAME record serialize, but writes to
 *     DIFFERENT records run in parallel. This is the scalability win over
 *     the single-file write queue.
 *   - `queueTypeIndexWrite(fn)` single-tail-chains writes to the type-level
 *     `index.json` so cross-record state (e.g. a shared `runs[]` log) doesn't
 *     race with itself.
 *   - `saveOne(id, record)` already wraps its write in `queueRecordWrite`, so
 *     plain mutations don't need explicit queueing. Wrap a custom RMW cycle
 *     in `queueRecordWrite(id, async () => { ... })` when an external read
 *     spans the load/save boundary, then call `saveOneNow` inside that queue
 *     so the queued function does not wait on itself.
 */

import { join } from 'path';
import { readdir, rm } from 'fs/promises';
import { atomicWrite, readJSONFile, readJSONFileStrict, unreadableStoreError, ensureDir } from './fileUtils.js';
import { createFileWriteQueue, createRecordWriteQueue } from './fileWriteQueue.js';
import { isPlainObject, POLLUTING_KEYS } from './objects.js';
import { isVitestRunner } from './runtimeEnv.js';

/**
 * The type-index `config` slot — cross-record state owned by the collection as
 * a whole. See the header doc block for the convention. All keys are optional;
 * a consumer that needs a slot not listed here MAY add its own (document it
 * next to the consumer). Each value should be an array or plain object so the
 * top-level shallow-merge in `saveTypeIndex` stays predictable.
 *
 * @typedef {object} TypeIndexConfig
 * @property {Array<object>} [runs]
 *   Append-mostly cross-record history/audit log. The consumer is responsible
 *   for capping growth (universeBuilder keeps the last 200) and for sanitizing
 *   each entry on read/write. The only slot with a shipped consumer today.
 */

/**
 * Create a per-type collection store.
 *
 * @param {object} opts
 * @param {string} opts.dir
 *   Absolute path to the type's root directory, e.g. `${PATHS.data}/universes`.
 * @param {string} opts.type
 *   Stable type label (e.g. `'universes'`). Persisted in the type-level index;
 *   used in error/log messages so a misregistered store is obvious.
 * @param {number} opts.schemaVersion
 *   The layout version the code currently expects. `verifySchemaVersion()`
 *   asserts the on-disk type index matches this.
 * @param {(record: any) => any} [opts.sanitizeRecord]
 *   Optional record-level normalizer. Called on every `loadOne` result. Return
 *   `null` to treat the on-disk record as invalid (loadOne returns `null`).
 * @param {TypeIndexConfig} [opts.defaultTypeIndexConfig]
 *   Initial value for the `config` slot of a freshly-minted type index (and the
 *   fallback when an on-disk index is missing/has a non-object `config`).
 *   Defaults to `{}`. Seeds the slot on first read only — it is NOT re-applied
 *   on later `saveTypeIndex` writes, which shallow-merge over the current value.
 *   See the `TypeIndexConfig` typedef and the header `config`-slot convention.
 * @param {RegExp} [opts.idPattern]
 *   Allowlist for record-directory names. Defaults to a permissive pattern
 *   (`/^[A-Za-z0-9_-]{1,128}$/`) — services with stricter id rules should pass
 *   their own (e.g. UUID-only) so `listIds()` doesn't surface stray files.
 */
export function createCollectionStore({
  dir,
  type,
  schemaVersion,
  sanitizeRecord = null,
  defaultTypeIndexConfig = {},
  idPattern = /^[A-Za-z0-9_-]{1,128}$/,
} = {}) {
  if (typeof dir !== 'string' || !dir) {
    throw new Error('createCollectionStore: `dir` is required');
  }
  if (typeof type !== 'string' || !type) {
    throw new Error('createCollectionStore: `type` is required');
  }
  if (!Number.isInteger(schemaVersion) || schemaVersion < 1) {
    throw new Error('createCollectionStore: `schemaVersion` must be a positive integer');
  }

  const typeIndexPath = () => join(dir, 'index.json');
  const recordDir = (id) => join(dir, id);
  const recordPath = (id) => join(dir, id, 'index.json');
  // Process-local fallback for tests that mock fileUtils' read/write helpers
  // without backing them with a real directory tree. Production listIds still
  // uses readdir as the source of truth whenever the collection dir exists.
  const knownIds = new Set();

  // Reject prototype-polluting names even when they pass the id pattern, so
  // consumers do not each need to guard legacy JSON keys used as record ids.
  const isValidId = (id) => typeof id === 'string' && idPattern.test(id) && !POLLUTING_KEYS.has(id);

  // Per-record write queue. Tail-chained per `id` so two writes against the
  // same record serialize while writes against different records proceed in
  // parallel. The shared `createRecordWriteQueue` helper carries the id-keyed
  // silence/self-prune discipline; the assert throws synchronously on a bad id
  // before anything is enqueued.
  const queueRecordWrite = createRecordWriteQueue((id) => {
    if (!isValidId(id)) {
      throw new Error(`collectionStore[${type}]: invalid record id "${id}" — must match ${idPattern}`);
    }
  });

  // Type-index single-tail queue. Used by `saveTypeIndex` and by callers that
  // need a write-fence around config-slot updates (`runs[]`, feature flags).
  const queueTypeIndexWrite = createFileWriteQueue();

  const buildEmptyTypeIndex = () => ({
    schemaVersion,
    type,
    updatedAt: new Date().toISOString(),
    config: { ...defaultTypeIndexConfig },
  });

  /**
   * Load the type-level index.json. Returns the default shape (with the
   * code-expected `schemaVersion`) when the file is ABSENT — does NOT write
   * to disk. Use `saveTypeIndex` / `saveOne` to persist.
   *
   * Strict (#7261): every `config`-slot writer starts from this read and writes
   * the result back, so a swallowed present-but-unreadable index (EACCES, EIO,
   * truncated bytes, a cloud-sync conflict copy) would persist the shipped
   * `defaultTypeIndexConfig` over real user state AND stamp the code's current
   * `schemaVersion` over records still in the old on-disk layout. Throwing here
   * means `idealoomLists.updateSettings` and universeBuilder's run writers
   * reject BEFORE any byte is written, leaving the file intact for repair;
   * request-path callers surface it as a 5xx (`unreadableStoreError` carries
   * `status: 500`). Same posture as `settingsStore.get()`.
   */
  async function loadTypeIndex() {
    const { ok, value: raw } = await readJSONFileStrict(typeIndexPath(), null, { allowArray: false });
    if (!ok) throw unreadableStoreError(typeIndexPath());
    if (!isPlainObject(raw)) return buildEmptyTypeIndex();
    return {
      schemaVersion: Number.isInteger(raw.schemaVersion) ? raw.schemaVersion : schemaVersion,
      type: typeof raw.type === 'string' && raw.type ? raw.type : type,
      updatedAt: typeof raw.updatedAt === 'string' ? raw.updatedAt : null,
      config: isPlainObject(raw.config) ? raw.config : { ...defaultTypeIndexConfig },
    };
  }

  /**
   * Persist a patch into the type-level index. `schemaVersion` / `type` are
   * replaced only when the patch supplies a valid value (else preserved);
   * `config` is SHALLOW-merged one level deep (`{ ...current, ...patch.config }`,
   * so a patched slot like `runs` replaces the whole array — see the header
   * `config`-slot convention); `updatedAt` is restamped automatically.
   * Serialized through `queueTypeIndexWrite` so concurrent callers can't stomp
   * each other's config-slot edits. For a read-modify-write that must read the
   * current slot value before writing (e.g. appending to `runs`), do the load
   * and the write together inside one `queueTypeIndexWrite(fn)` rather than
   * leaning on this patch form.
   *
   * @param {{ schemaVersion?: number, type?: string, config?: TypeIndexConfig }} [patch]
   */
  function saveTypeIndex(patch = {}) {
    return queueTypeIndexWrite(async () => {
      await ensureDir(dir);
      const current = await loadTypeIndex();
      const next = {
        schemaVersion: Number.isInteger(patch.schemaVersion) ? patch.schemaVersion : current.schemaVersion,
        type: typeof patch.type === 'string' && patch.type ? patch.type : current.type,
        config: isPlainObject(patch.config) ? { ...current.config, ...patch.config } : current.config,
        updatedAt: new Date().toISOString(),
      };
      await atomicWrite(typeIndexPath(), next);
      return next;
    });
  }

  /**
   * List the on-disk record ids. Source of truth is `readdir(dir)`, not a
   * manifest — keeps `index.json` from drifting against the actual directory
   * contents. Filters out the type-level `index.json`, hidden entries, and
   * anything that doesn't match `idPattern` or isn't a directory.
   */
  async function listIds() {
    const entries = await readdir(dir, { withFileTypes: true }).catch((err) => {
      if (err?.code === 'ENOENT') return null;
      throw err;
    });
    if (entries === null) return [...knownIds].filter(isValidId);
    // Dirent types exclude symlinks without an explicit lstat per record.
    return entries.filter((entry) =>
      entry.name !== 'index.json'
      && !entry.name.startsWith('.')
      && isValidId(entry.name)
      && entry.isDirectory()
    ).map((entry) => entry.name);
  }

  /**
   * Load one record by id. Returns `null` when the record dir is missing or
   * the on-disk JSON fails to parse; runs the configured `sanitizeRecord` on
   * the parsed payload (sanitizer can also return `null` to reject).
   *
   * Non-strict by design (unlike `loadTypeIndex` above, #7261): a per-record
   * read is not a read-modify-write of shared state — `saveOneNow` writes a
   * record the caller already holds in full, so an unreadable neighbor can
   * never be overwritten by defaults. The "which ids failed" signal a strict
   * read would carry is already preserved by `loadAllResult` below, and a
   * corrupt single record must not take down a collection-wide load. Callers
   * that need to tell "absent" from "corrupt" use `loadAllResult`.
   */
  async function loadOne(id) {
    if (!isValidId(id)) return null;
    const parsed = await readJSONFile(recordPath(id), null, { allowArray: false, logError: false });
    if (!isPlainObject(parsed)) return null;
    if (typeof sanitizeRecord === 'function') {
      return sanitizeRecord(parsed);
    }
    return parsed;
  }

  /**
   * Load one record WITHOUT running `sanitizeRecord`. Some callers need to
   * compare the raw on-disk shape to the sanitized shape (e.g. detect "this
   * record's nested ids need persisting" — sanitize mints missing ids on the
   * fly but doesn't persist them until the next write). Returns the parsed
   * JSON or `null`; ignores the sanitizer entirely. Non-strict for the same
   * reason as `loadOne` above.
   */
  async function loadOneRaw(id) {
    if (!isValidId(id)) return null;
    const parsed = await readJSONFile(recordPath(id), null, { allowArray: false, logError: false });
    return isPlainObject(parsed) ? parsed : null;
  }

  /**
   * Load every record in parallel, keeping the "which ids failed to load" signal
   * that `loadAll` throws away. A record `id` returned by `listIds()` whose
   * `loadOne(id)` resolves to `null` means the on-disk file is
   * missing/unreadable/unparseable or was rejected by `sanitizeRecord` — i.e. a
   * corrupt record in an otherwise-readable collection. Callers that reason over
   * the whole set (merge-rate stats, adoption reports) need to distinguish
   * "N records" from "N-of-M records, K corrupt" so a silent undercount can't
   * masquerade as complete truth — the same sentinel-vs-empty rule the AGENTS.md
   * conventions call out. Mirrors the `loadAll`/`listOutcomesResult` precedent.
   *
   * @returns {Promise<{ records: any[], failedIds: string[] }>}
   *   `records` — the successfully-loaded, sanitized records (nulls dropped).
   *   `failedIds` — the ids `listIds()` surfaced that failed to load (empty when
   *   every record loaded cleanly).
   */
  async function loadAllResult() {
    const ids = await listIds();
    const loaded = await Promise.all(ids.map((id) => loadOne(id)));
    const records = [];
    const failedIds = [];
    ids.forEach((id, i) => {
      if (loaded[i] != null) records.push(loaded[i]);
      else failedIds.push(id);
    });
    return { records, failedIds };
  }

  /**
   * Load every record in parallel. O(N) cost — acceptable while record counts
   * are bounded (typical PortOS collections sit in the tens-to-hundreds), but
   * callers iterating very large collections should prefer `listIds` +
   * targeted `loadOne` rather than slurping the whole set. Back-compat flattening
   * wrapper over `loadAllResult` — silently drops corrupt records; reach for
   * `loadAllResult` when the caller must be able to tell a partial set apart.
   */
  async function loadAll() {
    return (await loadAllResult()).records;
  }

  /**
   * Persist a record. Serialized per-id so two concurrent writes against the
   * same id chain on each other; writes against different ids run in parallel.
   * Throws on missing/invalid id; sanitizer is NOT applied on write (callers
   * are expected to pre-sanitize before calling — same convention as the
   * monolithic-state services).
   */
  async function saveOneNow(id, record) {
    if (!isValidId(id)) {
      throw new Error(`collectionStore[${type}]: invalid record id "${id}" — must match ${idPattern}`);
    }
    if (!isPlainObject(record)) {
      throw new Error(`collectionStore[${type}]: record must be a plain object`);
    }
    await ensureDir(recordDir(id));
    await atomicWrite(recordPath(id), record);
    knownIds.add(id);
    return record;
  }

  function saveOne(id, record) {
    if (!isValidId(id)) {
      throw new Error(`collectionStore[${type}]: invalid record id "${id}" — must match ${idPattern}`);
    }
    if (!isPlainObject(record)) {
      throw new Error(`collectionStore[${type}]: record must be a plain object`);
    }
    return queueRecordWrite(id, async () => {
      return saveOneNow(id, record);
    });
  }

  /**
   * Hard-delete a record WITHOUT the per-id queue. Removes the entire
   * `{dir}/{id}/` subtree AND drops the id from `knownIds` (the readdir-ENOENT
   * fallback source). The delete-side counterpart to `saveOneNow`: call it
   * INSIDE a `queueRecordWrite(id, …)` block when an external recheck must span
   * the load→delete boundary atomically (e.g. tombstone GC re-checking that the
   * record is still deleted before removing it). Idempotent — missing dir is a
   * no-op. Outside a queue, prefer `deleteOne`.
   */
  async function deleteOneNow(id) {
    if (!isValidId(id)) {
      throw new Error(`collectionStore[${type}]: invalid record id "${id}" — must match ${idPattern}`);
    }
    // #6176 — a recursive delete is the most destructive form of a data-root
    // leak, so it takes the same guard the write primitives do. Loaded lazily
    // for the same import-budget reason as fileCore.js.
    if (isVitestRunner()) {
      const { assertNotRealDataWrite } = await import('./testDataIsolation.js');
      assertNotRealDataWrite(recordDir(id), 'collectionStore delete');
    }
    await rm(recordDir(id), { recursive: true, force: true });
    knownIds.delete(id);
  }

  /**
   * Hard-delete a record. Removes the entire `{dir}/{id}/` subtree so any
   * sidecar files (per-record images, derived assets) go with it. Serialized
   * on the per-record queue so a delete can't race with an in-flight write.
   * Idempotent — missing dir is a no-op.
   */
  function deleteOne(id) {
    if (!isValidId(id)) {
      throw new Error(`collectionStore[${type}]: invalid record id "${id}" — must match ${idPattern}`);
    }
    return queueRecordWrite(id, () => deleteOneNow(id));
  }

  /**
   * Boot-time verifier — confirms the on-disk type index matches the code's
   * expected `schemaVersion`. Returns a status object rather than throwing so
   * `server/index.js` can log all collections in one pass.
   *
   *   ok=true   — on-disk version matches code (`reason: 'match'`), or the type
   *               index is ABSENT (`reason: 'missing'`), which is treated as a
   *               fresh install — the next write stamps the correct version.
   *   ok=false  — on-disk version is older (`reason: 'older'`) or newer
   *               (`reason: 'newer'`) than code. Older → a pending migration
   *               didn't run. Newer → code rolled back below a forward-only
   *               migration. OR the file is present but could not be read
   *               (`reason: 'unreadable'`, `onDisk: null`).
   *
   * The `unreadable` verdict exists because this gate is the ONE boot-time check
   * that catches a skipped or rolled-back storage migration (#7261). Reporting
   * "fresh install" for a file we could not read would silence that alarm for a
   * collection whose records are still in the old layout — and the next write
   * would then stamp the current version over it, falsifying the marker
   * permanently.
   */
  async function verifySchemaVersion() {
    const { ok: readable, value: raw } = await readJSONFileStrict(typeIndexPath(), null, { allowArray: false, logError: false });
    if (!readable) {
      return {
        ok: false,
        onDisk: null,
        expected: schemaVersion,
        type,
        reason: 'unreadable',
        message: `collection "${type}": index.json at ${typeIndexPath()} exists but could not be read or parsed — repair or delete it and restart. Until then the storage-version check for this collection cannot run.`,
      };
    }
    if (!isPlainObject(raw) || raw.schemaVersion == null) {
      return {
        ok: true,
        onDisk: null,
        expected: schemaVersion,
        type,
        reason: 'missing',
        message: `collection "${type}": no index.json (fresh install) — first write will stamp schemaVersion=${schemaVersion}`,
      };
    }
    const onDisk = Number.isInteger(raw.schemaVersion) ? raw.schemaVersion : null;
    if (onDisk === schemaVersion) {
      return { ok: true, onDisk, expected: schemaVersion, type, reason: 'match', message: `collection "${type}" @ v${schemaVersion}` };
    }
    if (onDisk != null && onDisk < schemaVersion) {
      return {
        ok: false,
        onDisk,
        expected: schemaVersion,
        type,
        reason: 'older',
        message: `collection "${type}": on-disk v${onDisk}, code expects v${schemaVersion} — a migration didn't run. Check scripts/migrations/ and run \`npm run migrations\`.`,
      };
    }
    return {
      ok: false,
      onDisk,
      expected: schemaVersion,
      type,
      reason: 'newer',
      message: `collection "${type}": on-disk v${onDisk}, code expects v${schemaVersion} — code rolled back below a forward-only migration. Roll forward or restore from a backup.`,
    };
  }

  return {
    // Paths (mostly for tests + migration scripts)
    dir,
    type,
    schemaVersion,
    isValidId,
    typeIndexPath,
    recordDir,
    recordPath,
    // Type-level index
    loadTypeIndex,
    saveTypeIndex,
    // Per-record CRUD
    listIds,
    loadOne,
    loadOneRaw,
    loadAll,
    loadAllResult,
    saveOneNow,
    saveOne,
    deleteOneNow,
    deleteOne,
    // Concurrency primitives
    queueRecordWrite,
    queueTypeIndexWrite,
    // Boot-time verifier
    verifySchemaVersion,
  };
}

/**
 * Walk a list of stores, run `verifySchemaVersion()` on each, log the result,
 * and return the per-store status array. Used at server boot AFTER migrations
 * run so a missed migration produces a loud, single-line log per collection.
 *
 * PortOS is single-user — a hard exit on mismatch is worse than a noisy log,
 * so this never throws (the caller decides whether to keep booting).
 */
export async function verifyCollectionVersions(stores = []) {
  const statuses = [];
  for (const store of stores) {
    const status = await store.verifySchemaVersion();
    if (status.ok) {
      console.log(`📋 ${status.message}`);
    } else {
      console.error(`❌ ${status.message}`);
    }
    statuses.push(status);
  }
  return statuses;
}