Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
test: protect migration-derived data when seed exclusions are removed (#11994)
Adam Eivy5dc344a2ac70
diff --git a/AGENTS.md b/AGENTS.mdindex 88cc7058b9..23c1cd9574 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -102,7 +102,7 @@ Because both are off by default, **never treat "the password is set" as an avail- **Never delete or skip migration / compatibility code on the grounds that "there's only one install."** Other people and other machines run this code and update it independently.- On-disk format changes need a migration in `scripts/migrations/` (applied-list tracked per install in `data/migrations.applied.json`). Seed files ship in `data.reference/`.-- **A migration that DERIVES `data/<x>` from an install's existing records ships NO `data.reference/<x>` seed, and gates on the presence of its INPUT — never the absence of its output.** `setup-data.js` runs before `run-migrations.js` and copies every `data.reference/` file the install lacks, so a seed lands first; a migration that skips because its output already exists then leaves shipped defaults where the user's data was, and the next write destroys the original. Declare such a path in `scripts/lib/migrationOwnedPaths.js` (its test fails if a seed reappears). The applied-list already provides re-run safety. This replaced CoS settings with shipped defaults on installs crossing #6182 — see `scripts/migrations/340-cos-config-seed-repair.js`.+- **A migration that DERIVES `data/<x>` from an install's existing records ships NO `data.reference/<x>` seed, and gates on the presence of its INPUT — never the absence of its output.** `setup-data.js` runs before `run-migrations.js` and copies every `data.reference/` file the install lacks, so a seed lands first; a migration that skips because its output already exists then leaves shipped defaults where the user's data was, and the next write destroys the original. Declare such a path in `scripts/lib/migrationOwnedPaths.js` and export the same `data/`-relative path in the deriving migration's `OWNED_OUTPUT_PATHS` array (the guard checks both declarations and fails if a seed reappears). The applied-list already provides re-run safety. This replaced CoS settings with shipped defaults on installs crossing #6182 — see `scripts/migrations/340-cos-config-seed-repair.js`.- Prompt-default changes are two steps: bump the prompt's `PROMPT_VERSIONS` entry, then run `node scripts/regen-prompt-integrity-snapshot.js`. The tool retires the outgoing default's hash into `server/services/taskPromptDefaults/integrity.snapshot.json` — how other installs recognize a stored prompt as a shipped default and auto-upgrade it (`promptMatchesShippedDefault`) — and refuses a body change with no bump. Retired bodies live only in git history; never paste one back as recognition data (the one literal in `taskPromptDefaults/retiredPromptFixtures.js` is a test fixture).- Cross-machine sync payloads stay version-gated (`server/lib/schemaVersions.js`) so a newer peer can't corrupt an older one.- The self-update path stays fork-aware — other users run forks. **Never add `--force` to the server-side `gh repo sync`** (it would discard a user's fork commits; the 409 `FORK_DIVERGED` message points them at running it themselves). Don't re-hardcode the upstream slug outside `server/lib/gitRemote.js`, and new UI claiming "you are running PortOS" must read `remoteInfo.isUpstream`, not just `currentVersion`. Mechanism: [docs/SELF_UPDATE.md](docs/SELF_UPDATE.md).diff --git a/scripts/lib/migrationOwnedPaths.js b/scripts/lib/migrationOwnedPaths.jsindex 85ba2d94f9..d8fb3b3824 100644--- a/scripts/lib/migrationOwnedPaths.js+++ b/scripts/lib/migrationOwnedPaths.js@@ -2,7 +2,8 @@* `data/` paths that a MIGRATION derives from an install's existing records,* and that must therefore ship no `data.reference/` seed.*- * The enforcement is the seed's ABSENCE, asserted by the sibling test. The+ * The sibling test checks migration OWNED_OUTPUT_PATHS declarations against+ * this runtime set, then enforces every seed's ABSENCE. The* seed filter below — used by `scripts/setup-data.js` and the boot smoke's* disposable install (`scripts/smoke-boot.js`) — is defense in depth for a* seed someone re-adds. Full rationale — and thediff --git a/scripts/lib/migrationOwnedPaths.test.js b/scripts/lib/migrationOwnedPaths.test.jsindex 109173ad93..5f21b72e86 100644--- a/scripts/lib/migrationOwnedPaths.test.js+++ b/scripts/lib/migrationOwnedPaths.test.js@@ -9,15 +9,39 @@* `scripts/migrations/340-cos-config-seed-repair.js`.*/import { describe, it, expect } from 'vitest';-import { existsSync } from 'fs';+import { existsSync, readdirSync, readFileSync } from 'fs';import { dirname, join } from 'path';-import { fileURLToPath } from 'url';+import { fileURLToPath, pathToFileURL } from 'url';import { MIGRATION_OWNED_PATHS } from './migrationOwnedPaths.js';const repoRoot = join(dirname(fileURLToPath(import.meta.url)), '..', '..');+const migrationsDir = join(repoRoot, 'scripts', 'migrations');+// Import declarations only: unrelated historical migrations may have boot-time+// dependencies, and none of their up() functions should run in this guard.+const declaringMigrations = readdirSync(migrationsDir).filter((file) =>+ file.endsWith('.js') && !file.endsWith('.test.js')+ && /\bOWNED_OUTPUT_PATHS\b/.test(readFileSync(join(migrationsDir, file), 'utf8')),+);+const migrationOutputs = await Promise.all(declaringMigrations.map(async (file) => ({+ file,+ paths: (await import(pathToFileURL(join(migrationsDir, file)).href)).OWNED_OUTPUT_PATHS,+})));+describe('migration-owned data paths', () => {+ it('protects every output declared by a deriving migration in the runtime set', () => {+ expect(migrationOutputs.length).toBeGreaterThan(0);+ for (const { file, paths } of migrationOutputs) {+ expect(Array.isArray(paths), `${file}: OWNED_OUTPUT_PATHS must be an array`).toBe(true);+ expect(paths.length, `${file}: declare at least one derived output`).toBeGreaterThan(0);+ for (const path of paths) {+ expect(typeof path, `${file}: output paths must be strings`).toBe('string');+ expect(MIGRATION_OWNED_PATHS.has(path), `${file}: ${path} is missing from MIGRATION_OWNED_PATHS`).toBe(true);+ }+ }+ });+it('ships no data.reference seed for any migration-owned path', () => {const seeded = [...MIGRATION_OWNED_PATHS].filter((relPath) =>existsSync(join(repoRoot, 'data.reference', ...relPath.split('/'))),diff --git a/scripts/migrations/323-eidoverse-world-design-v2.js b/scripts/migrations/323-eidoverse-world-design-v2.jsindex 7d1d3c875d..331241e210 100644--- a/scripts/migrations/323-eidoverse-world-design-v2.js+++ b/scripts/migrations/323-eidoverse-world-design-v2.js@@ -12,6 +12,9 @@ import { join } from 'path';import { atomicWrite, safeJSONParse } from '../../server/lib/fileUtils.js';import { migrateEidoverseWorldState, EIDOVERSE_WORLD_DESIGN_VERSION, EIDOVERSE_WORLD_STATE_SCHEMA_VERSION } from '../../server/lib/eidoverseWorldDesign.js';+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['eidoverse/portos-world.json'];+export default {async up({ rootDir }) {const statePath = join(rootDir, 'data', 'eidoverse', 'portos-world.json');diff --git a/scripts/migrations/339-cos-config-split.js b/scripts/migrations/339-cos-config-split.jsindex afdd48f13b..a81e2fde2b 100644--- a/scripts/migrations/339-cos-config-split.js+++ b/scripts/migrations/339-cos-config-split.js@@ -49,6 +49,9 @@ async function removeRetiredBackup(rootDir) {return removed;}+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['cos/config.json'];+export default {async up({ rootDir }) {// Runs on every path below, including the already-split no-op: an installdiff --git a/scripts/migrations/343-eidoverse-label-aliases.js b/scripts/migrations/343-eidoverse-label-aliases.jsindex 9a71198ba6..8e1e587ea9 100644--- a/scripts/migrations/343-eidoverse-label-aliases.js+++ b/scripts/migrations/343-eidoverse-label-aliases.js@@ -4,6 +4,9 @@ import { join } from 'node:path';import { atomicWrite, safeJSONParse } from '../../server/lib/fileUtils.js';import { migrateEidoverseWorldState } from '../../server/lib/eidoverseWorldDesign.js';+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['eidoverse/portos-world.json'];+export default {async up({ rootDir }) {const path = join(rootDir, 'data', 'eidoverse', 'portos-world.json');diff --git a/scripts/migrations/349-eidoverse-commons.js b/scripts/migrations/349-eidoverse-commons.jsindex 309e65ebc5..2ab5451621 100644--- a/scripts/migrations/349-eidoverse-commons.js+++ b/scripts/migrations/349-eidoverse-commons.js@@ -4,6 +4,9 @@ import { join } from 'node:path';import { atomicWrite, safeJSONParse } from '../../server/lib/fileUtils.js';import { migrateEidoverseWorldState } from '../../server/lib/eidoverseWorldDesign.js';+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['eidoverse/portos-world.json'];+export default {async up({ rootDir }) {const path = join(rootDir, 'data', 'eidoverse', 'portos-world.json');diff --git a/scripts/migrations/357-private-api-keys.js b/scripts/migrations/357-private-api-keys.jsindex 613d720c32..66b92bd2ad 100644--- a/scripts/migrations/357-private-api-keys.js+++ b/scripts/migrations/357-private-api-keys.js@@ -2,6 +2,9 @@ import { join } from 'node:path';import { atomicWrite, readJSONFileStrict } from '../../server/lib/fileUtils.js';import { persistPrivateKeys } from '../../server/services/privateKeyStore.js';+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['private/api-keys.json'];+export default {async up({ rootDir }) {const file = join(rootDir, 'data/settings.json');diff --git a/scripts/migrations/358-provider-connection-graph.js b/scripts/migrations/358-provider-connection-graph.jsindex eeda6d9e9d..e309e2f9f3 100644--- a/scripts/migrations/358-provider-connection-graph.js+++ b/scripts/migrations/358-provider-connection-graph.js@@ -27,6 +27,9 @@ import { atomicWrite, readJSONFileStrict, tryReadFile } from '../../server/lib/f* the applied-list plus the explicit "already parked" check below — a second* run must never overwrite the pre-graph copy with a post-graph file.*/+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['private/providers.pre-graph.json'];+export default {async up({ rootDir }) {const input = join(rootDir, 'data/providers.json');diff --git a/scripts/migrations/359-quota-burn-task-references.js b/scripts/migrations/359-quota-burn-task-references.jsindex 9d5ae58773..520a32633a 100644--- a/scripts/migrations/359-quota-burn-task-references.js+++ b/scripts/migrations/359-quota-burn-task-references.js@@ -85,6 +85,9 @@ async function readJson(fullPath, label) {const writeJson = (fullPath, value) => atomicWrite(fullPath, `${JSON.stringify(value, null, 2)}\n`);+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['cos/quota-burn.json', 'cos/quota-burn.pre-359.json'];+export default {async up({ rootDir }) {const planPath = join(rootDir, QUOTA_BURN_PATH);diff --git a/scripts/migrations/370-tailcat-remote-ingress.js b/scripts/migrations/370-tailcat-remote-ingress.jsindex 19abaa5b65..c433552a3a 100644--- a/scripts/migrations/370-tailcat-remote-ingress.js+++ b/scripts/migrations/370-tailcat-remote-ingress.js@@ -5,6 +5,9 @@ import { PORTS } from '../../server/lib/ports.js';// Derived from the existing machine-local config. No seed and no forward-port// rewrite: a stored forward may still target an independently upgraded peer.+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['tailcat-serve.json'];+export default {async up({ rootDir }) {const path = join(rootDir, 'data', 'tailcat-serve.json');diff --git a/scripts/migrations/397-eidoverse-foundation-derived-assay.js b/scripts/migrations/397-eidoverse-foundation-derived-assay.jsindex c02f847d0d..d4a8f780ff 100644--- a/scripts/migrations/397-eidoverse-foundation-derived-assay.js+++ b/scripts/migrations/397-eidoverse-foundation-derived-assay.js@@ -40,6 +40,9 @@ import { derivedContributionId } from '../../server/lib/eidoverseFoundations.js'import { foundationSandbox } from '../../server/lib/eidoverseFoundationSandbox.js';import { findControllerDefinitionById } from '../../server/services/eidoverseControllerRegistry.js';+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['eidoverse/foundations.json'];+export default {async up({ rootDir }) {const file = join(rootDir, 'data', 'eidoverse', 'foundations.json');diff --git a/scripts/migrations/406-cos-agent-completion-order-index.js b/scripts/migrations/406-cos-agent-completion-order-index.jsindex 844416888f..c506cea523 100644--- a/scripts/migrations/406-cos-agent-completion-order-index.js+++ b/scripts/migrations/406-cos-agent-completion-order-index.js@@ -45,6 +45,9 @@ const readJSON = async (path) => {try { return JSON.parse(raw); } catch { return null; }};+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['cos/agents/index.order.json'];+export default {async up({ rootDir }) {const agentsDir = join(rootDir, AGENTS_REL);diff --git a/scripts/migrations/412-prune-config-seed-duplicates.js b/scripts/migrations/412-prune-config-seed-duplicates.jsindex a630d055bb..9358501c00 100644--- a/scripts/migrations/412-prune-config-seed-duplicates.js+++ b/scripts/migrations/412-prune-config-seed-duplicates.js@@ -81,6 +81,9 @@ function pruneConfig(config, retiredSeed) {return Object.keys(pruned).length > 0 ? pruned : null;}+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['memory-classifier-config.json', 'browser-config.json'];+export default {async up({ rootDir }) {const results = {diff --git a/scripts/migrations/425-eidoverse-controller-completed-delivery.js b/scripts/migrations/425-eidoverse-controller-completed-delivery.jsindex cf473260f3..09b179d9c5 100644--- a/scripts/migrations/425-eidoverse-controller-completed-delivery.js+++ b/scripts/migrations/425-eidoverse-controller-completed-delivery.js@@ -7,6 +7,9 @@ import { readFile } from 'node:fs/promises';import { join } from 'node:path';import { writeJsonAtomic } from './_lib.js';+/** Derived paths relative to `data/`; never ship reference seeds. */+export const OWNED_OUTPUT_PATHS = ['eidoverse/controllers.json'];+export default {async up({ rootDir }) {const file = join(rootDir, 'data', 'eidoverse', 'controllers.json');