Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: read the migration purge marker statically and keep new migrations off server/services
[._.]/ Adam Eivy5b26433aea00
diff --git a/scripts/migrations/migrationConventions.guard.test.js b/scripts/migrations/migrationConventions.guard.test.jsindex 4529c7544e..aa6114ca30 100644--- a/scripts/migrations/migrationConventions.guard.test.js+++ b/scripts/migrations/migrationConventions.guard.test.js@@ -11,12 +11,26 @@* `collidesWith` 5 each before `_lib.js` exported them. A declaration* whose name matches a `_lib.js` export — or one of the local spellings* in `LOCAL_ALIASES` — must import the export instead.+ *+ * 2. No import from `server/services` (#12049). A migration is a frozen+ * snapshot of a transform: an install upgrading across many versions runs+ * it months later, and a transform that calls into live service code+ * then does whatever today's service does, not what the migration did+ * when it shipped. Seventy-odd applied migrations import services and+ * stay as they are; a new one uses `_lib.js` primitives, `server/lib`+ * pure helpers, or a frozen copy of the transform.+ *+ * 3. The purge marker stays statically visible. `run-migrations.js` reads+ * `purge: true` from source text (`sourceDeclaresPurge`) so a fresh+ * install's first boot does not import every pending module up front;+ * every shipped migration's module must agree with that probe.*/import { describe, it, expect } from 'vitest';import { readdirSync, readFileSync } from 'node:fs';import { resolve } from 'node:path';import { fileURLToPath } from 'node:url';import * as lib from './_lib.js';+import { sourceDeclaresPurge } from '../run-migrations.js';const HERE = resolve(fileURLToPath(import.meta.url), '..');@@ -51,6 +65,11 @@ const redeclaredLibHelpers = (source) => [...stripComments(source).matchAll(DECL.filter((name) => LIB_EXPORTS.has(name) || name in LOCAL_ALIASES).map((name) => ({ name, use: LOCAL_ALIASES[name] ?? name }));+// Static `from '…/server/services/…'` and dynamic `import('…/server/services/…')`.+const SERVICE_IMPORT = /(?:\bfrom\s+|\bimport\s*\(\s*)['"][^'"]*\/server\/services\//;++const importsServerServices = (source) => SERVICE_IMPORT.test(stripComments(source));+const migrationNumber = (file) => Number.parseInt(file.slice(0, 3), 10);const migrationFiles = () => readdirSync(HERE)@@ -73,6 +92,44 @@ describe('new-migration conventions', () => {expect(offenders).toEqual([]);});+ it('no guarded migration imports from server/services', () => {+ const offenders = guardedSources()+ .filter(([, source]) => importsServerServices(source))+ .map(([file]) => `${file}: imports server/services — use _lib.js, server/lib, or a frozen copy of the transform`);+ expect(offenders).toEqual([]);+ });++ it('every shipped migration agrees with the static purge probe', async () => {+ // Only modules that mention purge at all can disagree; importing those+ // few keeps the drift check cheap while still covering every purge+ // migration and every near-miss spelling.+ const candidates = migrationFiles()+ .map((file) => [file, readFileSync(resolve(HERE, file), 'utf8')])+ .filter(([, source]) => /purge/i.test(source));+ expect(candidates.length).toBeGreaterThan(0);+ for (const [file, source] of candidates) {+ const mod = await import(`./${file}`);+ const migration = mod?.default && typeof mod.default.up === 'function' ? mod.default : mod;+ expect({ file, purge: sourceDeclaresPurge(source) }).toEqual({ file, purge: migration?.purge === true });+ }+ });++ describe('service import matching', () => {+ it('flags static and dynamic imports from server/services', () => {+ expect(importsServerServices("import { listUniverses } from '../../server/services/universeBuilder.js';")).toBe(true);+ expect(importsServerServices("const { x } = await import('../../server/services/universeCanon.js');")).toBe(true);+ });++ it('passes _lib.js, server/lib and a comment that only mentions the path', () => {+ const source = [+ "import { readJsonOrNull } from './_lib.js';",+ "import { atomicWrite } from '../../server/lib/fileUtils.js';",+ '// a frozen copy of server/services/universeCanon.js#backfill, not an import',+ ].join('\n');+ expect(importsServerServices(source)).toBe(false);+ });+ });+describe('helper redeclaration matching', () => {it('flags the local readJson spelling and points at readJsonOrNull', () => {const source = [diff --git a/scripts/run-migrations.js b/scripts/run-migrations.jsindex e3e7ae345e..b5cbe9d868 100644--- a/scripts/run-migrations.js+++ b/scripts/run-migrations.js@@ -17,6 +17,32 @@ const DEFAULT_MIGRATIONS_DIR = join(__dirname, 'migrations');const appliedFilePath = (rootDir) => join(rootDir, 'data', 'migrations.applied.json');+// ---- purge-class probe (#12049) ----+//+// The empty-ledger guard below has to know which PENDING migrations are purge+// migrations before any of them runs. Importing each module to read its+// `purge` export loaded every pending migration's import graph up front — on a+// fresh install that is every migration, dozens of which pull in live server+// services before the toolkit has booted. The marker is read from the source+// text instead: a `purge: true` property at the two-space indentation of the+// default export's own properties (how every purge migration writes it), or+// an `export const purge = true`. Comments are stripped first so the rationale+// block every purge migration carries cannot match. A `purge: true` nested+// deeper inside a data literal is data, not the marker.+// `migrationConventions.guard.test.js` checks every shipped migration's module+// against this probe so the two cannot drift.+const stripComments = (source) => source+ .replace(/\/\*[\s\S]*?\*\//g, '')+ .replace(/(^|[^:'"`\\])\/\/.*$/gm, '$1');+const PURGE_PROPERTY = /^ {2}purge\s*:\s*true\s*,?\s*$/m;+const PURGE_EXPORT = /^export\s+const\s+purge\s*=\s*true\b/m;++/** Whether a migration's source text opts into the runner's purge class. */+export const sourceDeclaresPurge = (source) => {+ const code = stripComments(source);+ return PURGE_PROPERTY.test(code) || PURGE_EXPORT.test(code);+};+// ---- runner lock (#12046) ----//// The ledger is read-then-written by every runner: the booting server@@ -237,14 +263,13 @@ async function runPendingMigrations({ rootDir, migrationsDir, appliedFile }) {// the run loop: if this rebuilt-from-empty run aborted on an earlier// throwing migration (the documented repair-and-reboot flow), the partial// ledger would no longer "start empty" on the next boot — and the purge- // would then execute destructively against the rebuilt install.+ // would then execute destructively against the rebuilt install. The probe+ // reads source text, never importing a module before its turn in the loop.if (ledgerStartedEmpty) {let disarmed = 0;for (const file of files) {if (applied.includes(file)) continue;- const mod = await import(pathToFileURL(join(migrationsDir, file)).href);- const migration = (mod?.default && typeof mod.default.up === 'function') ? mod.default : mod;- if (!migration?.purge) continue;+ if (!sourceDeclaresPurge(await readFile(join(migrationsDir, file), 'utf-8'))) continue;console.warn(`⏭️ Skipping purge migration ${file}: applied-list started empty/rebuilt — recorded as applied without running to avoid a destructive rerun (#2770)`);applied.push(file);disarmed++;diff --git a/scripts/run-migrations.test.js b/scripts/run-migrations.test.jsindex 92f55f28fd..8283d200e3 100644--- a/scripts/run-migrations.test.js+++ b/scripts/run-migrations.test.js@@ -4,7 +4,7 @@ import * as fsPromises from 'fs/promises';import { tmpdir } from 'os';import { join } from 'path';-import { runMigrations, listPendingMigrations } from './run-migrations.js';+import { runMigrations, listPendingMigrations, sourceDeclaresPurge } from './run-migrations.js';const fsMocks = vi.hoisted(() => ({ readFailures: new Map(), realRename: null }));@@ -309,6 +309,26 @@ export default {.toEqual(['001-normal.js', '197-purge.js']);});+ it('disarms a pending purge migration without importing its module (#12049)', async () => {+ // A purge migration whose import graph cannot even load: the old probe+ // imported every pending module to read `purge`, which on a fresh install+ // pulled in dozens of live service graphs before boot. The static probe+ // must disarm it from the source text alone.+ writeFileSync(join(migrationsDir, '197-purge.js'), `+import { neverInstalled } from 'portos-missing-dependency-for-test';+export default {+ purge: true,+ async up() { neverInstalled(); }+};+`);++ const ran = await runMigrations({ rootDir, migrationsDir });++ expect(ran).toBe(0);+ expect(JSON.parse(readFileSync(appliedFile, 'utf-8'))).toEqual(['197-purge.js']);+ expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('Skipping purge migration'));+ });+it('disarms purge migrations even when an earlier migration aborts the rebuilt-empty run', async () => {// The re-arm hole: a rebuilt-from-empty run that throws BEFORE reaching the// purge migration persists a partial ledger. The next boot's ledger no@@ -450,6 +470,21 @@ export default {});});+describe('sourceDeclaresPurge', () => {+ it('reads the marker from the default export or a named export', () => {+ expect(sourceDeclaresPurge('export default {\n purge: true,\n async up() {}\n};\n')).toBe(true);+ expect(sourceDeclaresPurge('export default {\n purge: true\n};\n')).toBe(true);+ expect(sourceDeclaresPurge('export const purge = true;\nexport async function up() {}\n')).toBe(true);+ });++ it('ignores prose, data literals and other flags', () => {+ expect(sourceDeclaresPurge('/*\n * opts into the purge class (purge: true)\n */\nexport default { async up() {} };')).toBe(false);+ expect(sourceDeclaresPurge('// purge: true\nexport default { async up() {} };')).toBe(false);+ expect(sourceDeclaresPurge('const SEED = {\n retention: {\n purge: true,\n },\n};\nexport default { async up() {} };')).toBe(false);+ expect(sourceDeclaresPurge('export default {\n autoPurge: true,\n purge: false,\n async up() {}\n};')).toBe(false);+ });+});+describe('runMigrations worktree backstop (#1947)', () => {let baseDir;let migrationsDir;diff --git a/server/AGENTS.md b/server/AGENTS.mdindex 962aa9314a..f147921f95 100644--- a/server/AGENTS.md+++ b/server/AGENTS.md@@ -52,5 +52,6 @@ These apply to Express/service/model code. Universal constraints (no try/catch o## Prompt templates- **Stage-prompt template changes need a migration.** `scripts/setup-data.js` only copies *missing* prompt files to `data/prompts/stages/` — existing installs keep their old templates. When you add a `{{template.variable}}` reference to a `data.reference/prompts/stages/*.md` file, also add an entry to `scripts/migrations/NNN-…js` that updates the corresponding installed prompt when its hash still matches the pre-change shipped version. Normalize line endings (`\r\n` → `\n`, bare `\r` → `\n`) before hashing so the comparison is correct on Windows checkouts. Build it with `makePromptReplaceMigration` and export `ACCEPTED_OLD_MD5` / `NEW_SHIPPED_MD5` — **never hand-mirror a hash table into `scripts/setup-data.js`**, which sweeps those exports itself via `buildPromptDriftTables`. Worked example: `scripts/migrations/394-factual-extraction-lens-prompts.js`. A `makeSeedMigration` only copies MISSING files and will NOT upgrade an installed template. Re-point every earlier migration naming the same file at the new hash (`_shippedHashDrift.test.js` fails otherwise) and keep the superseded hash in the new migration's accepted-old set. Migration scripts live in `scripts/migrations/`; the per-install applied-list is `data/migrations.applied.json` (gitignored runtime state). See `scripts/migrations/003-update-pipeline-stage-prompts.js`.+- **A new migration is a frozen snapshot of its transform.** Applied migrations are never edited; a new one (numbered above the current head) imports `_lib.js` primitives, `server/lib` pure helpers, or carries a frozen copy of the transform — never `server/services`, which an install upgrading months later would run under today's semantics. It declares no local copy of a `_lib.js` helper (`readJsonOrNull`, `writeJsonAtomic`, `writeTextAtomic`, the dashboard-grid helpers, …), and a purge migration writes its `purge: true` marker as a top-level property of the default export so `scripts/run-migrations.js` can read it from source without importing the module. `scripts/migrations/migrationConventions.guard.test.js` enforces all three.- **A `PROMPT_VERSIONS` bump for a scheduled-task prompt (`server/services/taskPromptDefaults.js`) needs NO migration** — its second step is `node scripts/regen-prompt-integrity-snapshot.js`, which retires the outgoing hash (root `AGENTS.md` "Distribution model"). `readSchedule()` in `server/services/taskScheduleStore.js` already upgrades every uncustomized stored prompt to the shipped default on the next `loadSchedule()` and persists the result — the first schedule read after the update converges the install regardless of whether a migration ran. Six migrations re-implemented that same loop against a `task-schedule.json` at the data root — a path the store never reads; `readSchedule()` reads `data/cos/task-schedule.json` — before this was written down: `281-reconcile-missing-releases-prompt.js`, `293-agents-md-prompt-rename.js`, `297-release-check-slashdo-review-prompt.js`, `308-release-check-advisory-review-prompt.js`, `332-release-check-fix-blockers-prompt.js`, `338-claim-issue-volunteer-claim-contract.js`. They stay — the distribution model forbids deleting applied migrations — but they are closed precedent, not a pattern to repeat. `scripts/migrations/promptBumpMigrations.guard.test.js` fails if a seventh copy imports `taskPromptDefaults.js` and writes `task-schedule.json`.- **Creative prompts carry the IP-latitude clause automatically — classify the stage or source rather than wrapping a call site.** PortOS is a professional artist's tool; a model that silently renames, genericizes, or disclaimers a deliberate reference to existing IP produces defective output, and rights/clearance is the operator's responsibility, not the model's. `server/lib/creativeLatitude.js` owns the clause and the single table that classifies what gets it, applied at each of the three LLM transports: `promptService.buildPrompt` (stage templates, keyed on the stage name — fails SAFE, so a stage the tables never saw is stamped), `promptRunner.runPromptThroughProvider` (every hand-rolled prompt, keyed on the run `source` tag it already requires — an allowlist, since ops calls dominate there), and `aiProvider.callProviderAISimple` (its own fetch, so creative callers pass `creative: true`). So **a new creative stage or LLM-calling service is covered only once its stage name / `source` is classified in that module** — add the prefix or exact name there rather than wrapping a call site. Two guard tests force the decision: `creativeLatitude.test.js` walks `data.reference/prompts/stage-config.json`, and `creativeLatitude.sources.test.js` scans every `source:` literal handed to the shared runners — both fail on an unclassified entry. A caller that budgets a context window before the stamp lands (the inline editorial check) must reserve `CREATIVE_LATITUDE_TOKENS`. **The clause is shown, never hidden (#11317):** where the user typed the text the LLM receives (prompt Enhance/Refine, Music Designer, series review notes, mood board Generate more), the service stamps with `withCreativeLatitude` itself (idempotent, so the transport passes it through once) and returns the exact sent text plus `CREATIVE_LATITUDE_ADDITION` and its instructions as `promptAdditions`, which the client lists with `SentPromptPanel`. `services/mediaPromptRefiner.js` is the worked example.