Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: read the migration purge marker statically and keep new migrations off server/services

[._.]/ Adam Eivy5b26433aea00
diff --git a/scripts/migrations/migrationConventions.guard.test.js b/scripts/migrations/migrationConventions.guard.test.js
index 4529c7544e..aa6114ca30 100644
--- a/scripts/migrations/migrationConventions.guard.test.js
+++ b/scripts/migrations/migrationConventions.guard.test.js
@@ -11,12 +11,26 @@
* `collidesWith` 5 each before `_lib.js` exported them. A declaration
* whose name matches a `_lib.js` export — or one of the local spellings
* in `LOCAL_ALIASES` — must import the export instead.
+ *
+ * 2. No import from `server/services` (#12049). A migration is a frozen
+ * snapshot of a transform: an install upgrading across many versions runs
+ * it months later, and a transform that calls into live service code
+ * then does whatever today's service does, not what the migration did
+ * when it shipped. Seventy-odd applied migrations import services and
+ * stay as they are; a new one uses `_lib.js` primitives, `server/lib`
+ * pure helpers, or a frozen copy of the transform.
+ *
+ * 3. The purge marker stays statically visible. `run-migrations.js` reads
+ * `purge: true` from source text (`sourceDeclaresPurge`) so a fresh
+ * install's first boot does not import every pending module up front;
+ * every shipped migration's module must agree with that probe.
*/
import { describe, it, expect } from 'vitest';
import { readdirSync, readFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import * as lib from './_lib.js';
+import { sourceDeclaresPurge } from '../run-migrations.js';
const HERE = resolve(fileURLToPath(import.meta.url), '..');
@@ -51,6 +65,11 @@ const redeclaredLibHelpers = (source) => [...stripComments(source).matchAll(DECL
.filter((name) => LIB_EXPORTS.has(name) || name in LOCAL_ALIASES)
.map((name) => ({ name, use: LOCAL_ALIASES[name] ?? name }));
+// Static `from '…/server/services/…'` and dynamic `import('…/server/services/…')`.
+const SERVICE_IMPORT = /(?:\bfrom\s+|\bimport\s*\(\s*)['"][^'"]*\/server\/services\//;
+
+const importsServerServices = (source) => SERVICE_IMPORT.test(stripComments(source));
+
const migrationNumber = (file) => Number.parseInt(file.slice(0, 3), 10);
const migrationFiles = () => readdirSync(HERE)
@@ -73,6 +92,44 @@ describe('new-migration conventions', () => {
expect(offenders).toEqual([]);
});
+ it('no guarded migration imports from server/services', () => {
+ const offenders = guardedSources()
+ .filter(([, source]) => importsServerServices(source))
+ .map(([file]) => `${file}: imports server/services — use _lib.js, server/lib, or a frozen copy of the transform`);
+ expect(offenders).toEqual([]);
+ });
+
+ it('every shipped migration agrees with the static purge probe', async () => {
+ // Only modules that mention purge at all can disagree; importing those
+ // few keeps the drift check cheap while still covering every purge
+ // migration and every near-miss spelling.
+ const candidates = migrationFiles()
+ .map((file) => [file, readFileSync(resolve(HERE, file), 'utf8')])
+ .filter(([, source]) => /purge/i.test(source));
+ expect(candidates.length).toBeGreaterThan(0);
+ for (const [file, source] of candidates) {
+ const mod = await import(`./${file}`);
+ const migration = mod?.default && typeof mod.default.up === 'function' ? mod.default : mod;
+ expect({ file, purge: sourceDeclaresPurge(source) }).toEqual({ file, purge: migration?.purge === true });
+ }
+ });
+
+ describe('service import matching', () => {
+ it('flags static and dynamic imports from server/services', () => {
+ expect(importsServerServices("import { listUniverses } from '../../server/services/universeBuilder.js';")).toBe(true);
+ expect(importsServerServices("const { x } = await import('../../server/services/universeCanon.js');")).toBe(true);
+ });
+
+ it('passes _lib.js, server/lib and a comment that only mentions the path', () => {
+ const source = [
+ "import { readJsonOrNull } from './_lib.js';",
+ "import { atomicWrite } from '../../server/lib/fileUtils.js';",
+ '// a frozen copy of server/services/universeCanon.js#backfill, not an import',
+ ].join('\n');
+ expect(importsServerServices(source)).toBe(false);
+ });
+ });
+
describe('helper redeclaration matching', () => {
it('flags the local readJson spelling and points at readJsonOrNull', () => {
const source = [
diff --git a/scripts/run-migrations.js b/scripts/run-migrations.js
index e3e7ae345e..b5cbe9d868 100644
--- a/scripts/run-migrations.js
+++ b/scripts/run-migrations.js
@@ -17,6 +17,32 @@ const DEFAULT_MIGRATIONS_DIR = join(__dirname, 'migrations');
const appliedFilePath = (rootDir) => join(rootDir, 'data', 'migrations.applied.json');
+// ---- purge-class probe (#12049) ----
+//
+// The empty-ledger guard below has to know which PENDING migrations are purge
+// migrations before any of them runs. Importing each module to read its
+// `purge` export loaded every pending migration's import graph up front — on a
+// fresh install that is every migration, dozens of which pull in live server
+// services before the toolkit has booted. The marker is read from the source
+// text instead: a `purge: true` property at the two-space indentation of the
+// default export's own properties (how every purge migration writes it), or
+// an `export const purge = true`. Comments are stripped first so the rationale
+// block every purge migration carries cannot match. A `purge: true` nested
+// deeper inside a data literal is data, not the marker.
+// `migrationConventions.guard.test.js` checks every shipped migration's module
+// against this probe so the two cannot drift.
+const stripComments = (source) => source
+ .replace(/\/\*[\s\S]*?\*\//g, '')
+ .replace(/(^|[^:'"`\\])\/\/.*$/gm, '$1');
+const PURGE_PROPERTY = /^ {2}purge\s*:\s*true\s*,?\s*$/m;
+const PURGE_EXPORT = /^export\s+const\s+purge\s*=\s*true\b/m;
+
+/** Whether a migration's source text opts into the runner's purge class. */
+export const sourceDeclaresPurge = (source) => {
+ const code = stripComments(source);
+ return PURGE_PROPERTY.test(code) || PURGE_EXPORT.test(code);
+};
+
// ---- runner lock (#12046) ----
//
// The ledger is read-then-written by every runner: the booting server
@@ -237,14 +263,13 @@ async function runPendingMigrations({ rootDir, migrationsDir, appliedFile }) {
// the run loop: if this rebuilt-from-empty run aborted on an earlier
// throwing migration (the documented repair-and-reboot flow), the partial
// ledger would no longer "start empty" on the next boot — and the purge
- // would then execute destructively against the rebuilt install.
+ // would then execute destructively against the rebuilt install. The probe
+ // reads source text, never importing a module before its turn in the loop.
if (ledgerStartedEmpty) {
let disarmed = 0;
for (const file of files) {
if (applied.includes(file)) continue;
- const mod = await import(pathToFileURL(join(migrationsDir, file)).href);
- const migration = (mod?.default && typeof mod.default.up === 'function') ? mod.default : mod;
- if (!migration?.purge) continue;
+ if (!sourceDeclaresPurge(await readFile(join(migrationsDir, file), 'utf-8'))) continue;
console.warn(`⏭️ Skipping purge migration ${file}: applied-list started empty/rebuilt — recorded as applied without running to avoid a destructive rerun (#2770)`);
applied.push(file);
disarmed++;
diff --git a/scripts/run-migrations.test.js b/scripts/run-migrations.test.js
index 92f55f28fd..8283d200e3 100644
--- a/scripts/run-migrations.test.js
+++ b/scripts/run-migrations.test.js
@@ -4,7 +4,7 @@ import * as fsPromises from 'fs/promises';
import { tmpdir } from 'os';
import { join } from 'path';
-import { runMigrations, listPendingMigrations } from './run-migrations.js';
+import { runMigrations, listPendingMigrations, sourceDeclaresPurge } from './run-migrations.js';
const fsMocks = vi.hoisted(() => ({ readFailures: new Map(), realRename: null }));
@@ -309,6 +309,26 @@ export default {
.toEqual(['001-normal.js', '197-purge.js']);
});
+ it('disarms a pending purge migration without importing its module (#12049)', async () => {
+ // A purge migration whose import graph cannot even load: the old probe
+ // imported every pending module to read `purge`, which on a fresh install
+ // pulled in dozens of live service graphs before boot. The static probe
+ // must disarm it from the source text alone.
+ writeFileSync(join(migrationsDir, '197-purge.js'), `
+import { neverInstalled } from 'portos-missing-dependency-for-test';
+export default {
+ purge: true,
+ async up() { neverInstalled(); }
+};
+`);
+
+ const ran = await runMigrations({ rootDir, migrationsDir });
+
+ expect(ran).toBe(0);
+ expect(JSON.parse(readFileSync(appliedFile, 'utf-8'))).toEqual(['197-purge.js']);
+ expect(warnSpy).toHaveBeenCalledWith(expect.stringContaining('Skipping purge migration'));
+ });
+
it('disarms purge migrations even when an earlier migration aborts the rebuilt-empty run', async () => {
// The re-arm hole: a rebuilt-from-empty run that throws BEFORE reaching the
// purge migration persists a partial ledger. The next boot's ledger no
@@ -450,6 +470,21 @@ export default {
});
});
+describe('sourceDeclaresPurge', () => {
+ it('reads the marker from the default export or a named export', () => {
+ expect(sourceDeclaresPurge('export default {\n purge: true,\n async up() {}\n};\n')).toBe(true);
+ expect(sourceDeclaresPurge('export default {\n purge: true\n};\n')).toBe(true);
+ expect(sourceDeclaresPurge('export const purge = true;\nexport async function up() {}\n')).toBe(true);
+ });
+
+ it('ignores prose, data literals and other flags', () => {
+ expect(sourceDeclaresPurge('/*\n * opts into the purge class (purge: true)\n */\nexport default { async up() {} };')).toBe(false);
+ expect(sourceDeclaresPurge('// purge: true\nexport default { async up() {} };')).toBe(false);
+ expect(sourceDeclaresPurge('const SEED = {\n retention: {\n purge: true,\n },\n};\nexport default { async up() {} };')).toBe(false);
+ expect(sourceDeclaresPurge('export default {\n autoPurge: true,\n purge: false,\n async up() {}\n};')).toBe(false);
+ });
+});
+
describe('runMigrations worktree backstop (#1947)', () => {
let baseDir;
let migrationsDir;
diff --git a/server/AGENTS.md b/server/AGENTS.md
index 962aa9314a..f147921f95 100644
--- a/server/AGENTS.md
+++ b/server/AGENTS.md
@@ -52,5 +52,6 @@ These apply to Express/service/model code. Universal constraints (no try/catch o
## Prompt templates
- **Stage-prompt template changes need a migration.** `scripts/setup-data.js` only copies *missing* prompt files to `data/prompts/stages/` — existing installs keep their old templates. When you add a `{{template.variable}}` reference to a `data.reference/prompts/stages/*.md` file, also add an entry to `scripts/migrations/NNN-…js` that updates the corresponding installed prompt when its hash still matches the pre-change shipped version. Normalize line endings (`\r\n` → `\n`, bare `\r` → `\n`) before hashing so the comparison is correct on Windows checkouts. Build it with `makePromptReplaceMigration` and export `ACCEPTED_OLD_MD5` / `NEW_SHIPPED_MD5` — **never hand-mirror a hash table into `scripts/setup-data.js`**, which sweeps those exports itself via `buildPromptDriftTables`. Worked example: `scripts/migrations/394-factual-extraction-lens-prompts.js`. A `makeSeedMigration` only copies MISSING files and will NOT upgrade an installed template. Re-point every earlier migration naming the same file at the new hash (`_shippedHashDrift.test.js` fails otherwise) and keep the superseded hash in the new migration's accepted-old set. Migration scripts live in `scripts/migrations/`; the per-install applied-list is `data/migrations.applied.json` (gitignored runtime state). See `scripts/migrations/003-update-pipeline-stage-prompts.js`.
+- **A new migration is a frozen snapshot of its transform.** Applied migrations are never edited; a new one (numbered above the current head) imports `_lib.js` primitives, `server/lib` pure helpers, or carries a frozen copy of the transform — never `server/services`, which an install upgrading months later would run under today's semantics. It declares no local copy of a `_lib.js` helper (`readJsonOrNull`, `writeJsonAtomic`, `writeTextAtomic`, the dashboard-grid helpers, …), and a purge migration writes its `purge: true` marker as a top-level property of the default export so `scripts/run-migrations.js` can read it from source without importing the module. `scripts/migrations/migrationConventions.guard.test.js` enforces all three.
- **A `PROMPT_VERSIONS` bump for a scheduled-task prompt (`server/services/taskPromptDefaults.js`) needs NO migration** — its second step is `node scripts/regen-prompt-integrity-snapshot.js`, which retires the outgoing hash (root `AGENTS.md` "Distribution model"). `readSchedule()` in `server/services/taskScheduleStore.js` already upgrades every uncustomized stored prompt to the shipped default on the next `loadSchedule()` and persists the result — the first schedule read after the update converges the install regardless of whether a migration ran. Six migrations re-implemented that same loop against a `task-schedule.json` at the data root — a path the store never reads; `readSchedule()` reads `data/cos/task-schedule.json` — before this was written down: `281-reconcile-missing-releases-prompt.js`, `293-agents-md-prompt-rename.js`, `297-release-check-slashdo-review-prompt.js`, `308-release-check-advisory-review-prompt.js`, `332-release-check-fix-blockers-prompt.js`, `338-claim-issue-volunteer-claim-contract.js`. They stay — the distribution model forbids deleting applied migrations — but they are closed precedent, not a pattern to repeat. `scripts/migrations/promptBumpMigrations.guard.test.js` fails if a seventh copy imports `taskPromptDefaults.js` and writes `task-schedule.json`.
- **Creative prompts carry the IP-latitude clause automatically — classify the stage or source rather than wrapping a call site.** PortOS is a professional artist's tool; a model that silently renames, genericizes, or disclaimers a deliberate reference to existing IP produces defective output, and rights/clearance is the operator's responsibility, not the model's. `server/lib/creativeLatitude.js` owns the clause and the single table that classifies what gets it, applied at each of the three LLM transports: `promptService.buildPrompt` (stage templates, keyed on the stage name — fails SAFE, so a stage the tables never saw is stamped), `promptRunner.runPromptThroughProvider` (every hand-rolled prompt, keyed on the run `source` tag it already requires — an allowlist, since ops calls dominate there), and `aiProvider.callProviderAISimple` (its own fetch, so creative callers pass `creative: true`). So **a new creative stage or LLM-calling service is covered only once its stage name / `source` is classified in that module** — add the prefix or exact name there rather than wrapping a call site. Two guard tests force the decision: `creativeLatitude.test.js` walks `data.reference/prompts/stage-config.json`, and `creativeLatitude.sources.test.js` scans every `source:` literal handed to the shared runners — both fail on an unclassified entry. A caller that budgets a context window before the stamp lands (the inline editorial check) must reserve `CREATIVE_LATITUDE_TOKENS`. **The clause is shown, never hidden (#11317):** where the user typed the text the LLM receives (prompt Enhance/Refine, Music Designer, series review notes, mood board Generate more), the service stamps with `withCreativeLatitude` itself (idempotent, so the transport passes it through once) and returns the exact sent text plus `CREATIVE_LATITUDE_ADDITION` and its instructions as `promptAdditions`, which the client lists with `SentPromptPanel`. `services/mediaPromptRefiner.js` is the worked example.