Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

Bound large collection audit fixture startup and preserve cleanup evidence (#11573)

[._.]/c3394a3fda4d
diff --git a/scripts/perf/README.md b/scripts/perf/README.md
index 5e2d1bca9c..6d419bd806 100644
--- a/scripts/perf/README.md
+++ b/scripts/perf/README.md
@@ -14,12 +14,31 @@ temporary directory. Initialization failures also clean up. As with any process,
SIGKILL or machine power loss cannot run cleanup; use graceful shutdown.
A future CDP runner can import startCollectionFixture() and use its returned
-{ url, cardinalities, close }. Always await close() in a finally block. It is
+{ url, cardinalities, startup, close }. Always await close() in a finally block. It is
idempotent. /__fixture/ready returns the same readiness/cardinality information
without database identifiers, credentials, paths, or record payloads. The optional
clientDist argument selects an already-built client; it defaults to client/dist.
No development proxy is started.
+Startup preserves the 30,000 ms overall worker readiness budget. A separate
+10,000 ms no-progress deadline detects a stalled worker sooner; only an advancing
+allowlisted phase or completed-record count resets it. Repeated heartbeats do not.
+The optional `startupLimits: { overallTimeoutMs, noProgressTimeoutMs }` argument
+accepts integer milliseconds from 100 through 120,000, with the no-progress limit
+at most the overall limit. These are explicit experiment limits, not automatic
+retries or pass thresholds. The worker budgets start after the parent source copy
+and schema setup; startup diagnostics include those parent phases as well.
+
+`startup` contains fixed phase names, elapsed/phase milliseconds, completed
+image/video/account/message counts, and cumulative milliseconds awaiting file
+writes and canonical media-index inserts. Eight records seed concurrently, with
+two sequential file writes and one canonical upsert per record. A failed batch
+drains its outstanding writes before cleanup. The time aggregates sum work across
+records and can exceed elapsed wall time with concurrent seeding. Each phase
+retains only its latest progress sample, bounding report size. Rejected startup
+errors carry the same sanitized `fixtureStartup` snapshot and a fixed
+`fixtureFailure` reason; exception text and worker stderr never enter reports.
+
The fixed dataset contains 2,400 PNG images, 1,200 one-second MP4 videos and their
PNG thumbnails, and 4,000 messages across two disabled synthetic accounts. Every
tenth media record is hidden (240 images and 120 videos). Prompts and message
@@ -67,10 +86,13 @@ Tests, from server/:
~~~sh
npm test -- ../scripts/perf/collectionFixture.test.js
+npm test -- ../scripts/perf/collectionFixtureStartup.test.js ../scripts/perf/collectionFixtureLifecycle.test.js ../scripts/perf/collectionFixtureSeed.test.js
npm run test:db -- ../scripts/perf/collectionFixture.db.test.js
~~~
-The first covers pre-import isolation rejection. The second covers actual HTTP
+The non-DB tests cover pre-import isolation rejection, bounded concurrency,
+failed-batch drainage, independent total/inactivity deadlines, sanitized IPC and
+stalled-child cleanup with injected resources/time. The DB suite covers actual HTTP
pagination, projections, unavailable actions, and success/failure cleanup against
portos_test.
@@ -93,6 +115,8 @@ gate is evidence to fix, never a reason to raise the limit until the run passes.
The report records:
+- Fixture startup phases, aggregate seed work and counts on success or failure;
+ startup failures retain specific fixed reasons instead of a generic audit error.
- Time from navigation start until the first visible synthetic collection row,
then a two-second observation tail for eager fetches; 60 media cards / 50 inbox
rows maximum, checked against response counts before client rendering.
@@ -138,8 +162,8 @@ single-record hydration on open; the fixture serves the same bounded image
lookup and video-history record readers the migrated UI calls. The media
scenario now passes with a 32,212-byte first page, visible/hidden totals of
3,240/360, no cold detail reads, and exactly one detail read when a row opens.
-Image variant lineage (`/api/image-gen/:filename/variants`) remains outside the
-fixture and is counted as unavailable. Inbox summary/detail and Contacts idle
+The fixture also serves image variant lineage (`/api/image-gen/:filename/variants`)
+for the lightbox's current lazy-detail read. Inbox summary/detail and Contacts idle
gates passed, including a full idle minute with zero HTTP transfer and four
bytes of socket heartbeat payload. This runner does not claim all #8232
contracts are met.
diff --git a/scripts/perf/browserCollectionAudit.mjs b/scripts/perf/browserCollectionAudit.mjs
index 7b4b4669f7..3713c26d83 100644
--- a/scripts/perf/browserCollectionAudit.mjs
+++ b/scripts/perf/browserCollectionAudit.mjs
@@ -144,13 +144,16 @@ async function scenario(route) {
try {
fixture = await startCollectionFixture();
+ report.startup = fixture.startup;
report.cardinalities = fixture.cardinalities;
browser = await chromium.launch({ headless: true, ...(process.env.COLLECTION_AUDIT_BROWSER
? { executablePath: process.env.COLLECTION_AUDIT_BROWSER } : { channel: 'chrome' }) });
await scenario('/media/history');
await scenario('/messages/inbox');
-} catch {
- report.failures.push({ endpoint: 'fixture', reason: 'audit-startup-or-shutdown', bytes: 0 });
+} catch (error) {
+ if (error.fixtureStartup) report.startup = error.fixtureStartup;
+ report.failures.push({ endpoint: fixture ? 'browser' : 'fixture',
+ reason: error.fixtureFailure || 'audit-startup-or-shutdown', bytes: 0 });
} finally {
await browser?.close().catch(() => report.failures.push({ endpoint: 'browser', reason: 'cleanup', bytes: 0 }));
await fixture?.close().catch(() => report.failures.push({ endpoint: 'fixture', reason: 'cleanup', bytes: 0 }));
diff --git a/scripts/perf/collectionFixture.js b/scripts/perf/collectionFixture.js
index c7ac682d43..c10d158353 100644
--- a/scripts/perf/collectionFixture.js
+++ b/scripts/perf/collectionFixture.js
@@ -12,6 +12,7 @@ import { promisify } from 'node:util';
import { fileURLToPath } from 'node:url';
import { isDirectlyInvoked } from '../lib/directInvocation.js';
+import { createFixtureStartup } from './collectionFixtureStartup.js';
const codeRoot = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
@@ -25,11 +26,13 @@ export function validateFixtureEnvironment(env = process.env, host = '127.0.0.1'
if (env.PGSERVICE || env.PGSERVICEFILE || env.DATABASE_URL) throw new Error('Remove alternate database configuration');
}
-/** Each invocation returns { url, cardinalities, close }; close is idempotent. */
+/** Returns { url, cardinalities, startup, close }; close is idempotent. */
export async function startCollectionFixture({
env = process.env, host = '127.0.0.1', clientDist = join(codeRoot, 'client/dist'),
+ startupLimits,
} = {}) {
validateFixtureEnvironment(env, host);
+ const startup = createFixtureStartup(startupLimits);
await access(join(clientDist, 'index.html'));
const require = createRequire(join(codeRoot, 'server/package.json'));
const { Client } = require('pg');
@@ -71,6 +74,7 @@ export async function startCollectionFixture({
process.once('SIGINT', onSignal);
process.once('SIGTERM', onSignal);
try {
+ startup.record({ phase: 'source-copy' });
root = await realpath(await mkdtemp(join(tmpdir(), 'portos-collection-audit-')));
// Copy code, never data, configuration, credentials, or the live checkout's
// symlinks. A physical temp copy also preserves dataRoot's worktree guard.
@@ -82,12 +86,13 @@ export async function startCollectionFixture({
await cp(join(codeRoot, file), join(root, file), { dereference: false });
}
await mkdir(join(root, 'scripts/perf'), { recursive: true });
- for (const file of ['collectionFixtureWorker.js', 'collectionFixtureData.js']) {
+ for (const file of ['collectionFixtureWorker.js', 'collectionFixtureData.js', 'collectionFixtureSeed.js']) {
await cp(join(codeRoot, 'scripts/perf', file), join(root, 'scripts/perf', file));
}
await cp(join(codeRoot, 'scripts/perf/assets'), join(root, 'scripts/perf/assets'), { recursive: true });
await writeFile(join(root, 'package.json'), '{"type":"module"}\n');
await symlink(join(codeRoot, 'server/node_modules'), join(root, 'server/node_modules'), 'junction');
+ startup.record({ phase: 'parent-database' });
await db.connect();
const identity = await db.query('SELECT current_database() AS name');
if (identity.rows[0].name !== 'portos_test') throw new Error('Connected database is not portos_test');
@@ -109,31 +114,16 @@ export async function startCollectionFixture({
});
// Drain diagnostics without leaking paths, payloads, or credentials into reports.
child.stderr.resume();
- const ready = await new Promise((resolveReady, rejectReady) => {
- const timer = setTimeout(() => rejectReady(new Error('Collection fixture readiness timed out')), 30000);
- const finish = (error, value) => {
- clearTimeout(timer);
- child.removeListener('error', onError);
- child.removeListener('exit', onExit);
- child.removeListener('message', onMessage);
- error ? rejectReady(error) : resolveReady(value);
- };
- const onError = () => finish(new Error('Collection fixture child failed to start'));
- const onExit = () => finish(new Error('Collection fixture child exited before readiness'));
- const onMessage = message => {
- if (message?.type === 'ready') finish(null, message);
- if (message?.type === 'failed') finish(new Error('Collection fixture initialization failed at ' + message.phase));
- };
- child.once('error', onError);
- child.once('exit', onExit);
- child.on('message', onMessage);
- });
+ const ready = await startup.wait(child);
if (stopped) throw new Error('Fixture startup interrupted');
finishSetup();
child.once('exit', () => { close().catch(() => { process.exitCode = 1; }); });
- return { url: ready.url, cardinalities: ready.cardinalities, close };
+ return { url: ready.url, cardinalities: ready.cardinalities, startup: startup.snapshot(), close };
} catch (error) {
finishSetup();
+ // Capture only sanitized diagnostics; raw I/O/DB errors may contain secrets.
+ error.fixtureStartup ||= startup.snapshot();
+ error.fixtureFailure ||= 'setup-failed';
await close();
throw error;
}
diff --git a/scripts/perf/collectionFixtureLifecycle.test.js b/scripts/perf/collectionFixtureLifecycle.test.js
new file mode 100644
index 0000000000..965fd86a53
--- /dev/null
+++ b/scripts/perf/collectionFixtureLifecycle.test.js
@@ -0,0 +1,56 @@
+import { EventEmitter } from 'node:events';
+import { afterEach, expect, it, vi } from 'vitest';
+
+const owned = vi.hoisted(() => ({ queries: [], removed: [], child: null, ended: false }));
+vi.mock('node:fs/promises', () => ({
+ access: vi.fn(), cp: vi.fn(), mkdir: vi.fn(), symlink: vi.fn(), writeFile: vi.fn(),
+ lstat: async () => ({ isFile: () => true }),
+ mkdtemp: async () => '/tmp/portos-collection-audit-synthetic-lifecycle', realpath: async path => path,
+ rm: async path => { owned.removed.push(path); },
+}));
+vi.mock('node:module', () => ({ createRequire: () => () => ({ Client: function () {
+ return { on: vi.fn(), connect: vi.fn(), end: async () => { owned.ended = true; },
+ query: async sql => {
+ owned.queries.push(sql);
+ return { rows: [{ name: 'portos_test' }] };
+ } };
+} }) }));
+vi.mock('node:child_process', () => {
+ const execFile = vi.fn();
+ execFile[Symbol.for('nodejs.util.promisify.custom')] = async () => ({ stdout: 'server/package.json\0' });
+ return { execFile, fork: () => {
+ const child = new EventEmitter();
+ child.exitCode = null;
+ child.signalCode = null;
+ child.stderr = { resume: vi.fn() };
+ child.kill = vi.fn(signal => {
+ child.signalCode = signal;
+ queueMicrotask(() => child.emit('exit', null, signal));
+ });
+ owned.child = child;
+ return child;
+ } };
+});
+import { startCollectionFixture } from './collectionFixture.js';
+
+afterEach(() => vi.useRealTimers());
+
+it('a stalled fixture rejects and removes only its owned schema, root, child and listeners', async () => {
+ vi.useFakeTimers();
+ const beforeSignals = ['SIGINT', 'SIGTERM'].map(signal => process.listenerCount(signal));
+ const ready = startCollectionFixture({ env: {}, clientDist: '/synthetic-client' });
+ const failure = expect(ready).rejects.toMatchObject({ fixtureFailure: 'startup-stalled',
+ fixtureStartup: { noProgressTimeoutMs: 10000, overallTimeoutMs: 30000 } });
+ await vi.advanceTimersByTimeAsync(10000);
+ await failure;
+ const create = owned.queries.find(sql => sql.startsWith('CREATE SCHEMA'));
+ expect(create).toMatch(/^CREATE SCHEMA "collection_audit_[a-f0-9]{32}"$/);
+ expect(owned.queries).toContain(create.replace('CREATE SCHEMA', 'DROP SCHEMA') + ' CASCADE');
+ expect(owned.queries).toHaveLength(3);
+ expect(owned.removed).toEqual(['/tmp/portos-collection-audit-synthetic-lifecycle']);
+ expect(owned.ended).toBe(true);
+ expect(owned.child.kill).toHaveBeenCalledExactlyOnceWith('SIGTERM');
+ expect(owned.child.eventNames()).toEqual([]);
+ expect(['SIGINT', 'SIGTERM'].map(signal => process.listenerCount(signal))).toEqual(beforeSignals);
+ expect(vi.getTimerCount()).toBe(0);
+});
diff --git a/scripts/perf/collectionFixtureSeed.js b/scripts/perf/collectionFixtureSeed.js
new file mode 100644
index 0000000000..a8ff28afa0
--- /dev/null
+++ b/scripts/perf/collectionFixtureSeed.js
@@ -0,0 +1,16 @@
+// Bound both filesystem pressure and PostgreSQL concurrency. Every record still
+// writes its own files and passes through the canonical single-row index writer.
+export const SEED_BATCH_SIZE = 8;
+
+export async function seedFixtureRecords(total, writeRecord, completeBatch) {
+ for (let offset = 0; offset < total; offset += SEED_BATCH_SIZE) {
+ const end = Math.min(offset + SEED_BATCH_SIZE, total);
+ const results = await Promise.allSettled(Array.from({ length: end - offset },
+ (_, index) => Promise.resolve().then(() => writeRecord(offset + index))));
+ // Drain the whole bounded batch before propagating failure: cleanup must
+ // never race sibling writes that still own resources below the fixture root.
+ const failed = results.find(result => result.status === 'rejected');
+ if (failed) throw failed.reason;
+ completeBatch(end);
+ }
+}
diff --git a/scripts/perf/collectionFixtureSeed.test.js b/scripts/perf/collectionFixtureSeed.test.js
new file mode 100644
index 0000000000..4b40d63486
--- /dev/null
+++ b/scripts/perf/collectionFixtureSeed.test.js
@@ -0,0 +1,42 @@
+import { expect, it } from 'vitest';
+import { seedFixtureRecords, SEED_BATCH_SIZE } from './collectionFixtureSeed.js';
+
+it('bounds concurrent records and reports only fully completed batches', async () => {
+ const pending = [];
+ const started = [];
+ const completed = [];
+ const seed = seedFixtureRecords(SEED_BATCH_SIZE + 1, index => {
+ started.push(index);
+ return new Promise(resolve => pending.push(resolve));
+ }, count => completed.push(count));
+ await Promise.resolve();
+ expect(started).toHaveLength(SEED_BATCH_SIZE);
+ expect(completed).toEqual([]);
+ for (const resolve of pending.splice(0)) resolve();
+ await new Promise(resolve => setImmediate(resolve));
+ expect(started).toEqual(Array.from({ length: SEED_BATCH_SIZE + 1 }, (_, index) => index));
+ expect(completed).toEqual([SEED_BATCH_SIZE]);
+ pending[0]();
+ await seed;
+ expect(completed).toEqual([SEED_BATCH_SIZE, SEED_BATCH_SIZE + 1]);
+});
+
+it('drains a failed batch before rejecting, and starts no later records', async () => {
+ const pending = [];
+ const started = [];
+ const completed = [];
+ const failure = new Error('Synthetic seed write failure');
+ let rejected = false;
+ const seed = seedFixtureRecords(SEED_BATCH_SIZE + 1, index => {
+ started.push(index);
+ if (index === 0) throw failure;
+ return new Promise(resolve => pending.push(resolve));
+ }, count => completed.push(count)).catch(error => { rejected = true; throw error; });
+ const assertion = expect(seed).rejects.toBe(failure);
+ await new Promise(resolve => setImmediate(resolve));
+ expect(rejected).toBe(false);
+ for (const resolve of pending) resolve();
+ await assertion;
+ expect(started).toHaveLength(SEED_BATCH_SIZE);
+ expect(completed).toEqual([]);
+});
diff --git a/scripts/perf/collectionFixtureStartup.js b/scripts/perf/collectionFixtureStartup.js
new file mode 100644
index 0000000000..acaaa421b6
--- /dev/null
+++ b/scripts/perf/collectionFixtureStartup.js
@@ -0,0 +1,96 @@
+import { performance } from 'node:perf_hooks';
+import { CARDINALITIES } from './collectionFixtureData.js';
+import { SEED_BATCH_SIZE } from './collectionFixtureSeed.js';
+
+const phases = ['source-copy', 'parent-database', 'imports', 'database', 'image-seed', 'video-seed', 'inbox-seed', 'routes', 'ready'];
+const limits = { images: CARDINALITIES.images, videos: CARDINALITIES.videos,
+ accounts: CARDINALITIES.accounts, messages: CARDINALITIES.messages };
+
+/** Only fixed stages and bounded aggregates cross the worker/report boundary. */
+export function createFixtureStartup({ overallTimeoutMs = 30000, noProgressTimeoutMs = 10000 } = {}) {
+ if (!Number.isInteger(overallTimeoutMs) || overallTimeoutMs < 100 || overallTimeoutMs > 120000
+ || !Number.isInteger(noProgressTimeoutMs) || noProgressTimeoutMs < 100
+ || noProgressTimeoutMs > overallTimeoutMs) throw new Error('Invalid collection fixture startup limits');
+ const started = performance.now();
+ let workerStarted;
+ const samples = [];
+ let phaseIndex = -1;
+ let counts = { images: 0, videos: 0, accounts: 0, messages: 0 };
+ let workMs = { files: 0, index: 0 };
+ const record = message => {
+ const nextPhase = phases.indexOf(message?.phase);
+ if (nextPhase < phaseIndex || nextPhase < 0) return false;
+ const nextCounts = {};
+ for (const [key, max] of Object.entries(limits)) {
+ const value = message.counts?.[key] ?? counts[key];
+ if (!Number.isInteger(value) || value < counts[key] || value > max) return false;
+ nextCounts[key] = value;
+ }
+ // Repeated heartbeats cannot keep a stuck worker alive.
+ if (nextPhase === phaseIndex && Object.keys(limits).every(key => nextCounts[key] === counts[key])) return false;
+ const nextWork = {};
+ for (const key of ['files', 'index']) {
+ const value = message.workMs?.[key] ?? workMs[key];
+ if (!Number.isFinite(value) || value < workMs[key] || value > 120000 * SEED_BATCH_SIZE) return false;
+ nextWork[key] = Math.round(value);
+ }
+ phaseIndex = nextPhase;
+ counts = nextCounts;
+ workMs = nextWork;
+ const elapsedMs = Math.round(performance.now() - started);
+ const samePhase = samples.at(-1)?.phase === phases[phaseIndex];
+ if (samples.length && !samePhase) {
+ samples.at(-1).durationMs = elapsedMs - samples.at(-1).startedMs;
+ }
+ const startedMs = samePhase ? samples.at(-1).startedMs : elapsedMs;
+ const sample = { phase: phases[phaseIndex], startedMs, elapsedMs,
+ durationMs: elapsedMs - startedMs, counts, workMs };
+ // One latest aggregate per phase bounds report size even for noisy IPC.
+ if (samples.at(-1)?.phase === sample.phase) samples[samples.length - 1] = sample;
+ else samples.push(sample);
+ return true;
+ };
+ const snapshot = () => {
+ const elapsedMs = Math.round(performance.now() - started);
+ return { overallTimeoutMs, noProgressTimeoutMs, elapsedMs,
+ workerElapsedMs: workerStarted === undefined ? null : Math.round(performance.now() - workerStarted),
+ phases: samples.map((sample, index) => ({ ...sample,
+ durationMs: index === samples.length - 1 ? elapsedMs - sample.startedMs : sample.durationMs,
+ counts: { ...sample.counts }, workMs: { ...sample.workMs } })) };
+ };
+ const failure = reason => Object.assign(new Error('Collection fixture ' + reason), {
+ fixtureFailure: reason, fixtureStartup: snapshot(),
+ });
+ const wait = child => new Promise((resolveReady, rejectReady) => {
+ workerStarted = performance.now();
+ let idleTimer;
+ const finish = (error, value) => {
+ clearTimeout(overallTimer);
+ clearTimeout(idleTimer);
+ child.removeListener('error', onError);
+ child.removeListener('exit', onExit);
+ child.removeListener('message', onMessage);
+ error ? rejectReady(error) : resolveReady(value);
+ };
+ const armIdle = () => {
+ clearTimeout(idleTimer);
+ idleTimer = setTimeout(() => finish(failure('startup-stalled')), noProgressTimeoutMs);
+ };
+ const onError = () => finish(failure('child-start-failed'));
+ const onExit = () => finish(failure('child-exited'));
+ const onMessage = message => {
+ if (message?.type === 'progress' && record(message)) armIdle();
+ if (message?.type === 'failed') finish(failure('initialization-failed'));
+ if (message?.type === 'ready') {
+ record({ phase: 'ready' });
+ finish(null, message);
+ }
+ };
+ const overallTimer = setTimeout(() => finish(failure('startup-timeout')), overallTimeoutMs);
+ child.once('error', onError);
+ child.once('exit', onExit);
+ child.on('message', onMessage);
+ armIdle();
+ });
+ return { record, snapshot, wait };
+}
diff --git a/scripts/perf/collectionFixtureStartup.test.js b/scripts/perf/collectionFixtureStartup.test.js
new file mode 100644
index 0000000000..0099b48675
--- /dev/null
+++ b/scripts/perf/collectionFixtureStartup.test.js
@@ -0,0 +1,68 @@
+import { EventEmitter } from 'node:events';
+import { afterEach, describe, expect, it, vi } from 'vitest';
+import { createFixtureStartup } from './collectionFixtureStartup.js';
+
+afterEach(() => vi.useRealTimers());
+
+describe('fixture startup IPC and deadlines', () => {
+ it('does not let repeated or invalid IPC keep a stalled child alive or leak payloads', async () => {
+ vi.useFakeTimers();
+ const child = new EventEmitter();
+ const startup = createFixtureStartup();
+ const ready = startup.wait(child);
+ const failure = expect(ready).rejects.toMatchObject({ fixtureFailure: 'startup-stalled' });
+ child.emit('message', { type: 'progress', phase: 'image-seed', counts: { images: 100 }, private: 'secret' });
+ await vi.advanceTimersByTimeAsync(9000);
+ child.emit('message', { type: 'progress', phase: 'image-seed', counts: { images: 100 }, workMs: { files: 9999 } });
+ child.emit('message', { type: 'progress', phase: 'secret', counts: { images: 200 } });
+ child.emit('message', { type: 'progress', phase: 'image-seed', counts: { images: 2401 } });
+ await vi.advanceTimersByTimeAsync(1000);
+ await failure;
+ expect(startup.snapshot().phases).toEqual([{ phase: 'image-seed', startedMs: expect.any(Number), elapsedMs: expect.any(Number), durationMs: expect.any(Number),
+ counts: { images: 100, videos: 0, accounts: 0, messages: 0 }, workMs: { files: 0, index: 0 } }]);
+ expect(JSON.stringify(startup.snapshot())).not.toContain('secret');
+ expect(child.eventNames()).toEqual([]);
+ expect(vi.getTimerCount()).toBe(0);
+ });
+
+ it('enforces the total budget even while counts advance', async () => {
+ vi.useFakeTimers();
+ const child = new EventEmitter();
+ const startup = createFixtureStartup();
+ const ready = startup.wait(child);
+ const failure = expect(ready).rejects.toMatchObject({ fixtureFailure: 'startup-timeout' });
+ for (let images = 100; images <= 400; images += 100) {
+ child.emit('message', { type: 'progress', phase: 'image-seed', counts: { images } });
+ await vi.advanceTimersByTimeAsync(images === 400 ? 3000 : 9000);
+ }
+ await failure;
+ expect(child.eventNames()).toEqual([]);
+ expect(vi.getTimerCount()).toBe(0);
+ });
+
+ it('extends only the inactivity deadline for real progress, then disposes listeners on ready or exit', async () => {
+ vi.useFakeTimers();
+ const child = new EventEmitter();
+ const startup = createFixtureStartup();
+ const ready = startup.wait(child);
+ child.emit('message', { type: 'progress', phase: 'image-seed', counts: { images: 100 } });
+ await vi.advanceTimersByTimeAsync(9000);
+ child.emit('message', { type: 'progress', phase: 'image-seed', counts: { images: 200 } });
+ await vi.advanceTimersByTimeAsync(9000);
+ child.emit('message', { type: 'ready' });
+ await expect(ready).resolves.toMatchObject({ type: 'ready' });
+ expect(child.eventNames()).toEqual([]);
+ expect(vi.getTimerCount()).toBe(0);
+ const exited = startup.wait(child);
+ child.emit('exit', 1);
+ await expect(exited).rejects.toMatchObject({ fixtureFailure: 'child-exited' });
+ expect(child.eventNames()).toEqual([]);
+ });
+
+ it('rejects invalid limits before resource creation', () => {
+ for (const startupLimits of [{ overallTimeoutMs: 120001 }, { overallTimeoutMs: NaN },
+ { noProgressTimeoutMs: 0 }, { overallTimeoutMs: 1000, noProgressTimeoutMs: 1001 }]) {
+ expect(() => createFixtureStartup(startupLimits)).toThrow('Invalid collection fixture startup limits');
+ }
+ });
+});
diff --git a/scripts/perf/collectionFixtureWorker.js b/scripts/perf/collectionFixtureWorker.js
index 9dc4b3f15a..53ebdd08fa 100644
--- a/scripts/perf/collectionFixtureWorker.js
+++ b/scripts/perf/collectionFixtureWorker.js
@@ -1,7 +1,9 @@
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { createRequire } from 'node:module';
import { join, resolve } from 'node:path';
+import { performance } from 'node:perf_hooks';
import { CARDINALITIES, ACCOUNT_IDS, mediaFixture, messageFixture } from './collectionFixtureData.js';
+import { seedFixtureRecords } from './collectionFixtureSeed.js';
// This is the only worker entrypoint. Validate BEFORE dynamic server imports.
const root = process.cwd();
@@ -18,6 +20,18 @@ let io;
let db;
let closing;
let phase = 'imports';
+const counts = { images: 0, videos: 0, accounts: 0, messages: 0 };
+const workMs = { files: 0, index: 0 };
+const progress = nextPhase => {
+ phase = nextPhase;
+ if (process.connected) process.send({ type: 'progress', phase, counts,
+ workMs: { files: Math.round(workMs.files), index: Math.round(workMs.index) } });
+};
+const writeSeedFile = async (path, contents) => {
+ const started = performance.now();
+ await writeFile(path, contents);
+ workMs.files += performance.now() - started;
+};
const close = () => closing ||= (async () => {
if (server) {
if (io) await new Promise(resolveIo => io.close(resolveIo));
@@ -33,13 +47,14 @@ process.once('SIGTERM', stop);
process.once('disconnect', stop);
async function start() {
+ progress('imports');
const require = createRequire(join(root, 'server/package.json'));
const express = require('express');
const sharp = require('sharp');
db = await import('../../server/lib/db.js');
const { PATHS } = await import('../../server/lib/paths.js');
if (resolve(PATHS.data) !== join(root, 'data')) throw new Error('Fixture data root mismatch');
- phase = 'database';
+ progress('database');
const identity = await db.query('SELECT current_database() AS db, current_schema() AS schema');
if (identity.rows[0].db !== 'portos_test' || identity.rows[0].schema !== schema) throw new Error('Fixture DB isolation mismatch');
const { mediaDdl } = await import('../../server/lib/db/schema/media.js');
@@ -51,36 +66,50 @@ async function start() {
const { imageToRow, videoToRow } = await import('../../server/services/mediaAssetIndex/logic.js');
await Promise.all(['images', 'videos', 'video-thumbnails', 'messages/cache'].map(dir => mkdir(join(PATHS.data, dir), { recursive: true })));
const png = await sharp({ create: { width: 640, height: 360, channels: 3, background: '#335577' } }).png().toBuffer();
- phase = 'media-seed';
- for (let index = 0; index < CARDINALITIES.images; index++) {
+ progress('image-seed');
+ await seedFixtureRecords(CARDINALITIES.images, async index => {
const image = mediaFixture('image', index);
- await writeFile(join(PATHS.images, image.filename), png);
- await writeFile(join(PATHS.images, image.filename.replace('.png', '.metadata.json')), JSON.stringify(image));
+ await writeSeedFile(join(PATHS.images, image.filename), png);
+ await writeSeedFile(join(PATHS.images, image.filename.replace('.png', '.metadata.json')), JSON.stringify(image));
+ const started = performance.now();
await upsertAsset(imageToRow(image));
- }
+ workMs.index += performance.now() - started;
+ }, completed => {
+ counts.images = completed;
+ progress('image-seed');
+ });
const clip = await readFile(join(root, 'scripts/perf/assets/synthetic.mp4'));
const videos = [];
- for (let index = 0; index < CARDINALITIES.videos; index++) {
+ progress('video-seed');
+ await seedFixtureRecords(CARDINALITIES.videos, async index => {
const video = mediaFixture('video', index);
- videos.push(video);
- await writeFile(join(PATHS.data, 'videos', video.filename), clip);
- await writeFile(join(PATHS.data, 'video-thumbnails', video.thumbnail), png);
+ videos[index] = video;
+ await writeSeedFile(join(PATHS.data, 'videos', video.filename), clip);
+ await writeSeedFile(join(PATHS.data, 'video-thumbnails', video.thumbnail), png);
+ const started = performance.now();
await upsertAsset(videoToRow(video));
- }
- await writeFile(join(PATHS.data, 'video-history.json'), JSON.stringify(videos));
- phase = 'inbox-seed';
+ workMs.index += performance.now() - started;
+ }, completed => {
+ counts.videos = completed;
+ progress('video-seed');
+ });
+ await writeSeedFile(join(PATHS.data, 'video-history.json'), JSON.stringify(videos));
+ progress('inbox-seed');
const accounts = {};
for (const [accountIndex, id] of ACCOUNT_IDS.entries()) {
accounts[id] = { id, name: 'Synthetic inbox ' + (accountIndex + 1), type: 'gmail', provider: 'api',
email: 'fixture' + accountIndex + '@example.com', enabled: false, syncConfig: {}, lastSyncAt: null };
const messages = Array.from({ length: CARDINALITIES.messages / CARDINALITIES.accounts },
(_, index) => messageFixture(id, index + accountIndex * CARDINALITIES.messages / CARDINALITIES.accounts));
- await writeFile(join(PATHS.messages, 'cache', id + '.json'), JSON.stringify({ syncCursor: null, messages }));
+ await writeSeedFile(join(PATHS.messages, 'cache', id + '.json'), JSON.stringify({ syncCursor: null, messages }));
+ counts.accounts++;
+ counts.messages += messages.length;
+ progress('inbox-seed');
}
await writeFile(join(PATHS.messages, 'accounts.json'), JSON.stringify(accounts));
await writeFile(join(PATHS.data, 'settings.json'), JSON.stringify({ timezone: 'UTC', instances: [] }));
- phase = 'routes';
+ progress('routes');
const { createImageGalleryHandlers } = await import('../../server/routes/imageGalleryRead.js');
const { createVideoHistoryItemRead } = await import('../../server/routes/videoHistoryRead.js');
const { messageInboxRead, messageDetailRead } = await import('../../server/routes/messageInboxRead.js');