Multiplayer agent harness for work.
Copy legacy tape steps automatically before the tape drop (#2349)
Regan Bell9b4540944680
diff --git a/docs/SPEC.md b/docs/SPEC.mdindex 33363ab9d..df23042cb 100644--- a/docs/SPEC.md+++ b/docs/SPEC.md@@ -175,7 +175,7 @@ Growing the codebase to keep every path it ever took alive. The canonical case:- **Sidebar surface** (2026-10-14): The web sidebar's `surfaceOf` (`plugins/web-ui/src/sessions.ts`) is a conditional that string-matched `threadRef` prefixes (`web:`, `dm:`, `ch:`) because older sessions lacked a typed `surface`, and trusted the stored field only for subagents. When automation sessions landed in the wrong group, the proposed fix added more prefixes. Every session now stores its surface; read that field alone. _Status:_ still present on main.- **Credential capture redesign** (2026-10-08): The login-as-a-skill handoff planned that "a stale skill calling /v1/keychain/use gets a clear refusal" and that "logins saved under the old capture system still load." Both keep the old system on life support. Remove the endpoint and the old capture system outright; stale skills fail and get fixed. _Status:_ [qm#2146](https://github.com/yc-software/qm/pull/2146) deletes `/v1/keychain/use` (open).- **Sandbox routing retirement** (2026-10-02): Retiring `sandbox_routing` came with a locked, marker-guarded one-time upgrade import that preserved legacy IDs, explicit nulls, and machine IDs, kept the old table as rollback input, and told operators to keep credentials for legacy-mapped providers. A plain migration would have done. _Status:_ merged in [qm#2019](https://github.com/yc-software/qm/pull/2019).-- **Session tape beside session entries** (2026-09-16): The tape was meant to replace `session_entries`, but caution kept both. `SESSION_TAPE_MODE` shadow/serve chose between them, exact-transcript annotations were added so the tape could reproduce every historical field, and a verifying, lease-aware operator backfill with dry runs and partial-scan statuses guarded the cutover. We got stuck here for weeks; one migration and a hard cutover would have left one system instead of two. _Status:_ code reads and writes only `session_entries` since [qm#2273](https://github.com/yc-software/qm/pull/2273); `session_tape` is dropped by a gated migration once its step metadata has been copied out ([qm#2280](https://github.com/yc-software/qm/pull/2280)).+- **Session tape beside session entries** (2026-09-16): The tape was meant to replace `session_entries`, but caution kept both. `SESSION_TAPE_MODE` shadow/serve chose between them, exact-transcript annotations were added so the tape could reproduce every historical field, and a verifying, lease-aware operator backfill with dry runs and partial-scan statuses guarded the cutover. We got stuck here for weeks; one migration and a hard cutover would have left one system instead of two. _Status:_ code reads and writes only `session_entries` since [qm#2273](https://github.com/yc-software/qm/pull/2273); startup migrations copy `session_tape`'s step metadata out ([qm#2280](https://github.com/yc-software/qm/pull/2280)) and then drop it behind a gate.- **Legacy Inbox loop:** `src/loops/inbox-migration.ts` migrates each owner's legacy Inbox loop at runtime, with its own phase record in UI-state preferences, a pause-and-disable sequence, and a repair path for half-finished runs. _Status:_ still on main; deletion approved for the loops rebuild.### Over-testingdiff --git a/docs/upgrading-runtime.md b/docs/upgrading-runtime.mdindex e381dad14..4f1052f5c 100644--- a/docs/upgrading-runtime.md+++ b/docs/upgrading-runtime.md@@ -43,6 +43,17 @@ retention also removes expired entries from the legacy table, preventing a laterrestart from importing them again. The legacy table remains available for olderbinaries.+Upgrading from a release that still writes `session_tape` copies its step metadata+into `legacy_tape_steps` during startup migrations, then drops the tape. Stop every+older core and worker before starting the new release: a tape write after the copy+starts makes the drop refuse, and that startup fails with the reset to run. The copy+runs on the direct migration connection without a statement timeout and resumes+after an interrupted startup. It walks every session at roughly 30 sessions per+second, so a few thousand sessions take a couple of minutes; above that, run+`node src/migrate-main.ts` with the release's configuration before starting the+servers so readiness checks do not time out. Instances that start meanwhile wait for the copy on+the migration lock, so a database or role `statement_timeout` must allow that wait.+Rolling back the runtime restores code and configuration, not database contents.Older binaries do not display cron history written only to the new journal; thoserecords remain stored in `cron_fires`. Validate both old and new readers against thediff --git a/src/persistence/pg-pool.ts b/src/persistence/pg-pool.tsindex 467feaaa7..c199ce0d1 100644--- a/src/persistence/pg-pool.ts+++ b/src/persistence/pg-pool.ts@@ -102,6 +102,7 @@ export interface PgMigrationDefinition {statements: readonly string[];expectedChecksum?: string;legacyId?: string;+ prepare?: readonly string[];}export interface PgMigration extends PgMigrationDefinition {@@ -189,6 +190,7 @@ export function definePgMigration(statements: readonly string[],expectedChecksum?: string,legacyId?: string,+ prepare: readonly string[] = [],): PgMigration {if (!/^[a-z0-9][a-z0-9._/-]*$/i.test(id) || id.includes("..")) {throw new Error(`pg-pool: invalid migration id ${JSON.stringify(id)}`);@@ -201,7 +203,15 @@ export function definePgMigration(`pg-pool: migration ${id} source checksum mismatch (expected=${expectedChecksum}, actual=${checksum})`,);}- return { id, statements: normalized, checksum, ...(legacyId ? { legacyId } : {}) };+ const preparation = prepare.map((statement) => statement.trim()).filter((statement) => statement.length > 0);+ for (const statement of preparation) assertOneStatement(statement);+ return {+ id,+ statements: normalized,+ checksum,+ ...(legacyId ? { legacyId } : {}),+ ...(preparation.length ? { prepare: preparation } : {}),+ };}export async function applyPgMigrations(pool: Pool, migrations: readonly PgMigration[]): Promise<void> {@@ -229,6 +239,18 @@ export async function applyPgMigrations(pool: Pool, migrations: readonly PgMigra}continue;}+ if (migration.prepare) {+ await client.query("SET statement_timeout = 0");+ try {+ for (const statement of migration.prepare) await client.query(statement);+ } catch (error) {+ await client+ .query("RESET statement_timeout")+ .catch(swallowAs(`pg-pool: reset statement timeout after preparing ${migration.id}`, undefined));+ throw error;+ }+ await client.query("RESET statement_timeout");+ }if (migration.statements.some((statement) => concurrentIndexName(statement))) {if (migration.legacyId || !migration.statements.every((statement) => concurrentIndexName(statement))) {throw new Error(`pg-pool: concurrent index migration ${migration.id} must contain only concurrent indexes`);@@ -419,7 +441,13 @@ export function createPgPool(? [{ id: idOrDefinitions, statements: statementsOrMaintenance as readonly string[] }]: idOrDefinitions;const migrations = definitions.map((definition) =>- definePgMigration(definition.id, definition.statements, definition.expectedChecksum, definition.legacyId),+ definePgMigration(+ definition.id,+ definition.statements,+ definition.expectedChecksum,+ definition.legacyId,+ definition.prepare,+ ),);for (const migration of migrations) registerPgMigration(connectionString, migration);const maintenanceSource =@@ -558,6 +586,7 @@ export function createPgPool(definition.statements,definition.expectedChecksum,definition.legacyId,+ definition.prepare,);registerPgMigration(connectionString, migration);if (closed) throw new Error("Postgres store is closed");@@ -567,7 +596,13 @@ export function createPgPool(function registerMigration(definition: PgMigrationDefinition): void {registerPgMigration(connectionString,- definePgMigration(definition.id, definition.statements, definition.expectedChecksum, definition.legacyId),+ definePgMigration(+ definition.id,+ definition.statements,+ definition.expectedChecksum,+ definition.legacyId,+ definition.prepare,+ ),);}return { pool, sessionPool, q, query, registerMigration, migrate, close };diff --git a/src/sessions/legacy-tape-steps.ts b/src/sessions/legacy-tape-steps.tsindex f3b945d63..9f3b4c79d 100644--- a/src/sessions/legacy-tape-steps.ts+++ b/src/sessions/legacy-tape-steps.ts@@ -273,6 +273,21 @@ END $batch$`,skippingProcedure(),];+export const PREPARE_TAPE_DROP_SQL = [+ `DO $prepare_tape_drop$+BEGIN+ IF to_regclass('session_tape') IS NULL OR (SELECT done FROM legacy_tape_steps_progress) THEN+ RETURN;+ END IF;+ IF (SELECT started_at FROM legacy_tape_steps_progress) IS NULL THEN+ DELETE FROM session_tape t WHERE NOT EXISTS (SELECT 1 FROM sessions s WHERE s.id = t.session_id);+ END IF;+ IF EXISTS (SELECT 1 FROM session_tape WHERE kind = 'message') THEN+ CALL prepare_legacy_tape_steps(50);+ END IF;+END $prepare_tape_drop$`,+];+export const DROP_SESSION_TAPE_SQL = [`SET LOCAL lock_timeout = '5s'`,`DO $tape_drop_gate$diff --git a/src/sessions/postgres-session-store.ts b/src/sessions/postgres-session-store.tsindex f0bab5c23..c05297897 100644--- a/src/sessions/postgres-session-store.ts+++ b/src/sessions/postgres-session-store.ts@@ -39,7 +39,12 @@ import type {StoreOptions,} from "./session-store.ts";import { tsPrefixQuery } from "./entry-search.ts";-import { DROP_SESSION_TAPE_SQL, LEGACY_TAPE_STEPS_SIZE_GUARD_SQL, LEGACY_TAPE_STEPS_SQL } from "./legacy-tape-steps.ts";+import {+ DROP_SESSION_TAPE_SQL,+ LEGACY_TAPE_STEPS_SIZE_GUARD_SQL,+ LEGACY_TAPE_STEPS_SQL,+ PREPARE_TAPE_DROP_SQL,+} from "./legacy-tape-steps.ts";import {cronIdOf,legacyOriginPattern,@@ -814,6 +819,7 @@ export function createPostgresSessionStore(connectionString: string, opts: Store},{id: "sessions/store/0031-drop-session-tape",+ prepare: PREPARE_TAPE_DROP_SQL,statements: DROP_SESSION_TAPE_SQL,},],diff --git a/test/postgres-legacy-tape-steps.test.ts b/test/postgres-legacy-tape-steps.test.tsindex d5a628c42..a609fc466 100644--- a/test/postgres-legacy-tape-steps.test.ts+++ b/test/postgres-legacy-tape-steps.test.ts@@ -78,6 +78,85 @@ const progress = async (raw: import("pg").Pool) =>)).rows[0];+const toolCallSession = (raw: import("pg").Pool, id: string) =>+ seedSession(+ raw,+ id,+ [message([{ type: "toolCall", id: `call-${id}`, name: "execute", arguments: { z: id, a: 1 } }])],+ [["tool_call", { tool: "execute", callId: `call-${id}`, a: 1, z: id }]],+ );++const tapeState = async (raw: import("pg").Pool) => ({+ tape: (await raw.query("SELECT to_regclass('session_tape') AS t")).rows[0].t,+ steps: (await raw.query("SELECT session_id, wire FROM legacy_tape_steps ORDER BY session_id")).rows,+ done: (await raw.query("SELECT done FROM legacy_tape_steps_progress")).rows[0].done,+});++test(+ "pg migrations copy legacy tape steps on their own before dropping the tape, past deleted sessions",+ { skip },+ async () => {+ await withSchema(async (raw, url) => {+ await toolCallSession(raw, "legacy");+ await toolCallSession(raw, "zdeleted");+ await raw.query("DELETE FROM sessions WHERE id = 'zdeleted'");+ await seedSession(raw, "tapeless", [], [["user", { text: "hi" }]]);+ const pg = (await import("pg")).default;+ const timed = new globalThis.URL(url);+ timed.searchParams.set("options", `${timed.searchParams.get("options")} -c statement_timeout=4321`);+ const startup = new pg.Pool({ connectionString: timed.toString(), max: 1 });+ try {+ await applyPgMigrations(startup, registeredPgMigrations(url));+ assert.equal((await startup.query("SHOW statement_timeout")).rows[0].statement_timeout, "4321ms");+ } finally {+ await startup.end();+ }+ assert.deepEqual(await tapeState(raw), {+ tape: null,+ steps: [{ session_id: "legacy", wire: { argumentsJson: '{"z":"legacy","a":1}' } }],+ done: true,+ });+ });+ },+);++test("pg migrations drop a tape whose sessions were all deleted", { skip }, async () => {+ await withSchema(async (raw, url) => {+ await toolCallSession(raw, "gone");+ await raw.query("DELETE FROM sessions");+ await applyPgMigrations(raw, registeredPgMigrations(url));+ assert.deepEqual(await tapeState(raw), { tape: null, steps: [], done: false });+ });+});++test("pg migrations resume an interrupted tape copy once when two instances start together", { skip }, async () => {+ await withSchema(async (raw, url) => {+ for (const id of ["s1", "s2", "s3"]) await toolCallSession(raw, id);+ await raw.query("CALL prepare_legacy_tape_steps(1)");+ await raw.query("UPDATE legacy_tape_steps_progress SET last_session = 's1', done = false");+ await raw.query("DELETE FROM legacy_tape_steps WHERE session_id <> 's1'");++ const pg = (await import("pg")).default;+ const other = new pg.Pool({ connectionString: url });+ try {+ await Promise.all([+ applyPgMigrations(raw, registeredPgMigrations(url)),+ applyPgMigrations(other, registeredPgMigrations(url)),+ ]);+ } finally {+ await other.end();+ }+ assert.deepEqual(await tapeState(raw), {+ tape: null,+ steps: ["s1", "s2", "s3"].map((id) => ({+ session_id: id,+ wire: { argumentsJson: `{"z":"${id}","a":1}` },+ })),+ done: true,+ });+ });+});+test("pg legacy tape steps keep each step's model and exact block wire, and map turns to runs", { skip }, async () => {await withSchema(async (raw) => {await seedSession(@@ -221,8 +300,6 @@ test([["tool_call", { tool: "execute", callId: "c1", a: 2, b: 1 }]],);const migrations = registeredPgMigrations(url);- await assert.rejects(applyPgMigrations(raw, migrations), /run CALL prepare_legacy_tape_steps\(\) to completion/);-await raw.query("CALL prepare_legacy_tape_steps()");await raw.query("UPDATE legacy_tape_steps_progress SET skipped_sessions = 101");await assert.rejects(