Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: repair text PostgreSQL refuses (split emoji, NUL) before it reaches jsonb/text (#12815)
Adam Eivy580dc8e3c819
diff --git a/server/AGENTS.md b/server/AGENTS.mdindex e45e23ccc0..df4f826a34 100644--- a/server/AGENTS.md+++ b/server/AGENTS.md@@ -19,6 +19,10 @@ These apply to Express/service/model code. Universal constraints (no try/catch o- **Async PATCH races on shared records — serialize writes server-side.** When two write paths can mutate the same record concurrently (e.g. a blur-save plus an explicit "Render" button against `stages.comicPages.cover`), client-side guards (refs, `onMouseDown`, status checks) are unreliable — keyboard activation, status==='unknown' stalls, and `loadState → modify → saveState` interleaving all defeat them. The correct fix is to serialize writes at the *file* level on the server (every PATCH awaits the previous one to settle before reading state, so it merges against the freshest persisted record). See `queueSeriesIssuesWrite` in `server/services/pipeline/issuesShared.js` (built on `createKeyCachedQueue` in `server/lib/createKeyCachedQueue.js`). A `Map<recordId, Promise>` is *not* enough — two writes to different record ids share the same JSON state file and can still clobber each other; collapse the queue to a single tail per shared file.- **High-frequency state writes must batch.** Per-line state mutations that round-trip through `withStateLock → loadState → saveState` (e.g. `appendAgentOutput` in `server/services/cosAgentLifecycle.js`) are fine for human-pace events but catastrophic when called from a hot loop — PTY output streams, AI tool-call streams, or any producer that can emit dozens of events per second. When wiring a new streaming producer, add a batched variant that takes an array (see `appendAgentOutputLines` for the pattern) and flush from the caller on a ~250ms debounce. Always drain the pending buffer in the producer's `finish`/`cleanup` path before the final state write so completion events don't beat the last output batch to disk.+## PostgreSQL parameters++- **A parameter bound to a `json`/`jsonb` column or `::jsonb` cast is serialized with `toPgJson`, never bare `JSON.stringify`.** PostgreSQL refuses a lone UTF-16 surrogate (an emoji `.slice()`d in half) and NUL inside `jsonb`, and NUL in `text`; every file store the migrated records came from accepted both, so an unrepaired write throws and the record is lost (a promoted CoS task, an automatic investigation task, a legacy import retained for retry forever, #12815). `toPgJson` and `normalizePgString` live in `server/lib/pgJson.js` and repair a bad unit to U+FFFD, the same lossy fix a UTF-8 file write applied. `lib/db.js` applies `normalizePgParams` to every `query()` and `withTransaction` client call (strings, array elements and plain-object parameters; it never unescapes a string, so a `text` value holding the characters `\^@` is stored as written), but a pre-stringified JSON string looks like ordinary text to it, so the call site still uses `toPgJson`. `server/pgJsonConventions.test.js` scans the tree for this and carries no allowlist; applied `scripts/db-migrations/` are frozen and skipped.+## Timers- **An `async` timer callback must own its rejections.** Inside `setTimeout(async …)` / `setInterval(async …)`, every `await` needs either an enclosing `try { … } catch` or a `.catch(…)` on the awaited chain. A timer fires outside the request lifecycle, so nothing holds the promise the callback returns and a rejection escapes as an unhandled rejection — fatal on Node >= 15. The CoS runner shipped that at boot: its orphan-cleanup timer awaited a state-file read/write, so one truncated `agents` file killed the runner seconds after start and PM2 restart-looped it against the same file (#5668). Log and continue in the catch unless the callback genuinely owns a failure verdict. `server/timerCallbackConventions.test.js` scans the whole server tree for this and carries no allowlist; the worked example is the watchdog in `server/services/mediaJobQueue/index.js`.diff --git a/server/lib/README.md b/server/lib/README.mdindex 72c1fb7216..e42194b95d 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -942,3 +942,4 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub- `nativeTrashXdg.js`: `trashXdg(path)` moves one file into the freedesktop Trash 1.0 directory serving its volume on `node:fs` alone (original names, RFC 2396 `Path=`, `O_EXCL` info reservation, NAME_MAX-bounded names, mount identity from `/proc/self/mountinfo` with a `st_dev` fallback so a bind mount gets its own volume trash instead of an `EXDEV`, physical-parent mount discovery, an owned trash directory is tightened to 0700 and `TRASH_DIR_UNSAFE` refuses a symlinked or foreign-owned one or a volume root whose entries other users could rename, a sticky shared `.Trash` counts only when root or the caller owns it and falls back to `.Trash-<uid>` when it cannot hold the per-user directory, refuses instead of copying across devices); `_topDirTrash(topDir, uid)`, `_parseMountInfo(text)` and `_mountOf(realDir, mounts)` are test hooks.- `nativeTrash.js`: `moveToNativeTrash(paths)` executes the pinned package native recycling helpers on macOS/Windows with hidden windows and a deadline; Linux uses the package XDG adapter. No local-directory or permanent-delete fallback.- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329); `systemClock` (globals resolved at call time), `__createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).+- `pgJson.js`: text repair for values bound to PostgreSQL, which refuses a lone UTF-16 surrogate in `jsonb` and NUL in `jsonb`/`text` where every file store accepted both. `normalizePgString(s)` → `toWellFormed()` with NUL → U+FFFD (the same lossy repair a UTF-8 file write applied). `toPgJson(value)` → `JSON.stringify` that repairs string values and object keys; **use it, never bare `JSON.stringify`, for a parameter bound to a `json`/`jsonb` column or `::jsonb` cast** (`pgJsonConventions.test.js` enforces this). `normalizePgParams(params)` is applied by `db.js` `query()` and the `withTransaction` client: repairs string and array-element params (never unescapes `\^@`) and serializes plain-object params via `toPgJson`.diff --git a/server/lib/db.js b/server/lib/db.jsindex 1c4309a217..7d40b22141 100644--- a/server/lib/db.js+++ b/server/lib/db.js@@ -11,6 +11,7 @@ import { isTestRunner } from './runtimeEnv.js';import { assertDatabaseAdmission, createDatabaseMaintenanceJournal } from './databaseMaintenanceJournal.js';import { assertDatabasePoolAuthority } from './databaseAuthority.js';import { createDatabaseRestoreRecovery, databaseRestoreRecoveryError } from './databaseRestoreRecovery.js';+import { normalizePgParams } from './pgJson.js';const { Pool } = pg;@@ -207,6 +208,11 @@ const GUARDED_CLIENT_HANDLER = {return (config, ...rest) => {const sql = typeof config === 'string' ? config : config?.text;assertWriteAllowed(sql);+ // A Submittable (pg-cursor) also carries `values`; only a plain config is rewritten.+ if (typeof config === 'string' && Array.isArray(rest[0])) return target.query(config, normalizePgParams(rest[0]), ...rest.slice(1));+ if (Array.isArray(config?.values) && Object.getPrototypeOf(config) === Object.prototype) {+ return target.query({ ...config, values: normalizePgParams(config.values) }, ...rest);+ }return target.query(config, ...rest);};}@@ -295,7 +301,7 @@ export async function withDatabaseMaintenance(fn, { restoreRecoveryId } = {}) {/*** Execute a query against the connection pool.* @param {string} text - SQL query text with $1, $2, etc. placeholders- * @param {Array} params - Parameter values+ * @param {Array} params - Parameter values; strings are repaired for PostgreSQL (see pgJson.js) and plain objects serialized, here and on the withTransaction client* @returns {Promise<pg.QueryResult>}*/export async function query(text, params) {@@ -304,7 +310,7 @@ export async function query(text, params) {// author calls query('INSERT …') directly, or a backend selector mis-chose// Postgres because NODE_ENV wasn't 'test').assertWriteAllowed(text);- return databaseOperation(() => pool.query(text, params));+ return databaseOperation(() => pool.query(text, normalizePgParams(params)));}/**diff --git a/server/lib/db.test.js b/server/lib/db.test.jsindex 151de495c1..556c84ad53 100644--- a/server/lib/db.test.js+++ b/server/lib/db.test.js@@ -33,7 +33,7 @@ vi.mock('./db/schema/index.js', async () => {};});-const { checkHealth, ensureSchema, close, POOL_CONFIG, SCHEMA_DDL_ADVISORY_LOCK_KEY } = await import('./db.js');+const { checkHealth, ensureSchema, close, query, withTransaction, POOL_CONFIG, SCHEMA_DDL_ADVISORY_LOCK_KEY } = await import('./db.js');let dbReady = false;let skipReason = '';@@ -110,3 +110,31 @@ describe.skipIf(!runDb)('ensureSchema() boot DDL advisory lock (#5977)', () => {await expect(ensureSchema()).resolves.toBeUndefined();});});++// PostgreSQL refuses a lone surrogate in jsonb and NUL in jsonb/text; the pool+// and the withTransaction client repair bound parameters in one place (#12815).+describe.skipIf(!runDb)('bound parameter repair', () => {+ const split = '😀'.slice(0, 1);+ const runners = {+ 'query()': (sql, params) => query(sql, params),+ 'withTransaction client': (sql, params) => withTransaction((client) => client.query(sql, params)),+ };++ describe.each(Object.entries(runners))('%s', (_name, run) => {+ it('repairs a string parameter and leaves the characters \\^@ alone', async () => {+ const { rows: [row] } = await run('SELECT $1::text AS bad, $2::text AS literal', [`x${split}y\0z`, 'a\\^@b']);+ expect(row.bad).toBe('x\uFFFDy\uFFFDz');+ expect(row.literal).toBe('a\\^@b');+ });++ it('repairs every string in an array parameter', async () => {+ const { rows: [row] } = await run('SELECT $1::text[] AS items', [['ok', `a${split}`, 'b\0']]);+ expect(row.items).toEqual(['ok', 'a\uFFFD', 'b\uFFFD']);+ });++ it('serializes a plain-object parameter as repaired jsonb, keys included', async () => {+ const { rows: [row] } = await run('SELECT $1::jsonb AS doc', [{ [`k${split}`]: [`v${split}`, 'n\0'], nested: { ok: 1 } }]);+ expect(row.doc).toEqual({ 'k\uFFFD': ['v\uFFFD', 'n\uFFFD'], nested: { ok: 1 } });+ });+ });+});diff --git a/server/lib/index.js b/server/lib/index.jsindex 5b9e2db34d..2d7ad7429f 100644--- a/server/lib/index.js+++ b/server/lib/index.js@@ -928,3 +928,4 @@ export * from './reconcileRetirementEvidence.js';export * from './checkoutRetirementAdmission.js';export * from './moltworldExplore.js';+export * from './pgJson.js';diff --git a/server/lib/pgJson.js b/server/lib/pgJson.jsnew file mode 100644index 0000000000..94fb355fa5--- /dev/null+++ b/server/lib/pgJson.js@@ -0,0 +1,55 @@+/**+ * Text repair for values bound to PostgreSQL.+ *+ * `jsonb` refuses a lone UTF-16 surrogate (`\ud83d`, e.g. an emoji cut in half by+ * `.slice()`) and both `jsonb` and `text` refuse NUL. Every file store these+ * records came from tolerated both, so a write that used to succeed started+ * throwing after the move to PostgreSQL. The repair is the same lossy one a+ * UTF-8 file write always applied: a bad unit becomes U+FFFD.+ */++const REPLACEMENT = '�';++const isPlainObject = (value) => {+ if (value === null || typeof value !== 'object') return false;+ const proto = Object.getPrototypeOf(value);+ return proto === Object.prototype || proto === null;+};++export function normalizePgString(s) {+ return s.toWellFormed().replaceAll('\0', REPLACEMENT);+}++// JSON.stringify never shows its replacer an object key, so a key holding a bad+// unit is repaired by handing back a copy with normalized keys.+function normalizeKeys(value) {+ const keys = Object.keys(value);+ if (keys.every((key) => normalizePgString(key) === key)) return value;+ return Object.fromEntries(keys.map((key) => [normalizePgString(key), value[key]]));+}++function repairJsonValue(_key, value) {+ if (typeof value === 'string') return normalizePgString(value);+ return isPlainObject(value) ? normalizeKeys(value) : value;+}++/** `JSON.stringify` for a `json`/`jsonb` parameter: strings and object keys are repaired first. */+export function toPgJson(value) {+ return JSON.stringify(value, repairJsonValue);+}++const normalizeParam = (param) => {+ if (typeof param === 'string') return normalizePgString(param);+ if (Array.isArray(param)) return param.map(normalizeParam);+ // Mirrors node-postgres, which JSON-serializes a plain object parameter itself.+ return isPlainObject(param) ? toPgJson(param) : param;+};++/**+ * Repair a query's bound parameters in one place. Strings are repaired as-is+ * (never unescaped: a `text` value may legitimately contain the characters `\^@`),+ * array elements likewise, and plain objects become repaired JSON text.+ */+export function normalizePgParams(params) {+ return Array.isArray(params) ? params.map(normalizeParam) : params;+}diff --git a/server/pgJsonConventions.test.js b/server/pgJsonConventions.test.jsnew file mode 100644index 0000000000..a4d59474dd--- /dev/null+++ b/server/pgJsonConventions.test.js@@ -0,0 +1,164 @@+/**+ * Repo-wide guard: a value bound to a `json`/`jsonb` parameter is serialized with+ * `toPgJson` (lib/pgJson.js), never bare `JSON.stringify`.+ *+ * PostgreSQL refuses a lone UTF-16 surrogate and NUL inside `jsonb`, which every+ * file store these records came from accepted. `JSON.stringify` writes a lone+ * surrogate as the ASCII escape `\ud83d`, so the value reaches the server intact+ * and is refused — an emoji cut in half by `.slice()` lost a user-promoted CoS+ * task (#12815). `toPgJson` applies the repair the old UTF-8 file writes did.+ *+ * ## The rule+ *+ * A `JSON.stringify(` call is a violation when it feeds a statement that mentions+ * `::json`/`::jsonb`, in either shape:+ *+ * 1. it sits anywhere inside a `query(…)`/`execute(…)`-style call whose text+ * mentions `::json`/`::jsonb` (the common case, including a `.map(…)` that+ * builds an `unnest` column), or+ * 2. its innermost bracket is an array literal or a `.push(…)` — a bind-parameter+ * list assembled apart from the SQL — and the enclosing block mentions+ * `::json`/`::jsonb`.+ *+ * Hashing, logging, HTTP bodies and advisory-lock keys are not in either shape.+ *+ * ## Allowlist+ *+ * There is none. Applied migrations (`scripts/db-migrations/`) are frozen and+ * skipped wholesale: `020-*` binds to `json` (exact text) on purpose and verifies+ * byte equality. Anything else: swap in `toPgJson`.+ *+ * ## What this guard CANNOT see+ *+ * A lexer-assisted source scan (see lib/sourceScan.js), not a type-aware one. It+ * misses a stringified value bound to a `json`/`jsonb` COLUMN with no `::json`+ * cast anywhere in its block, and a value stringified in one function and bound+ * in another. `db.js` repairs plain-object parameters regardless, which covers+ * the call sites that never stringified at all.+ */+import { describe, it, expect } from 'vitest';+import { execFileSync } from 'child_process';+import { readFileSync } from 'fs';+import { dirname, join } from 'path';+import { fileURLToPath } from 'url';+import { blankCommentBodies, blankLiterals, matchBracket } from './lib/sourceScan.js';++const SERVER_ROOT = dirname(fileURLToPath(import.meta.url));+const SCANNED_DIRS = ['services/', 'lib/', 'routes/', 'scripts/'];+const JSON_CAST = /::jsonb?\b/;+const STRINGIFY = /JSON\.stringify\s*\(/g;+const QUERY_CALLEE = /(?:^|[^\w$])(?:query|queryFn|execute|exec|run|q)\s*$/;++/** Every violating `JSON.stringify(` in `src`, as `line N`. */+export function findPgJsonViolations(src) {+ const code = blankLiterals(src);+ const text = blankCommentBodies(src);+ const line = (index) => code.slice(0, index).split('\n').length;+ const calleeOf = (open) => code.slice(Math.max(0, open - 40), open);+ const hits = [];++ // One pass keeps the stack of enclosing brackets for every stringify call.+ const stack = [];+ const stringifyStarts = new Set([...code.matchAll(STRINGIFY)].map((match) => match.index));+ for (let i = 0; i < code.length; i += 1) {+ const ch = code[i];+ if (stringifyStarts.has(i) && violates(stack)) hits.push(`line ${line(i)}`);+ if (ch === '(' || ch === '[' || ch === '{') stack.push(i);+ else if (ch === ')' || ch === ']' || ch === '}') stack.pop();+ }+ return hits;++ function violates(enclosing) {+ // Shape 1: inside a query-style call, whose own text is the statement.+ const queryCall = enclosing.find((open) => code[open] === '(' && QUERY_CALLEE.test(calleeOf(open)));+ if (queryCall !== undefined) return JSON_CAST.test(text.slice(queryCall, matchBracket(code, queryCall)));+ // Shape 2: a bind-parameter list assembled apart from the SQL.+ const innermost = enclosing[enclosing.length - 1];+ if (innermost === undefined) return false;+ const isParamList = code[innermost] === '['+ || (code[innermost] === '(' && /\.push\s*$/.test(calleeOf(innermost)));+ if (!isParamList) return false;+ const block = [...enclosing].reverse().find((open) => code[open] === '{');+ if (block === undefined) return JSON_CAST.test(text);+ const end = matchBracket(code, block);+ return end !== -1 && JSON_CAST.test(text.slice(block, end));+ }+}++const trackedSources = () => execFileSync('git', ['ls-files', '*.js', '*.mjs'], {+ cwd: SERVER_ROOT,+ encoding: 'utf8',+ maxBuffer: 64 * 1024 * 1024,+}).split('\n').filter((f) => f+ && SCANNED_DIRS.some((dir) => f.startsWith(dir))+ && !f.includes('.test.')+ && !f.includes('db-migrations/'));++describe('json/jsonb parameters are serialized with toPgJson (#12815)', () => {+ it('scans the server tree', () => {+ // A broken `git ls-files` would otherwise let every assertion pass by scanning nothing.+ expect(trackedSources().length).toBeGreaterThan(200);+ });++ it('has no bare JSON.stringify feeding a json/jsonb parameter', () => {+ const violations = [];+ for (const file of trackedSources()) {+ const src = readFileSync(join(SERVER_ROOT, file), 'utf8');+ if (!src.includes('JSON.stringify') || !JSON_CAST.test(src)) continue;+ for (const hit of findPgJsonViolations(src)) violations.push(`server/${file} ${hit}`);+ }+ expect(+ violations,+ 'PostgreSQL rejects a lone surrogate and NUL inside jsonb, so a bare JSON.stringify parameter '+ + 'loses the write when a string was cut mid-emoji (#12815).\n'+ + 'Fix: import { toPgJson } from the lib/pgJson.js module and use it in place of JSON.stringify.\n'+ + `Offenders:\n ${violations.join('\n ')}`,+ ).toEqual([]);+ });+});++// Guards the guard: if the recognizer stops seeing the bad shape the scan above goes green.+describe('the pgJson recognizer', () => {+ it('flags a stringified value inside a jsonb query', () => {+ expect(findPgJsonViolations("await query('INSERT INTO t (d) VALUES ($1::jsonb)', [JSON.stringify(record)]);"))+ .toEqual(['line 1']);+ expect(findPgJsonViolations(`await client.query(+ 'INSERT INTO t SELECT u.d::jsonb FROM unnest($1::text[]) AS u(d)',+ [rows.map((row) => JSON.stringify(row.task))],+ );`)).toEqual(['line 3']);+ });++ it('flags a parameter list assembled apart from the SQL', () => {+ expect(findPgJsonViolations(`function update(patch) {+ const sets = ['payload = $2::jsonb'];+ const params = [];+ params.push(JSON.stringify(patch));+ return run(sets, params);+ }`)).toEqual(['line 4']);+ expect(findPgJsonViolations(`function insert(id, doc) {+ const sql = 'INSERT INTO t (id, doc) VALUES ($1, $2::jsonb)';+ const values = [id, JSON.stringify(doc)];+ return pool.query(sql, values);+ }`)).toEqual(['line 3']);+ });++ it('accepts toPgJson and stringification that is not a bind parameter', () => {+ expect(findPgJsonViolations("await query('INSERT INTO t (d) VALUES ($1::jsonb)', [toPgJson(record)]);")).toEqual([]);+ expect(findPgJsonViolations(`function bind(doc, a, b) {+ const sql = 'INSERT INTO t (doc) VALUES ($1::jsonb)';+ const key = createHash('sha256').update(JSON.stringify(doc)).digest('hex');+ const same = JSON.stringify(a) === JSON.stringify(b);+ return query(sql, [toPgJson(doc), key, same]);+ }`)).toEqual([]);+ // A stringified lock key in a query with no json cast, beside a jsonb statement.+ expect(findPgJsonViolations(`async function tx(client) {+ await client.query('SELECT pg_advisory_xact_lock(hashtext($1))', [JSON.stringify([a, b])]);+ await client.query("UPDATE t SET d = '{}'::jsonb");+ }`)).toEqual([]);+ });++ it('ignores JSON.stringify named in a comment or a string', () => {+ expect(findPgJsonViolations(`// params.push(JSON.stringify(x)) with ::jsonb+ const note = 'values = [JSON.stringify(x)] ::jsonb';`)).toEqual([]);+ });+});diff --git a/server/scripts/migrateCatalogPayload.js b/server/scripts/migrateCatalogPayload.jsindex c1a9bfd168..a0fdadeef8 100644--- a/server/scripts/migrateCatalogPayload.js+++ b/server/scripts/migrateCatalogPayload.js@@ -30,6 +30,7 @@import { readMarker, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { CATALOG_TYPES, currentPayloadSchemaVersion, upgradePayload } from '../lib/catalogTypes.js';+import { toPgJson } from '../lib/pgJson.js';const MARKER_FILENAME = 'catalog-payload.applied.json';@@ -70,7 +71,7 @@ async function migrateType(type) {const next = upgradePayload(type.id, row.payload || {});await query(`UPDATE catalog_ingredients SET payload = $2::jsonb WHERE id = $1`,- [row.id, JSON.stringify(next)],+ [row.id, toPgJson(next)],);upgraded++;}diff --git a/server/scripts/migrateCatalogUserTypesToDB.js b/server/scripts/migrateCatalogUserTypesToDB.jsindex 4a12534f72..cf95f2547a 100644--- a/server/scripts/migrateCatalogUserTypesToDB.js+++ b/server/scripts/migrateCatalogUserTypesToDB.js@@ -34,6 +34,7 @@import { query } from '../lib/db.js';import { mirrorTimestamp } from '../lib/pgTimestamp.js';import { getSettings, updateSettingsWith } from '../services/settings.js';+import { toPgJson } from '../lib/pgJson.js';const LEGACY_KEY = 'catalogUserTypes';const IMPORTED_KEY = 'catalogUserTypes_imported';@@ -79,7 +80,7 @@ export async function migrateCatalogUserTypesToDB() {`INSERT INTO catalog_user_types (id, data, updated_at, deleted_at)VALUES ($1, $2::jsonb, COALESCE($3::timestamptz, NOW()), $4)ON CONFLICT (id) DO NOTHING`,- [type.id, JSON.stringify(type), mirrorTimestamp(type.updatedAt, null), mirrorTimestamp(type.deletedAt, null)],+ [type.id, toPgJson(type), mirrorTimestamp(type.updatedAt, null), mirrorTimestamp(type.deletedAt, null)],);if (result.rowCount > 0) imported += 1;else skipped += 1;diff --git a/server/scripts/migrateCreativeDirectorToDB.js b/server/scripts/migrateCreativeDirectorToDB.jsindex 0d0b06d858..c9565b3d02 100644--- a/server/scripts/migrateCreativeDirectorToDB.js+++ b/server/scripts/migrateCreativeDirectorToDB.js@@ -35,6 +35,7 @@ import { PATHS } from '../lib/fileUtils.js';import { markerExists, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { mirrorStatus, mirrorTimestamp } from '../services/creativeDirector/projectsLogic.js';+import { toPgJson } from '../lib/pgJson.js';const LEGACY_FILENAME = 'creative-director-projects.json';const IMPORTED_SUFFIX = '.imported';@@ -91,7 +92,7 @@ export async function migrateCreativeDirectorToDB() {[project.id,mirrorStatus(project.status),- JSON.stringify(project),+ toPgJson(project),createdAt,mirrorTimestamp(project.updatedAt, createdAt),],diff --git a/server/scripts/migrateIssuesToDB.js b/server/scripts/migrateIssuesToDB.jsindex 9c505a8b0f..29ce3bec8d 100644--- a/server/scripts/migrateIssuesToDB.js+++ b/server/scripts/migrateIssuesToDB.js@@ -26,6 +26,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { readLegacyJSON, legacyDirectory, incompleteImport } from './legacyImport.js';import { mirrorTimestamp } from '../lib/pgTimestamp.js';+import { toPgJson } from '../lib/pgJson.js';const LEGACY_DIRNAME = 'pipeline-issues';const IMPORTED_DIRNAME = 'pipeline-issues.imported';@@ -47,7 +48,7 @@ export async function importRecord(record, execute = query) {typeof record.seasonId === 'string' && record.seasonId ? record.seasonId : null,Number.isFinite(record.number) ? Math.floor(record.number) : null,typeof record.status === 'string' && record.status ? record.status.slice(0, 32) : null,- JSON.stringify(record),+ toPgJson(record),record.ephemeral === true,createdAt,mirrorTimestamp(record.updatedAt, createdAt),diff --git a/server/scripts/migrateSeriesToDB.js b/server/scripts/migrateSeriesToDB.jsindex 0ce9d060b6..5399e83c37 100644--- a/server/scripts/migrateSeriesToDB.js+++ b/server/scripts/migrateSeriesToDB.js@@ -37,6 +37,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { readLegacyJSON, legacyDirectory, parkLegacyFile, incompleteImport } from './legacyImport.js';import { mirrorTimestamp } from '../lib/pgTimestamp.js';+import { toPgJson } from '../lib/pgJson.js';const LEGACY_DIRNAME = 'pipeline-series';const MARKER_FILENAME = 'pipeline-series.migrated.json';@@ -58,7 +59,7 @@ export async function importRecord(record, execute = query) {typeof record.name === 'string' ? record.name : '',typeof record.universeId === 'string' && record.universeId ? record.universeId : null,typeof record.writersRoomWorkId === 'string' && record.writersRoomWorkId ? record.writersRoomWorkId : null,- JSON.stringify(record),+ toPgJson(record),record.ephemeral === true,createdAt,mirrorTimestamp(record.updatedAt, createdAt),diff --git a/server/scripts/migrateStoryBuilderToDB.js b/server/scripts/migrateStoryBuilderToDB.jsindex 1689374794..9efbe0cfc6 100644--- a/server/scripts/migrateStoryBuilderToDB.js+++ b/server/scripts/migrateStoryBuilderToDB.js@@ -27,6 +27,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { readLegacyJSON, legacyDirectory, incompleteImport } from './legacyImport.js';import { mirrorTimestamp } from '../lib/pgTimestamp.js';+import { toPgJson } from '../lib/pgJson.js';const LEGACY_DIRNAME = 'story-builder';const IMPORTED_DIRNAME = 'story-builder.imported';@@ -49,7 +50,7 @@ export async function importRecord(record, execute = query) {typeof record.universeId === 'string' && record.universeId ? record.universeId : null,typeof record.seriesId === 'string' && record.seriesId ? record.seriesId : null,record.sync === true,- JSON.stringify(record),+ toPgJson(record),record.ephemeral === true,createdAt,mirrorTimestamp(record.updatedAt, createdAt),diff --git a/server/scripts/migrateUniversesToDB.js b/server/scripts/migrateUniversesToDB.jsindex d19617cd27..c1f486e1c7 100644--- a/server/scripts/migrateUniversesToDB.js+++ b/server/scripts/migrateUniversesToDB.js@@ -35,6 +35,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { readLegacyJSON, legacyDirectory, incompleteImport } from './legacyImport.js';import { mirrorTimestamp } from '../lib/pgTimestamp.js';+import { toPgJson } from '../lib/pgJson.js';const LEGACY_DIRNAME = 'universes';const IMPORTED_DIRNAME = 'universes.imported';@@ -55,7 +56,7 @@ export async function importRecord(record, execute = query) {[record.id,typeof record.name === 'string' ? record.name : '',- JSON.stringify(record),+ toPgJson(record),schemaVersion,record.ephemeral === true,createdAt,@@ -77,7 +78,7 @@ export async function importRun(run, execute = query) {run.id,run.universeId,typeof run.collectionId === 'string' ? run.collectionId : null,- JSON.stringify(run),+ toPgJson(run),mirrorTimestamp(run.createdAt, new Date().toISOString()),],);diff --git a/server/scripts/migrateUniversesToDB.test.js b/server/scripts/migrateUniversesToDB.test.jsindex d316767464..013f66b210 100644--- a/server/scripts/migrateUniversesToDB.test.js+++ b/server/scripts/migrateUniversesToDB.test.js@@ -59,14 +59,21 @@ vi.mock('fs/promises', () => ({let uTable;let rTable;+// jsonb refuses the ASCII escape of an unpaired surrogate (`\ud83d`), as PostgreSQL does (#12815).+const assertJsonbAccepts = (text) => {+ const unpaired = text.replace(/\\ud[89ab][0-9a-f]{2}\\ud[c-f][0-9a-f]{2}/gi, '').match(/\\ud[89a-f][0-9a-f]{2}/i);+ if (unpaired) throw new Error(`invalid input syntax for type json: lone surrogate ${unpaired[0]}`);+};const query = vi.fn(async (sql, params) => {if (/INSERT INTO universes/.test(sql)) {+ assertJsonbAccepts(params[2]);const id = params[0];if (uTable.has(id)) return { rowCount: 0, rows: [] };uTable.set(id, { id, name: params[1], data: JSON.parse(params[2]) });return { rowCount: 1, rows: [] };}if (/INSERT INTO universe_runs/.test(sql)) {+ assertJsonbAccepts(params[3]);const id = params[0];if (rTable.has(id)) return { rowCount: 0, rows: [] };rTable.set(id, { id, universeId: params[1] });@@ -130,6 +137,16 @@ describe('migrateUniversesToDB', () => {expect(MARKER in files).toBe(true);});+ it('imports a record holding a lone-surrogate \\ud83d escape, repaired, instead of retaining it for retry', async () => {+ dirEntries = ['u-1'];+ recordsByDir['/fake/data/universes/u-1/index.json'] = rec('u-1', { logline: 'cut\ud83d', credit: '\ud83d\ude00 kept' });+ const result = await importer();+ expect(result).toMatchObject({ ok: true, reason: 'imported', imported: 1 });+ expect(uTable.get('u-1').data).toMatchObject({ logline: 'cut\uFFFD', credit: '😀 kept' });+ expect(renamed).toEqual([['/fake/data/universes', '/fake/data/universes.imported']]);+ expect(MARKER in files).toBe(true);+ });+it('is idempotent: a re-run over the same rows imports 0 (ON CONFLICT DO NOTHING)', async () => {dirEntries = ['u-1'];recordsByDir['/fake/data/universes/u-1/index.json'] = rec('u-1');diff --git a/server/scripts/migrateWritersRoomToDB.js b/server/scripts/migrateWritersRoomToDB.jsindex e492ded1aa..5e0e0f6c86 100644--- a/server/scripts/migrateWritersRoomToDB.js+++ b/server/scripts/migrateWritersRoomToDB.js@@ -35,6 +35,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';import { query } from '../lib/db.js';import { readLegacyJSON, legacyDirectory, parkLegacyFile, incompleteImport } from './legacyImport.js';import { mirrorTimestamp } from '../lib/pgTimestamp.js';+import { toPgJson } from '../lib/pgJson.js';const ROOT_DIRNAME = 'writers-room';const MARKER_FILENAME = 'writers-room.migrated.json';@@ -53,7 +54,7 @@ export async function importFolder(folder, execute = query) {typeof folder.parentId === 'string' && folder.parentId ? folder.parentId : null,String(folder.name ?? ''),Number.isInteger(folder.sortOrder) ? folder.sortOrder : 0,- JSON.stringify(folder),+ toPgJson(folder),createdAt,mirrorTimestamp(folder.updatedAt, createdAt),],@@ -71,7 +72,7 @@ export async function importExercise(exercise, execute = query) {exercise.id,typeof exercise.workId === 'string' && exercise.workId ? exercise.workId : null,typeof exercise.status === 'string' ? exercise.status.slice(0, 16) : null,- JSON.stringify(exercise),+ toPgJson(exercise),mirrorTimestamp(exercise.startedAt, null),mirrorTimestamp(exercise.finishedAt, null),],@@ -106,7 +107,7 @@ export async function importWork(manifest, execute = query) {typeof manifest.pipelineIssueId === 'string' ? manifest.pipelineIssueId : null,typeof manifest.cdProjectId === 'string' ? manifest.cdProjectId : null,typeof manifest.mediaCollectionId === 'string' ? manifest.mediaCollectionId : null,- JSON.stringify(workData),+ toPgJson(workData),createdAt,mirrorTimestamp(manifest.updatedAt, createdAt),],@@ -125,9 +126,9 @@ export async function importWork(manifest, execute = query) {String(draft.contentFile ?? ''),typeof draft.contentHash === 'string' ? draft.contentHash : null,Number.isInteger(draft.wordCount) ? draft.wordCount : 0,- JSON.stringify(Array.isArray(draft.segmentIndex) ? draft.segmentIndex : []),+ toPgJson(Array.isArray(draft.segmentIndex) ? draft.segmentIndex : []),typeof draft.createdFromVersionId === 'string' ? draft.createdFromVersionId : null,- JSON.stringify(draft),+ toPgJson(draft),mirrorTimestamp(draft.createdAt, createdAt),],);diff --git a/server/services/albums/db.js b/server/services/albums/db.jsindex 6717e8c6c6..a53f993ee8 100644--- a/server/services/albums/db.js+++ b/server/services/albums/db.js@@ -16,6 +16,7 @@ import {maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,withBaseHashFlushBatch,} from '../../lib/conflictJournal.js';+import { toPgJson } from '../../lib/pgJson.js';function rowToAlbum(row) {return row ? row.data : null;@@ -36,7 +37,7 @@ async function persist(exec, album) {[album.id,typeof album.title === 'string' ? album.title : '',- JSON.stringify(album),+ toPgJson(album),createdAt,mirrorTimestamp(album.updatedAt, createdAt),album.deleted === true,diff --git a/server/services/appQuality.js b/server/services/appQuality.jsindex eeb1138a6f..d755582037 100644--- a/server/services/appQuality.js+++ b/server/services/appQuality.js@@ -6,6 +6,7 @@ import { tryReadFile } from '../lib/jsonIo.js';import { normalizeAuditTaskType } from '../lib/auditCatalog.js';import { AUDIT_FRESHNESS_MS, parseAuditQualityReport, parseAuditSlice, summarizeAppQuality, buildAppQualityHistory, dailyQualityRecords } from '../lib/auditQuality.js';import { collectAppQuality, qualityRecord, readQualityRecords, readReleaseQuality } from './appQualityFederation.js';+import { toPgJson } from '../lib/pgJson.js';export async function recordAuditQuality({ task, taskType, agentId, workspacePath, success, assessedAt, deepDiscoveryComplete = false }, deps = {}) {if (!success || !workspacePath || !task?.metadata?.app || !agentId) return false;@@ -31,7 +32,7 @@ export async function recordAuditQuality({ task, taskType, agentId, workspacePat`INSERT INTO app_quality_measurements (app_id, category, agent_id, assessed_at, report, slice)VALUES ($1, $2, $3, $4, $5::jsonb, $6)ON CONFLICT (app_id, category, agent_id) DO NOTHING`,- [task.metadata.app, category, agentId, assessedAt, JSON.stringify(report), slice]+ [task.metadata.app, category, agentId, assessedAt, toPgJson(report), slice]);return true;}diff --git a/server/services/artists/db.js b/server/services/artists/db.jsindex 4c659fe2b7..fa67368a12 100644--- a/server/services/artists/db.js+++ b/server/services/artists/db.js@@ -18,6 +18,7 @@ import {maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,withBaseHashFlushBatch,} from '../../lib/conflictJournal.js';+import { toPgJson } from '../../lib/pgJson.js';function rowToArtist(row) {return row ? row.data : null;@@ -38,7 +39,7 @@ async function persist(exec, artist) {[artist.id,typeof artist.name === 'string' ? artist.name : '',- JSON.stringify(artist),+ toPgJson(artist),createdAt,mirrorTimestamp(artist.updatedAt, createdAt),artist.deleted === true,diff --git a/server/services/authors/db.js b/server/services/authors/db.jsindex 9633b1ea38..0bef60d678 100644--- a/server/services/authors/db.js+++ b/server/services/authors/db.js@@ -21,6 +21,7 @@ import {maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,withBaseHashFlushBatch,} from '../../lib/conflictJournal.js';+import { toPgJson } from '../../lib/pgJson.js';function rowToAuthor(row) {return row ? row.data : null;@@ -41,7 +42,7 @@ async function persist(exec, author) {[author.id,typeof author.name === 'string' ? author.name : '',- JSON.stringify(author),+ toPgJson(author),createdAt,mirrorTimestamp(author.updatedAt, createdAt),author.deleted === true,diff --git a/server/services/catalogDB/commit.js b/server/services/catalogDB/commit.jsindex 7f4ff68359..20e851377d 100644--- a/server/services/catalogDB/commit.js+++ b/server/services/catalogDB/commit.js@@ -12,6 +12,7 @@ import { catalogScrapCommitSchema } from '../../lib/catalogValidation.js';import { query, withTransaction } from '../../lib/db.js';import { createIngredient } from './ingredients.js';import { linkIngredientToSource, linkIngredientToRef, linkIngredientRelation, universeRefRoleForType } from './refs.js';+import { toPgJson } from '../../lib/pgJson.js';// Legacy same-batch `related-to` edges connect a single scrap's extractions into one// cluster instead of N isolated nodes (#7615). Bounded: a batch above this@@ -125,7 +126,7 @@ export async function commitScrap({ scrapId, accepted = [], embeds = [], universif (operationKey) {await client.query('UPDATE catalog_commit_receipts SET ingredients = $2::jsonb WHERE operation_key = $1',- [operationKey, JSON.stringify(created)],+ [operationKey, toPgJson(created)],);}return created;diff --git a/server/services/catalogDB/ingredients.js b/server/services/catalogDB/ingredients.jsindex 000798b76b..f4ff9dc1fb 100644--- a/server/services/catalogDB/ingredients.js+++ b/server/services/catalogDB/ingredients.js@@ -30,6 +30,7 @@ import {} from './shared.js';import { normalizeTags } from './tags.js';import { assertCatalogIdentityWritable } from './identityAccess.js';+import { toPgJson } from '../../lib/pgJson.js';// `{ client }` is optional — when supplied, SQL runs on the caller's transaction// client (so the write rolls back if a later step in the same `withTransaction`@@ -81,7 +82,7 @@ export async function createIngredient({ id: explicitId, type, name, payload = {id,type,String(name).trim(),- JSON.stringify(storedPayload),+ toPgJson(storedPayload),normalizedTags,embedding ? arrayToPgvector(embedding) : null,embeddingModel,@@ -187,7 +188,7 @@ export async function updateIngredient(id, patch = {}, { client, source = 'user'if (normalizedPatch[jsField] === undefined) continue;if (jsField === 'payload') {fields.push(`${dbField} = $${idx++}::jsonb`);- params.push(JSON.stringify(normalizedPatch.payload || {}));+ params.push(toPgJson(normalizedPatch.payload || {}));} else if (jsField === 'embedding') {fields.push(`${dbField} = $${idx++}`);params.push(normalizedPatch.embedding ? arrayToPgvector(normalizedPatch.embedding) : null);@@ -270,7 +271,7 @@ export async function recordIngredientRevision(ingredient, { source = 'user', acnewRevisionId(),ingredient.id,ingredient.name,- JSON.stringify(ingredient.payload || {}),+ toPgJson(ingredient.payload || {}),ingredient.tags || [],src,actor ? String(actor).slice(0, 120) : null,@@ -346,7 +347,7 @@ export async function reviveDeletedIngredient(id, { type, name, payload = {}, tatype = $2, name = $3, payload = $4::jsonb, tags = $5WHERE id = $1 AND deleted = trueRETURNING *`,- [id, type, String(name).trim(), JSON.stringify(storedPayload), tags || []],+ [id, type, String(name).trim(), toPgJson(storedPayload), tags || []],);return result.rows.length > 0 ? rowToIngredient(result.rows[0]) : null;}diff --git a/server/services/catalogDB/media.js b/server/services/catalogDB/media.jsindex 35220220b3..dbd18e7474 100644--- a/server/services/catalogDB/media.js+++ b/server/services/catalogDB/media.js@@ -13,6 +13,7 @@ import { resolveImageInputPath } from '../../lib/fileUtils.js';import { rowToMedia } from './shared.js';import { MAX_CAST_GUIDES, normalizeLibrary } from '../../lib/catalogMediaLibrary.js';import { assertCatalogIdentityWritable, assertCatalogAssociationRemovable } from './identityAccess.js';+import { toPgJson } from '../../lib/pgJson.js';const hasOwn = (value, key) => Object.prototype.hasOwnProperty.call(value || {}, key);const hasMeaningfulMetadata = (value) => value && typeof value === 'object'@@ -44,7 +45,7 @@ async function insertMedia(runQuery, ingredientId, mediaKey, kind, options) {RETURNING *`,[ingredientId, mediaKey, kind, role, caption,- ...(hasMetadata ? [JSON.stringify(options.metadata || {})] : []),+ ...(hasMetadata ? [toPgJson(options.metadata || {})] : []),],);return rowToMedia(result.rows[0]);@@ -176,7 +177,7 @@ export async function updateMediaLibrary(ingredientId, mediaKey, kind, patch = {`UPDATE catalog_ingredient_media SET library = $4::jsonbWHERE ingredient_id = catalog_resolve_ingredient_id($1) AND media_key = $2 AND kind = $3 AND deleted = falseRETURNING *`,- [ingredientId, mediaKey, kind, JSON.stringify(next)],+ [ingredientId, mediaKey, kind, toPgJson(next)],);return rowToMedia(result.rows[0]);});diff --git a/server/services/catalogDB/merge.js b/server/services/catalogDB/merge.jsindex a06b22076c..0d34ea27db 100644--- a/server/services/catalogDB/merge.js+++ b/server/services/catalogDB/merge.js@@ -8,6 +8,7 @@ import { getInstanceId, UNKNOWN_INSTANCE_ID } from '../instanceIdentity.js';import { listSheetPointers } from '../../lib/sheetPointers.js';import { sanitizeBiblePayload } from './shared.js';import { catalogMergeSchema } from '../../lib/catalogValidation.js';+import { toPgJson } from '../../lib/pgJson.js';const TABLES = Object.freeze({ingredients: ['catalog_ingredients', 'id = ANY($1)'],@@ -123,7 +124,7 @@ export async function mergeIngredients(input) {await client.query(`UPDATE catalog_ingredients SET name = $2, payload = $3::jsonb, tags = $4,embedding = NULL, embedding_model = NULL WHERE id = $1`,- [survivorId, proposed.name, JSON.stringify(proposed.payload), proposed.tags]);+ [survivorId, proposed.name, toPgJson(proposed.payload), proposed.tags]);// Copy only live associations; retain the original rows and collision// provenance in the receipt rather than overwriting survivor captions/spans.await client.query(`INSERT INTO catalog_ingredient_refs (ingredient_id, ref_kind, ref_id, role, created_at)@@ -158,7 +159,7 @@ export async function mergeIngredients(input) {await client.query(`INSERT INTO catalog_ingredient_media (ingredient_id, media_key, kind, role, caption, metadata, created_at, library)VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8::jsonb)ON CONFLICT (ingredient_id, media_key, kind) DO UPDATE SET deleted = false, deleted_at = NULL`,- [survivorId, row.media_key, row.kind, row.role, row.caption, JSON.stringify(row.metadata), row.created_at, JSON.stringify(row.library || {})]);+ [survivorId, row.media_key, row.kind, row.role, row.caption, toPgJson(row.metadata), row.created_at, toPgJson(row.library || {})]);}await client.query(`INSERT INTO catalog_ingredient_relations (from_id, to_id, kind, created_at)SELECT CASE WHEN from_id = $1 THEN $2 ELSE from_id END,@@ -168,12 +169,12 @@ export async function mergeIngredients(input) {ON CONFLICT (from_id, to_id, kind) DO UPDATE SET deleted = false, deleted_at = NULL`, [sourceId, survivorId, ids]);await client.query(`INSERT INTO catalog_ingredient_revisions (id, ingredient_id, name, payload, tags, source, actor)VALUES ($1, $2, $3, $4::jsonb, $5, 'user', 'catalog-merge')`,- [`cat-rev-${randomUUID()}`, survivorId, proposed.name, JSON.stringify(proposed.payload), proposed.tags]);+ [`cat-rev-${randomUUID()}`, survivorId, proposed.name, toPgJson(proposed.payload), proposed.tags]);const after = await snapshot(client, ids);const id = randomUUID();await client.query(`INSERT INTO catalog_ingredient_merges(id, source_id, survivor_id, before_state, after_fingerprint) VALUES ($1, $2, $3, $4::jsonb, $5)`,- [id, sourceId, survivorId, JSON.stringify({ ...before, decision: { scope: 'local', fieldChoices, proposed } }), fingerprint(after)]);+ [id, sourceId, survivorId, toPgJson({ ...before, decision: { scope: 'local', fieldChoices, proposed } }), fingerprint(after)]);return { id, sourceId, survivorId, merged: true, scope: 'local' };});}@@ -227,19 +228,19 @@ export async function recoverIngredientMerge(id) {const updates = columns.filter(column => !keys.includes(column)).map(column => `${column} = EXCLUDED.${column}`);await client.query(`INSERT INTO ${table} (${columns.join(', ')})SELECT ${columns.join(', ')} FROM jsonb_populate_recordset(NULL::${table}, $1::jsonb)- ON CONFLICT (${keys.join(', ')}) DO UPDATE SET ${updates.join(', ')}`, [JSON.stringify(rows)]);+ ON CONFLICT (${keys.join(', ')}) DO UPDATE SET ${updates.join(', ')}`, [toPgJson(rows)]);}// Keep the merge revision as an audit event. Recovery adds a new snapshot;// the receipt retains the complete before/after identity history.for (const row of receipt.before_state.ingredients) {await client.query(`UPDATE catalog_ingredients SET name = $2, payload = $3::jsonb, tags = $4,embedding = $5, embedding_model = $6, deleted = $7, deleted_at = $8 WHERE id = $1`,- [row.id, row.name, JSON.stringify(row.payload), row.tags, row.embedding, row.embedding_model, row.deleted, row.deleted_at]);+ [row.id, row.name, toPgJson(row.payload), row.tags, row.embedding, row.embedding_model, row.deleted, row.deleted_at]);}for (const row of receipt.before_state.ingredients) {await client.query(`INSERT INTO catalog_ingredient_revisions (id, ingredient_id, name, payload, tags, source, actor)VALUES ($1, $2, $3, $4::jsonb, $5, 'user', 'catalog-merge-recovery')`,- [`cat-rev-${randomUUID()}`, row.id, row.name, JSON.stringify(row.payload), row.tags]);+ [`cat-rev-${randomUUID()}`, row.id, row.name, toPgJson(row.payload), row.tags]);}return { id, restored: true };});diff --git a/server/services/catalogDB/refs.js b/server/services/catalogDB/refs.jsindex 5460a2d081..09211a246a 100644--- a/server/services/catalogDB/refs.js+++ b/server/services/catalogDB/refs.js@@ -14,6 +14,7 @@ import { query } from '../../lib/db.js';import { resolveRefs } from '../catalogRefResolver.js';import { assertCatalogIdentityWritable, assertCatalogAssociationRemovable, assertCatalogRelationRemovable } from './identityAccess.js';import { rowToRef, rowToSource, rowToIngredient } from './shared.js';+import { toPgJson } from '../../lib/pgJson.js';// Canonical detail views omit local-merge originals; original-ID user exports// retain those raw rows alongside the bilateral receipt-backed evidence.@@ -32,7 +33,7 @@ export async function linkIngredientToSource(ingredientId, scrapId, span = null,`INSERT INTO catalog_ingredient_sources (ingredient_id, scrap_id, span)VALUES (catalog_resolve_ingredient_id($1), $2, $3::jsonb)ON CONFLICT (ingredient_id, scrap_id) ${preserveExisting ? 'DO NOTHING' : 'DO UPDATE SET span = EXCLUDED.span'}`,- [ingredientId, scrapId, span ? JSON.stringify(span) : null],+ [ingredientId, scrapId, span ? toPgJson(span) : null],);}diff --git a/server/services/catalogDB/scraps.js b/server/services/catalogDB/scraps.jsindex c26354216d..46010ae07a 100644--- a/server/services/catalogDB/scraps.js+++ b/server/services/catalogDB/scraps.js@@ -10,6 +10,7 @@ import { query, withTransaction, arrayToPgvector } from '../../lib/db.js';import { chunkRawText } from '../../lib/catalogChunking.js';import { getInstanceId } from '../instanceIdentity.js';import { newScrapId, rowToScrap } from './shared.js';+import { toPgJson } from '../../lib/pgJson.js';// `{ client }` is optional — when supplied, the INSERT runs on the caller's// transaction client so a bulk-import batch can commit-or-rollback its scraps@@ -31,7 +32,7 @@ export async function createScrap({ id: explicitId, title, rawText, sourceKind =title || null,rawText,sourceKind,- JSON.stringify(metadata || {}),+ toPgJson(metadata || {}),embedding ? arrayToPgvector(embedding) : null,embeddingModel,originInstanceId,@@ -141,7 +142,7 @@ export async function updateScrap(id, patch = {}) {if (patch[jsField] === undefined) continue;if (jsField === 'metadata') {fields.push(`${dbField} = $${idx++}::jsonb`);- params.push(JSON.stringify(patch.metadata || {}));+ params.push(toPgJson(patch.metadata || {}));} else if (jsField === 'embedding') {fields.push(`${dbField} = $${idx++}`);params.push(patch.embedding ? arrayToPgvector(patch.embedding) : null);diff --git a/server/services/catalogDB/sync.js b/server/services/catalogDB/sync.jsindex 12ee3eb6d4..02d92e3cc0 100644--- a/server/services/catalogDB/sync.js+++ b/server/services/catalogDB/sync.js@@ -28,6 +28,7 @@ import {rowToMedia,rowToTag,} from './shared.js';+import { toPgJson } from '../../lib/pgJson.js';// One page of a table's feed: rows whose committed feed position is past the// peer's cursor, in position order. `syncSequence` on each item carries the@@ -122,14 +123,14 @@ export async function upsertMediaFromPeer(media) {const metadataColumn = hasMetadata ? ', metadata' : '';const metadataPlaceholder = hasMetadata ? `, $${hasTombstoneFields ? 10 : 7}` : '';const metadataUpdate = hasMetadata ? ', metadata = EXCLUDED.metadata' : '';- const metadataParam = hasMetadata ? [JSON.stringify(media.metadata || {})] : [];+ const metadataParam = hasMetadata ? [toPgJson(media.metadata || {})] : [];// Casting reference library (catalog v12): absent = no opinion, same as metadata.const hasLibrary = Object.prototype.hasOwnProperty.call(media, 'library')&& media.library && typeof media.library === 'object' && !Array.isArray(media.library);const libraryColumn = hasLibrary ? ', library' : '';const libraryPlaceholder = hasLibrary ? `, $${(hasTombstoneFields ? 10 : 7) + (hasMetadata ? 1 : 0)}` : '';const libraryUpdate = hasLibrary ? ', library = EXCLUDED.library' : '';- const libraryParam = hasLibrary ? [JSON.stringify(normalizeLibrary(media.library))] : [];+ const libraryParam = hasLibrary ? [toPgJson(normalizeLibrary(media.library))] : [];if (hasTombstoneFields) {// #8347 revival guard — see upsertRelationFromPeer for the rationale.// `updated_at` also moves on a role/caption/metadata-only edit (the@@ -267,7 +268,7 @@ export async function upsertScrapFromPeer(scrap) {scrap.title || null,scrap.rawText,scrap.sourceKind || 'paste',- JSON.stringify(scrap.metadata || {}),+ toPgJson(scrap.metadata || {}),scrap.embedding ? arrayToPgvector(scrap.embedding) : null,scrap.embeddingModel || null,scrap.originInstanceId || null,@@ -319,7 +320,7 @@ export async function upsertIngredientFromPeer(ing) {ing.id,ing.type,ing.name,- JSON.stringify(ing.payload || {}),+ toPgJson(ing.payload || {}),ing.tags || [],ing.embedding ? arrayToPgvector(ing.embedding) : null,ing.embeddingModel || null,@@ -340,7 +341,7 @@ export async function upsertSourceFromPeer(src) {`INSERT INTO catalog_ingredient_sources (ingredient_id, scrap_id, span, extracted_at)VALUES ($1, $2, $3::jsonb, $4)ON CONFLICT (ingredient_id, scrap_id) DO UPDATE SET span = EXCLUDED.span`,- [src.ingredientId, src.scrapId, src.span ? JSON.stringify(src.span) : null, src.extractedAt],+ [src.ingredientId, src.scrapId, src.span ? toPgJson(src.span) : null, src.extractedAt],);});}@@ -422,7 +423,7 @@ async function deferParentApply(kind, row, parentId, updatedAt) {source_updated_at = EXCLUDED.source_updated_at,payload = EXCLUDED.payloadWHERE EXCLUDED.source_updated_at > catalog_pending_applies.source_updated_at`,- [kind, row.id, parentId, updatedAt, JSON.stringify(row)],+ [kind, row.id, parentId, updatedAt, toPgJson(row)],);}@@ -453,7 +454,7 @@ export async function drainPendingCatalogApplies() {const removed = await query(`DELETE FROM catalog_pending_appliesWHERE kind = $1 AND id = $2 AND payload = $3::jsonb`,- [kind, pending.id, JSON.stringify(pending.payload)],+ [kind, pending.id, toPgJson(pending.payload)],);progressed ||= removed.rowCount > 0;}diff --git a/server/services/catalogIdentity/index.js b/server/services/catalogIdentity/index.jsindex 819a04ae4b..959b413495 100644--- a/server/services/catalogIdentity/index.js+++ b/server/services/catalogIdentity/index.js@@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto';import { query, withTransaction } from '../../lib/db.js';import { snapshot, lockCatalog, ingredient, assertUnlocked, buildPlan, hash, fail, validateSnapshot } from './snapshots.js';import { localIdentity, identityPeer, readEvidence, revalidateEvidence } from './transport.js';+import { toPgJson } from '../../lib/pgJson.js';const db = { query };const idsOf = row => [row.source_id, row.survivor_id];@@ -49,7 +50,7 @@ async function update(client, row, state, epoch, extra = {}) {const columns = { state, epoch, ...extra };const values = [row.id];const sets = Object.entries(columns).map(([key, value]) => {- values.push(value && typeof value === 'object' ? JSON.stringify(value) : value);+ values.push(value && typeof value === 'object' ? toPgJson(value) : value);return `${key} = $${values.length}`;});await client.query(`UPDATE catalog_identity_operations SET ${sets.join(', ')}, updated_at = NOW() WHERE id = $1`, values);@@ -60,10 +61,10 @@ async function writeContent(client, row, original = false) {const content = original ? ingredient(row.before_state, row.survivor_id) : row.plan.proposed;await client.query(`UPDATE catalog_ingredients SET name = $2, payload = $3::jsonb, tags = $4,embedding = NULL, embedding_model = NULL WHERE id = $1`,- [row.survivor_id, content.name, JSON.stringify(content.payload), content.tags]);+ [row.survivor_id, content.name, toPgJson(content.payload), content.tags]);await client.query(`INSERT INTO catalog_ingredient_revisions (id, ingredient_id, name, payload, tags, source, actor)VALUES ($1, $2, $3, $4::jsonb, $5, 'user', $6)`,- [`cat-rev-${randomUUID()}`, row.survivor_id, content.name, JSON.stringify(content.payload), content.tags,+ [`cat-rev-${randomUUID()}`, row.survivor_id, content.name, toPgJson(content.payload), content.tags,original ? 'catalog-identity-recovery' : 'catalog-shared-identity']);}async function adoptContent(client, row, content) {@@ -72,7 +73,7 @@ async function adoptContent(client, row, content) {// snapshot retain the actual revision identity and owner timestamp.await client.query(`UPDATE catalog_ingredients SET name=$2,payload=$3::jsonb,tags=$4,updated_at=$5,embedding=NULL,embedding_model=NULL WHERE id=$1`,- [row.survivor_id, content.name, JSON.stringify(content.payload), content.tags, content.updated_at]);+ [row.survivor_id, content.name, toPgJson(content.payload), content.tags, content.updated_at]);}export async function listIdentityOperations({ ingredientId, limit = 20, offset = 0 } = {}) {const { rows } = await query(`SELECT id, source_id, survivor_id, source_owner, survivor_owner, local_owner,@@ -127,7 +128,7 @@ export async function createIdentity(input) {await client.query(`INSERT INTO catalog_identity_operations(id, source_id, survivor_id, source_owner, survivor_owner, local_owner, state, plan, plan_hash)VALUES ($1,$2,$3,$4,$5,$6,'proposed',$7::jsonb,$8)`,- [id, plan.sourceId, plan.survivorId, plan.sourceOwner, plan.survivorOwner, self, JSON.stringify(plan), preview.planHash]);+ [id, plan.sourceId, plan.survivorId, plan.sourceOwner, plan.survivorOwner, self, toPgJson(plan), preview.planHash]);return present(await load(client, id));});}@@ -156,7 +157,7 @@ export async function importIdentity({ peerId, operationId }) {await client.query(`INSERT INTO catalog_identity_operations(id,source_id,survivor_id,source_owner,survivor_owner,local_owner,state,plan,plan_hash)VALUES ($1,$2,$3,$4,$5,$6,'proposed',$7::jsonb,$8)`,- [operationId, remote.sourceId, remote.survivorId, self, peerId, self, JSON.stringify(remote.plan), remote.planHash]);+ [operationId, remote.sourceId, remote.survivorId, self, peerId, self, toPgJson(remote.plan), remote.planHash]);return present(await load(client, operationId));});}diff --git a/server/services/catalogUserTypes/db.js b/server/services/catalogUserTypes/db.jsindex ad7ca0dcd0..9a9f33b701 100644--- a/server/services/catalogUserTypes/db.js+++ b/server/services/catalogUserTypes/db.js@@ -16,6 +16,7 @@import { query, withTransaction } from '../../lib/db.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';function rowToType(row) {// `data` already holds id/label/primaryContentKey/fields/updatedAt/deletedAt;@@ -86,7 +87,7 @@ async function applyWrite(exec, list) {data = EXCLUDED.data,updated_at = EXCLUDED.updated_at,deleted_at = EXCLUDED.deleted_at`,- [t.id, JSON.stringify(t), updatedAt, deletedAt],+ [t.id, toPgJson(t), updatedAt, deletedAt],);}// Prune rows that left the desired slice. With keepIds empty the wholediff --git a/server/services/codeAnimation/jobStore.js b/server/services/codeAnimation/jobStore.jsindex 3c86e131a7..481bab8914 100644--- a/server/services/codeAnimation/jobStore.js+++ b/server/services/codeAnimation/jobStore.js@@ -9,6 +9,7 @@ import { query } from '../../lib/db.js';import { PATHS } from '../../lib/paths.js';import { atomicWrite } from '../../lib/fileUtils.js';import { ServerError } from '../../lib/errorHandler.js';+import { toPgJson } from '../../lib/pgJson.js';const JOB_ID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;const ANIMATIONS_DIR = join(PATHS.data, 'code-animations');@@ -81,7 +82,7 @@ export async function saveCodeAnimationJobRecord(job) {model = EXCLUDED.model,data = EXCLUDED.data,updated_at = EXCLUDED.updated_at`,- [job.id, job.status, job.title, job.input?.concept || '', job.providerId, job.model, JSON.stringify(job), job.createdAt, job.updatedAt],+ [job.id, job.status, job.title, job.input?.concept || '', job.providerId, job.model, toPgJson(job), job.createdAt, job.updatedAt],);return job;}diff --git a/server/services/codeAnimation/projectStore.js b/server/services/codeAnimation/projectStore.jsindex db444af9e5..3a3478db4e 100644--- a/server/services/codeAnimation/projectStore.js+++ b/server/services/codeAnimation/projectStore.js@@ -2,6 +2,7 @@import { randomUUID } from 'crypto';import { query, withTransaction } from '../../lib/db.js';import { ServerError } from '../../lib/errorHandler.js';+import { toPgJson } from '../../lib/pgJson.js';const columns = `id, title, data, accepted_revision_id AS "acceptedRevisionId",candidate_revision_id AS "candidateRevisionId", created_at AS "createdAt", updated_at AS "updatedAt"`;@@ -210,8 +211,8 @@ export async function setAcceptedOutput(projectId, accepted) {const result = await client.query(`UPDATE code_animation_projects SET data = data || jsonb_build_object('acceptedOutput', $2::jsonb, 'acceptanceProjection', $5::jsonb),accepted_revision_id = CASE WHEN $4 THEN $3 ELSE accepted_revision_id END,candidate_revision_id = CASE WHEN $4 THEN NULL ELSE candidate_revision_id END, updated_at = NOW()- WHERE id = $1 RETURNING ${columns}`, [projectId, JSON.stringify(accepted), accepted.revisionId, promoteSource,- JSON.stringify({ decisionId: randomUUID(), status: 'pending' })]);+ WHERE id = $1 RETURNING ${columns}`, [projectId, toPgJson(accepted), accepted.revisionId, promoteSource,+ toPgJson({ decisionId: randomUUID(), status: 'pending' })]);return present(result.rows[0]);});}diff --git a/server/services/cosTaskRepository.db.test.js b/server/services/cosTaskRepository.db.test.jsindex f9ceb5a47e..f6f428d97b 100644--- a/server/services/cosTaskRepository.db.test.js+++ b/server/services/cosTaskRepository.db.test.js@@ -3,11 +3,12 @@* _cosTaskRepositoryContract.js. Runs via `npm run test:db` (→ `portos_test`)* ONLY; skips cleanly when no database is reachable.*/-import { afterAll, describe } from 'vitest';+import { afterAll, describe, expect, it } from 'vitest';import { checkHealth, close, ensureSchema, query, withTransaction } from '../lib/db.js';import { requireDbOrSkip } from '../lib/dbTestGate.js';import { __makePgCosTaskBackend } from './cosTaskRepository.js';import { __runCosTaskRepositoryContract } from './_cosTaskRepositoryContract.js';+import { planListSave } from '../lib/cosTaskRows.js';const health = await checkHealth().catch(() => ({ connected: false }));if (health.connected) await ensureSchema();@@ -28,3 +29,30 @@ describe.skipIf(!runDb)('cos_tasks repository on PostgreSQL', () => {},});});++// PostgreSQL refuses a lone surrogate in jsonb and NUL in jsonb/text; the file+// stores these queues came from accepted both (#12815).+describe.skipIf(!runDb)('cos_tasks repository text PostgreSQL would refuse', () => {+ const task = (n, description) => ({+ id: `task-example-${n}`, status: 'pending', priority: 'MEDIUM', priorityValue: 2, description,+ approvalRequired: false, autoApproved: true, metadata: {},+ });++ it('repairs a split emoji and a NUL to U+FFFD instead of failing the save', async () => {+ await query('DELETE FROM cos_tasks');+ await query('DELETE FROM cos_task_lists');+ const backend = __makePgCosTaskBackend({ query, withTransaction });+ // The cut askPromote makes: 279 characters, then half of an emoji's surrogate pair.+ const split = `${'a'.repeat(279)}${'😀'.slice(0, 1)}`;+ const next = [task(1, split), task(2, 'before\0after')];+ const { rows } = await backend.loadWorkingSet('user');+ const plan = planListSave({ previous: rows, next, source: 'user' });+ await backend.apply('user', { upserts: plan.upserts, deletes: plan.deletes });++ const saved = Object.fromEntries((await backend.loadWorkingSet('user')).rows.map((row) => [row.task.id, row.task]));+ expect(saved['task-example-1'].description).toBe(`${'a'.repeat(279)}\uFFFD`);+ expect(saved['task-example-2'].description).toBe('before\uFFFDafter');+ const { rows: [column] } = await query('SELECT description FROM cos_tasks WHERE task_id = $1', ['task-example-2']);+ expect(column.description).toBe('before\uFFFDafter');+ });+});diff --git a/server/services/cosTaskRepository.js b/server/services/cosTaskRepository.jsindex faab401a43..affe704dd4 100644--- a/server/services/cosTaskRepository.js+++ b/server/services/cosTaskRepository.js@@ -20,6 +20,7 @@ import { isTestRunner } from '../lib/runtimeEnv.js';import {WORKING_SET_COMPLETED_WINDOW_MS, WORKING_SET_KEEP_COMPLETED, completionStamp, rowColumns, selectWorkingSet,} from '../lib/cosTaskRows.js';+import { toPgJson } from '../lib/pgJson.js';const SOURCES = ['user', 'internal'];// Bounds a single statement's parameter payload (prompts can be MBs).@@ -65,7 +66,7 @@ function makePgBackend(db) {position = EXCLUDED.position, completed_at = EXCLUDED.completed_at, description = EXCLUDED.description, task = EXCLUDED.task`,[source, columns.map((c) => c.task.id), columns.map((c) => c.status), columns.map((c) => c.priorityValue),columns.map((c) => c.position), columns.map((c) => c.completedAt), columns.map((c) => c.description),- columns.map((c) => JSON.stringify(c.task))],+ columns.map((c) => toPgJson(c.task))],);}return bumpVersions(client, [source]);diff --git a/server/services/creativeCommissions/db.js b/server/services/creativeCommissions/db.jsindex cf4c67627a..26f515efd1 100644--- a/server/services/creativeCommissions/db.js+++ b/server/services/creativeCommissions/db.js@@ -23,6 +23,7 @@import { query } from '../../lib/db.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';/** Raw stored record (the `data` JSONB), or null. Live only unless includeDeleted (#2686). */export async function readRaw(id, { includeDeleted = false } = {}) {@@ -81,7 +82,7 @@ export async function writeRaw(id, record) {id,typeof record?.name === 'string' ? record.name : '',record?.enabled !== false,- JSON.stringify(record),+ toPgJson(record),createdAt,mirrorTimestamp(record?.updatedAt, createdAt),deleted,diff --git a/server/services/creativeCommissions/feedbackDb.js b/server/services/creativeCommissions/feedbackDb.jsindex 3b51e20be8..723d307bbc 100644--- a/server/services/creativeCommissions/feedbackDb.js+++ b/server/services/creativeCommissions/feedbackDb.js@@ -24,6 +24,7 @@import { query } from '../../lib/db.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';/** Raw stored record (the `data` JSONB), or null. Live only unless includeDeleted. */export async function readRaw(id, { includeDeleted = false } = {}) {@@ -90,7 +91,7 @@ export async function writeRaw(id, record) {id,typeof record?.commissionId === 'string' && record.commissionId ? record.commissionId : null,typeof record?.runId === 'string' && record.runId ? record.runId : null,- JSON.stringify(record),+ toPgJson(record),createdAt,mirrorTimestamp(record?.updatedAt, createdAt),deleted,diff --git a/server/services/deepAudit.js b/server/services/deepAudit.jsindex ec0f8dcb51..f87e2f9f5d 100644--- a/server/services/deepAudit.js+++ b/server/services/deepAudit.js@@ -10,6 +10,7 @@ import { normalizeAuditTaskType, isAuditTaskType } from '../lib/auditCatalog.js'import { deepAuditProgress, mergeDeepAuditReport, DEEP_AUDIT_CONTRACT_VERSION } from '../lib/deepAudit.js';import { PROMPT_VERSIONS } from './taskPromptDefaults/versions.js';import { resolveTaskHookType } from './taskTypeHooks.js';+import { toPgJson } from '../lib/pgJson.js';function deepAuditId(task) {const meta = task.metadata || {};@@ -59,11 +60,11 @@ async function mutateLedger(id, initial, mutate) {await ensureSchema();return withTransaction(async client => {if (initial) await client.query(`INSERT INTO deep_audit_ledgers (id, app_id, category, ledger)- VALUES ($1,$2,$3,$4::jsonb) ON CONFLICT (id) DO NOTHING`, [id, initial.appId, initial.category, JSON.stringify(initial)]);+ VALUES ($1,$2,$3,$4::jsonb) ON CONFLICT (id) DO NOTHING`, [id, initial.appId, initial.category, toPgJson(initial)]);const { rows } = await client.query('SELECT ledger FROM deep_audit_ledgers WHERE id=$1 FOR UPDATE', [id]);if (!rows.length) throw new Error('Deep audit ledger is unavailable');const updated = await mutate(rows[0].ledger);- await client.query('UPDATE deep_audit_ledgers SET ledger=$2::jsonb, updated_at=NOW() WHERE id=$1', [id, JSON.stringify(updated)]);+ await client.query('UPDATE deep_audit_ledgers SET ledger=$2::jsonb, updated_at=NOW() WHERE id=$1', [id, toPgJson(updated)]);return updated;});}diff --git a/server/services/fableLoom/db.js b/server/services/fableLoom/db.jsindex a29efc6e0f..8f7c6a9778 100644--- a/server/services/fableLoom/db.js+++ b/server/services/fableLoom/db.js@@ -6,6 +6,7 @@import { query } from '../../lib/db.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';export async function readRaw(id) {const { rows } = await query('SELECT data FROM fableloom_stories WHERE id = $1', [id]);@@ -69,7 +70,7 @@ export async function writeRaw(id, record) {record.name,record.universeId ?? null,record.seriesId ?? null,- JSON.stringify(record),+ toPgJson(record),createdAt,updatedAt,record.deleted === true,diff --git a/server/services/games/db.js b/server/services/games/db.jsindex da5ff7fce7..a7fafbaa85 100644--- a/server/services/games/db.js+++ b/server/services/games/db.js@@ -5,6 +5,7 @@import { query } from '../../lib/db.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';export async function readRaw(id) {const { rows } = await query('SELECT data FROM games WHERE id = $1', [id]);@@ -28,7 +29,7 @@ export async function writeRaw(id, record) {name = EXCLUDED.name,data = EXCLUDED.data,updated_at = EXCLUDED.updated_at`,- [id, record.appId, record.name, JSON.stringify(record), createdAt, updatedAt],+ [id, record.appId, record.name, toPgJson(record), createdAt, updatedAt],);return record;}diff --git a/server/services/humanActivity.js b/server/services/humanActivity.jsindex 1ed2720c1c..ef48735cf1 100644--- a/server/services/humanActivity.js+++ b/server/services/humanActivity.js@@ -26,6 +26,7 @@ import { ensureSchema, query, withTransaction } from '../lib/db.js';import { normalizeIdentifier } from '../lib/tribeMatch.js';import { getLocalParts, localDayRangeUtc, todayInTimezone, anchorLocalMidnightUtc } from '../lib/timezone.js';import { getUserTimezone } from './userTimezone.js';+import { toPgJson } from '../lib/pgJson.js';export { localDayRangeUtc };@@ -337,7 +338,7 @@ export async function recordEvents(candidates = []) {// Insert one chunk of normalized rows; returns the count actually inserted.// 12 bound params per row (created_at is a NOW() literal, not a param);-// participants and metadata are JSON.stringify'd + ::jsonb-cast (node-pg would+// participants and metadata are toPgJson-serialized + ::jsonb-cast (node-pg would// otherwise coerce a JS array to a Postgres array literal and fail on jsonb).async function insertActivityChunk(rows) {const paramsPerRow = 12;@@ -346,8 +347,8 @@ async function insertActivityChunk(rows) {const b = i * paramsPerRow;values.push(r.id, r.source, r.accountId, r.kind, r.happenedAt, r.durationS,- r.title, r.summary, r.url, JSON.stringify(r.participants),- JSON.stringify(r.metadata), r.dedupeKey,+ r.title, r.summary, r.url, toPgJson(r.participants),+ toPgJson(r.metadata), r.dedupeKey,);return `($${b + 1}, $${b + 2}, $${b + 3}, $${b + 4}, $${b + 5}, $${b + 6}, $${b + 7}, $${b + 8}, $${b + 9}, $${b + 10}::jsonb, $${b + 11}::jsonb, $${b + 12}, NOW())`;});@@ -447,7 +448,7 @@ export async function listEvents({ from, to, source, accountId, kind, personId,}if (kind) { params.push(kind); clauses.push(`kind = $${params.length}`); }if (personId) {- params.push(JSON.stringify([{ personId: String(personId) }]));+ params.push(toPgJson([{ personId: String(personId) }]));clauses.push(`participants @> $${params.length}::jsonb`);}if (chatGuid != null && String(chatGuid).length > 0) {diff --git a/server/services/imageTo3d/db.js b/server/services/imageTo3d/db.jsindex 179399cec2..c9d6c412c8 100644--- a/server/services/imageTo3d/db.js+++ b/server/services/imageTo3d/db.js@@ -12,6 +12,7 @@ import { modelLifecycleEvents } from '../modelLifecycleEvents.js';import { randomUUID } from 'crypto';import { query, withTransaction } from '../../lib/db.js';import { ServerError } from '../../lib/errorHandler.js';+import { toPgJson } from '../../lib/pgJson.js';const rowToModel = (row) => row?.data ?? null;@@ -30,7 +31,7 @@ async function persist(exec, model) {model.id,model.name,model.status,- JSON.stringify(model),+ toPgJson(model),model.createdAt,model.updatedAt,model.deleted === true,diff --git a/server/services/loraTraining/db.js b/server/services/loraTraining/db.jsindex fa3cfaa194..5655753658 100644--- a/server/services/loraTraining/db.js+++ b/server/services/loraTraining/db.js@@ -18,6 +18,7 @@import { query } from '../../lib/db.js';import { ServerError } from '../../lib/errorHandler.js';import { createKeyCachedQueue } from '../../lib/createKeyCachedQueue.js';+import { toPgJson } from '../../lib/pgJson.js';const rowToRun = (row) => row?.data ?? null;@@ -39,7 +40,7 @@ async function persist(run) {run.id,run.status,run.character?.entryId || null,- JSON.stringify(run),+ toPgJson(run),run.createdAt || new Date().toISOString(),new Date().toISOString(),],diff --git a/server/services/mediaAssetIndex/db.js b/server/services/mediaAssetIndex/db.jsindex 1a013969dd..bee32c9abb 100644--- a/server/services/mediaAssetIndex/db.js+++ b/server/services/mediaAssetIndex/db.js@@ -35,6 +35,7 @@ import {imageToRow, videoToRow, compactGalleryRecord, compactSource, compactSourceToRow,COMPACT_SOURCE_FIELDS, COMPACT_PREVIEW_KEY, COMPACT_PROMPT_CHARS,} from './logic.js';+import { toPgJson } from '../../lib/pgJson.js';function rowToAsset(row) {if (!row) return null;@@ -57,7 +58,7 @@ export async function upsertAsset(row) {`INSERT INTO media_assets (media_key, kind, ref, data, created_at, indexed_at)VALUES ($1, $2, $3, $4::jsonb, $5, NOW())${UPSERT_CONFLICT}`,- [row.mediaKey, row.kind, row.ref, JSON.stringify(row.data), row.createdAt],+ [row.mediaKey, row.kind, row.ref, toPgJson(row.data), row.createdAt],);}@@ -81,7 +82,7 @@ async function upsertAssets(rows, runQuery = query) {const b = j * 5;// NOW() for indexed_at is a literal, not a param.values.push(`($${b + 1}, $${b + 2}, $${b + 3}, $${b + 4}::jsonb, $${b + 5}, NOW())`);- params.push(row.mediaKey, row.kind, row.ref, JSON.stringify(row.data), row.createdAt);+ params.push(row.mediaKey, row.kind, row.ref, toPgJson(row.data), row.createdAt);});await runQuery(`INSERT INTO media_assets (media_key, kind, ref, data, created_at, indexed_at)@@ -181,7 +182,7 @@ const rowShape = (compact, projected) => !compact ? data => datafunction assetSource(params, videos, { materialized = false, compact = false } = {}) {const data = compact ? `${compactSourceSql('data')} AS data` : 'data';if (videos === undefined) return { cte: '', table: 'media_assets', data };- params.push(JSON.stringify(videos.map(video => ({ data: compact ? compactSource(video) : video,+ params.push(toPgJson(videos.map(video => ({ data: compact ? compactSource(video) : video,createdAt: Number.isFinite(Date.parse(video.createdAt)) ? new Date(video.createdAt).toISOString() : new Date(0).toISOString() }))));return {data: 'data',@@ -305,7 +306,7 @@ export async function collectionCovers(collections, videos) {SELECT 'unsorted',(SELECT cover FROM unsorted WHERE kind = 'image' OR cover IS NOT NULL ORDER BY created_at DESC, media_key ASC LIMIT 1),COUNT(*) FILTER (WHERE kind = 'image'), COUNT(*) FILTER (WHERE kind = 'video')- FROM unsorted`, [JSON.stringify(collections), JSON.stringify(videoRows)]);+ FROM unsorted`, [toPgJson(collections), toPgJson(videoRows)]);return result.rows;}@@ -316,7 +317,7 @@ export async function visibleImageCollections(collections) {const result = await query(`SELECT DISTINCT m.idFROM jsonb_to_recordset($1::jsonb) AS m(id text, key text)JOIN media_assets a ON a.media_key = m.key- WHERE a.kind = 'image' AND COALESCE(a.data->>'hidden', 'false') <> 'true'`, [JSON.stringify(memberships)]);+ WHERE a.kind = 'image' AND COALESCE(a.data->>'hidden', 'false') <> 'true'`, [toPgJson(memberships)]);const populated = new Set(result.rows.map(row => row.id));return collections.filter(collection => populated.has(collection.id)).map(({ id, name }) => ({ id, name }));}diff --git a/server/services/memoryDB.js b/server/services/memoryDB.jsindex 11595624b9..6b528e053a 100644--- a/server/services/memoryDB.js+++ b/server/services/memoryDB.js@@ -17,6 +17,7 @@ import * as notifications from './notifications.js';import { DEFAULT_MEMORY_CONFIG, generateSummary, decrementAgentPendingApproval } from './memoryConfig.js';import { getInstanceId } from './instanceIdentity.js';import { DEFAULT_MEMORY_LINK_TYPE, isSymmetricMemoryLinkType } from '../lib/memoryLinkTypes.js';+import { toPgJson } from '../lib/pgJson.js';const protectedMindMemoryTags = Object.values(PERSISTENT_MIND_MEMORY_PROTECTION_TAGS);@@ -173,7 +174,7 @@ export async function getBrainMemoryLinks(legacyMap = {}, { readOnly = false, reSELECT legacy.key, m.id FROM jsonb_each_text($1::jsonb) legacyJOIN memories m ON m.id::text = legacy.value AND m.source_app_id = 'brain'ORDER BY legacy.key- ON CONFLICT (bridge_key) DO NOTHING`, [JSON.stringify(legacyMap)]);+ ON CONFLICT (bridge_key) DO NOTHING`, [toPgJson(legacyMap)]);}const result = await client.query(`SELECT l.bridge_key, l.memory_id FROM brain_memory_links l@@ -182,7 +183,7 @@ export async function getBrainMemoryLinks(legacyMap = {}, { readOnly = false, reSELECT legacy.key, m.id FROM jsonb_each_text($1::jsonb) legacyJOIN memories m ON m.id::text = legacy.value AND m.source_app_id = 'brain'WHERE NOT EXISTS (SELECT 1 FROM brain_memory_links l WHERE l.bridge_key = legacy.key)`,- [JSON.stringify(legacyMap)]);+ [toPgJson(legacyMap)]);return Object.fromEntries(result.rows.map(row => [row.bridge_key, row.memory_id]));});}diff --git a/server/services/moodBoard/db.js b/server/services/moodBoard/db.jsindex 98630537d2..b8e0a382ee 100644--- a/server/services/moodBoard/db.js+++ b/server/services/moodBoard/db.js@@ -47,6 +47,7 @@ import {maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,withBaseHashFlushBatch,} from '../../lib/conflictJournal.js';+import { toPgJson } from '../../lib/pgJson.js';// `data` JSONB is the whole record; name/created_at/updated_at mirror into// columns (kept in lockstep on every write) for the live-list sort. Reads@@ -71,7 +72,7 @@ async function persist(exec, board) {[board.id,board.name,- JSON.stringify(board),+ toPgJson(board),createdAt,mirrorTimestamp(board.updatedAt, createdAt),board.deleted === true,diff --git a/server/services/peerExecutionLedger.js b/server/services/peerExecutionLedger.jsindex 20cafea6c0..748bed6af1 100644--- a/server/services/peerExecutionLedger.js+++ b/server/services/peerExecutionLedger.js@@ -7,6 +7,7 @@ import * as fs from 'node:fs';import { z } from 'zod';import { peerAdminIntentSchema, peerAdminActionSchema } from '../lib/peerAdminValidation.js';import { createPeerExecutionAuthority, executionAuthorityError } from '../lib/peerExecutionAuthority.js';+import { toPgJson } from '../lib/pgJson.js';const uuid = z.string().uuid().toLowerCase();const digest = z.string().regex(/^[a-f0-9]{64}$/);@@ -90,7 +91,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerEconst { rows: [row] } = await client.query(`INSERT INTO peer_execution_operations(operation_id, host_instance_id, peer_instance_id, request_id, fingerprint, binding, execution_epoch, state)VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, 'queued') RETURNING *`,- [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), JSON.stringify(binding), binding.executionEpoch]);+ [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), toPgJson(binding), binding.executionEpoch]);await client.query(`INSERT INTO peer_execution_generation_floors (host_instance_id, peer_instance_id, action, generation)VALUES ($1, $2, $3, $4) ON CONFLICT (host_instance_id, peer_instance_id, action)DO UPDATE SET generation = GREATEST(peer_execution_generation_floors.generation, EXCLUDED.generation)`,@@ -113,7 +114,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerEconst { rows: [row] } = await client.query(`INSERT INTO peer_execution_operations(operation_id, host_instance_id, peer_instance_id, request_id, fingerprint, binding, execution_epoch, state, receipt)VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, 'failed', $8::jsonb) RETURNING *`,- [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), JSON.stringify(binding), binding.executionEpoch, JSON.stringify(receipt)]);+ [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), toPgJson(binding), binding.executionEpoch, toPgJson(receipt)]);authority.requireReady(epoch);return fromRow(row);});@@ -173,7 +174,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerEconst next = operationSchema.parse({ ...current, state, revision: revision + 1, claim: claim ?? current.claim, receipt });const { rows: [updated] } = await client.query(`UPDATE peer_execution_operationsSET state = $2, revision = $3, claim = $4::jsonb, receipt = $5::jsonb, updated_at = NOW()- WHERE operation_id = $1 RETURNING *`, [id, next.state, next.revision, JSON.stringify(next.claim), JSON.stringify(next.receipt)]);+ WHERE operation_id = $1 RETURNING *`, [id, next.state, next.revision, toPgJson(next.claim), toPgJson(next.receipt)]);authority.requireReady(epoch);return fromRow(updated);});@@ -322,7 +323,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerEON CONFLICT (operation_id) DO UPDATE SET state = EXCLUDED.state,revision = GREATEST(peer_execution_operations.revision, EXCLUDED.revision), claim = EXCLUDED.claim, receipt = EXCLUDED.receipt, updated_at = NOW()`,[saved.operationId, saved.binding.hostInstanceId, saved.binding.peerInstanceId, saved.binding.requestId, saved.fingerprint,- JSON.stringify(saved.binding), saved.binding.executionEpoch, state, saved.revision + 1, JSON.stringify(owner), JSON.stringify(receipt)]);+ toPgJson(saved.binding), saved.binding.executionEpoch, state, saved.revision + 1, toPgJson(owner), toPgJson(receipt)]);});if (replayed.digest !== proof.digest || replayed.count !== proof.count)throw executionAuthorityError('Execution recovery evidence changed during reconciliation.');diff --git a/server/services/pipeline/issuesStore/db.js b/server/services/pipeline/issuesStore/db.jsindex 9c3a584f96..b6d232b473 100644--- a/server/services/pipeline/issuesStore/db.js+++ b/server/services/pipeline/issuesStore/db.js@@ -17,6 +17,7 @@import { query } from '../../../lib/db.js';import { mirrorTimestamp } from '../../../lib/pgTimestamp.js';import { PIPELINE_STAGE_IDS } from '../../../lib/pipelineStages.js';+import { toPgJson } from '../../../lib/pgJson.js';// Only trusted, closed stage IDs form SQL paths. PostgreSQL removes history// before serializing JSONB for Node; every other field remains lossless.@@ -124,7 +125,7 @@ export async function writeRaw(id, record) {typeof record?.seasonId === 'string' && record.seasonId ? record.seasonId : null,Number.isFinite(record?.number) ? Math.floor(record.number) : null,typeof record?.status === 'string' && record.status ? record.status.slice(0, 32) : null,- JSON.stringify(record),+ toPgJson(record),record?.ephemeral === true,createdAt,mirrorTimestamp(record?.updatedAt, createdAt),diff --git a/server/services/pipeline/seriesStore/db.js b/server/services/pipeline/seriesStore/db.jsindex 5e637e80d3..fef5a5daa3 100644--- a/server/services/pipeline/seriesStore/db.js+++ b/server/services/pipeline/seriesStore/db.js@@ -21,6 +21,7 @@import { query } from '../../../lib/db.js';import { mirrorTimestamp } from '../../../lib/pgTimestamp.js';+import { toPgJson } from '../../../lib/pgJson.js';/** Raw on-disk-equivalent record (the `data` JSONB), or null. No sanitize. */export async function readRaw(id) {@@ -115,7 +116,7 @@ export async function writeRaw(id, record) {typeof record?.name === 'string' ? record.name : '',typeof record?.universeId === 'string' && record.universeId ? record.universeId : null,typeof record?.writersRoomWorkId === 'string' && record.writersRoomWorkId ? record.writersRoomWorkId : null,- JSON.stringify(record),+ toPgJson(record),record?.ephemeral === true,createdAt,mirrorTimestamp(record?.updatedAt, createdAt),diff --git a/server/services/postRunDb.js b/server/services/postRunDb.jsindex 9bfa88c390..6d819e1529 100644--- a/server/services/postRunDb.js+++ b/server/services/postRunDb.js@@ -6,6 +6,8 @@* rolls the complete save back instead of leaving a partial training session.*/+import { toPgJson } from '../lib/pgJson.js';+const asIso = (value) => value instanceof Date ? value.toISOString() : value;const asDay = (value) => value instanceof Date ? value.toISOString().slice(0, 10) : String(value || '').slice(0, 10);@@ -58,8 +60,8 @@ export async function saveNormalizedRunWithClient(client, input) {updated_at = NOW()`,[run.id, run.mode, run.localDay, run.startedAt, run.completedAt || null,- run.status || 'completed', JSON.stringify(run.planned || {}),- JSON.stringify(run.data || {}), run.legacy === true,+ run.status || 'completed', toPgJson(run.planned || {}),+ toPgJson(run.data || {}), run.legacy === true,],);@@ -107,11 +109,11 @@ export async function saveNormalizedRunWithClient(client, input) {updated_at = NOW()`,[attempt.id, run.id, position, attempt.module, attempt.drillType,- attempt.difficulty == null ? null : JSON.stringify(attempt.difficulty),+ attempt.difficulty == null ? null : toPgJson(attempt.difficulty),attempt.configVersion || null, attempt.correct ?? null, attempt.score ?? null,attempt.latencyMs || 0, attempt.completion ?? null, attempt.hintUsed === true,attempt.confidence ?? null, attempt.inputMode || 'unknown',- attempt.scorerProvenance || 'unknown', JSON.stringify(attempt.data || {}),+ attempt.scorerProvenance || 'unknown', toPgJson(attempt.data || {}),attempt.legacy === true,],);diff --git a/server/services/privacyBrokers.js b/server/services/privacyBrokers.jsindex 9d0d1a6979..9fd9a59916 100644--- a/server/services/privacyBrokers.js+++ b/server/services/privacyBrokers.js@@ -30,6 +30,7 @@ import { PATHS } from '../lib/fileUtils.js';import { fetchWithTimeout } from '../lib/fetchWithTimeout.js';import { resolveSubjectId } from './privacySubjects.js';import { encryptValue, decryptValue, ensureVaultKey } from '../lib/vaultCrypto.js';+import { toPgJson } from '../lib/pgJson.js';// Cap on each registry fetch so a hung broker source can't stall a// user-triggered refresh indefinitely (both fetchers run under Promise.all).@@ -239,7 +240,7 @@ async function upsertBroker(client, b, { onlyIfNotCurated = false, onlyIfCuratedupdated_at = NOW()${guard}`,[- b.id, b.name, JSON.stringify(b.urls), JSON.stringify(b.optout), b.tier,+ b.id, b.name, toPgJson(b.urls), toPgJson(b.optout), b.tier,b.disclosureFields, b.clusterParent, b.preferSuppression, b.antibot,b.source, b.confidence, b.lastVerified, b.enabled,],@@ -517,7 +518,7 @@ async function sealLegacyCaseEvidence() {for (const row of rows) {const evidence = parseEvidence(row.evidence);const stored = await prepareEvidenceForStorage(evidence, { existingEnvelope: evidence[SEALED_EVIDENCE_KEY] ?? null });- await client.query(`UPDATE privacy_broker_cases SET evidence = $1 WHERE id = $2`, [JSON.stringify(stored), row.id]);+ await client.query(`UPDATE privacy_broker_cases SET evidence = $1 WHERE id = $2`, [toPgJson(stored), row.id]);}if (rows.length) console.log(`🔐 Sealed identity evidence on ${rows.length} legacy broker case(s)`);return { converted: rows.length };@@ -643,7 +644,7 @@ export async function recordScanVerdict(brokerId, verdict, { evidence = {}, founconst resolvedSubjectId = resolveSubjectId(subjectId);// A fresh scan REPLACES the identity match (a re-scan that no longer finds a// listing must not keep the old one), so no existing envelope is carried.- const storedEvidence = JSON.stringify(await prepareEvidenceForStorage(evidence));+ const storedEvidence = toPgJson(await prepareEvidenceForStorage(evidence));return withTransaction(async (client) => {const broker = await client.query(`SELECT id FROM privacy_brokers WHERE id = $1`, [brokerId]);if (!broker.rows[0]) throw new ServerError('Broker not found', { status: 404, code: 'NOT_FOUND' });@@ -741,7 +742,7 @@ export async function transitionCase(caseId, toState, patch = {}) {const clearIdentity = toState === 'confirmed_removed';if (patch.evidence !== undefined || clearIdentity) {const stored = parseEvidence(existing.rows[0].evidence);- add('evidence', JSON.stringify(await prepareEvidenceForStorage(patch.evidence ?? stored, {+ add('evidence', toPgJson(await prepareEvidenceForStorage(patch.evidence ?? stored, {existingEnvelope: stored[SEALED_EVIDENCE_KEY] ?? null, clearIdentity,})));}diff --git a/server/services/privacyOrgs.js b/server/services/privacyOrgs.jsindex 8ceccb2362..bf8678d0b1 100644--- a/server/services/privacyOrgs.js+++ b/server/services/privacyOrgs.js@@ -22,6 +22,7 @@ import { randomUUID } from 'crypto';import { query, withTransaction } from '../lib/db.js';import { ServerError } from '../lib/errorHandler.js';import { resolveSubjectId, assertSubject } from './privacySubjects.js';+import { toPgJson } from '../lib/pgJson.js';const ORG_COLUMNS = `id, subject_id, name, category, website, trust, status, contact,social_account_id, notes, created_at, updated_at`;@@ -73,7 +74,7 @@ export async function createOrg(input) {id, subjectId, input.name,input.category ?? 'other', input.website ?? '',input.trust ?? 'trusted', input.status ?? 'active',- JSON.stringify(input.contact ?? {}),+ toPgJson(input.contact ?? {}),input.socialAccountId ?? null,input.notes ?? '',],@@ -117,7 +118,7 @@ export async function updateOrg(id, patch) {if (patch.website !== undefined) add('website', patch.website);if (patch.trust !== undefined) add('trust', patch.trust);if (patch.status !== undefined) add('status', patch.status);- if (patch.contact !== undefined) add('contact', JSON.stringify(patch.contact));+ if (patch.contact !== undefined) add('contact', toPgJson(patch.contact));if (patch.socialAccountId !== undefined) add('social_account_id', patch.socialAccountId);if (patch.notes !== undefined) add('notes', patch.notes);params.push(id);diff --git a/server/services/projectDbStore.js b/server/services/projectDbStore.jsindex 6081d3d4f9..1ad6f05f14 100644--- a/server/services/projectDbStore.js+++ b/server/services/projectDbStore.js@@ -10,6 +10,7 @@ import {deleteSyncBaseHash,withBaseHashFlushBatch,} from '../lib/conflictJournal.js';+import { toPgJson } from '../lib/pgJson.js';export function createProjectDbStore({ table, kind, idPrefix, logEmoji, logLabel, logic }) {if (!/^[a-z][a-z0-9_]*$/.test(table)) throw new Error(`Invalid project table: ${table}`);@@ -43,7 +44,7 @@ export function createProjectDbStore({ table, kind, idPrefix, logEmoji, logLabel[prepared.id,logic.mirrorStatus(prepared.status),- JSON.stringify(prepared),+ toPgJson(prepared),createdAt,logic.mirrorTimestamp(prepared.updatedAt, createdAt),prepared.deleted === true,diff --git a/server/services/providerGraphStore.js b/server/services/providerGraphStore.jsindex 501e12e1dd..46267fc116 100644--- a/server/services/providerGraphStore.js+++ b/server/services/providerGraphStore.js@@ -1,4 +1,5 @@import { query, withTransaction } from '../lib/db.js';+import { toPgJson } from '../lib/pgJson.js';/*** Postgres adapter for the provider connection graph — `ai_connections`,@@ -97,8 +98,8 @@ const upsertConnection = (client, connection) => client.query(credential_via = COALESCE($12, ai_connections.credential_via),updated_at = NOW()`,[connection.id, connection.revision ?? 1, connection.kind, connection.label ?? '',- JSON.stringify(connection.transports || {}), JSON.stringify(connection.credentials || {}),- JSON.stringify(connection.catalog || { state: 'unknown', models: [] }),+ toPgJson(connection.transports || {}), toPgJson(connection.credentials || {}),+ toPgJson(connection.catalog || { state: 'unknown', models: [] }),connection.slug ?? null, connection.definitionId ?? null, connection.plan ?? null,typeof connection.enabled === 'boolean' ? connection.enabled : null, connection.credentialVia ?? null],);@@ -114,7 +115,7 @@ const upsertBinding = (client, binding) => client.query(enabled = EXCLUDED.enabled, selected_models = EXCLUDED.selected_models, updated_at = NOW()`,[binding.id, binding.revision ?? 1, binding.connectionId, binding.harnessId ?? null,binding.variantKey, binding.label ?? '', binding.enabled === true,- JSON.stringify(binding.selectedModels || [])],+ toPgJson(binding.selectedModels || [])],);// `model_alias_overrides` is set on INSERT and deliberately absent from the@@ -130,10 +131,10 @@ const upsertRoute = (client, route) => client.query(binding_id = EXCLUDED.binding_id, mode = EXCLUDED.mode, model_map = EXCLUDED.model_map,projected = EXCLUDED.projected, pending = EXCLUDED.pending,pending_revision = EXCLUDED.pending_revision, updated_at = NOW()`,- [route.providerId, route.bindingId, route.mode, JSON.stringify(route.modelMap || {}),- JSON.stringify(route.modelAliasOverrides || {}),- JSON.stringify(route.projected || {}),- route.pending == null ? null : JSON.stringify(route.pending),+ [route.providerId, route.bindingId, route.mode, toPgJson(route.modelMap || {}),+ toPgJson(route.modelAliasOverrides || {}),+ toPgJson(route.projected || {}),+ route.pending == null ? null : toPgJson(route.pending),route.pendingRevision ?? null],);@@ -191,7 +192,7 @@ export async function applyReconciliation(plan) {for (const { providerId, owned } of plan.snapshots) {await client.query('UPDATE ai_route_bindings SET projected = $1, updated_at = NOW() WHERE provider_id = $2',- [JSON.stringify(owned), providerId],+ [toPgJson(owned), providerId],);}@@ -222,7 +223,7 @@ export async function commitPendingProjection(projections) {const { rows } = await client.query(`UPDATE ai_route_bindings SET pending = $1, updated_at = NOW() WHERE provider_id = $2RETURNING binding_id`,- [JSON.stringify(owned), providerId],+ [toPgJson(owned), providerId],);if (rows.length === 0) continue;const { rows: bumped } = await client.query(@@ -325,8 +326,8 @@ export async function saveConnectionSettings({ id, label, transports, credentialrevision = revision + 1, updated_at = NOW()WHERE id = $1RETURNING revision`,- [id, label ?? '', JSON.stringify(transports || {}), JSON.stringify(credentials || {}),- JSON.stringify(catalog || { state: 'unknown', models: [] }),+ [id, label ?? '', toPgJson(transports || {}), toPgJson(credentials || {}),+ toPgJson(catalog || { state: 'unknown', models: [] }),plan ?? null, typeof enabled === 'boolean' ? enabled : null, credentialVia ?? null],);return rows[0]?.revision ?? null;@@ -366,7 +367,7 @@ export async function saveBindingSettings({ id, label, selectedModels }) {SET label = $2, selected_models = $3, revision = revision + 1, updated_at = NOW()WHERE id = $1RETURNING revision`,- [id, label ?? '', JSON.stringify(selectedModels || [])],+ [id, label ?? '', toPgJson(selectedModels || [])],);return rows[0]?.revision ?? null;}@@ -374,7 +375,7 @@ export async function saveBindingSettings({ id, label, selectedModels }) {/** Record the canonical→executable model aliases a refresh observed on a route. */export async function saveRouteModelMap(providerId, modelMap) {await query('UPDATE ai_route_bindings SET model_map = $2, updated_at = NOW() WHERE provider_id = $1',- [providerId, JSON.stringify(modelMap || {})]);+ [providerId, toPgJson(modelMap || {})]);}/**@@ -388,6 +389,6 @@ export async function saveRouteModelMap(providerId, modelMap) {export async function saveRouteModelAliases(providerId, overrides) {await query('UPDATE ai_route_bindings SET model_alias_overrides = $2, updated_at = NOW() WHERE provider_id = $1',- [providerId, JSON.stringify(overrides || {})],+ [providerId, toPgJson(overrides || {})],);}diff --git a/server/services/sprites/recordsDB.js b/server/services/sprites/recordsDB.jsindex 4a174eb2e4..3cba1a9eda 100644--- a/server/services/sprites/recordsDB.js+++ b/server/services/sprites/recordsDB.js@@ -19,6 +19,7 @@ import {mergeImportedRecord,mirrorTimestamp,} from './recordsLogic.js';+import { toPgJson } from '../../lib/pgJson.js';function rowToRecord(row) {return row ? row.data : null;@@ -41,7 +42,7 @@ async function persist(exec, record) {record.id,record.kind,record.status,- JSON.stringify(record),+ toPgJson(record),createdAt,mirrorTimestamp(record.updatedAt, createdAt),record.deleted === true,diff --git a/server/services/storyBuilderStore/db.js b/server/services/storyBuilderStore/db.jsindex 856e29f781..9c8554d3aa 100644--- a/server/services/storyBuilderStore/db.js+++ b/server/services/storyBuilderStore/db.js@@ -17,6 +17,7 @@import { query } from '../../lib/db.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';/** Raw on-disk-equivalent record (the `data` JSONB), or null. No sanitize. */export async function readRaw(id) {@@ -63,7 +64,7 @@ export async function writeRaw(id, record) {typeof record?.universeId === 'string' && record.universeId ? record.universeId : null,typeof record?.seriesId === 'string' && record.seriesId ? record.seriesId : null,record?.sync === true,- JSON.stringify(record),+ toPgJson(record),record?.ephemeral === true,createdAt,mirrorTimestamp(record?.updatedAt, createdAt),diff --git a/server/services/threejsModels/db.js b/server/services/threejsModels/db.jsindex 662abd1924..eec08165fd 100644--- a/server/services/threejsModels/db.js+++ b/server/services/threejsModels/db.js@@ -10,6 +10,7 @@ import { randomUUID } from 'crypto';import { query, withTransaction } from '../../lib/db.js';import { ServerError } from '../../lib/errorHandler.js';import { GENERAL_FAMILY_ID } from '../../lib/threejsModelFamilies.js';+import { toPgJson } from '../../lib/pgJson.js';const rowToModel = (row) => row?.data ?? null;@@ -28,7 +29,7 @@ async function persist(exec, model) {model.id,model.name,model.status,- JSON.stringify(model),+ toPgJson(model),model.createdAt,model.updatedAt,model.deleted === true,diff --git a/server/services/tracks/db.js b/server/services/tracks/db.jsindex fa40783be8..f424329900 100644--- a/server/services/tracks/db.js+++ b/server/services/tracks/db.js@@ -16,6 +16,7 @@ import {maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,withBaseHashFlushBatch,} from '../../lib/conflictJournal.js';+import { toPgJson } from '../../lib/pgJson.js';// Re-sanitize the stored JSONB on read (mirrors the file backend's loadAll →// sanitizeTrack). The record was already sanitized on write, so for an in-shape@@ -42,7 +43,7 @@ async function persist(exec, track) {[track.id,typeof track.title === 'string' ? track.title : '',- JSON.stringify(track),+ toPgJson(track),createdAt,mirrorTimestamp(track.updatedAt, createdAt),track.deleted === true,diff --git a/server/services/universeBuilder/db.js b/server/services/universeBuilder/db.jsindex 7393589838..4816a2e49c 100644--- a/server/services/universeBuilder/db.js+++ b/server/services/universeBuilder/db.js@@ -23,6 +23,7 @@import { query, withTransaction } from '../../lib/db.js';import { LOGLINE_MAX } from '../../lib/universeBibleLimits.js';import { mirrorTimestamp } from '../../lib/pgTimestamp.js';+import { toPgJson } from '../../lib/pgJson.js';// Keep the runs[] log bounded to the most-recent 200 — matches the file// backend's cap (universeBuilder.recordRun) so behavior is identical.@@ -219,7 +220,7 @@ export async function writeRaw(id, record) {[id,typeof record?.name === 'string' ? record.name : '',- JSON.stringify(record),+ toPgJson(record),schemaVersion,record?.ephemeral === true,createdAt,@@ -264,7 +265,7 @@ export async function appendRun(run) {run.id,run.universeId,typeof run.collectionId === 'string' ? run.collectionId : null,- JSON.stringify(run),+ toPgJson(run),mirrorTimestamp(run.createdAt, new Date().toISOString()),],);diff --git a/server/services/userActionsDb.js b/server/services/userActionsDb.jsindex 163dc661ed..f1708d4ea0 100644--- a/server/services/userActionsDb.js+++ b/server/services/userActionsDb.js@@ -9,6 +9,8 @@* drive it against `portos_test` under `npm run test:db`.*/+import { toPgJson } from '../lib/pgJson.js';+const iso = (value) => (value instanceof Date ? value.toISOString() : value);/** Map a `user_action_events` row to the camelCase event shape the API returns. */@@ -42,8 +44,8 @@ export async function insertUserActionEvent(db, event) {RETURNING id`,[event.id, event.type, event.actor, event.happenedAt, event.target, event.targetName,- event.success, event.summary, JSON.stringify(event.payload),- JSON.stringify(event.source), event.dedupeKey,+ event.success, event.summary, toPgJson(event.payload),+ toPgJson(event.source), event.dedupeKey,],);return (result.rowCount || 0) > 0;diff --git a/server/services/voice/profiles.js b/server/services/voice/profiles.jsindex e0798e1c8a..74c875a6f4 100644--- a/server/services/voice/profiles.js+++ b/server/services/voice/profiles.js@@ -16,6 +16,7 @@ import { ServerError } from '../../lib/errorHandler.js';import { PATHS } from '../../lib/paths.js';import { writeFileGuarded } from '../../lib/fileUtils.js';import { trimTo } from '../../lib/textUtils.js';+import { toPgJson } from '../../lib/pgJson.js';export const VOICE_PROFILE_ENGINES = new Set(['kokoro', 'piper', 'qwen3-tts', 'auk']);export const VOICE_PROFILE_KINDS = new Set(['preset', 'designed', 'cloned', 'fine-tuned']);@@ -203,7 +204,7 @@ export const persistVoiceProfile = async (profile, execute = query) => {profile.binding.universeId,profile.binding.characterId,profile.approval.status,- JSON.stringify(profile),+ toPgJson(profile),profile.createdAt,profile.updatedAt,],@@ -597,7 +598,7 @@ export async function saveProfileBenchmark(profile, benchmark, { interactive } =updated_at = $5WHERE id = $1 AND (data->>'version')::int = $3RETURNING data`,- [profile.id, JSON.stringify(patch), profile.version, route ? JSON.stringify(route) : null, patch.updatedAt],+ [profile.id, toPgJson(patch), profile.version, route ? toPgJson(route) : null, patch.updatedAt],);if (!rows.length) {throw new ServerError('Voice profile changed during benchmarking; run the benchmark again', {@@ -655,7 +656,7 @@ export async function recordVoiceProfileRender({profile_revision = EXCLUDED.profile_revision,data = EXCLUDED.data,updated_at = EXCLUDED.updated_at`,- [issue, line, profileId, profileRevision, JSON.stringify(data), now, now],+ [issue, line, profileId, profileRevision, toPgJson(data), now, now],);}diff --git a/server/services/writersRoom/db.js b/server/services/writersRoom/db.jsindex 12d417a416..b1c75ca81e 100644--- a/server/services/writersRoom/db.js+++ b/server/services/writersRoom/db.js@@ -42,6 +42,7 @@ import {WRITERS_ROOM_FOLDER_KIND, mergeFolderRecord,WRITERS_ROOM_EXERCISE_KIND, mergeExerciseRecord,} from './syncLogic.js';+import { toPgJson } from '../../lib/pgJson.js';/*** Shared per-record LWW merge for a body-less Writers Room record (folder /@@ -151,7 +152,7 @@ function persistFolderTx(client, folder) {typeof folder?.parentId === 'string' && folder.parentId ? folder.parentId : null,String(folder?.name ?? ''),Number.isInteger(folder?.sortOrder) ? folder.sortOrder : 0,- JSON.stringify(folderData),+ toPgJson(folderData),createdAt,mirrorTimestamp(folder?.updatedAt, createdAt),deleted,@@ -241,7 +242,7 @@ function persistExerciseTx(client, exercise) {exercise.id,typeof exercise?.workId === 'string' && exercise.workId ? exercise.workId : null,typeof exercise?.status === 'string' ? exercise.status.slice(0, 16) : null,- JSON.stringify(exerciseData),+ toPgJson(exerciseData),mirrorTimestamp(exercise?.startedAt, null),mirrorTimestamp(exercise?.finishedAt, null),deleted,@@ -377,9 +378,9 @@ function draftBinds(workId, draft) {String(draft?.contentFile ?? ''),typeof draft?.contentHash === 'string' ? draft.contentHash : null,Number.isInteger(draft?.wordCount) ? draft.wordCount : 0,- JSON.stringify(Array.isArray(draft?.segmentIndex) ? draft.segmentIndex : []),+ toPgJson(Array.isArray(draft?.segmentIndex) ? draft.segmentIndex : []),typeof draft?.createdFromVersionId === 'string' ? draft.createdFromVersionId : null,- JSON.stringify(draft),+ toPgJson(draft),mirrorTimestamp(draft?.createdAt, new Date().toISOString()),];}@@ -453,7 +454,7 @@ async function persistWorkTx(client, manifest) {typeof manifest?.pipelineIssueId === 'string' ? manifest.pipelineIssueId : null,typeof manifest?.cdProjectId === 'string' ? manifest.cdProjectId : null,typeof manifest?.mediaCollectionId === 'string' ? manifest.mediaCollectionId : null,- JSON.stringify(workData),+ toPgJson(workData),createdAt,mirrorTimestamp(manifest?.updatedAt, createdAt),deleted,