Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: repair text PostgreSQL refuses (split emoji, NUL) before it reaches jsonb/text (#12815)

Adam Eivy580dc8e3c819
diff --git a/server/AGENTS.md b/server/AGENTS.md
index e45e23ccc0..df4f826a34 100644
--- a/server/AGENTS.md
+++ b/server/AGENTS.md
@@ -19,6 +19,10 @@ These apply to Express/service/model code. Universal constraints (no try/catch o
- **Async PATCH races on shared records — serialize writes server-side.** When two write paths can mutate the same record concurrently (e.g. a blur-save plus an explicit "Render" button against `stages.comicPages.cover`), client-side guards (refs, `onMouseDown`, status checks) are unreliable — keyboard activation, status==='unknown' stalls, and `loadState → modify → saveState` interleaving all defeat them. The correct fix is to serialize writes at the *file* level on the server (every PATCH awaits the previous one to settle before reading state, so it merges against the freshest persisted record). See `queueSeriesIssuesWrite` in `server/services/pipeline/issuesShared.js` (built on `createKeyCachedQueue` in `server/lib/createKeyCachedQueue.js`). A `Map<recordId, Promise>` is *not* enough — two writes to different record ids share the same JSON state file and can still clobber each other; collapse the queue to a single tail per shared file.
- **High-frequency state writes must batch.** Per-line state mutations that round-trip through `withStateLock → loadState → saveState` (e.g. `appendAgentOutput` in `server/services/cosAgentLifecycle.js`) are fine for human-pace events but catastrophic when called from a hot loop — PTY output streams, AI tool-call streams, or any producer that can emit dozens of events per second. When wiring a new streaming producer, add a batched variant that takes an array (see `appendAgentOutputLines` for the pattern) and flush from the caller on a ~250ms debounce. Always drain the pending buffer in the producer's `finish`/`cleanup` path before the final state write so completion events don't beat the last output batch to disk.
+## PostgreSQL parameters
+
+- **A parameter bound to a `json`/`jsonb` column or `::jsonb` cast is serialized with `toPgJson`, never bare `JSON.stringify`.** PostgreSQL refuses a lone UTF-16 surrogate (an emoji `.slice()`d in half) and NUL inside `jsonb`, and NUL in `text`; every file store the migrated records came from accepted both, so an unrepaired write throws and the record is lost (a promoted CoS task, an automatic investigation task, a legacy import retained for retry forever, #12815). `toPgJson` and `normalizePgString` live in `server/lib/pgJson.js` and repair a bad unit to U+FFFD, the same lossy fix a UTF-8 file write applied. `lib/db.js` applies `normalizePgParams` to every `query()` and `withTransaction` client call (strings, array elements and plain-object parameters; it never unescapes a string, so a `text` value holding the characters `\^@` is stored as written), but a pre-stringified JSON string looks like ordinary text to it, so the call site still uses `toPgJson`. `server/pgJsonConventions.test.js` scans the tree for this and carries no allowlist; applied `scripts/db-migrations/` are frozen and skipped.
+
## Timers
- **An `async` timer callback must own its rejections.** Inside `setTimeout(async …)` / `setInterval(async …)`, every `await` needs either an enclosing `try { … } catch` or a `.catch(…)` on the awaited chain. A timer fires outside the request lifecycle, so nothing holds the promise the callback returns and a rejection escapes as an unhandled rejection — fatal on Node >= 15. The CoS runner shipped that at boot: its orphan-cleanup timer awaited a state-file read/write, so one truncated `agents` file killed the runner seconds after start and PM2 restart-looped it against the same file (#5668). Log and continue in the catch unless the callback genuinely owns a failure verdict. `server/timerCallbackConventions.test.js` scans the whole server tree for this and carries no allowlist; the worked example is the watchdog in `server/services/mediaJobQueue/index.js`.
diff --git a/server/lib/README.md b/server/lib/README.md
index 72c1fb7216..e42194b95d 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -942,3 +942,4 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
- `nativeTrashXdg.js`: `trashXdg(path)` moves one file into the freedesktop Trash 1.0 directory serving its volume on `node:fs` alone (original names, RFC 2396 `Path=`, `O_EXCL` info reservation, NAME_MAX-bounded names, mount identity from `/proc/self/mountinfo` with a `st_dev` fallback so a bind mount gets its own volume trash instead of an `EXDEV`, physical-parent mount discovery, an owned trash directory is tightened to 0700 and `TRASH_DIR_UNSAFE` refuses a symlinked or foreign-owned one or a volume root whose entries other users could rename, a sticky shared `.Trash` counts only when root or the caller owns it and falls back to `.Trash-<uid>` when it cannot hold the per-user directory, refuses instead of copying across devices); `_topDirTrash(topDir, uid)`, `_parseMountInfo(text)` and `_mountOf(realDir, mounts)` are test hooks.
- `nativeTrash.js`: `moveToNativeTrash(paths)` executes the pinned package native recycling helpers on macOS/Windows with hidden windows and a deadline; Linux uses the package XDG adapter. No local-directory or permanent-delete fallback.
- `eventLoopLagProbe.js`: `createEventLoopLagProbe({ clock, intervalMs, toleranceMs })` opens phase-tracking spans that charge overdue time of one shared, span-scoped tick to the current phase, so a slow-operation diagnostic splits each phase into event-loop/host scheduling lag and asynchronous wait (#12329); `systemClock` (globals resolved at call time), `__createManualClock()` (deterministic `advance` = on-time timers vs `stall` = blocked loop, for tests), `roundMs`, and `phaseGroup(summary, name)` (a phase plus its `name.` sub-phases).
+- `pgJson.js`: text repair for values bound to PostgreSQL, which refuses a lone UTF-16 surrogate in `jsonb` and NUL in `jsonb`/`text` where every file store accepted both. `normalizePgString(s)` → `toWellFormed()` with NUL → U+FFFD (the same lossy repair a UTF-8 file write applied). `toPgJson(value)` → `JSON.stringify` that repairs string values and object keys; **use it, never bare `JSON.stringify`, for a parameter bound to a `json`/`jsonb` column or `::jsonb` cast** (`pgJsonConventions.test.js` enforces this). `normalizePgParams(params)` is applied by `db.js` `query()` and the `withTransaction` client: repairs string and array-element params (never unescapes `\^@`) and serializes plain-object params via `toPgJson`.
diff --git a/server/lib/db.js b/server/lib/db.js
index 1c4309a217..7d40b22141 100644
--- a/server/lib/db.js
+++ b/server/lib/db.js
@@ -11,6 +11,7 @@ import { isTestRunner } from './runtimeEnv.js';
import { assertDatabaseAdmission, createDatabaseMaintenanceJournal } from './databaseMaintenanceJournal.js';
import { assertDatabasePoolAuthority } from './databaseAuthority.js';
import { createDatabaseRestoreRecovery, databaseRestoreRecoveryError } from './databaseRestoreRecovery.js';
+import { normalizePgParams } from './pgJson.js';
const { Pool } = pg;
@@ -207,6 +208,11 @@ const GUARDED_CLIENT_HANDLER = {
return (config, ...rest) => {
const sql = typeof config === 'string' ? config : config?.text;
assertWriteAllowed(sql);
+ // A Submittable (pg-cursor) also carries `values`; only a plain config is rewritten.
+ if (typeof config === 'string' && Array.isArray(rest[0])) return target.query(config, normalizePgParams(rest[0]), ...rest.slice(1));
+ if (Array.isArray(config?.values) && Object.getPrototypeOf(config) === Object.prototype) {
+ return target.query({ ...config, values: normalizePgParams(config.values) }, ...rest);
+ }
return target.query(config, ...rest);
};
}
@@ -295,7 +301,7 @@ export async function withDatabaseMaintenance(fn, { restoreRecoveryId } = {}) {
/**
* Execute a query against the connection pool.
* @param {string} text - SQL query text with $1, $2, etc. placeholders
- * @param {Array} params - Parameter values
+ * @param {Array} params - Parameter values; strings are repaired for PostgreSQL (see pgJson.js) and plain objects serialized, here and on the withTransaction client
* @returns {Promise<pg.QueryResult>}
*/
export async function query(text, params) {
@@ -304,7 +310,7 @@ export async function query(text, params) {
// author calls query('INSERT …') directly, or a backend selector mis-chose
// Postgres because NODE_ENV wasn't 'test').
assertWriteAllowed(text);
- return databaseOperation(() => pool.query(text, params));
+ return databaseOperation(() => pool.query(text, normalizePgParams(params)));
}
/**
diff --git a/server/lib/db.test.js b/server/lib/db.test.js
index 151de495c1..556c84ad53 100644
--- a/server/lib/db.test.js
+++ b/server/lib/db.test.js
@@ -33,7 +33,7 @@ vi.mock('./db/schema/index.js', async () => {
};
});
-const { checkHealth, ensureSchema, close, POOL_CONFIG, SCHEMA_DDL_ADVISORY_LOCK_KEY } = await import('./db.js');
+const { checkHealth, ensureSchema, close, query, withTransaction, POOL_CONFIG, SCHEMA_DDL_ADVISORY_LOCK_KEY } = await import('./db.js');
let dbReady = false;
let skipReason = '';
@@ -110,3 +110,31 @@ describe.skipIf(!runDb)('ensureSchema() boot DDL advisory lock (#5977)', () => {
await expect(ensureSchema()).resolves.toBeUndefined();
});
});
+
+// PostgreSQL refuses a lone surrogate in jsonb and NUL in jsonb/text; the pool
+// and the withTransaction client repair bound parameters in one place (#12815).
+describe.skipIf(!runDb)('bound parameter repair', () => {
+ const split = '😀'.slice(0, 1);
+ const runners = {
+ 'query()': (sql, params) => query(sql, params),
+ 'withTransaction client': (sql, params) => withTransaction((client) => client.query(sql, params)),
+ };
+
+ describe.each(Object.entries(runners))('%s', (_name, run) => {
+ it('repairs a string parameter and leaves the characters \\^@ alone', async () => {
+ const { rows: [row] } = await run('SELECT $1::text AS bad, $2::text AS literal', [`x${split}y\0z`, 'a\\^@b']);
+ expect(row.bad).toBe('x\uFFFDy\uFFFDz');
+ expect(row.literal).toBe('a\\^@b');
+ });
+
+ it('repairs every string in an array parameter', async () => {
+ const { rows: [row] } = await run('SELECT $1::text[] AS items', [['ok', `a${split}`, 'b\0']]);
+ expect(row.items).toEqual(['ok', 'a\uFFFD', 'b\uFFFD']);
+ });
+
+ it('serializes a plain-object parameter as repaired jsonb, keys included', async () => {
+ const { rows: [row] } = await run('SELECT $1::jsonb AS doc', [{ [`k${split}`]: [`v${split}`, 'n\0'], nested: { ok: 1 } }]);
+ expect(row.doc).toEqual({ 'k\uFFFD': ['v\uFFFD', 'n\uFFFD'], nested: { ok: 1 } });
+ });
+ });
+});
diff --git a/server/lib/index.js b/server/lib/index.js
index 5b9e2db34d..2d7ad7429f 100644
--- a/server/lib/index.js
+++ b/server/lib/index.js
@@ -928,3 +928,4 @@ export * from './reconcileRetirementEvidence.js';
export * from './checkoutRetirementAdmission.js';
export * from './moltworldExplore.js';
+export * from './pgJson.js';
diff --git a/server/lib/pgJson.js b/server/lib/pgJson.js
new file mode 100644
index 0000000000..94fb355fa5
--- /dev/null
+++ b/server/lib/pgJson.js
@@ -0,0 +1,55 @@
+/**
+ * Text repair for values bound to PostgreSQL.
+ *
+ * `jsonb` refuses a lone UTF-16 surrogate (`\ud83d`, e.g. an emoji cut in half by
+ * `.slice()`) and both `jsonb` and `text` refuse NUL. Every file store these
+ * records came from tolerated both, so a write that used to succeed started
+ * throwing after the move to PostgreSQL. The repair is the same lossy one a
+ * UTF-8 file write always applied: a bad unit becomes U+FFFD.
+ */
+
+const REPLACEMENT = '�';
+
+const isPlainObject = (value) => {
+ if (value === null || typeof value !== 'object') return false;
+ const proto = Object.getPrototypeOf(value);
+ return proto === Object.prototype || proto === null;
+};
+
+export function normalizePgString(s) {
+ return s.toWellFormed().replaceAll('\0', REPLACEMENT);
+}
+
+// JSON.stringify never shows its replacer an object key, so a key holding a bad
+// unit is repaired by handing back a copy with normalized keys.
+function normalizeKeys(value) {
+ const keys = Object.keys(value);
+ if (keys.every((key) => normalizePgString(key) === key)) return value;
+ return Object.fromEntries(keys.map((key) => [normalizePgString(key), value[key]]));
+}
+
+function repairJsonValue(_key, value) {
+ if (typeof value === 'string') return normalizePgString(value);
+ return isPlainObject(value) ? normalizeKeys(value) : value;
+}
+
+/** `JSON.stringify` for a `json`/`jsonb` parameter: strings and object keys are repaired first. */
+export function toPgJson(value) {
+ return JSON.stringify(value, repairJsonValue);
+}
+
+const normalizeParam = (param) => {
+ if (typeof param === 'string') return normalizePgString(param);
+ if (Array.isArray(param)) return param.map(normalizeParam);
+ // Mirrors node-postgres, which JSON-serializes a plain object parameter itself.
+ return isPlainObject(param) ? toPgJson(param) : param;
+};
+
+/**
+ * Repair a query's bound parameters in one place. Strings are repaired as-is
+ * (never unescaped: a `text` value may legitimately contain the characters `\^@`),
+ * array elements likewise, and plain objects become repaired JSON text.
+ */
+export function normalizePgParams(params) {
+ return Array.isArray(params) ? params.map(normalizeParam) : params;
+}
diff --git a/server/pgJsonConventions.test.js b/server/pgJsonConventions.test.js
new file mode 100644
index 0000000000..a4d59474dd
--- /dev/null
+++ b/server/pgJsonConventions.test.js
@@ -0,0 +1,164 @@
+/**
+ * Repo-wide guard: a value bound to a `json`/`jsonb` parameter is serialized with
+ * `toPgJson` (lib/pgJson.js), never bare `JSON.stringify`.
+ *
+ * PostgreSQL refuses a lone UTF-16 surrogate and NUL inside `jsonb`, which every
+ * file store these records came from accepted. `JSON.stringify` writes a lone
+ * surrogate as the ASCII escape `\ud83d`, so the value reaches the server intact
+ * and is refused — an emoji cut in half by `.slice()` lost a user-promoted CoS
+ * task (#12815). `toPgJson` applies the repair the old UTF-8 file writes did.
+ *
+ * ## The rule
+ *
+ * A `JSON.stringify(` call is a violation when it feeds a statement that mentions
+ * `::json`/`::jsonb`, in either shape:
+ *
+ * 1. it sits anywhere inside a `query(…)`/`execute(…)`-style call whose text
+ * mentions `::json`/`::jsonb` (the common case, including a `.map(…)` that
+ * builds an `unnest` column), or
+ * 2. its innermost bracket is an array literal or a `.push(…)` — a bind-parameter
+ * list assembled apart from the SQL — and the enclosing block mentions
+ * `::json`/`::jsonb`.
+ *
+ * Hashing, logging, HTTP bodies and advisory-lock keys are not in either shape.
+ *
+ * ## Allowlist
+ *
+ * There is none. Applied migrations (`scripts/db-migrations/`) are frozen and
+ * skipped wholesale: `020-*` binds to `json` (exact text) on purpose and verifies
+ * byte equality. Anything else: swap in `toPgJson`.
+ *
+ * ## What this guard CANNOT see
+ *
+ * A lexer-assisted source scan (see lib/sourceScan.js), not a type-aware one. It
+ * misses a stringified value bound to a `json`/`jsonb` COLUMN with no `::json`
+ * cast anywhere in its block, and a value stringified in one function and bound
+ * in another. `db.js` repairs plain-object parameters regardless, which covers
+ * the call sites that never stringified at all.
+ */
+import { describe, it, expect } from 'vitest';
+import { execFileSync } from 'child_process';
+import { readFileSync } from 'fs';
+import { dirname, join } from 'path';
+import { fileURLToPath } from 'url';
+import { blankCommentBodies, blankLiterals, matchBracket } from './lib/sourceScan.js';
+
+const SERVER_ROOT = dirname(fileURLToPath(import.meta.url));
+const SCANNED_DIRS = ['services/', 'lib/', 'routes/', 'scripts/'];
+const JSON_CAST = /::jsonb?\b/;
+const STRINGIFY = /JSON\.stringify\s*\(/g;
+const QUERY_CALLEE = /(?:^|[^\w$])(?:query|queryFn|execute|exec|run|q)\s*$/;
+
+/** Every violating `JSON.stringify(` in `src`, as `line N`. */
+export function findPgJsonViolations(src) {
+ const code = blankLiterals(src);
+ const text = blankCommentBodies(src);
+ const line = (index) => code.slice(0, index).split('\n').length;
+ const calleeOf = (open) => code.slice(Math.max(0, open - 40), open);
+ const hits = [];
+
+ // One pass keeps the stack of enclosing brackets for every stringify call.
+ const stack = [];
+ const stringifyStarts = new Set([...code.matchAll(STRINGIFY)].map((match) => match.index));
+ for (let i = 0; i < code.length; i += 1) {
+ const ch = code[i];
+ if (stringifyStarts.has(i) && violates(stack)) hits.push(`line ${line(i)}`);
+ if (ch === '(' || ch === '[' || ch === '{') stack.push(i);
+ else if (ch === ')' || ch === ']' || ch === '}') stack.pop();
+ }
+ return hits;
+
+ function violates(enclosing) {
+ // Shape 1: inside a query-style call, whose own text is the statement.
+ const queryCall = enclosing.find((open) => code[open] === '(' && QUERY_CALLEE.test(calleeOf(open)));
+ if (queryCall !== undefined) return JSON_CAST.test(text.slice(queryCall, matchBracket(code, queryCall)));
+ // Shape 2: a bind-parameter list assembled apart from the SQL.
+ const innermost = enclosing[enclosing.length - 1];
+ if (innermost === undefined) return false;
+ const isParamList = code[innermost] === '['
+ || (code[innermost] === '(' && /\.push\s*$/.test(calleeOf(innermost)));
+ if (!isParamList) return false;
+ const block = [...enclosing].reverse().find((open) => code[open] === '{');
+ if (block === undefined) return JSON_CAST.test(text);
+ const end = matchBracket(code, block);
+ return end !== -1 && JSON_CAST.test(text.slice(block, end));
+ }
+}
+
+const trackedSources = () => execFileSync('git', ['ls-files', '*.js', '*.mjs'], {
+ cwd: SERVER_ROOT,
+ encoding: 'utf8',
+ maxBuffer: 64 * 1024 * 1024,
+}).split('\n').filter((f) => f
+ && SCANNED_DIRS.some((dir) => f.startsWith(dir))
+ && !f.includes('.test.')
+ && !f.includes('db-migrations/'));
+
+describe('json/jsonb parameters are serialized with toPgJson (#12815)', () => {
+ it('scans the server tree', () => {
+ // A broken `git ls-files` would otherwise let every assertion pass by scanning nothing.
+ expect(trackedSources().length).toBeGreaterThan(200);
+ });
+
+ it('has no bare JSON.stringify feeding a json/jsonb parameter', () => {
+ const violations = [];
+ for (const file of trackedSources()) {
+ const src = readFileSync(join(SERVER_ROOT, file), 'utf8');
+ if (!src.includes('JSON.stringify') || !JSON_CAST.test(src)) continue;
+ for (const hit of findPgJsonViolations(src)) violations.push(`server/${file} ${hit}`);
+ }
+ expect(
+ violations,
+ 'PostgreSQL rejects a lone surrogate and NUL inside jsonb, so a bare JSON.stringify parameter '
+ + 'loses the write when a string was cut mid-emoji (#12815).\n'
+ + 'Fix: import { toPgJson } from the lib/pgJson.js module and use it in place of JSON.stringify.\n'
+ + `Offenders:\n ${violations.join('\n ')}`,
+ ).toEqual([]);
+ });
+});
+
+// Guards the guard: if the recognizer stops seeing the bad shape the scan above goes green.
+describe('the pgJson recognizer', () => {
+ it('flags a stringified value inside a jsonb query', () => {
+ expect(findPgJsonViolations("await query('INSERT INTO t (d) VALUES ($1::jsonb)', [JSON.stringify(record)]);"))
+ .toEqual(['line 1']);
+ expect(findPgJsonViolations(`await client.query(
+ 'INSERT INTO t SELECT u.d::jsonb FROM unnest($1::text[]) AS u(d)',
+ [rows.map((row) => JSON.stringify(row.task))],
+ );`)).toEqual(['line 3']);
+ });
+
+ it('flags a parameter list assembled apart from the SQL', () => {
+ expect(findPgJsonViolations(`function update(patch) {
+ const sets = ['payload = $2::jsonb'];
+ const params = [];
+ params.push(JSON.stringify(patch));
+ return run(sets, params);
+ }`)).toEqual(['line 4']);
+ expect(findPgJsonViolations(`function insert(id, doc) {
+ const sql = 'INSERT INTO t (id, doc) VALUES ($1, $2::jsonb)';
+ const values = [id, JSON.stringify(doc)];
+ return pool.query(sql, values);
+ }`)).toEqual(['line 3']);
+ });
+
+ it('accepts toPgJson and stringification that is not a bind parameter', () => {
+ expect(findPgJsonViolations("await query('INSERT INTO t (d) VALUES ($1::jsonb)', [toPgJson(record)]);")).toEqual([]);
+ expect(findPgJsonViolations(`function bind(doc, a, b) {
+ const sql = 'INSERT INTO t (doc) VALUES ($1::jsonb)';
+ const key = createHash('sha256').update(JSON.stringify(doc)).digest('hex');
+ const same = JSON.stringify(a) === JSON.stringify(b);
+ return query(sql, [toPgJson(doc), key, same]);
+ }`)).toEqual([]);
+ // A stringified lock key in a query with no json cast, beside a jsonb statement.
+ expect(findPgJsonViolations(`async function tx(client) {
+ await client.query('SELECT pg_advisory_xact_lock(hashtext($1))', [JSON.stringify([a, b])]);
+ await client.query("UPDATE t SET d = '{}'::jsonb");
+ }`)).toEqual([]);
+ });
+
+ it('ignores JSON.stringify named in a comment or a string', () => {
+ expect(findPgJsonViolations(`// params.push(JSON.stringify(x)) with ::jsonb
+ const note = 'values = [JSON.stringify(x)] ::jsonb';`)).toEqual([]);
+ });
+});
diff --git a/server/scripts/migrateCatalogPayload.js b/server/scripts/migrateCatalogPayload.js
index c1a9bfd168..a0fdadeef8 100644
--- a/server/scripts/migrateCatalogPayload.js
+++ b/server/scripts/migrateCatalogPayload.js
@@ -30,6 +30,7 @@
import { readMarker, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { CATALOG_TYPES, currentPayloadSchemaVersion, upgradePayload } from '../lib/catalogTypes.js';
+import { toPgJson } from '../lib/pgJson.js';
const MARKER_FILENAME = 'catalog-payload.applied.json';
@@ -70,7 +71,7 @@ async function migrateType(type) {
const next = upgradePayload(type.id, row.payload || {});
await query(
`UPDATE catalog_ingredients SET payload = $2::jsonb WHERE id = $1`,
- [row.id, JSON.stringify(next)],
+ [row.id, toPgJson(next)],
);
upgraded++;
}
diff --git a/server/scripts/migrateCatalogUserTypesToDB.js b/server/scripts/migrateCatalogUserTypesToDB.js
index 4a12534f72..cf95f2547a 100644
--- a/server/scripts/migrateCatalogUserTypesToDB.js
+++ b/server/scripts/migrateCatalogUserTypesToDB.js
@@ -34,6 +34,7 @@
import { query } from '../lib/db.js';
import { mirrorTimestamp } from '../lib/pgTimestamp.js';
import { getSettings, updateSettingsWith } from '../services/settings.js';
+import { toPgJson } from '../lib/pgJson.js';
const LEGACY_KEY = 'catalogUserTypes';
const IMPORTED_KEY = 'catalogUserTypes_imported';
@@ -79,7 +80,7 @@ export async function migrateCatalogUserTypesToDB() {
`INSERT INTO catalog_user_types (id, data, updated_at, deleted_at)
VALUES ($1, $2::jsonb, COALESCE($3::timestamptz, NOW()), $4)
ON CONFLICT (id) DO NOTHING`,
- [type.id, JSON.stringify(type), mirrorTimestamp(type.updatedAt, null), mirrorTimestamp(type.deletedAt, null)],
+ [type.id, toPgJson(type), mirrorTimestamp(type.updatedAt, null), mirrorTimestamp(type.deletedAt, null)],
);
if (result.rowCount > 0) imported += 1;
else skipped += 1;
diff --git a/server/scripts/migrateCreativeDirectorToDB.js b/server/scripts/migrateCreativeDirectorToDB.js
index 0d0b06d858..c9565b3d02 100644
--- a/server/scripts/migrateCreativeDirectorToDB.js
+++ b/server/scripts/migrateCreativeDirectorToDB.js
@@ -35,6 +35,7 @@ import { PATHS } from '../lib/fileUtils.js';
import { markerExists, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { mirrorStatus, mirrorTimestamp } from '../services/creativeDirector/projectsLogic.js';
+import { toPgJson } from '../lib/pgJson.js';
const LEGACY_FILENAME = 'creative-director-projects.json';
const IMPORTED_SUFFIX = '.imported';
@@ -91,7 +92,7 @@ export async function migrateCreativeDirectorToDB() {
[
project.id,
mirrorStatus(project.status),
- JSON.stringify(project),
+ toPgJson(project),
createdAt,
mirrorTimestamp(project.updatedAt, createdAt),
],
diff --git a/server/scripts/migrateIssuesToDB.js b/server/scripts/migrateIssuesToDB.js
index 9c505a8b0f..29ce3bec8d 100644
--- a/server/scripts/migrateIssuesToDB.js
+++ b/server/scripts/migrateIssuesToDB.js
@@ -26,6 +26,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { readLegacyJSON, legacyDirectory, incompleteImport } from './legacyImport.js';
import { mirrorTimestamp } from '../lib/pgTimestamp.js';
+import { toPgJson } from '../lib/pgJson.js';
const LEGACY_DIRNAME = 'pipeline-issues';
const IMPORTED_DIRNAME = 'pipeline-issues.imported';
@@ -47,7 +48,7 @@ export async function importRecord(record, execute = query) {
typeof record.seasonId === 'string' && record.seasonId ? record.seasonId : null,
Number.isFinite(record.number) ? Math.floor(record.number) : null,
typeof record.status === 'string' && record.status ? record.status.slice(0, 32) : null,
- JSON.stringify(record),
+ toPgJson(record),
record.ephemeral === true,
createdAt,
mirrorTimestamp(record.updatedAt, createdAt),
diff --git a/server/scripts/migrateSeriesToDB.js b/server/scripts/migrateSeriesToDB.js
index 0ce9d060b6..5399e83c37 100644
--- a/server/scripts/migrateSeriesToDB.js
+++ b/server/scripts/migrateSeriesToDB.js
@@ -37,6 +37,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { readLegacyJSON, legacyDirectory, parkLegacyFile, incompleteImport } from './legacyImport.js';
import { mirrorTimestamp } from '../lib/pgTimestamp.js';
+import { toPgJson } from '../lib/pgJson.js';
const LEGACY_DIRNAME = 'pipeline-series';
const MARKER_FILENAME = 'pipeline-series.migrated.json';
@@ -58,7 +59,7 @@ export async function importRecord(record, execute = query) {
typeof record.name === 'string' ? record.name : '',
typeof record.universeId === 'string' && record.universeId ? record.universeId : null,
typeof record.writersRoomWorkId === 'string' && record.writersRoomWorkId ? record.writersRoomWorkId : null,
- JSON.stringify(record),
+ toPgJson(record),
record.ephemeral === true,
createdAt,
mirrorTimestamp(record.updatedAt, createdAt),
diff --git a/server/scripts/migrateStoryBuilderToDB.js b/server/scripts/migrateStoryBuilderToDB.js
index 1689374794..9efbe0cfc6 100644
--- a/server/scripts/migrateStoryBuilderToDB.js
+++ b/server/scripts/migrateStoryBuilderToDB.js
@@ -27,6 +27,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { readLegacyJSON, legacyDirectory, incompleteImport } from './legacyImport.js';
import { mirrorTimestamp } from '../lib/pgTimestamp.js';
+import { toPgJson } from '../lib/pgJson.js';
const LEGACY_DIRNAME = 'story-builder';
const IMPORTED_DIRNAME = 'story-builder.imported';
@@ -49,7 +50,7 @@ export async function importRecord(record, execute = query) {
typeof record.universeId === 'string' && record.universeId ? record.universeId : null,
typeof record.seriesId === 'string' && record.seriesId ? record.seriesId : null,
record.sync === true,
- JSON.stringify(record),
+ toPgJson(record),
record.ephemeral === true,
createdAt,
mirrorTimestamp(record.updatedAt, createdAt),
diff --git a/server/scripts/migrateUniversesToDB.js b/server/scripts/migrateUniversesToDB.js
index d19617cd27..c1f486e1c7 100644
--- a/server/scripts/migrateUniversesToDB.js
+++ b/server/scripts/migrateUniversesToDB.js
@@ -35,6 +35,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { readLegacyJSON, legacyDirectory, incompleteImport } from './legacyImport.js';
import { mirrorTimestamp } from '../lib/pgTimestamp.js';
+import { toPgJson } from '../lib/pgJson.js';
const LEGACY_DIRNAME = 'universes';
const IMPORTED_DIRNAME = 'universes.imported';
@@ -55,7 +56,7 @@ export async function importRecord(record, execute = query) {
[
record.id,
typeof record.name === 'string' ? record.name : '',
- JSON.stringify(record),
+ toPgJson(record),
schemaVersion,
record.ephemeral === true,
createdAt,
@@ -77,7 +78,7 @@ export async function importRun(run, execute = query) {
run.id,
run.universeId,
typeof run.collectionId === 'string' ? run.collectionId : null,
- JSON.stringify(run),
+ toPgJson(run),
mirrorTimestamp(run.createdAt, new Date().toISOString()),
],
);
diff --git a/server/scripts/migrateUniversesToDB.test.js b/server/scripts/migrateUniversesToDB.test.js
index d316767464..013f66b210 100644
--- a/server/scripts/migrateUniversesToDB.test.js
+++ b/server/scripts/migrateUniversesToDB.test.js
@@ -59,14 +59,21 @@ vi.mock('fs/promises', () => ({
let uTable;
let rTable;
+// jsonb refuses the ASCII escape of an unpaired surrogate (`\ud83d`), as PostgreSQL does (#12815).
+const assertJsonbAccepts = (text) => {
+ const unpaired = text.replace(/\\ud[89ab][0-9a-f]{2}\\ud[c-f][0-9a-f]{2}/gi, '').match(/\\ud[89a-f][0-9a-f]{2}/i);
+ if (unpaired) throw new Error(`invalid input syntax for type json: lone surrogate ${unpaired[0]}`);
+};
const query = vi.fn(async (sql, params) => {
if (/INSERT INTO universes/.test(sql)) {
+ assertJsonbAccepts(params[2]);
const id = params[0];
if (uTable.has(id)) return { rowCount: 0, rows: [] };
uTable.set(id, { id, name: params[1], data: JSON.parse(params[2]) });
return { rowCount: 1, rows: [] };
}
if (/INSERT INTO universe_runs/.test(sql)) {
+ assertJsonbAccepts(params[3]);
const id = params[0];
if (rTable.has(id)) return { rowCount: 0, rows: [] };
rTable.set(id, { id, universeId: params[1] });
@@ -130,6 +137,16 @@ describe('migrateUniversesToDB', () => {
expect(MARKER in files).toBe(true);
});
+ it('imports a record holding a lone-surrogate \\ud83d escape, repaired, instead of retaining it for retry', async () => {
+ dirEntries = ['u-1'];
+ recordsByDir['/fake/data/universes/u-1/index.json'] = rec('u-1', { logline: 'cut\ud83d', credit: '\ud83d\ude00 kept' });
+ const result = await importer();
+ expect(result).toMatchObject({ ok: true, reason: 'imported', imported: 1 });
+ expect(uTable.get('u-1').data).toMatchObject({ logline: 'cut\uFFFD', credit: '😀 kept' });
+ expect(renamed).toEqual([['/fake/data/universes', '/fake/data/universes.imported']]);
+ expect(MARKER in files).toBe(true);
+ });
+
it('is idempotent: a re-run over the same rows imports 0 (ON CONFLICT DO NOTHING)', async () => {
dirEntries = ['u-1'];
recordsByDir['/fake/data/universes/u-1/index.json'] = rec('u-1');
diff --git a/server/scripts/migrateWritersRoomToDB.js b/server/scripts/migrateWritersRoomToDB.js
index e492ded1aa..5e0e0f6c86 100644
--- a/server/scripts/migrateWritersRoomToDB.js
+++ b/server/scripts/migrateWritersRoomToDB.js
@@ -35,6 +35,7 @@ import { markerExists, writeMarker } from '../lib/migrationMarker.js';
import { query } from '../lib/db.js';
import { readLegacyJSON, legacyDirectory, parkLegacyFile, incompleteImport } from './legacyImport.js';
import { mirrorTimestamp } from '../lib/pgTimestamp.js';
+import { toPgJson } from '../lib/pgJson.js';
const ROOT_DIRNAME = 'writers-room';
const MARKER_FILENAME = 'writers-room.migrated.json';
@@ -53,7 +54,7 @@ export async function importFolder(folder, execute = query) {
typeof folder.parentId === 'string' && folder.parentId ? folder.parentId : null,
String(folder.name ?? ''),
Number.isInteger(folder.sortOrder) ? folder.sortOrder : 0,
- JSON.stringify(folder),
+ toPgJson(folder),
createdAt,
mirrorTimestamp(folder.updatedAt, createdAt),
],
@@ -71,7 +72,7 @@ export async function importExercise(exercise, execute = query) {
exercise.id,
typeof exercise.workId === 'string' && exercise.workId ? exercise.workId : null,
typeof exercise.status === 'string' ? exercise.status.slice(0, 16) : null,
- JSON.stringify(exercise),
+ toPgJson(exercise),
mirrorTimestamp(exercise.startedAt, null),
mirrorTimestamp(exercise.finishedAt, null),
],
@@ -106,7 +107,7 @@ export async function importWork(manifest, execute = query) {
typeof manifest.pipelineIssueId === 'string' ? manifest.pipelineIssueId : null,
typeof manifest.cdProjectId === 'string' ? manifest.cdProjectId : null,
typeof manifest.mediaCollectionId === 'string' ? manifest.mediaCollectionId : null,
- JSON.stringify(workData),
+ toPgJson(workData),
createdAt,
mirrorTimestamp(manifest.updatedAt, createdAt),
],
@@ -125,9 +126,9 @@ export async function importWork(manifest, execute = query) {
String(draft.contentFile ?? ''),
typeof draft.contentHash === 'string' ? draft.contentHash : null,
Number.isInteger(draft.wordCount) ? draft.wordCount : 0,
- JSON.stringify(Array.isArray(draft.segmentIndex) ? draft.segmentIndex : []),
+ toPgJson(Array.isArray(draft.segmentIndex) ? draft.segmentIndex : []),
typeof draft.createdFromVersionId === 'string' ? draft.createdFromVersionId : null,
- JSON.stringify(draft),
+ toPgJson(draft),
mirrorTimestamp(draft.createdAt, createdAt),
],
);
diff --git a/server/services/albums/db.js b/server/services/albums/db.js
index 6717e8c6c6..a53f993ee8 100644
--- a/server/services/albums/db.js
+++ b/server/services/albums/db.js
@@ -16,6 +16,7 @@ import {
maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,
withBaseHashFlushBatch,
} from '../../lib/conflictJournal.js';
+import { toPgJson } from '../../lib/pgJson.js';
function rowToAlbum(row) {
return row ? row.data : null;
@@ -36,7 +37,7 @@ async function persist(exec, album) {
[
album.id,
typeof album.title === 'string' ? album.title : '',
- JSON.stringify(album),
+ toPgJson(album),
createdAt,
mirrorTimestamp(album.updatedAt, createdAt),
album.deleted === true,
diff --git a/server/services/appQuality.js b/server/services/appQuality.js
index eeb1138a6f..d755582037 100644
--- a/server/services/appQuality.js
+++ b/server/services/appQuality.js
@@ -6,6 +6,7 @@ import { tryReadFile } from '../lib/jsonIo.js';
import { normalizeAuditTaskType } from '../lib/auditCatalog.js';
import { AUDIT_FRESHNESS_MS, parseAuditQualityReport, parseAuditSlice, summarizeAppQuality, buildAppQualityHistory, dailyQualityRecords } from '../lib/auditQuality.js';
import { collectAppQuality, qualityRecord, readQualityRecords, readReleaseQuality } from './appQualityFederation.js';
+import { toPgJson } from '../lib/pgJson.js';
export async function recordAuditQuality({ task, taskType, agentId, workspacePath, success, assessedAt, deepDiscoveryComplete = false }, deps = {}) {
if (!success || !workspacePath || !task?.metadata?.app || !agentId) return false;
@@ -31,7 +32,7 @@ export async function recordAuditQuality({ task, taskType, agentId, workspacePat
`INSERT INTO app_quality_measurements (app_id, category, agent_id, assessed_at, report, slice)
VALUES ($1, $2, $3, $4, $5::jsonb, $6)
ON CONFLICT (app_id, category, agent_id) DO NOTHING`,
- [task.metadata.app, category, agentId, assessedAt, JSON.stringify(report), slice]
+ [task.metadata.app, category, agentId, assessedAt, toPgJson(report), slice]
);
return true;
}
diff --git a/server/services/artists/db.js b/server/services/artists/db.js
index 4c659fe2b7..fa67368a12 100644
--- a/server/services/artists/db.js
+++ b/server/services/artists/db.js
@@ -18,6 +18,7 @@ import {
maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,
withBaseHashFlushBatch,
} from '../../lib/conflictJournal.js';
+import { toPgJson } from '../../lib/pgJson.js';
function rowToArtist(row) {
return row ? row.data : null;
@@ -38,7 +39,7 @@ async function persist(exec, artist) {
[
artist.id,
typeof artist.name === 'string' ? artist.name : '',
- JSON.stringify(artist),
+ toPgJson(artist),
createdAt,
mirrorTimestamp(artist.updatedAt, createdAt),
artist.deleted === true,
diff --git a/server/services/authors/db.js b/server/services/authors/db.js
index 9633b1ea38..0bef60d678 100644
--- a/server/services/authors/db.js
+++ b/server/services/authors/db.js
@@ -21,6 +21,7 @@ import {
maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,
withBaseHashFlushBatch,
} from '../../lib/conflictJournal.js';
+import { toPgJson } from '../../lib/pgJson.js';
function rowToAuthor(row) {
return row ? row.data : null;
@@ -41,7 +42,7 @@ async function persist(exec, author) {
[
author.id,
typeof author.name === 'string' ? author.name : '',
- JSON.stringify(author),
+ toPgJson(author),
createdAt,
mirrorTimestamp(author.updatedAt, createdAt),
author.deleted === true,
diff --git a/server/services/catalogDB/commit.js b/server/services/catalogDB/commit.js
index 7f4ff68359..20e851377d 100644
--- a/server/services/catalogDB/commit.js
+++ b/server/services/catalogDB/commit.js
@@ -12,6 +12,7 @@ import { catalogScrapCommitSchema } from '../../lib/catalogValidation.js';
import { query, withTransaction } from '../../lib/db.js';
import { createIngredient } from './ingredients.js';
import { linkIngredientToSource, linkIngredientToRef, linkIngredientRelation, universeRefRoleForType } from './refs.js';
+import { toPgJson } from '../../lib/pgJson.js';
// Legacy same-batch `related-to` edges connect a single scrap's extractions into one
// cluster instead of N isolated nodes (#7615). Bounded: a batch above this
@@ -125,7 +126,7 @@ export async function commitScrap({ scrapId, accepted = [], embeds = [], univers
if (operationKey) {
await client.query(
'UPDATE catalog_commit_receipts SET ingredients = $2::jsonb WHERE operation_key = $1',
- [operationKey, JSON.stringify(created)],
+ [operationKey, toPgJson(created)],
);
}
return created;
diff --git a/server/services/catalogDB/ingredients.js b/server/services/catalogDB/ingredients.js
index 000798b76b..f4ff9dc1fb 100644
--- a/server/services/catalogDB/ingredients.js
+++ b/server/services/catalogDB/ingredients.js
@@ -30,6 +30,7 @@ import {
} from './shared.js';
import { normalizeTags } from './tags.js';
import { assertCatalogIdentityWritable } from './identityAccess.js';
+import { toPgJson } from '../../lib/pgJson.js';
// `{ client }` is optional — when supplied, SQL runs on the caller's transaction
// client (so the write rolls back if a later step in the same `withTransaction`
@@ -81,7 +82,7 @@ export async function createIngredient({ id: explicitId, type, name, payload = {
id,
type,
String(name).trim(),
- JSON.stringify(storedPayload),
+ toPgJson(storedPayload),
normalizedTags,
embedding ? arrayToPgvector(embedding) : null,
embeddingModel,
@@ -187,7 +188,7 @@ export async function updateIngredient(id, patch = {}, { client, source = 'user'
if (normalizedPatch[jsField] === undefined) continue;
if (jsField === 'payload') {
fields.push(`${dbField} = $${idx++}::jsonb`);
- params.push(JSON.stringify(normalizedPatch.payload || {}));
+ params.push(toPgJson(normalizedPatch.payload || {}));
} else if (jsField === 'embedding') {
fields.push(`${dbField} = $${idx++}`);
params.push(normalizedPatch.embedding ? arrayToPgvector(normalizedPatch.embedding) : null);
@@ -270,7 +271,7 @@ export async function recordIngredientRevision(ingredient, { source = 'user', ac
newRevisionId(),
ingredient.id,
ingredient.name,
- JSON.stringify(ingredient.payload || {}),
+ toPgJson(ingredient.payload || {}),
ingredient.tags || [],
src,
actor ? String(actor).slice(0, 120) : null,
@@ -346,7 +347,7 @@ export async function reviveDeletedIngredient(id, { type, name, payload = {}, ta
type = $2, name = $3, payload = $4::jsonb, tags = $5
WHERE id = $1 AND deleted = true
RETURNING *`,
- [id, type, String(name).trim(), JSON.stringify(storedPayload), tags || []],
+ [id, type, String(name).trim(), toPgJson(storedPayload), tags || []],
);
return result.rows.length > 0 ? rowToIngredient(result.rows[0]) : null;
}
diff --git a/server/services/catalogDB/media.js b/server/services/catalogDB/media.js
index 35220220b3..dbd18e7474 100644
--- a/server/services/catalogDB/media.js
+++ b/server/services/catalogDB/media.js
@@ -13,6 +13,7 @@ import { resolveImageInputPath } from '../../lib/fileUtils.js';
import { rowToMedia } from './shared.js';
import { MAX_CAST_GUIDES, normalizeLibrary } from '../../lib/catalogMediaLibrary.js';
import { assertCatalogIdentityWritable, assertCatalogAssociationRemovable } from './identityAccess.js';
+import { toPgJson } from '../../lib/pgJson.js';
const hasOwn = (value, key) => Object.prototype.hasOwnProperty.call(value || {}, key);
const hasMeaningfulMetadata = (value) => value && typeof value === 'object'
@@ -44,7 +45,7 @@ async function insertMedia(runQuery, ingredientId, mediaKey, kind, options) {
RETURNING *`,
[
ingredientId, mediaKey, kind, role, caption,
- ...(hasMetadata ? [JSON.stringify(options.metadata || {})] : []),
+ ...(hasMetadata ? [toPgJson(options.metadata || {})] : []),
],
);
return rowToMedia(result.rows[0]);
@@ -176,7 +177,7 @@ export async function updateMediaLibrary(ingredientId, mediaKey, kind, patch = {
`UPDATE catalog_ingredient_media SET library = $4::jsonb
WHERE ingredient_id = catalog_resolve_ingredient_id($1) AND media_key = $2 AND kind = $3 AND deleted = false
RETURNING *`,
- [ingredientId, mediaKey, kind, JSON.stringify(next)],
+ [ingredientId, mediaKey, kind, toPgJson(next)],
);
return rowToMedia(result.rows[0]);
});
diff --git a/server/services/catalogDB/merge.js b/server/services/catalogDB/merge.js
index a06b22076c..0d34ea27db 100644
--- a/server/services/catalogDB/merge.js
+++ b/server/services/catalogDB/merge.js
@@ -8,6 +8,7 @@ import { getInstanceId, UNKNOWN_INSTANCE_ID } from '../instanceIdentity.js';
import { listSheetPointers } from '../../lib/sheetPointers.js';
import { sanitizeBiblePayload } from './shared.js';
import { catalogMergeSchema } from '../../lib/catalogValidation.js';
+import { toPgJson } from '../../lib/pgJson.js';
const TABLES = Object.freeze({
ingredients: ['catalog_ingredients', 'id = ANY($1)'],
@@ -123,7 +124,7 @@ export async function mergeIngredients(input) {
await client.query(`UPDATE catalog_ingredients SET name = $2, payload = $3::jsonb, tags = $4,
embedding = NULL, embedding_model = NULL WHERE id = $1`,
- [survivorId, proposed.name, JSON.stringify(proposed.payload), proposed.tags]);
+ [survivorId, proposed.name, toPgJson(proposed.payload), proposed.tags]);
// Copy only live associations; retain the original rows and collision
// provenance in the receipt rather than overwriting survivor captions/spans.
await client.query(`INSERT INTO catalog_ingredient_refs (ingredient_id, ref_kind, ref_id, role, created_at)
@@ -158,7 +159,7 @@ export async function mergeIngredients(input) {
await client.query(`INSERT INTO catalog_ingredient_media (ingredient_id, media_key, kind, role, caption, metadata, created_at, library)
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8::jsonb)
ON CONFLICT (ingredient_id, media_key, kind) DO UPDATE SET deleted = false, deleted_at = NULL`,
- [survivorId, row.media_key, row.kind, row.role, row.caption, JSON.stringify(row.metadata), row.created_at, JSON.stringify(row.library || {})]);
+ [survivorId, row.media_key, row.kind, row.role, row.caption, toPgJson(row.metadata), row.created_at, toPgJson(row.library || {})]);
}
await client.query(`INSERT INTO catalog_ingredient_relations (from_id, to_id, kind, created_at)
SELECT CASE WHEN from_id = $1 THEN $2 ELSE from_id END,
@@ -168,12 +169,12 @@ export async function mergeIngredients(input) {
ON CONFLICT (from_id, to_id, kind) DO UPDATE SET deleted = false, deleted_at = NULL`, [sourceId, survivorId, ids]);
await client.query(`INSERT INTO catalog_ingredient_revisions (id, ingredient_id, name, payload, tags, source, actor)
VALUES ($1, $2, $3, $4::jsonb, $5, 'user', 'catalog-merge')`,
- [`cat-rev-${randomUUID()}`, survivorId, proposed.name, JSON.stringify(proposed.payload), proposed.tags]);
+ [`cat-rev-${randomUUID()}`, survivorId, proposed.name, toPgJson(proposed.payload), proposed.tags]);
const after = await snapshot(client, ids);
const id = randomUUID();
await client.query(`INSERT INTO catalog_ingredient_merges
(id, source_id, survivor_id, before_state, after_fingerprint) VALUES ($1, $2, $3, $4::jsonb, $5)`,
- [id, sourceId, survivorId, JSON.stringify({ ...before, decision: { scope: 'local', fieldChoices, proposed } }), fingerprint(after)]);
+ [id, sourceId, survivorId, toPgJson({ ...before, decision: { scope: 'local', fieldChoices, proposed } }), fingerprint(after)]);
return { id, sourceId, survivorId, merged: true, scope: 'local' };
});
}
@@ -227,19 +228,19 @@ export async function recoverIngredientMerge(id) {
const updates = columns.filter(column => !keys.includes(column)).map(column => `${column} = EXCLUDED.${column}`);
await client.query(`INSERT INTO ${table} (${columns.join(', ')})
SELECT ${columns.join(', ')} FROM jsonb_populate_recordset(NULL::${table}, $1::jsonb)
- ON CONFLICT (${keys.join(', ')}) DO UPDATE SET ${updates.join(', ')}`, [JSON.stringify(rows)]);
+ ON CONFLICT (${keys.join(', ')}) DO UPDATE SET ${updates.join(', ')}`, [toPgJson(rows)]);
}
// Keep the merge revision as an audit event. Recovery adds a new snapshot;
// the receipt retains the complete before/after identity history.
for (const row of receipt.before_state.ingredients) {
await client.query(`UPDATE catalog_ingredients SET name = $2, payload = $3::jsonb, tags = $4,
embedding = $5, embedding_model = $6, deleted = $7, deleted_at = $8 WHERE id = $1`,
- [row.id, row.name, JSON.stringify(row.payload), row.tags, row.embedding, row.embedding_model, row.deleted, row.deleted_at]);
+ [row.id, row.name, toPgJson(row.payload), row.tags, row.embedding, row.embedding_model, row.deleted, row.deleted_at]);
}
for (const row of receipt.before_state.ingredients) {
await client.query(`INSERT INTO catalog_ingredient_revisions (id, ingredient_id, name, payload, tags, source, actor)
VALUES ($1, $2, $3, $4::jsonb, $5, 'user', 'catalog-merge-recovery')`,
- [`cat-rev-${randomUUID()}`, row.id, row.name, JSON.stringify(row.payload), row.tags]);
+ [`cat-rev-${randomUUID()}`, row.id, row.name, toPgJson(row.payload), row.tags]);
}
return { id, restored: true };
});
diff --git a/server/services/catalogDB/refs.js b/server/services/catalogDB/refs.js
index 5460a2d081..09211a246a 100644
--- a/server/services/catalogDB/refs.js
+++ b/server/services/catalogDB/refs.js
@@ -14,6 +14,7 @@ import { query } from '../../lib/db.js';
import { resolveRefs } from '../catalogRefResolver.js';
import { assertCatalogIdentityWritable, assertCatalogAssociationRemovable, assertCatalogRelationRemovable } from './identityAccess.js';
import { rowToRef, rowToSource, rowToIngredient } from './shared.js';
+import { toPgJson } from '../../lib/pgJson.js';
// Canonical detail views omit local-merge originals; original-ID user exports
// retain those raw rows alongside the bilateral receipt-backed evidence.
@@ -32,7 +33,7 @@ export async function linkIngredientToSource(ingredientId, scrapId, span = null,
`INSERT INTO catalog_ingredient_sources (ingredient_id, scrap_id, span)
VALUES (catalog_resolve_ingredient_id($1), $2, $3::jsonb)
ON CONFLICT (ingredient_id, scrap_id) ${preserveExisting ? 'DO NOTHING' : 'DO UPDATE SET span = EXCLUDED.span'}`,
- [ingredientId, scrapId, span ? JSON.stringify(span) : null],
+ [ingredientId, scrapId, span ? toPgJson(span) : null],
);
}
diff --git a/server/services/catalogDB/scraps.js b/server/services/catalogDB/scraps.js
index c26354216d..46010ae07a 100644
--- a/server/services/catalogDB/scraps.js
+++ b/server/services/catalogDB/scraps.js
@@ -10,6 +10,7 @@ import { query, withTransaction, arrayToPgvector } from '../../lib/db.js';
import { chunkRawText } from '../../lib/catalogChunking.js';
import { getInstanceId } from '../instanceIdentity.js';
import { newScrapId, rowToScrap } from './shared.js';
+import { toPgJson } from '../../lib/pgJson.js';
// `{ client }` is optional — when supplied, the INSERT runs on the caller's
// transaction client so a bulk-import batch can commit-or-rollback its scraps
@@ -31,7 +32,7 @@ export async function createScrap({ id: explicitId, title, rawText, sourceKind =
title || null,
rawText,
sourceKind,
- JSON.stringify(metadata || {}),
+ toPgJson(metadata || {}),
embedding ? arrayToPgvector(embedding) : null,
embeddingModel,
originInstanceId,
@@ -141,7 +142,7 @@ export async function updateScrap(id, patch = {}) {
if (patch[jsField] === undefined) continue;
if (jsField === 'metadata') {
fields.push(`${dbField} = $${idx++}::jsonb`);
- params.push(JSON.stringify(patch.metadata || {}));
+ params.push(toPgJson(patch.metadata || {}));
} else if (jsField === 'embedding') {
fields.push(`${dbField} = $${idx++}`);
params.push(patch.embedding ? arrayToPgvector(patch.embedding) : null);
diff --git a/server/services/catalogDB/sync.js b/server/services/catalogDB/sync.js
index 12ee3eb6d4..02d92e3cc0 100644
--- a/server/services/catalogDB/sync.js
+++ b/server/services/catalogDB/sync.js
@@ -28,6 +28,7 @@ import {
rowToMedia,
rowToTag,
} from './shared.js';
+import { toPgJson } from '../../lib/pgJson.js';
// One page of a table's feed: rows whose committed feed position is past the
// peer's cursor, in position order. `syncSequence` on each item carries the
@@ -122,14 +123,14 @@ export async function upsertMediaFromPeer(media) {
const metadataColumn = hasMetadata ? ', metadata' : '';
const metadataPlaceholder = hasMetadata ? `, $${hasTombstoneFields ? 10 : 7}` : '';
const metadataUpdate = hasMetadata ? ', metadata = EXCLUDED.metadata' : '';
- const metadataParam = hasMetadata ? [JSON.stringify(media.metadata || {})] : [];
+ const metadataParam = hasMetadata ? [toPgJson(media.metadata || {})] : [];
// Casting reference library (catalog v12): absent = no opinion, same as metadata.
const hasLibrary = Object.prototype.hasOwnProperty.call(media, 'library')
&& media.library && typeof media.library === 'object' && !Array.isArray(media.library);
const libraryColumn = hasLibrary ? ', library' : '';
const libraryPlaceholder = hasLibrary ? `, $${(hasTombstoneFields ? 10 : 7) + (hasMetadata ? 1 : 0)}` : '';
const libraryUpdate = hasLibrary ? ', library = EXCLUDED.library' : '';
- const libraryParam = hasLibrary ? [JSON.stringify(normalizeLibrary(media.library))] : [];
+ const libraryParam = hasLibrary ? [toPgJson(normalizeLibrary(media.library))] : [];
if (hasTombstoneFields) {
// #8347 revival guard — see upsertRelationFromPeer for the rationale.
// `updated_at` also moves on a role/caption/metadata-only edit (the
@@ -267,7 +268,7 @@ export async function upsertScrapFromPeer(scrap) {
scrap.title || null,
scrap.rawText,
scrap.sourceKind || 'paste',
- JSON.stringify(scrap.metadata || {}),
+ toPgJson(scrap.metadata || {}),
scrap.embedding ? arrayToPgvector(scrap.embedding) : null,
scrap.embeddingModel || null,
scrap.originInstanceId || null,
@@ -319,7 +320,7 @@ export async function upsertIngredientFromPeer(ing) {
ing.id,
ing.type,
ing.name,
- JSON.stringify(ing.payload || {}),
+ toPgJson(ing.payload || {}),
ing.tags || [],
ing.embedding ? arrayToPgvector(ing.embedding) : null,
ing.embeddingModel || null,
@@ -340,7 +341,7 @@ export async function upsertSourceFromPeer(src) {
`INSERT INTO catalog_ingredient_sources (ingredient_id, scrap_id, span, extracted_at)
VALUES ($1, $2, $3::jsonb, $4)
ON CONFLICT (ingredient_id, scrap_id) DO UPDATE SET span = EXCLUDED.span`,
- [src.ingredientId, src.scrapId, src.span ? JSON.stringify(src.span) : null, src.extractedAt],
+ [src.ingredientId, src.scrapId, src.span ? toPgJson(src.span) : null, src.extractedAt],
);
});
}
@@ -422,7 +423,7 @@ async function deferParentApply(kind, row, parentId, updatedAt) {
source_updated_at = EXCLUDED.source_updated_at,
payload = EXCLUDED.payload
WHERE EXCLUDED.source_updated_at > catalog_pending_applies.source_updated_at`,
- [kind, row.id, parentId, updatedAt, JSON.stringify(row)],
+ [kind, row.id, parentId, updatedAt, toPgJson(row)],
);
}
@@ -453,7 +454,7 @@ export async function drainPendingCatalogApplies() {
const removed = await query(
`DELETE FROM catalog_pending_applies
WHERE kind = $1 AND id = $2 AND payload = $3::jsonb`,
- [kind, pending.id, JSON.stringify(pending.payload)],
+ [kind, pending.id, toPgJson(pending.payload)],
);
progressed ||= removed.rowCount > 0;
}
diff --git a/server/services/catalogIdentity/index.js b/server/services/catalogIdentity/index.js
index 819a04ae4b..959b413495 100644
--- a/server/services/catalogIdentity/index.js
+++ b/server/services/catalogIdentity/index.js
@@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto';
import { query, withTransaction } from '../../lib/db.js';
import { snapshot, lockCatalog, ingredient, assertUnlocked, buildPlan, hash, fail, validateSnapshot } from './snapshots.js';
import { localIdentity, identityPeer, readEvidence, revalidateEvidence } from './transport.js';
+import { toPgJson } from '../../lib/pgJson.js';
const db = { query };
const idsOf = row => [row.source_id, row.survivor_id];
@@ -49,7 +50,7 @@ async function update(client, row, state, epoch, extra = {}) {
const columns = { state, epoch, ...extra };
const values = [row.id];
const sets = Object.entries(columns).map(([key, value]) => {
- values.push(value && typeof value === 'object' ? JSON.stringify(value) : value);
+ values.push(value && typeof value === 'object' ? toPgJson(value) : value);
return `${key} = $${values.length}`;
});
await client.query(`UPDATE catalog_identity_operations SET ${sets.join(', ')}, updated_at = NOW() WHERE id = $1`, values);
@@ -60,10 +61,10 @@ async function writeContent(client, row, original = false) {
const content = original ? ingredient(row.before_state, row.survivor_id) : row.plan.proposed;
await client.query(`UPDATE catalog_ingredients SET name = $2, payload = $3::jsonb, tags = $4,
embedding = NULL, embedding_model = NULL WHERE id = $1`,
- [row.survivor_id, content.name, JSON.stringify(content.payload), content.tags]);
+ [row.survivor_id, content.name, toPgJson(content.payload), content.tags]);
await client.query(`INSERT INTO catalog_ingredient_revisions (id, ingredient_id, name, payload, tags, source, actor)
VALUES ($1, $2, $3, $4::jsonb, $5, 'user', $6)`,
- [`cat-rev-${randomUUID()}`, row.survivor_id, content.name, JSON.stringify(content.payload), content.tags,
+ [`cat-rev-${randomUUID()}`, row.survivor_id, content.name, toPgJson(content.payload), content.tags,
original ? 'catalog-identity-recovery' : 'catalog-shared-identity']);
}
async function adoptContent(client, row, content) {
@@ -72,7 +73,7 @@ async function adoptContent(client, row, content) {
// snapshot retain the actual revision identity and owner timestamp.
await client.query(`UPDATE catalog_ingredients SET name=$2,payload=$3::jsonb,tags=$4,updated_at=$5,
embedding=NULL,embedding_model=NULL WHERE id=$1`,
- [row.survivor_id, content.name, JSON.stringify(content.payload), content.tags, content.updated_at]);
+ [row.survivor_id, content.name, toPgJson(content.payload), content.tags, content.updated_at]);
}
export async function listIdentityOperations({ ingredientId, limit = 20, offset = 0 } = {}) {
const { rows } = await query(`SELECT id, source_id, survivor_id, source_owner, survivor_owner, local_owner,
@@ -127,7 +128,7 @@ export async function createIdentity(input) {
await client.query(`INSERT INTO catalog_identity_operations
(id, source_id, survivor_id, source_owner, survivor_owner, local_owner, state, plan, plan_hash)
VALUES ($1,$2,$3,$4,$5,$6,'proposed',$7::jsonb,$8)`,
- [id, plan.sourceId, plan.survivorId, plan.sourceOwner, plan.survivorOwner, self, JSON.stringify(plan), preview.planHash]);
+ [id, plan.sourceId, plan.survivorId, plan.sourceOwner, plan.survivorOwner, self, toPgJson(plan), preview.planHash]);
return present(await load(client, id));
});
}
@@ -156,7 +157,7 @@ export async function importIdentity({ peerId, operationId }) {
await client.query(`INSERT INTO catalog_identity_operations
(id,source_id,survivor_id,source_owner,survivor_owner,local_owner,state,plan,plan_hash)
VALUES ($1,$2,$3,$4,$5,$6,'proposed',$7::jsonb,$8)`,
- [operationId, remote.sourceId, remote.survivorId, self, peerId, self, JSON.stringify(remote.plan), remote.planHash]);
+ [operationId, remote.sourceId, remote.survivorId, self, peerId, self, toPgJson(remote.plan), remote.planHash]);
return present(await load(client, operationId));
});
}
diff --git a/server/services/catalogUserTypes/db.js b/server/services/catalogUserTypes/db.js
index ad7ca0dcd0..9a9f33b701 100644
--- a/server/services/catalogUserTypes/db.js
+++ b/server/services/catalogUserTypes/db.js
@@ -16,6 +16,7 @@
import { query, withTransaction } from '../../lib/db.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
function rowToType(row) {
// `data` already holds id/label/primaryContentKey/fields/updatedAt/deletedAt;
@@ -86,7 +87,7 @@ async function applyWrite(exec, list) {
data = EXCLUDED.data,
updated_at = EXCLUDED.updated_at,
deleted_at = EXCLUDED.deleted_at`,
- [t.id, JSON.stringify(t), updatedAt, deletedAt],
+ [t.id, toPgJson(t), updatedAt, deletedAt],
);
}
// Prune rows that left the desired slice. With keepIds empty the whole
diff --git a/server/services/codeAnimation/jobStore.js b/server/services/codeAnimation/jobStore.js
index 3c86e131a7..481bab8914 100644
--- a/server/services/codeAnimation/jobStore.js
+++ b/server/services/codeAnimation/jobStore.js
@@ -9,6 +9,7 @@ import { query } from '../../lib/db.js';
import { PATHS } from '../../lib/paths.js';
import { atomicWrite } from '../../lib/fileUtils.js';
import { ServerError } from '../../lib/errorHandler.js';
+import { toPgJson } from '../../lib/pgJson.js';
const JOB_ID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const ANIMATIONS_DIR = join(PATHS.data, 'code-animations');
@@ -81,7 +82,7 @@ export async function saveCodeAnimationJobRecord(job) {
model = EXCLUDED.model,
data = EXCLUDED.data,
updated_at = EXCLUDED.updated_at`,
- [job.id, job.status, job.title, job.input?.concept || '', job.providerId, job.model, JSON.stringify(job), job.createdAt, job.updatedAt],
+ [job.id, job.status, job.title, job.input?.concept || '', job.providerId, job.model, toPgJson(job), job.createdAt, job.updatedAt],
);
return job;
}
diff --git a/server/services/codeAnimation/projectStore.js b/server/services/codeAnimation/projectStore.js
index db444af9e5..3a3478db4e 100644
--- a/server/services/codeAnimation/projectStore.js
+++ b/server/services/codeAnimation/projectStore.js
@@ -2,6 +2,7 @@
import { randomUUID } from 'crypto';
import { query, withTransaction } from '../../lib/db.js';
import { ServerError } from '../../lib/errorHandler.js';
+import { toPgJson } from '../../lib/pgJson.js';
const columns = `id, title, data, accepted_revision_id AS "acceptedRevisionId",
candidate_revision_id AS "candidateRevisionId", created_at AS "createdAt", updated_at AS "updatedAt"`;
@@ -210,8 +211,8 @@ export async function setAcceptedOutput(projectId, accepted) {
const result = await client.query(`UPDATE code_animation_projects SET data = data || jsonb_build_object('acceptedOutput', $2::jsonb, 'acceptanceProjection', $5::jsonb),
accepted_revision_id = CASE WHEN $4 THEN $3 ELSE accepted_revision_id END,
candidate_revision_id = CASE WHEN $4 THEN NULL ELSE candidate_revision_id END, updated_at = NOW()
- WHERE id = $1 RETURNING ${columns}`, [projectId, JSON.stringify(accepted), accepted.revisionId, promoteSource,
- JSON.stringify({ decisionId: randomUUID(), status: 'pending' })]);
+ WHERE id = $1 RETURNING ${columns}`, [projectId, toPgJson(accepted), accepted.revisionId, promoteSource,
+ toPgJson({ decisionId: randomUUID(), status: 'pending' })]);
return present(result.rows[0]);
});
}
diff --git a/server/services/cosTaskRepository.db.test.js b/server/services/cosTaskRepository.db.test.js
index f9ceb5a47e..f6f428d97b 100644
--- a/server/services/cosTaskRepository.db.test.js
+++ b/server/services/cosTaskRepository.db.test.js
@@ -3,11 +3,12 @@
* _cosTaskRepositoryContract.js. Runs via `npm run test:db` (→ `portos_test`)
* ONLY; skips cleanly when no database is reachable.
*/
-import { afterAll, describe } from 'vitest';
+import { afterAll, describe, expect, it } from 'vitest';
import { checkHealth, close, ensureSchema, query, withTransaction } from '../lib/db.js';
import { requireDbOrSkip } from '../lib/dbTestGate.js';
import { __makePgCosTaskBackend } from './cosTaskRepository.js';
import { __runCosTaskRepositoryContract } from './_cosTaskRepositoryContract.js';
+import { planListSave } from '../lib/cosTaskRows.js';
const health = await checkHealth().catch(() => ({ connected: false }));
if (health.connected) await ensureSchema();
@@ -28,3 +29,30 @@ describe.skipIf(!runDb)('cos_tasks repository on PostgreSQL', () => {
},
});
});
+
+// PostgreSQL refuses a lone surrogate in jsonb and NUL in jsonb/text; the file
+// stores these queues came from accepted both (#12815).
+describe.skipIf(!runDb)('cos_tasks repository text PostgreSQL would refuse', () => {
+ const task = (n, description) => ({
+ id: `task-example-${n}`, status: 'pending', priority: 'MEDIUM', priorityValue: 2, description,
+ approvalRequired: false, autoApproved: true, metadata: {},
+ });
+
+ it('repairs a split emoji and a NUL to U+FFFD instead of failing the save', async () => {
+ await query('DELETE FROM cos_tasks');
+ await query('DELETE FROM cos_task_lists');
+ const backend = __makePgCosTaskBackend({ query, withTransaction });
+ // The cut askPromote makes: 279 characters, then half of an emoji's surrogate pair.
+ const split = `${'a'.repeat(279)}${'😀'.slice(0, 1)}`;
+ const next = [task(1, split), task(2, 'before\0after')];
+ const { rows } = await backend.loadWorkingSet('user');
+ const plan = planListSave({ previous: rows, next, source: 'user' });
+ await backend.apply('user', { upserts: plan.upserts, deletes: plan.deletes });
+
+ const saved = Object.fromEntries((await backend.loadWorkingSet('user')).rows.map((row) => [row.task.id, row.task]));
+ expect(saved['task-example-1'].description).toBe(`${'a'.repeat(279)}\uFFFD`);
+ expect(saved['task-example-2'].description).toBe('before\uFFFDafter');
+ const { rows: [column] } = await query('SELECT description FROM cos_tasks WHERE task_id = $1', ['task-example-2']);
+ expect(column.description).toBe('before\uFFFDafter');
+ });
+});
diff --git a/server/services/cosTaskRepository.js b/server/services/cosTaskRepository.js
index faab401a43..affe704dd4 100644
--- a/server/services/cosTaskRepository.js
+++ b/server/services/cosTaskRepository.js
@@ -20,6 +20,7 @@ import { isTestRunner } from '../lib/runtimeEnv.js';
import {
WORKING_SET_COMPLETED_WINDOW_MS, WORKING_SET_KEEP_COMPLETED, completionStamp, rowColumns, selectWorkingSet,
} from '../lib/cosTaskRows.js';
+import { toPgJson } from '../lib/pgJson.js';
const SOURCES = ['user', 'internal'];
// Bounds a single statement's parameter payload (prompts can be MBs).
@@ -65,7 +66,7 @@ function makePgBackend(db) {
position = EXCLUDED.position, completed_at = EXCLUDED.completed_at, description = EXCLUDED.description, task = EXCLUDED.task`,
[source, columns.map((c) => c.task.id), columns.map((c) => c.status), columns.map((c) => c.priorityValue),
columns.map((c) => c.position), columns.map((c) => c.completedAt), columns.map((c) => c.description),
- columns.map((c) => JSON.stringify(c.task))],
+ columns.map((c) => toPgJson(c.task))],
);
}
return bumpVersions(client, [source]);
diff --git a/server/services/creativeCommissions/db.js b/server/services/creativeCommissions/db.js
index cf4c67627a..26f515efd1 100644
--- a/server/services/creativeCommissions/db.js
+++ b/server/services/creativeCommissions/db.js
@@ -23,6 +23,7 @@
import { query } from '../../lib/db.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
/** Raw stored record (the `data` JSONB), or null. Live only unless includeDeleted (#2686). */
export async function readRaw(id, { includeDeleted = false } = {}) {
@@ -81,7 +82,7 @@ export async function writeRaw(id, record) {
id,
typeof record?.name === 'string' ? record.name : '',
record?.enabled !== false,
- JSON.stringify(record),
+ toPgJson(record),
createdAt,
mirrorTimestamp(record?.updatedAt, createdAt),
deleted,
diff --git a/server/services/creativeCommissions/feedbackDb.js b/server/services/creativeCommissions/feedbackDb.js
index 3b51e20be8..723d307bbc 100644
--- a/server/services/creativeCommissions/feedbackDb.js
+++ b/server/services/creativeCommissions/feedbackDb.js
@@ -24,6 +24,7 @@
import { query } from '../../lib/db.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
/** Raw stored record (the `data` JSONB), or null. Live only unless includeDeleted. */
export async function readRaw(id, { includeDeleted = false } = {}) {
@@ -90,7 +91,7 @@ export async function writeRaw(id, record) {
id,
typeof record?.commissionId === 'string' && record.commissionId ? record.commissionId : null,
typeof record?.runId === 'string' && record.runId ? record.runId : null,
- JSON.stringify(record),
+ toPgJson(record),
createdAt,
mirrorTimestamp(record?.updatedAt, createdAt),
deleted,
diff --git a/server/services/deepAudit.js b/server/services/deepAudit.js
index ec0f8dcb51..f87e2f9f5d 100644
--- a/server/services/deepAudit.js
+++ b/server/services/deepAudit.js
@@ -10,6 +10,7 @@ import { normalizeAuditTaskType, isAuditTaskType } from '../lib/auditCatalog.js'
import { deepAuditProgress, mergeDeepAuditReport, DEEP_AUDIT_CONTRACT_VERSION } from '../lib/deepAudit.js';
import { PROMPT_VERSIONS } from './taskPromptDefaults/versions.js';
import { resolveTaskHookType } from './taskTypeHooks.js';
+import { toPgJson } from '../lib/pgJson.js';
function deepAuditId(task) {
const meta = task.metadata || {};
@@ -59,11 +60,11 @@ async function mutateLedger(id, initial, mutate) {
await ensureSchema();
return withTransaction(async client => {
if (initial) await client.query(`INSERT INTO deep_audit_ledgers (id, app_id, category, ledger)
- VALUES ($1,$2,$3,$4::jsonb) ON CONFLICT (id) DO NOTHING`, [id, initial.appId, initial.category, JSON.stringify(initial)]);
+ VALUES ($1,$2,$3,$4::jsonb) ON CONFLICT (id) DO NOTHING`, [id, initial.appId, initial.category, toPgJson(initial)]);
const { rows } = await client.query('SELECT ledger FROM deep_audit_ledgers WHERE id=$1 FOR UPDATE', [id]);
if (!rows.length) throw new Error('Deep audit ledger is unavailable');
const updated = await mutate(rows[0].ledger);
- await client.query('UPDATE deep_audit_ledgers SET ledger=$2::jsonb, updated_at=NOW() WHERE id=$1', [id, JSON.stringify(updated)]);
+ await client.query('UPDATE deep_audit_ledgers SET ledger=$2::jsonb, updated_at=NOW() WHERE id=$1', [id, toPgJson(updated)]);
return updated;
});
}
diff --git a/server/services/fableLoom/db.js b/server/services/fableLoom/db.js
index a29efc6e0f..8f7c6a9778 100644
--- a/server/services/fableLoom/db.js
+++ b/server/services/fableLoom/db.js
@@ -6,6 +6,7 @@
import { query } from '../../lib/db.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
export async function readRaw(id) {
const { rows } = await query('SELECT data FROM fableloom_stories WHERE id = $1', [id]);
@@ -69,7 +70,7 @@ export async function writeRaw(id, record) {
record.name,
record.universeId ?? null,
record.seriesId ?? null,
- JSON.stringify(record),
+ toPgJson(record),
createdAt,
updatedAt,
record.deleted === true,
diff --git a/server/services/games/db.js b/server/services/games/db.js
index da5ff7fce7..a7fafbaa85 100644
--- a/server/services/games/db.js
+++ b/server/services/games/db.js
@@ -5,6 +5,7 @@
import { query } from '../../lib/db.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
export async function readRaw(id) {
const { rows } = await query('SELECT data FROM games WHERE id = $1', [id]);
@@ -28,7 +29,7 @@ export async function writeRaw(id, record) {
name = EXCLUDED.name,
data = EXCLUDED.data,
updated_at = EXCLUDED.updated_at`,
- [id, record.appId, record.name, JSON.stringify(record), createdAt, updatedAt],
+ [id, record.appId, record.name, toPgJson(record), createdAt, updatedAt],
);
return record;
}
diff --git a/server/services/humanActivity.js b/server/services/humanActivity.js
index 1ed2720c1c..ef48735cf1 100644
--- a/server/services/humanActivity.js
+++ b/server/services/humanActivity.js
@@ -26,6 +26,7 @@ import { ensureSchema, query, withTransaction } from '../lib/db.js';
import { normalizeIdentifier } from '../lib/tribeMatch.js';
import { getLocalParts, localDayRangeUtc, todayInTimezone, anchorLocalMidnightUtc } from '../lib/timezone.js';
import { getUserTimezone } from './userTimezone.js';
+import { toPgJson } from '../lib/pgJson.js';
export { localDayRangeUtc };
@@ -337,7 +338,7 @@ export async function recordEvents(candidates = []) {
// Insert one chunk of normalized rows; returns the count actually inserted.
// 12 bound params per row (created_at is a NOW() literal, not a param);
-// participants and metadata are JSON.stringify'd + ::jsonb-cast (node-pg would
+// participants and metadata are toPgJson-serialized + ::jsonb-cast (node-pg would
// otherwise coerce a JS array to a Postgres array literal and fail on jsonb).
async function insertActivityChunk(rows) {
const paramsPerRow = 12;
@@ -346,8 +347,8 @@ async function insertActivityChunk(rows) {
const b = i * paramsPerRow;
values.push(
r.id, r.source, r.accountId, r.kind, r.happenedAt, r.durationS,
- r.title, r.summary, r.url, JSON.stringify(r.participants),
- JSON.stringify(r.metadata), r.dedupeKey,
+ r.title, r.summary, r.url, toPgJson(r.participants),
+ toPgJson(r.metadata), r.dedupeKey,
);
return `($${b + 1}, $${b + 2}, $${b + 3}, $${b + 4}, $${b + 5}, $${b + 6}, $${b + 7}, $${b + 8}, $${b + 9}, $${b + 10}::jsonb, $${b + 11}::jsonb, $${b + 12}, NOW())`;
});
@@ -447,7 +448,7 @@ export async function listEvents({ from, to, source, accountId, kind, personId,
}
if (kind) { params.push(kind); clauses.push(`kind = $${params.length}`); }
if (personId) {
- params.push(JSON.stringify([{ personId: String(personId) }]));
+ params.push(toPgJson([{ personId: String(personId) }]));
clauses.push(`participants @> $${params.length}::jsonb`);
}
if (chatGuid != null && String(chatGuid).length > 0) {
diff --git a/server/services/imageTo3d/db.js b/server/services/imageTo3d/db.js
index 179399cec2..c9d6c412c8 100644
--- a/server/services/imageTo3d/db.js
+++ b/server/services/imageTo3d/db.js
@@ -12,6 +12,7 @@ import { modelLifecycleEvents } from '../modelLifecycleEvents.js';
import { randomUUID } from 'crypto';
import { query, withTransaction } from '../../lib/db.js';
import { ServerError } from '../../lib/errorHandler.js';
+import { toPgJson } from '../../lib/pgJson.js';
const rowToModel = (row) => row?.data ?? null;
@@ -30,7 +31,7 @@ async function persist(exec, model) {
model.id,
model.name,
model.status,
- JSON.stringify(model),
+ toPgJson(model),
model.createdAt,
model.updatedAt,
model.deleted === true,
diff --git a/server/services/loraTraining/db.js b/server/services/loraTraining/db.js
index fa3cfaa194..5655753658 100644
--- a/server/services/loraTraining/db.js
+++ b/server/services/loraTraining/db.js
@@ -18,6 +18,7 @@
import { query } from '../../lib/db.js';
import { ServerError } from '../../lib/errorHandler.js';
import { createKeyCachedQueue } from '../../lib/createKeyCachedQueue.js';
+import { toPgJson } from '../../lib/pgJson.js';
const rowToRun = (row) => row?.data ?? null;
@@ -39,7 +40,7 @@ async function persist(run) {
run.id,
run.status,
run.character?.entryId || null,
- JSON.stringify(run),
+ toPgJson(run),
run.createdAt || new Date().toISOString(),
new Date().toISOString(),
],
diff --git a/server/services/mediaAssetIndex/db.js b/server/services/mediaAssetIndex/db.js
index 1a013969dd..bee32c9abb 100644
--- a/server/services/mediaAssetIndex/db.js
+++ b/server/services/mediaAssetIndex/db.js
@@ -35,6 +35,7 @@ import {
imageToRow, videoToRow, compactGalleryRecord, compactSource, compactSourceToRow,
COMPACT_SOURCE_FIELDS, COMPACT_PREVIEW_KEY, COMPACT_PROMPT_CHARS,
} from './logic.js';
+import { toPgJson } from '../../lib/pgJson.js';
function rowToAsset(row) {
if (!row) return null;
@@ -57,7 +58,7 @@ export async function upsertAsset(row) {
`INSERT INTO media_assets (media_key, kind, ref, data, created_at, indexed_at)
VALUES ($1, $2, $3, $4::jsonb, $5, NOW())
${UPSERT_CONFLICT}`,
- [row.mediaKey, row.kind, row.ref, JSON.stringify(row.data), row.createdAt],
+ [row.mediaKey, row.kind, row.ref, toPgJson(row.data), row.createdAt],
);
}
@@ -81,7 +82,7 @@ async function upsertAssets(rows, runQuery = query) {
const b = j * 5;
// NOW() for indexed_at is a literal, not a param.
values.push(`($${b + 1}, $${b + 2}, $${b + 3}, $${b + 4}::jsonb, $${b + 5}, NOW())`);
- params.push(row.mediaKey, row.kind, row.ref, JSON.stringify(row.data), row.createdAt);
+ params.push(row.mediaKey, row.kind, row.ref, toPgJson(row.data), row.createdAt);
});
await runQuery(
`INSERT INTO media_assets (media_key, kind, ref, data, created_at, indexed_at)
@@ -181,7 +182,7 @@ const rowShape = (compact, projected) => !compact ? data => data
function assetSource(params, videos, { materialized = false, compact = false } = {}) {
const data = compact ? `${compactSourceSql('data')} AS data` : 'data';
if (videos === undefined) return { cte: '', table: 'media_assets', data };
- params.push(JSON.stringify(videos.map(video => ({ data: compact ? compactSource(video) : video,
+ params.push(toPgJson(videos.map(video => ({ data: compact ? compactSource(video) : video,
createdAt: Number.isFinite(Date.parse(video.createdAt)) ? new Date(video.createdAt).toISOString() : new Date(0).toISOString() }))));
return {
data: 'data',
@@ -305,7 +306,7 @@ export async function collectionCovers(collections, videos) {
SELECT 'unsorted',
(SELECT cover FROM unsorted WHERE kind = 'image' OR cover IS NOT NULL ORDER BY created_at DESC, media_key ASC LIMIT 1),
COUNT(*) FILTER (WHERE kind = 'image'), COUNT(*) FILTER (WHERE kind = 'video')
- FROM unsorted`, [JSON.stringify(collections), JSON.stringify(videoRows)]);
+ FROM unsorted`, [toPgJson(collections), toPgJson(videoRows)]);
return result.rows;
}
@@ -316,7 +317,7 @@ export async function visibleImageCollections(collections) {
const result = await query(`SELECT DISTINCT m.id
FROM jsonb_to_recordset($1::jsonb) AS m(id text, key text)
JOIN media_assets a ON a.media_key = m.key
- WHERE a.kind = 'image' AND COALESCE(a.data->>'hidden', 'false') <> 'true'`, [JSON.stringify(memberships)]);
+ WHERE a.kind = 'image' AND COALESCE(a.data->>'hidden', 'false') <> 'true'`, [toPgJson(memberships)]);
const populated = new Set(result.rows.map(row => row.id));
return collections.filter(collection => populated.has(collection.id)).map(({ id, name }) => ({ id, name }));
}
diff --git a/server/services/memoryDB.js b/server/services/memoryDB.js
index 11595624b9..6b528e053a 100644
--- a/server/services/memoryDB.js
+++ b/server/services/memoryDB.js
@@ -17,6 +17,7 @@ import * as notifications from './notifications.js';
import { DEFAULT_MEMORY_CONFIG, generateSummary, decrementAgentPendingApproval } from './memoryConfig.js';
import { getInstanceId } from './instanceIdentity.js';
import { DEFAULT_MEMORY_LINK_TYPE, isSymmetricMemoryLinkType } from '../lib/memoryLinkTypes.js';
+import { toPgJson } from '../lib/pgJson.js';
const protectedMindMemoryTags = Object.values(PERSISTENT_MIND_MEMORY_PROTECTION_TAGS);
@@ -173,7 +174,7 @@ export async function getBrainMemoryLinks(legacyMap = {}, { readOnly = false, re
SELECT legacy.key, m.id FROM jsonb_each_text($1::jsonb) legacy
JOIN memories m ON m.id::text = legacy.value AND m.source_app_id = 'brain'
ORDER BY legacy.key
- ON CONFLICT (bridge_key) DO NOTHING`, [JSON.stringify(legacyMap)]);
+ ON CONFLICT (bridge_key) DO NOTHING`, [toPgJson(legacyMap)]);
}
const result = await client.query(`
SELECT l.bridge_key, l.memory_id FROM brain_memory_links l
@@ -182,7 +183,7 @@ export async function getBrainMemoryLinks(legacyMap = {}, { readOnly = false, re
SELECT legacy.key, m.id FROM jsonb_each_text($1::jsonb) legacy
JOIN memories m ON m.id::text = legacy.value AND m.source_app_id = 'brain'
WHERE NOT EXISTS (SELECT 1 FROM brain_memory_links l WHERE l.bridge_key = legacy.key)`,
- [JSON.stringify(legacyMap)]);
+ [toPgJson(legacyMap)]);
return Object.fromEntries(result.rows.map(row => [row.bridge_key, row.memory_id]));
});
}
diff --git a/server/services/moodBoard/db.js b/server/services/moodBoard/db.js
index 98630537d2..b8e0a382ee 100644
--- a/server/services/moodBoard/db.js
+++ b/server/services/moodBoard/db.js
@@ -47,6 +47,7 @@ import {
maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,
withBaseHashFlushBatch,
} from '../../lib/conflictJournal.js';
+import { toPgJson } from '../../lib/pgJson.js';
// `data` JSONB is the whole record; name/created_at/updated_at mirror into
// columns (kept in lockstep on every write) for the live-list sort. Reads
@@ -71,7 +72,7 @@ async function persist(exec, board) {
[
board.id,
board.name,
- JSON.stringify(board),
+ toPgJson(board),
createdAt,
mirrorTimestamp(board.updatedAt, createdAt),
board.deleted === true,
diff --git a/server/services/peerExecutionLedger.js b/server/services/peerExecutionLedger.js
index 20cafea6c0..748bed6af1 100644
--- a/server/services/peerExecutionLedger.js
+++ b/server/services/peerExecutionLedger.js
@@ -7,6 +7,7 @@ import * as fs from 'node:fs';
import { z } from 'zod';
import { peerAdminIntentSchema, peerAdminActionSchema } from '../lib/peerAdminValidation.js';
import { createPeerExecutionAuthority, executionAuthorityError } from '../lib/peerExecutionAuthority.js';
+import { toPgJson } from '../lib/pgJson.js';
const uuid = z.string().uuid().toLowerCase();
const digest = z.string().regex(/^[a-f0-9]{64}$/);
@@ -90,7 +91,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerE
const { rows: [row] } = await client.query(`INSERT INTO peer_execution_operations
(operation_id, host_instance_id, peer_instance_id, request_id, fingerprint, binding, execution_epoch, state)
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, 'queued') RETURNING *`,
- [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), JSON.stringify(binding), binding.executionEpoch]);
+ [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), toPgJson(binding), binding.executionEpoch]);
await client.query(`INSERT INTO peer_execution_generation_floors (host_instance_id, peer_instance_id, action, generation)
VALUES ($1, $2, $3, $4) ON CONFLICT (host_instance_id, peer_instance_id, action)
DO UPDATE SET generation = GREATEST(peer_execution_generation_floors.generation, EXCLUDED.generation)`,
@@ -113,7 +114,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerE
const { rows: [row] } = await client.query(`INSERT INTO peer_execution_operations
(operation_id, host_instance_id, peer_instance_id, request_id, fingerprint, binding, execution_epoch, state, receipt)
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, 'failed', $8::jsonb) RETURNING *`,
- [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), JSON.stringify(binding), binding.executionEpoch, JSON.stringify(receipt)]);
+ [makeId(), binding.hostInstanceId, binding.peerInstanceId, binding.requestId, fingerprint(binding), toPgJson(binding), binding.executionEpoch, toPgJson(receipt)]);
authority.requireReady(epoch);
return fromRow(row);
});
@@ -173,7 +174,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerE
const next = operationSchema.parse({ ...current, state, revision: revision + 1, claim: claim ?? current.claim, receipt });
const { rows: [updated] } = await client.query(`UPDATE peer_execution_operations
SET state = $2, revision = $3, claim = $4::jsonb, receipt = $5::jsonb, updated_at = NOW()
- WHERE operation_id = $1 RETURNING *`, [id, next.state, next.revision, JSON.stringify(next.claim), JSON.stringify(next.receipt)]);
+ WHERE operation_id = $1 RETURNING *`, [id, next.state, next.revision, toPgJson(next.claim), toPgJson(next.receipt)]);
authority.requireReady(epoch);
return fromRow(updated);
});
@@ -322,7 +323,7 @@ export function createPeerExecutionLedger({ db, dataDir, authority = createPeerE
ON CONFLICT (operation_id) DO UPDATE SET state = EXCLUDED.state,
revision = GREATEST(peer_execution_operations.revision, EXCLUDED.revision), claim = EXCLUDED.claim, receipt = EXCLUDED.receipt, updated_at = NOW()`,
[saved.operationId, saved.binding.hostInstanceId, saved.binding.peerInstanceId, saved.binding.requestId, saved.fingerprint,
- JSON.stringify(saved.binding), saved.binding.executionEpoch, state, saved.revision + 1, JSON.stringify(owner), JSON.stringify(receipt)]);
+ toPgJson(saved.binding), saved.binding.executionEpoch, state, saved.revision + 1, toPgJson(owner), toPgJson(receipt)]);
});
if (replayed.digest !== proof.digest || replayed.count !== proof.count)
throw executionAuthorityError('Execution recovery evidence changed during reconciliation.');
diff --git a/server/services/pipeline/issuesStore/db.js b/server/services/pipeline/issuesStore/db.js
index 9c3a584f96..b6d232b473 100644
--- a/server/services/pipeline/issuesStore/db.js
+++ b/server/services/pipeline/issuesStore/db.js
@@ -17,6 +17,7 @@
import { query } from '../../../lib/db.js';
import { mirrorTimestamp } from '../../../lib/pgTimestamp.js';
import { PIPELINE_STAGE_IDS } from '../../../lib/pipelineStages.js';
+import { toPgJson } from '../../../lib/pgJson.js';
// Only trusted, closed stage IDs form SQL paths. PostgreSQL removes history
// before serializing JSONB for Node; every other field remains lossless.
@@ -124,7 +125,7 @@ export async function writeRaw(id, record) {
typeof record?.seasonId === 'string' && record.seasonId ? record.seasonId : null,
Number.isFinite(record?.number) ? Math.floor(record.number) : null,
typeof record?.status === 'string' && record.status ? record.status.slice(0, 32) : null,
- JSON.stringify(record),
+ toPgJson(record),
record?.ephemeral === true,
createdAt,
mirrorTimestamp(record?.updatedAt, createdAt),
diff --git a/server/services/pipeline/seriesStore/db.js b/server/services/pipeline/seriesStore/db.js
index 5e637e80d3..fef5a5daa3 100644
--- a/server/services/pipeline/seriesStore/db.js
+++ b/server/services/pipeline/seriesStore/db.js
@@ -21,6 +21,7 @@
import { query } from '../../../lib/db.js';
import { mirrorTimestamp } from '../../../lib/pgTimestamp.js';
+import { toPgJson } from '../../../lib/pgJson.js';
/** Raw on-disk-equivalent record (the `data` JSONB), or null. No sanitize. */
export async function readRaw(id) {
@@ -115,7 +116,7 @@ export async function writeRaw(id, record) {
typeof record?.name === 'string' ? record.name : '',
typeof record?.universeId === 'string' && record.universeId ? record.universeId : null,
typeof record?.writersRoomWorkId === 'string' && record.writersRoomWorkId ? record.writersRoomWorkId : null,
- JSON.stringify(record),
+ toPgJson(record),
record?.ephemeral === true,
createdAt,
mirrorTimestamp(record?.updatedAt, createdAt),
diff --git a/server/services/postRunDb.js b/server/services/postRunDb.js
index 9bfa88c390..6d819e1529 100644
--- a/server/services/postRunDb.js
+++ b/server/services/postRunDb.js
@@ -6,6 +6,8 @@
* rolls the complete save back instead of leaving a partial training session.
*/
+import { toPgJson } from '../lib/pgJson.js';
+
const asIso = (value) => value instanceof Date ? value.toISOString() : value;
const asDay = (value) => value instanceof Date ? value.toISOString().slice(0, 10) : String(value || '').slice(0, 10);
@@ -58,8 +60,8 @@ export async function saveNormalizedRunWithClient(client, input) {
updated_at = NOW()`,
[
run.id, run.mode, run.localDay, run.startedAt, run.completedAt || null,
- run.status || 'completed', JSON.stringify(run.planned || {}),
- JSON.stringify(run.data || {}), run.legacy === true,
+ run.status || 'completed', toPgJson(run.planned || {}),
+ toPgJson(run.data || {}), run.legacy === true,
],
);
@@ -107,11 +109,11 @@ export async function saveNormalizedRunWithClient(client, input) {
updated_at = NOW()`,
[
attempt.id, run.id, position, attempt.module, attempt.drillType,
- attempt.difficulty == null ? null : JSON.stringify(attempt.difficulty),
+ attempt.difficulty == null ? null : toPgJson(attempt.difficulty),
attempt.configVersion || null, attempt.correct ?? null, attempt.score ?? null,
attempt.latencyMs || 0, attempt.completion ?? null, attempt.hintUsed === true,
attempt.confidence ?? null, attempt.inputMode || 'unknown',
- attempt.scorerProvenance || 'unknown', JSON.stringify(attempt.data || {}),
+ attempt.scorerProvenance || 'unknown', toPgJson(attempt.data || {}),
attempt.legacy === true,
],
);
diff --git a/server/services/privacyBrokers.js b/server/services/privacyBrokers.js
index 9d0d1a6979..9fd9a59916 100644
--- a/server/services/privacyBrokers.js
+++ b/server/services/privacyBrokers.js
@@ -30,6 +30,7 @@ import { PATHS } from '../lib/fileUtils.js';
import { fetchWithTimeout } from '../lib/fetchWithTimeout.js';
import { resolveSubjectId } from './privacySubjects.js';
import { encryptValue, decryptValue, ensureVaultKey } from '../lib/vaultCrypto.js';
+import { toPgJson } from '../lib/pgJson.js';
// Cap on each registry fetch so a hung broker source can't stall a
// user-triggered refresh indefinitely (both fetchers run under Promise.all).
@@ -239,7 +240,7 @@ async function upsertBroker(client, b, { onlyIfNotCurated = false, onlyIfCurated
updated_at = NOW()
${guard}`,
[
- b.id, b.name, JSON.stringify(b.urls), JSON.stringify(b.optout), b.tier,
+ b.id, b.name, toPgJson(b.urls), toPgJson(b.optout), b.tier,
b.disclosureFields, b.clusterParent, b.preferSuppression, b.antibot,
b.source, b.confidence, b.lastVerified, b.enabled,
],
@@ -517,7 +518,7 @@ async function sealLegacyCaseEvidence() {
for (const row of rows) {
const evidence = parseEvidence(row.evidence);
const stored = await prepareEvidenceForStorage(evidence, { existingEnvelope: evidence[SEALED_EVIDENCE_KEY] ?? null });
- await client.query(`UPDATE privacy_broker_cases SET evidence = $1 WHERE id = $2`, [JSON.stringify(stored), row.id]);
+ await client.query(`UPDATE privacy_broker_cases SET evidence = $1 WHERE id = $2`, [toPgJson(stored), row.id]);
}
if (rows.length) console.log(`🔐 Sealed identity evidence on ${rows.length} legacy broker case(s)`);
return { converted: rows.length };
@@ -643,7 +644,7 @@ export async function recordScanVerdict(brokerId, verdict, { evidence = {}, foun
const resolvedSubjectId = resolveSubjectId(subjectId);
// A fresh scan REPLACES the identity match (a re-scan that no longer finds a
// listing must not keep the old one), so no existing envelope is carried.
- const storedEvidence = JSON.stringify(await prepareEvidenceForStorage(evidence));
+ const storedEvidence = toPgJson(await prepareEvidenceForStorage(evidence));
return withTransaction(async (client) => {
const broker = await client.query(`SELECT id FROM privacy_brokers WHERE id = $1`, [brokerId]);
if (!broker.rows[0]) throw new ServerError('Broker not found', { status: 404, code: 'NOT_FOUND' });
@@ -741,7 +742,7 @@ export async function transitionCase(caseId, toState, patch = {}) {
const clearIdentity = toState === 'confirmed_removed';
if (patch.evidence !== undefined || clearIdentity) {
const stored = parseEvidence(existing.rows[0].evidence);
- add('evidence', JSON.stringify(await prepareEvidenceForStorage(patch.evidence ?? stored, {
+ add('evidence', toPgJson(await prepareEvidenceForStorage(patch.evidence ?? stored, {
existingEnvelope: stored[SEALED_EVIDENCE_KEY] ?? null, clearIdentity,
})));
}
diff --git a/server/services/privacyOrgs.js b/server/services/privacyOrgs.js
index 8ceccb2362..bf8678d0b1 100644
--- a/server/services/privacyOrgs.js
+++ b/server/services/privacyOrgs.js
@@ -22,6 +22,7 @@ import { randomUUID } from 'crypto';
import { query, withTransaction } from '../lib/db.js';
import { ServerError } from '../lib/errorHandler.js';
import { resolveSubjectId, assertSubject } from './privacySubjects.js';
+import { toPgJson } from '../lib/pgJson.js';
const ORG_COLUMNS = `id, subject_id, name, category, website, trust, status, contact,
social_account_id, notes, created_at, updated_at`;
@@ -73,7 +74,7 @@ export async function createOrg(input) {
id, subjectId, input.name,
input.category ?? 'other', input.website ?? '',
input.trust ?? 'trusted', input.status ?? 'active',
- JSON.stringify(input.contact ?? {}),
+ toPgJson(input.contact ?? {}),
input.socialAccountId ?? null,
input.notes ?? '',
],
@@ -117,7 +118,7 @@ export async function updateOrg(id, patch) {
if (patch.website !== undefined) add('website', patch.website);
if (patch.trust !== undefined) add('trust', patch.trust);
if (patch.status !== undefined) add('status', patch.status);
- if (patch.contact !== undefined) add('contact', JSON.stringify(patch.contact));
+ if (patch.contact !== undefined) add('contact', toPgJson(patch.contact));
if (patch.socialAccountId !== undefined) add('social_account_id', patch.socialAccountId);
if (patch.notes !== undefined) add('notes', patch.notes);
params.push(id);
diff --git a/server/services/projectDbStore.js b/server/services/projectDbStore.js
index 6081d3d4f9..1ad6f05f14 100644
--- a/server/services/projectDbStore.js
+++ b/server/services/projectDbStore.js
@@ -10,6 +10,7 @@ import {
deleteSyncBaseHash,
withBaseHashFlushBatch,
} from '../lib/conflictJournal.js';
+import { toPgJson } from '../lib/pgJson.js';
export function createProjectDbStore({ table, kind, idPrefix, logEmoji, logLabel, logic }) {
if (!/^[a-z][a-z0-9_]*$/.test(table)) throw new Error(`Invalid project table: ${table}`);
@@ -43,7 +44,7 @@ export function createProjectDbStore({ table, kind, idPrefix, logEmoji, logLabel
[
prepared.id,
logic.mirrorStatus(prepared.status),
- JSON.stringify(prepared),
+ toPgJson(prepared),
createdAt,
logic.mirrorTimestamp(prepared.updatedAt, createdAt),
prepared.deleted === true,
diff --git a/server/services/providerGraphStore.js b/server/services/providerGraphStore.js
index 501e12e1dd..46267fc116 100644
--- a/server/services/providerGraphStore.js
+++ b/server/services/providerGraphStore.js
@@ -1,4 +1,5 @@
import { query, withTransaction } from '../lib/db.js';
+import { toPgJson } from '../lib/pgJson.js';
/**
* Postgres adapter for the provider connection graph — `ai_connections`,
@@ -97,8 +98,8 @@ const upsertConnection = (client, connection) => client.query(
credential_via = COALESCE($12, ai_connections.credential_via),
updated_at = NOW()`,
[connection.id, connection.revision ?? 1, connection.kind, connection.label ?? '',
- JSON.stringify(connection.transports || {}), JSON.stringify(connection.credentials || {}),
- JSON.stringify(connection.catalog || { state: 'unknown', models: [] }),
+ toPgJson(connection.transports || {}), toPgJson(connection.credentials || {}),
+ toPgJson(connection.catalog || { state: 'unknown', models: [] }),
connection.slug ?? null, connection.definitionId ?? null, connection.plan ?? null,
typeof connection.enabled === 'boolean' ? connection.enabled : null, connection.credentialVia ?? null],
);
@@ -114,7 +115,7 @@ const upsertBinding = (client, binding) => client.query(
enabled = EXCLUDED.enabled, selected_models = EXCLUDED.selected_models, updated_at = NOW()`,
[binding.id, binding.revision ?? 1, binding.connectionId, binding.harnessId ?? null,
binding.variantKey, binding.label ?? '', binding.enabled === true,
- JSON.stringify(binding.selectedModels || [])],
+ toPgJson(binding.selectedModels || [])],
);
// `model_alias_overrides` is set on INSERT and deliberately absent from the
@@ -130,10 +131,10 @@ const upsertRoute = (client, route) => client.query(
binding_id = EXCLUDED.binding_id, mode = EXCLUDED.mode, model_map = EXCLUDED.model_map,
projected = EXCLUDED.projected, pending = EXCLUDED.pending,
pending_revision = EXCLUDED.pending_revision, updated_at = NOW()`,
- [route.providerId, route.bindingId, route.mode, JSON.stringify(route.modelMap || {}),
- JSON.stringify(route.modelAliasOverrides || {}),
- JSON.stringify(route.projected || {}),
- route.pending == null ? null : JSON.stringify(route.pending),
+ [route.providerId, route.bindingId, route.mode, toPgJson(route.modelMap || {}),
+ toPgJson(route.modelAliasOverrides || {}),
+ toPgJson(route.projected || {}),
+ route.pending == null ? null : toPgJson(route.pending),
route.pendingRevision ?? null],
);
@@ -191,7 +192,7 @@ export async function applyReconciliation(plan) {
for (const { providerId, owned } of plan.snapshots) {
await client.query(
'UPDATE ai_route_bindings SET projected = $1, updated_at = NOW() WHERE provider_id = $2',
- [JSON.stringify(owned), providerId],
+ [toPgJson(owned), providerId],
);
}
@@ -222,7 +223,7 @@ export async function commitPendingProjection(projections) {
const { rows } = await client.query(
`UPDATE ai_route_bindings SET pending = $1, updated_at = NOW() WHERE provider_id = $2
RETURNING binding_id`,
- [JSON.stringify(owned), providerId],
+ [toPgJson(owned), providerId],
);
if (rows.length === 0) continue;
const { rows: bumped } = await client.query(
@@ -325,8 +326,8 @@ export async function saveConnectionSettings({ id, label, transports, credential
revision = revision + 1, updated_at = NOW()
WHERE id = $1
RETURNING revision`,
- [id, label ?? '', JSON.stringify(transports || {}), JSON.stringify(credentials || {}),
- JSON.stringify(catalog || { state: 'unknown', models: [] }),
+ [id, label ?? '', toPgJson(transports || {}), toPgJson(credentials || {}),
+ toPgJson(catalog || { state: 'unknown', models: [] }),
plan ?? null, typeof enabled === 'boolean' ? enabled : null, credentialVia ?? null],
);
return rows[0]?.revision ?? null;
@@ -366,7 +367,7 @@ export async function saveBindingSettings({ id, label, selectedModels }) {
SET label = $2, selected_models = $3, revision = revision + 1, updated_at = NOW()
WHERE id = $1
RETURNING revision`,
- [id, label ?? '', JSON.stringify(selectedModels || [])],
+ [id, label ?? '', toPgJson(selectedModels || [])],
);
return rows[0]?.revision ?? null;
}
@@ -374,7 +375,7 @@ export async function saveBindingSettings({ id, label, selectedModels }) {
/** Record the canonical→executable model aliases a refresh observed on a route. */
export async function saveRouteModelMap(providerId, modelMap) {
await query('UPDATE ai_route_bindings SET model_map = $2, updated_at = NOW() WHERE provider_id = $1',
- [providerId, JSON.stringify(modelMap || {})]);
+ [providerId, toPgJson(modelMap || {})]);
}
/**
@@ -388,6 +389,6 @@ export async function saveRouteModelMap(providerId, modelMap) {
export async function saveRouteModelAliases(providerId, overrides) {
await query(
'UPDATE ai_route_bindings SET model_alias_overrides = $2, updated_at = NOW() WHERE provider_id = $1',
- [providerId, JSON.stringify(overrides || {})],
+ [providerId, toPgJson(overrides || {})],
);
}
diff --git a/server/services/sprites/recordsDB.js b/server/services/sprites/recordsDB.js
index 4a174eb2e4..3cba1a9eda 100644
--- a/server/services/sprites/recordsDB.js
+++ b/server/services/sprites/recordsDB.js
@@ -19,6 +19,7 @@ import {
mergeImportedRecord,
mirrorTimestamp,
} from './recordsLogic.js';
+import { toPgJson } from '../../lib/pgJson.js';
function rowToRecord(row) {
return row ? row.data : null;
@@ -41,7 +42,7 @@ async function persist(exec, record) {
record.id,
record.kind,
record.status,
- JSON.stringify(record),
+ toPgJson(record),
createdAt,
mirrorTimestamp(record.updatedAt, createdAt),
record.deleted === true,
diff --git a/server/services/storyBuilderStore/db.js b/server/services/storyBuilderStore/db.js
index 856e29f781..9c8554d3aa 100644
--- a/server/services/storyBuilderStore/db.js
+++ b/server/services/storyBuilderStore/db.js
@@ -17,6 +17,7 @@
import { query } from '../../lib/db.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
/** Raw on-disk-equivalent record (the `data` JSONB), or null. No sanitize. */
export async function readRaw(id) {
@@ -63,7 +64,7 @@ export async function writeRaw(id, record) {
typeof record?.universeId === 'string' && record.universeId ? record.universeId : null,
typeof record?.seriesId === 'string' && record.seriesId ? record.seriesId : null,
record?.sync === true,
- JSON.stringify(record),
+ toPgJson(record),
record?.ephemeral === true,
createdAt,
mirrorTimestamp(record?.updatedAt, createdAt),
diff --git a/server/services/threejsModels/db.js b/server/services/threejsModels/db.js
index 662abd1924..eec08165fd 100644
--- a/server/services/threejsModels/db.js
+++ b/server/services/threejsModels/db.js
@@ -10,6 +10,7 @@ import { randomUUID } from 'crypto';
import { query, withTransaction } from '../../lib/db.js';
import { ServerError } from '../../lib/errorHandler.js';
import { GENERAL_FAMILY_ID } from '../../lib/threejsModelFamilies.js';
+import { toPgJson } from '../../lib/pgJson.js';
const rowToModel = (row) => row?.data ?? null;
@@ -28,7 +29,7 @@ async function persist(exec, model) {
model.id,
model.name,
model.status,
- JSON.stringify(model),
+ toPgJson(model),
model.createdAt,
model.updatedAt,
model.deleted === true,
diff --git a/server/services/tracks/db.js b/server/services/tracks/db.js
index fa40783be8..f424329900 100644
--- a/server/services/tracks/db.js
+++ b/server/services/tracks/db.js
@@ -16,6 +16,7 @@ import {
maybeJournalBeforeOverwrite, setSyncBaseHash, contentHashForRecord, flushBaseHashes, deleteSyncBaseHash,
withBaseHashFlushBatch,
} from '../../lib/conflictJournal.js';
+import { toPgJson } from '../../lib/pgJson.js';
// Re-sanitize the stored JSONB on read (mirrors the file backend's loadAll →
// sanitizeTrack). The record was already sanitized on write, so for an in-shape
@@ -42,7 +43,7 @@ async function persist(exec, track) {
[
track.id,
typeof track.title === 'string' ? track.title : '',
- JSON.stringify(track),
+ toPgJson(track),
createdAt,
mirrorTimestamp(track.updatedAt, createdAt),
track.deleted === true,
diff --git a/server/services/universeBuilder/db.js b/server/services/universeBuilder/db.js
index 7393589838..4816a2e49c 100644
--- a/server/services/universeBuilder/db.js
+++ b/server/services/universeBuilder/db.js
@@ -23,6 +23,7 @@
import { query, withTransaction } from '../../lib/db.js';
import { LOGLINE_MAX } from '../../lib/universeBibleLimits.js';
import { mirrorTimestamp } from '../../lib/pgTimestamp.js';
+import { toPgJson } from '../../lib/pgJson.js';
// Keep the runs[] log bounded to the most-recent 200 — matches the file
// backend's cap (universeBuilder.recordRun) so behavior is identical.
@@ -219,7 +220,7 @@ export async function writeRaw(id, record) {
[
id,
typeof record?.name === 'string' ? record.name : '',
- JSON.stringify(record),
+ toPgJson(record),
schemaVersion,
record?.ephemeral === true,
createdAt,
@@ -264,7 +265,7 @@ export async function appendRun(run) {
run.id,
run.universeId,
typeof run.collectionId === 'string' ? run.collectionId : null,
- JSON.stringify(run),
+ toPgJson(run),
mirrorTimestamp(run.createdAt, new Date().toISOString()),
],
);
diff --git a/server/services/userActionsDb.js b/server/services/userActionsDb.js
index 163dc661ed..f1708d4ea0 100644
--- a/server/services/userActionsDb.js
+++ b/server/services/userActionsDb.js
@@ -9,6 +9,8 @@
* drive it against `portos_test` under `npm run test:db`.
*/
+import { toPgJson } from '../lib/pgJson.js';
+
const iso = (value) => (value instanceof Date ? value.toISOString() : value);
/** Map a `user_action_events` row to the camelCase event shape the API returns. */
@@ -42,8 +44,8 @@ export async function insertUserActionEvent(db, event) {
RETURNING id`,
[
event.id, event.type, event.actor, event.happenedAt, event.target, event.targetName,
- event.success, event.summary, JSON.stringify(event.payload),
- JSON.stringify(event.source), event.dedupeKey,
+ event.success, event.summary, toPgJson(event.payload),
+ toPgJson(event.source), event.dedupeKey,
],
);
return (result.rowCount || 0) > 0;
diff --git a/server/services/voice/profiles.js b/server/services/voice/profiles.js
index e0798e1c8a..74c875a6f4 100644
--- a/server/services/voice/profiles.js
+++ b/server/services/voice/profiles.js
@@ -16,6 +16,7 @@ import { ServerError } from '../../lib/errorHandler.js';
import { PATHS } from '../../lib/paths.js';
import { writeFileGuarded } from '../../lib/fileUtils.js';
import { trimTo } from '../../lib/textUtils.js';
+import { toPgJson } from '../../lib/pgJson.js';
export const VOICE_PROFILE_ENGINES = new Set(['kokoro', 'piper', 'qwen3-tts', 'auk']);
export const VOICE_PROFILE_KINDS = new Set(['preset', 'designed', 'cloned', 'fine-tuned']);
@@ -203,7 +204,7 @@ export const persistVoiceProfile = async (profile, execute = query) => {
profile.binding.universeId,
profile.binding.characterId,
profile.approval.status,
- JSON.stringify(profile),
+ toPgJson(profile),
profile.createdAt,
profile.updatedAt,
],
@@ -597,7 +598,7 @@ export async function saveProfileBenchmark(profile, benchmark, { interactive } =
updated_at = $5
WHERE id = $1 AND (data->>'version')::int = $3
RETURNING data`,
- [profile.id, JSON.stringify(patch), profile.version, route ? JSON.stringify(route) : null, patch.updatedAt],
+ [profile.id, toPgJson(patch), profile.version, route ? toPgJson(route) : null, patch.updatedAt],
);
if (!rows.length) {
throw new ServerError('Voice profile changed during benchmarking; run the benchmark again', {
@@ -655,7 +656,7 @@ export async function recordVoiceProfileRender({
profile_revision = EXCLUDED.profile_revision,
data = EXCLUDED.data,
updated_at = EXCLUDED.updated_at`,
- [issue, line, profileId, profileRevision, JSON.stringify(data), now, now],
+ [issue, line, profileId, profileRevision, toPgJson(data), now, now],
);
}
diff --git a/server/services/writersRoom/db.js b/server/services/writersRoom/db.js
index 12d417a416..b1c75ca81e 100644
--- a/server/services/writersRoom/db.js
+++ b/server/services/writersRoom/db.js
@@ -42,6 +42,7 @@ import {
WRITERS_ROOM_FOLDER_KIND, mergeFolderRecord,
WRITERS_ROOM_EXERCISE_KIND, mergeExerciseRecord,
} from './syncLogic.js';
+import { toPgJson } from '../../lib/pgJson.js';
/**
* Shared per-record LWW merge for a body-less Writers Room record (folder /
@@ -151,7 +152,7 @@ function persistFolderTx(client, folder) {
typeof folder?.parentId === 'string' && folder.parentId ? folder.parentId : null,
String(folder?.name ?? ''),
Number.isInteger(folder?.sortOrder) ? folder.sortOrder : 0,
- JSON.stringify(folderData),
+ toPgJson(folderData),
createdAt,
mirrorTimestamp(folder?.updatedAt, createdAt),
deleted,
@@ -241,7 +242,7 @@ function persistExerciseTx(client, exercise) {
exercise.id,
typeof exercise?.workId === 'string' && exercise.workId ? exercise.workId : null,
typeof exercise?.status === 'string' ? exercise.status.slice(0, 16) : null,
- JSON.stringify(exerciseData),
+ toPgJson(exerciseData),
mirrorTimestamp(exercise?.startedAt, null),
mirrorTimestamp(exercise?.finishedAt, null),
deleted,
@@ -377,9 +378,9 @@ function draftBinds(workId, draft) {
String(draft?.contentFile ?? ''),
typeof draft?.contentHash === 'string' ? draft.contentHash : null,
Number.isInteger(draft?.wordCount) ? draft.wordCount : 0,
- JSON.stringify(Array.isArray(draft?.segmentIndex) ? draft.segmentIndex : []),
+ toPgJson(Array.isArray(draft?.segmentIndex) ? draft.segmentIndex : []),
typeof draft?.createdFromVersionId === 'string' ? draft.createdFromVersionId : null,
- JSON.stringify(draft),
+ toPgJson(draft),
mirrorTimestamp(draft?.createdAt, new Date().toISOString()),
];
}
@@ -453,7 +454,7 @@ async function persistWorkTx(client, manifest) {
typeof manifest?.pipelineIssueId === 'string' ? manifest.pipelineIssueId : null,
typeof manifest?.cdProjectId === 'string' ? manifest.cdProjectId : null,
typeof manifest?.mediaCollectionId === 'string' ? manifest.mediaCollectionId : null,
- JSON.stringify(workData),
+ toPgJson(workData),
createdAt,
mirrorTimestamp(manifest?.updatedAt, createdAt),
deleted,