Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(databases): trace table-open phases and add a local profiling harness (#7496)
Wolf Mermelstein97892c20d29b
diff --git a/apps/web/scripts/database-profile/README.md b/apps/web/scripts/database-profile/README.mdnew file mode 100644index 0000000000..3392aa6e08--- /dev/null+++ b/apps/web/scripts/database-profile/README.md@@ -0,0 +1,94 @@+# Database profiling++Run the isolated stack with browser telemetry enabled, following+[Running locally](../../../../docs/development/running-locally.md). Keep the instance running+throughout the audit; `run_local` startup and its quit command rebuild/remove+instance data. Use the `r` hotkey to rebuild services without replacing data.++```sh+nix develop --command just run_local --no-doppler --instance database-profiling-audit --port-base 21000+```++Use a private Playwright browser to sign in and save its `storageState` outside+the repository. That file contains credentials: do not commit it or attach it to+a PR. Create dedicated synthetic databases through the app's supported APIs.+Keep existing fixtures. Test 1k, 5k, 10k, 25k, 50k and 100k **stored** rows, with+both unfiltered and selective views. Record columns, types, relationships,+filters, build SHA and browser version alongside each run.++From `apps/web`, measure without changing data:++```sh+bun scripts/database-profile/measure.mjs \+ --url https://localhost:21009/app/database/UUID \+ --state /tmp/database-profile-auth.json \+ --table Records --rows 1000 --entries 1000 \+ --output /tmp/database-profile-1000 --repeats 3 \+ --chrome /path/to/google-chrome+```++Use the actual proxy URL printed by `run_local`. The harness accepts loopback+URLs only, opens its own browser, and refuses to overwrite an output directory.+Each repetition starts with a fresh context containing only the saved login+state, then measures a warm reload in that same context. `--rows` is the number+of saved rows expected in the selected view; the blank insertion row is excluded.+`--entries` is the fixture's total stored row count. The engine currently caps+reads at 20,000 rows, so a 100k database does **not** imply 100k rendered rows.+Confirm the trace's `database_sql.truncated` attribute when interpreting results.++The JSON results include first complete grid and subsequent frame opportunity,+long tasks, CDP script/layout/task counters, heap size, HTTP timing/bytes, and+traceparents. The frame measurement is a two-frame rendering opportunity, not+an assertion that the browser painted. The 100ms row polling adds up to 100ms+measurement uncertainty; use the internal spans to attribute work precisely.+Requests completing within 1.5 seconds after that frame are included to expose+background refreshes; this is not a measurement of network quiescence.++Run `--profile` separately to save Chrome CPU profiles. Compare ordinary runs+first: profiling adds overhead. Use at least three repetitions and report+medians and ranges, not the fastest run. Vite measurements include development+overhead; repeat production-bundle measurements before claiming production+latency. Exported profiles may contain application source URLs and fixture IDs.++## Trace boundaries++| Span | What it measures |+| --- | --- |+| `database_sql.query` | Catalog through publishing, with reason, policy, target, row count, truncation and outcome |+| `database_sql.catalog` | WASM load if necessary and catalog construction |+| `database_sql.engine.open` / `.start` | Engine opening/compilation and initial planning |+| `database_sql.fetch` / `.fold` | Each existing 500-row source request and the engine consuming its answer |+| `database_sql.rows.decode` | GraphQL property values converted into engine cells, once per page |+| `database_sql.compare` | Equality comparison of the new catalog/result with the prior answer |+| `database_sql.publish` | Signal updates and synchronous downstream reactive work |+| `database_sql.frame` | Time after a changed result is published until a rendering opportunity; cancelled/hidden/timeout outcomes are not ready frames |+| `soup.pool.acquire` | Connection acquisition, with pool size and idle count at entry |+| `soup.sql.fetch_decode` | Query construction, SQL round trip, row decoding and flat-row conversion, excluding connection acquisition |+| `soup.rows.convert` | Grouped-row conversion after the SQL read |+| `graphql.serialize` | GraphQL response serialization and response construction, with uncompressed response bytes |++`graphql.execute` includes serialization. HTTP transfer, compression and proxy+time are outside `graphql.serialize`. Fetch/decode is not server-only SQL time:+use PostgreSQL execution plans, `pg_stat_statements`, and lock/I/O diagnostics+to separate server execution from transfer and client decoding. Cache worker+spans already measure cache reads, locks and queueing. No per-cell spans or+per-row conversion spans are emitted by these additions.++New local stacks preload `pg_stat_statements` (the existing migration installs+the extension). Verify `SHOW shared_preload_libraries`, then take snapshots of+`pg_stat_statements` before and after each measured workload, restricted to the+current database’s `dbid` from `pg_database`. Compare deltas in+`calls`, `total_exec_time`, `rows`, `shared_blks_hit` and `shared_blks_read` by+`dbid`, `userid`, `queryid` and `toplevel`; do not reset shared statistics. Statistics span all requests to this+database, so correlate the measured window and query shape with the trace.+Use `EXPLAIN (ANALYZE, BUFFERS)` only for the audit's read queries on synthetic+fixtures. Existing running stacks need a PostgreSQL restart to activate preload;+do not restart another session's stack or rerun `run_local` over retained data.++Use a request's traceparent to retrieve its complete trace from Tempo. Verify+the browser flag/exporter and server OTEL filter before an audit; do not infer+that absent spans mean zero cost. Correlate rows, bytes, page count and cache+policy across browser, GraphQL, authorization, properties and SQL. Validate+edits, added rows, picker latency, table/view switching and scrolling separately+on synthetic fixtures after every optimization; this read-only harness covers+opening and reloading only. Preserve the 500-row limits and grid behavior.diff --git a/apps/web/scripts/database-profile/measure.mjs b/apps/web/scripts/database-profile/measure.mjsnew file mode 100644index 0000000000..165d048899--- /dev/null+++ b/apps/web/scripts/database-profile/measure.mjs@@ -0,0 +1,215 @@+/** Read-only browser benchmark. Run with --help for arguments. */+import { mkdir, writeFile } from 'node:fs/promises';+import { resolve } from 'node:path';+import { parseArgs } from 'node:util';+import { chromium } from 'playwright';++const { values } = parseArgs({+ options: {+ url: { type: 'string' },+ state: { type: 'string' },+ table: { type: 'string' },+ rows: { type: 'string' },+ entries: { type: 'string' },+ repeats: { type: 'string', default: '3' },+ output: { type: 'string' },+ chrome: { type: 'string' },+ profile: { type: 'boolean', default: false },+ help: { type: 'boolean', default: false },+ },+});+if (values.help) {+ console.log(+ 'bun scripts/database-profile/measure.mjs --url <database URL> --state <Playwright auth JSON> --rows <expected loaded rows> --entries <stored rows> --output <new directory> [--table <name>] [--repeats 3] [--chrome <executable>] [--profile]'+ );+ process.exit(0);+}+if (+ !values.url ||+ !values.state ||+ !values.output ||+ !values.rows ||+ !values.entries+)+ throw new Error('Required: --url, --state, --rows, --entries, --output');+const url = new URL(values.url);+if (!['localhost', '127.0.0.1', '[::1]'].includes(url.hostname))+ throw new Error('Use an isolated local stack on localhost.');+const expectedRows = Number(values.rows);+const entries = Number(values.entries);+const repeats = Number(values.repeats);+if (+ ![expectedRows, entries, repeats].every(+ (n) => Number.isSafeInteger(n) && n > 0+ ) ||+ entries < expectedRows+)+ throw new Error(+ 'Row counts and repeats must be positive integers; entries must cover loaded rows.'+ );+const output = resolve(values.output);+await mkdir(output); // Refuse to overwrite a previous run.+const browser = await chromium.launch({+ executablePath: values.chrome,+ headless: true,+});+await writeFile(+ `${output}/environment.json`,+ JSON.stringify(+ {+ browser: browser.version(),+ startedAt: new Date().toISOString(),+ databasePath: url.pathname,+ entries,+ expectedRows,+ profile: values.profile,+ },+ null,+ 2+ )+);+const results = [];+try {+ for (let repeat = 1; repeat <= repeats; repeat++) {+ const context = await browser.newContext({+ storageState: values.state,+ ignoreHTTPSErrors: true,+ viewport: { width: 1600, height: 1000 },+ });+ await context.addInitScript(() => {+ window.__databaseProfile = { longTasks: [] };+ new PerformanceObserver((list) => {+ for (const entry of list.getEntries())+ window.__databaseProfile.longTasks.push({+ start: entry.startTime,+ duration: entry.duration,+ });+ }).observe({ type: 'longtask', buffered: true });+ });+ const page = await context.newPage();+ const session = await context.newCDPSession(page);+ await session.send('Performance.enable');+ for (const temperature of ['cold', 'warm']) {+ const requests = [];+ const pending = [];+ const finished = (request) => {+ const path = new URL(request.url()).pathname;+ if (+ path !== '/dss/items/soup/graphql' &&+ path !== '/dss/databases' &&+ !path.startsWith('/dss/databases/')+ )+ return;+ pending.push(+ (async () => {+ const response = await request.response();+ requests.push({+ path,+ status: response?.status(),+ traceparent: request.headers().traceparent,+ timing: request.timing(),+ sizes: await request.sizes().catch(() => undefined),+ });+ })()+ );+ };+ page.on('requestfinished', finished);+ if (values.profile) {+ await session.send('Profiler.enable');+ await session.send('Profiler.start');+ }+ const started = performance.now();+ await page.goto(values.url, {+ waitUntil: 'domcontentloaded',+ timeout: 180_000,+ });+ if (values.table)+ await page+ .locator('[aria-label="Database tables"]')+ .getByText(values.table, { exact: true })+ .click();+ await page.waitForFunction(+ (count) => {+ const grid = document.querySelector('[role="grid"]');+ return (+ grid &&+ grid.querySelectorAll(+ '[data-grid-row-id]:not([data-grid-row-id^="draft:"])'+ ).length >= count+ );+ },+ expectedRows,+ { timeout: 180_000, polling: 100 }+ );+ const dataReadyMs = performance.now() - started;+ const snapshot = await page.evaluate(async () => {+ if (document.hidden)+ throw new Error('The benchmark tab must be visible.');+ await new Promise((resolve) =>+ requestAnimationFrame(() => requestAnimationFrame(resolve))+ );+ const grid = document.querySelector('[role="grid"]');+ return {+ rows: grid.querySelectorAll(+ '[data-grid-row-id]:not([data-grid-row-id^="draft:"])'+ ).length,+ cells: grid.querySelectorAll('[role="gridcell"]').length,+ visibility: document.visibilityState,+ longTasks: window.__databaseProfile.longTasks,+ };+ });+ const frameReadyMs = performance.now() - started;+ const after = await session.send('Performance.getMetrics');+ // CDP navigation counters reset on document navigation.+ const metrics = Object.fromEntries(+ after.metrics.map(({ name, value }) => [name, value])+ );+ if (values.profile) {+ const { profile } = await session.send('Profiler.stop');+ await writeFile(+ `${output}/${repeat}-${temperature}.cpuprofile`,+ JSON.stringify(profile)+ );+ }+ // Capture refresh requests too, separately from the first rendering opportunity.+ await page.waitForTimeout(1500);+ page.off('requestfinished', finished);+ await Promise.all(pending);+ const result = {+ repeat,+ temperature,+ entries,+ expectedRows,+ dataReadyMs,+ frameReadyMs,+ ...snapshot,+ metrics,+ requests,+ };+ results.push(result);+ await writeFile(+ `${output}/results.json`,+ JSON.stringify(results, null, 2)+ );+ console.log(+ JSON.stringify({+ repeat,+ temperature,+ entries,+ loaded: snapshot.rows,+ dataReadyMs,+ frameReadyMs,+ })+ );+ }+ await context.close();+ }+} catch (error) {+ await writeFile(+ `${output}/failure.json`,+ JSON.stringify({ error: String(error), completed: results.length }, null, 2)+ );+ throw error;+} finally {+ await browser.close();+}diff --git a/apps/web/src/lib/core/database-sql/driver.ts b/apps/web/src/lib/core/database-sql/driver.tsindex c6d25a7517..7c167985c3 100644--- a/apps/web/src/lib/core/database-sql/driver.ts+++ b/apps/web/src/lib/core/database-sql/driver.ts@@ -147,7 +147,7 @@ async function steps(source: RowSource,trace: DatabaseSqlRunTrace): Promise<Result<Outcome, DatabaseSqlFailure>> {- let step = feed(() => query.start());+ let step = trace.start(() => feed(() => query.start()));while (step.isOk()) {const current = step.value;if (current.step === 'done') {@@ -164,7 +164,7 @@ function drive(source: RowSource,trace: DatabaseSqlRunTrace): ResultAsync<Outcome, DatabaseSqlFailure> {- return ResultAsync.fromPromise(opened(), engineFailure).andThen(+ return ResultAsync.fromPromise(trace.open(opened), engineFailure).andThen((query) =>new ResultAsync(steps(query, source, trace).finally(() => query.free())));diff --git a/apps/web/src/lib/core/database-sql/profile.test.ts b/apps/web/src/lib/core/database-sql/profile.test.tsnew file mode 100644index 0000000000..7b9f2ce455--- /dev/null+++ b/apps/web/src/lib/core/database-sql/profile.test.ts@@ -0,0 +1,89 @@+import type { Span } from '@macro-inc/observability';+import { afterEach, expect, it, vi } from 'vitest';+import { profileDatabasePhase, traceDatabaseFrame } from './profile';++function span() {+ return {+ run: <T>(operation: () => T) => operation(),+ setAttr: vi.fn(),+ end: vi.fn(),+ } as unknown as Span;+}++afterEach(() => vi.restoreAllMocks());++it('ends a synchronous phase even when it throws', () => {+ const phase = span();+ expect(() =>+ profileDatabasePhase(phase, () => {+ throw new Error('failed');+ })+ ).toThrow('failed');+ expect(phase.end).toHaveBeenCalledTimes(1);+});++it('waits for two frames and records a frame opportunity, not a paint timestamp', () => {+ const callbacks: FrameRequestCallback[] = [];+ vi.spyOn(window, 'requestAnimationFrame').mockImplementation((callback) =>+ callbacks.push(callback)+ );+ vi.spyOn(window, 'cancelAnimationFrame').mockImplementation(() => {});+ const phase = span();+ const cancel = traceDatabaseFrame(phase);+ callbacks[0](0);+ expect(phase.end).not.toHaveBeenCalled();+ callbacks[1](16);+ expect(phase.setAttr).toHaveBeenCalledWith(+ 'database_sql.frame_status',+ 'ready'+ );+ cancel();+ expect(phase.end).toHaveBeenCalledTimes(1);+});++it('cancels a superseded result before recording a ready frame', () => {+ vi.spyOn(window, 'requestAnimationFrame').mockReturnValue(7);+ const cancelled = vi+ .spyOn(window, 'cancelAnimationFrame')+ .mockImplementation(() => {});+ const phase = span();+ traceDatabaseFrame(phase)();+ expect(cancelled).toHaveBeenCalledWith(7);+ expect(phase.setAttr).toHaveBeenCalledWith(+ 'database_sql.frame_status',+ 'cancelled'+ );+ expect(phase.end).toHaveBeenCalledTimes(1);+});++it('does not call a hidden document a ready frame', () => {+ vi.spyOn(document, 'hidden', 'get').mockReturnValue(true);+ const request = vi.spyOn(window, 'requestAnimationFrame');+ const phase = span();+ traceDatabaseFrame(phase)();+ expect(request).not.toHaveBeenCalled();+ expect(phase.setAttr).toHaveBeenCalledWith(+ 'database_sql.frame_status',+ 'hidden'+ );+ expect(phase.end).toHaveBeenCalledTimes(1);+});++it('ends a frame span if animation frames never arrive', () => {+ vi.useFakeTimers();+ try {+ vi.spyOn(window, 'requestAnimationFrame').mockReturnValue(7);+ vi.spyOn(window, 'cancelAnimationFrame').mockImplementation(() => {});+ const phase = span();+ const cancel = traceDatabaseFrame(phase);+ vi.advanceTimersByTime(10_000);+ expect(phase.setAttr).toHaveBeenCalledWith(+ 'database_sql.frame_status',+ 'timeout'+ );+ cancel();+ expect(phase.end).toHaveBeenCalledTimes(1);+ } finally {+ vi.useRealTimers();+ }+});diff --git a/apps/web/src/lib/core/database-sql/profile.ts b/apps/web/src/lib/core/database-sql/profile.tsnew file mode 100644index 0000000000..60008d7101--- /dev/null+++ b/apps/web/src/lib/core/database-sql/profile.ts@@ -0,0 +1,36 @@+import type { Span } from '@macro-inc/observability';++/** Synchronous work, including Solid's downstream reactive updates. */+export function profileDatabasePhase<T>(span: Span, operation: () => T): T {+ try {+ return span.run(operation);+ } finally {+ span.end();+ }+}++/** Two animation frames provide a rendering opportunity, not proof of paint. */+export function traceDatabaseFrame(span: Span): () => void {+ let ended = false;+ let frame: number | undefined;+ const finish = (status: 'ready' | 'cancelled' | 'hidden' | 'timeout') => {+ if (ended) return;+ ended = true;+ if (frame !== undefined) cancelAnimationFrame(frame);+ clearTimeout(timeout);+ document.removeEventListener('visibilitychange', visibilityChanged);+ span.setAttr('database_sql.frame_status', status);+ span.end();+ };+ const visibilityChanged = () => {+ if (document.hidden) finish('hidden');+ };+ const timeout = setTimeout(() => finish('timeout'), 10_000);+ document.addEventListener('visibilitychange', visibilityChanged);+ if (document.hidden) finish('hidden');+ else+ frame = requestAnimationFrame(() => {+ frame = requestAnimationFrame(() => finish('ready'));+ });+ return () => finish('cancelled');+}diff --git a/apps/web/src/lib/core/database-sql/trace.test.ts b/apps/web/src/lib/core/database-sql/trace.test.tsindex b54ed8a1bb..d32f65fd4c 100644--- a/apps/web/src/lib/core/database-sql/trace.test.ts+++ b/apps/web/src/lib/core/database-sql/trace.test.ts@@ -1,4 +1,4 @@-import { errAsync, ok, okAsync } from 'neverthrow';+import { errAsync, ok, okAsync, ResultAsync } from 'neverthrow';import { afterEach, describe, expect, it, vi } from 'vitest';import type { Catalog, Outcome } from './generated/types';import { traceDatabaseSqlRun } from './trace';@@ -254,3 +254,54 @@ describe('database SQL run traces', () => {);});});++it('keeps engine and decoding phases under explicit parents across async work', async () => {+ vi.spyOn(console, 'groupCollapsed').mockImplementation(() => {});+ vi.spyOn(console, 'log').mockImplementation(() => {});+ vi.spyOn(console, 'groupEnd').mockImplementation(() => {});+ await traceDatabaseSqlRun(+ { kind: 'sql', sql: 'SELECT 1' },+ catalog,+ (trace) =>+ new ResultAsync(+ (async () => {+ await trace.open(async () => 1);+ trace.start(() => 2);+ await trace.fetch(+ {+ step: 'fetch',+ id: 0,+ query: {+ type: 'soup',+ table: 'guests',+ propf: null,+ keyHint: null,+ },+ needs: [],+ cursor: null,+ limit: 500,+ },+ (step) => okAsync(step.decode(2, () => [1, 2])),+ (rows) => rows.length+ );+ return ok(outcome);+ })()+ )+ );+ expect(recorded.roots).toHaveLength(1);+ const children = recorded.roots[0].children;+ expect(children.map((span) => span.name)).toEqual([+ 'database_sql.engine.open',+ 'database_sql.engine.start',+ 'database_sql.fetch',+ ]);+ expect(children.every((span) => span.ended)).toBe(true);+ expect(children[2].children).toEqual([+ {+ name: 'database_sql.rows.decode',+ attributes: { 'database_sql.rows': 2 },+ children: [],+ ended: true,+ },+ ]);+});diff --git a/apps/web/src/lib/core/database-sql/trace.ts b/apps/web/src/lib/core/database-sql/trace.tsindex c3ef05fa14..a51f27ca85 100644--- a/apps/web/src/lib/core/database-sql/trace.ts+++ b/apps/web/src/lib/core/database-sql/trace.ts@@ -14,6 +14,8 @@ import type {RunError,} from './generated/types';+import { profileDatabasePhase } from './profile';+const PREFIX = '[database-sql]';/** Long statements are cut to this many characters on the run span. */const SQL_ATTRIBUTE_LENGTH = 4000;@@ -46,6 +48,8 @@ export type DatabaseSqlReadContext = {/** What a row source tells a step about the GraphQL request it sends. */export interface DatabaseSqlStepTrace {+ /** Convert a page under its fetch span, independent of ambient async context. */+ decode<Value>(rows: number, operation: () => Value): Value;request(document: string, variables: unknown): void;/** W3C trace headers that put the request inside the step's span. */headers(): Record<string, string>;@@ -72,6 +76,9 @@ export type DatabaseSqlStepRecord = {/** What the driver asks of a run's trace while it drives the engine. */export interface DatabaseSqlRunTrace {+ /** Compile/open and start under this run even after an async boundary. */+ open<Value>(operation: () => Promise<Value>): Promise<Value>;+ start<Value>(operation: () => Value): Value;/** Read a step's rows inside its span; `count` says how many came back. */fetch<Value, Failure>(request: DatabaseSqlFetch,@@ -214,6 +221,20 @@ export function traceDatabaseSqlRun<const steps: DatabaseSqlStepRecord[] = [];const pages = new Map<string, number>();const trace: DatabaseSqlRunTrace = {+ async open(operation) {+ const span = runSpan.span('database_sql.engine.open');+ try {+ return await span.run(operation);+ } finally {+ span.end();+ }+ },+ start(operation) {+ return profileDatabasePhase(+ runSpan.span('database_sql.engine.start'),+ operation+ );+ },fetch(request, read, count) {const kind = fetchKind(request);const table = fetchTable(request);@@ -237,6 +258,11 @@ export function traceDatabaseSqlRun<span.setAttr('database_sql.page', page);span.setAttr('database_sql.hint_count', step.hints);const stepTrace: DatabaseSqlStepTrace = {+ decode(rows, operation) {+ const decodeSpan = span.span('database_sql.rows.decode');+ decodeSpan.setAttr('database_sql.rows', rows);+ return profileDatabasePhase(decodeSpan, operation);+ },request: (document, variables) =>step.requests.push({ document, variables }),headers: () => {diff --git a/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts b/apps/web/src/lib/queries/database-sql/create-database-sql-query.tsindex b610fd916a..70689f258e 100644--- a/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts+++ b/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts@@ -19,6 +19,10 @@ import type {Outcome,Schema,} from '@core/database-sql/generated/types';+import {+ profileDatabasePhase,+ traceDatabaseFrame,+} from '@core/database-sql/profile';import type {DatabaseSqlReadContext,DatabaseSqlReadReason,@@ -26,6 +30,7 @@ import type {import { buildDatabaseSqlCatalog } from '@core/database-sql/wasm-module';import { idToDisplayName, idToEmail } from '@core/user/util';import type { CacheHost } from '@graphql-cache/host/types';+import { Telemetry } from '@macro-inc/observability';import { queryClient } from '@queries/client';import { contactsQueryOptions } from '@queries/contacts/contacts';import { subscribeToVisibleCacheChanges } from '@queries/subscribe-to-visible-cache-changes';@@ -168,6 +173,7 @@ export function createDatabaseSqlQuery(const [error, setError] = createSignal<DatabaseSqlFailure>();const [loading, setLoading] = createSignal(false);let latest = 0;+ let cancelFrame: (() => void) | undefined;// The cache may not hold what an in-flight network read will bring, so a// cache change waits for it instead of answering from older rows.let networkRead: ResultAsync<DatabaseSqlRun, DatabaseSqlFailure> | undefined;@@ -183,40 +189,77 @@ export function createDatabaseSqlQuery(options: { reportFailure?: boolean; keepLoading?: boolean } = {}): ResultAsync<DatabaseSqlRun, DatabaseSqlFailure> => {const generation = ++latest;+ const querySpan = Telemetry.span('database_sql.query');+ querySpan.setAttr('database_sql.read_reason', reason);+ querySpan.setAttr('database_sql.request_policy', requestPolicy);+ if (current.scope)+ querySpan.setAttr('database_sql.scope_id', current.scope);+ if (current.view)+ querySpan.setAttr('database_sql.table_id', current.view.tableId);+ const catalogSpan = querySpan.span('database_sql.catalog');const host = capabilities.cacheHost();setLoading(true);- const answered = statementCatalog(current, capabilities)+ const answered = catalogSpan+ .run(() => statementCatalog(current, capabilities))+ .andTee(() => catalogSpan.end())+ .orTee(() => catalogSpan.end()).andThen((built) =>- runStatement(- built,- current,- createGraphqlRowSource({- client: capabilities.client(),- catalog: built,- requestPolicy,- cache: host,- people: capabilities.people,- membership: host ? { host, baselines, reconcile } : undefined,- }),- capabilities,- {- reason,- scope: current.scope,- requestPolicy,- reportFailure: options.reportFailure,- }- ).map((answer): DatabaseSqlRun => {- if (generation !== latest) return { landed: false };- // A cache change that left the answer alone keeps the same outcome.- batch(() => {- if (JSON.stringify(untrack(catalog)) !== JSON.stringify(built))- setCatalog(built);- if (JSON.stringify(untrack(outcome)) !== JSON.stringify(answer))- setOutcome(answer);- setError(undefined);- });- return { landed: true };- })+ querySpan+ .run(() =>+ runStatement(+ built,+ current,+ createGraphqlRowSource({+ client: capabilities.client(),+ catalog: built,+ requestPolicy,+ cache: host,+ people: capabilities.people,+ membership: host ? { host, baselines, reconcile } : undefined,+ }),+ capabilities,+ {+ reason,+ scope: current.scope,+ requestPolicy,+ reportFailure: options.reportFailure,+ }+ )+ )+ .map((answer): DatabaseSqlRun => {+ if (generation !== latest) return { landed: false };+ // A cache change that left the answer alone keeps the same outcome.+ querySpan.setAttr('database_sql.row_count', answer.rows.length);+ querySpan.setAttr('database_sql.truncated', answer.truncated);+ const changed = profileDatabasePhase(+ querySpan.span('database_sql.compare'),+ () => ({+ catalog:+ JSON.stringify(untrack(catalog)) !== JSON.stringify(built),+ outcome:+ JSON.stringify(untrack(outcome)) !== JSON.stringify(answer),+ })+ );+ querySpan.setAttr(+ 'database_sql.result_changed',+ changed.catalog || changed.outcome+ );+ const publishSpan = querySpan.span('database_sql.publish');+ profileDatabasePhase(publishSpan, () =>+ batch(() => {+ if (changed.catalog) setCatalog(built);+ if (changed.outcome) setOutcome(answer);+ setError(undefined);+ })+ );+ if (changed.catalog || changed.outcome) {+ cancelFrame?.();+ cancelFrame = traceDatabaseFrame(+ querySpan.span('database_sql.frame')+ );+ }+ return { landed: true };+ })).orElse((failure) => {if (generation !== latest) return okAsync({ landed: false });@@ -224,9 +267,21 @@ export function createDatabaseSqlQuery(return errAsync(failure);});const settle = async () => {- const result = await answered;- if (generation === latest && !options.keepLoading) setLoading(false);- return result;+ try {+ const result = await answered;+ if (generation === latest && !options.keepLoading) setLoading(false);+ querySpan.setAttr(+ 'database_sql.outcome',+ result.isErr()+ ? 'error'+ : result.value.landed+ ? 'published'+ : 'superseded'+ );+ return result;+ } finally {+ querySpan.end();+ }};return new ResultAsync(settle());};@@ -267,6 +322,7 @@ export function createDatabaseSqlQuery(createEffect(on(statement, (current, previous) => {+ cancelFrame?.();baselines = new Map();latest += 1;setError(undefined);@@ -305,6 +361,7 @@ export function createDatabaseSqlQuery(});onCleanup(() => {+ cancelFrame?.();latest += 1;});diff --git a/apps/web/src/lib/queries/database-sql/graphql-source.test.ts b/apps/web/src/lib/queries/database-sql/graphql-source.test.tsindex f4fc676fb8..84c82fe109 100644--- a/apps/web/src/lib/queries/database-sql/graphql-source.test.ts+++ b/apps/web/src/lib/queries/database-sql/graphql-source.test.ts@@ -365,7 +365,11 @@ describe('the GraphQL row source', () => {[],null,500,- { request: () => {}, headers: () => ({}) }+ {+ decode: (_rows, operation) => operation(),+ request: () => {},+ headers: () => ({}),+ });expect(page._unsafeUnwrap().rows).toEqual([diff --git a/apps/web/src/lib/queries/database-sql/graphql-source.ts b/apps/web/src/lib/queries/database-sql/graphql-source.tsindex 5f92de45a1..d57f693333 100644--- a/apps/web/src/lib/queries/database-sql/graphql-source.ts+++ b/apps/web/src/lib/queries/database-sql/graphql-source.ts@@ -376,7 +376,7 @@ function soupPage(const evidence = input.initial ? JSON.stringify(input) : undefined;const local: ResultAsync<Page | undefined, DatabaseSqlFetchFailure> =evidence && membership?.reconcile- ? reconciledPage(input, evidence, membership)+ ? reconciledPage(input, evidence, membership, step): okAsync(undefined);return local.andThen((page) =>page@@ -397,7 +397,10 @@ function soupPage(complete: nextCursor === null,});}- return tableRows(items).map((rows) => ({ rows, next: nextCursor }));+ return tableRows(items, step).map((rows) => ({+ rows,+ next: nextCursor,+ }));}));}@@ -416,7 +419,8 @@ function isNetworkRead(requestPolicy: RequestPolicy): boolean {function reconciledPage(input: SoupInput,evidence: string,- { host, baselines }: LocalMembership+ { host, baselines }: LocalMembership,+ step: DatabaseSqlStepTrace): ResultAsync<Page | undefined, DatabaseSqlFetchFailure> {const initial = input.initial;const baseline = baselines.get(evidence);@@ -448,7 +452,8 @@ function reconciledPage(thrownFetchFailure).andThen(({ records }) =>tableRows(- materializeReconciledSoup(result.keys, records, baseline.items)+ materializeReconciledSoup(result.keys, records, baseline.items),+ step).map((rows) => ({ rows, next: null })));}@@ -456,13 +461,16 @@ function reconciledPage(}function tableRows(- items: readonly DatabaseRowFieldsFragment[]+ items: readonly DatabaseRowFieldsFragment[],+ step: DatabaseSqlStepTrace): Result<Row[], DatabaseSqlFetchFailure> {- return Result.combine(- items.map((item) =>- item.__typename === 'GraphqlSoupDatabaseRow'- ? ok({ id: item.id, position: item.position, cells: rowCells(item) })- : err(fetchFailure(`a table query returned a ${item.__typename}`))+ return step.decode(items.length, () =>+ Result.combine(+ items.map((item) =>+ item.__typename === 'GraphqlSoupDatabaseRow'+ ? ok({ id: item.id, position: item.position, cells: rowCells(item) })+ : err(fetchFailure(`a table query returned a ${item.__typename}`))+ )));}diff --git a/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs b/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rsindex cd2a983f22..021cd20290 100644--- a/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs+++ b/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs@@ -48,6 +48,7 @@ use models_soup::{use recursion::CollapsibleExt;use sqlx::{PgPool, Postgres, QueryBuilder, Row, postgres::PgRow, prelude::FromRow};use system_properties::{StatusOption, SystemPropertyKey};+use tracing::Instrument;use uuid::Uuid;use crate::domain::models::{@@ -1832,7 +1833,6 @@ impl SoupRow {}}- #[tracing::instrument(err)]fn into_projection_hydration(self) -> Result<SoupProjectionHydration, sqlx::Error> {let document_server_facts = self.document_server_facts();Ok(SoupProjectionHydration {@@ -1841,7 +1841,6 @@ impl SoupRow {})}- #[tracing::instrument(err)]fn into_soup_item(self) -> Result<SoupItem<()>, sqlx::Error> {Ok(match self {SoupRow::Document(DocumentRow {@@ -2025,6 +2024,15 @@ async fn expanded_dynamic_cursor_soup_hydrated(for ids in document_ids.into_binds() {query = query.bind(ids);}+ let mut connection = db+ .acquire()+ .instrument(tracing::info_span!(+ "soup.pool.acquire",+ pool_size = db.size(),+ pool_idle = db.num_idle()+ ))+ .await?;+ let query_span = tracing::info_span!("soup.sql.fetch_decode", rows = tracing::field::Empty);let items = query// Unnamed statement: the SQL text varies per filter shape and per// interpolated literal (dates change daily), so a cached prepared@@ -2034,8 +2042,11 @@ async fn expanded_dynamic_cursor_soup_hydrated(// time costs ~1ms and keeps the plan stable..persistent(false).try_map(|row| SoupRow::from_row(&row)?.into_projection_hydration())- .fetch_all(db)+ .fetch_all(&mut *connection)+ .instrument(query_span.clone()).await?;+ query_span.record("rows", items.len());+ drop(query_span);Ok(items)}@@ -2454,27 +2465,41 @@ pub async fn expanded_dynamic_cursor_soup_grouped(query = query.bind(ids);}+ let mut connection = db+ .acquire()+ .instrument(tracing::info_span!(+ "soup.pool.acquire",+ pool_size = db.size(),+ pool_idle = db.num_idle()+ ))+ .await?;+ let query_span = tracing::info_span!("soup.sql.fetch_decode", rows = tracing::field::Empty);let rows: Vec<GroupedSoupRow> = query// Unnamed statement — same rationale as expanded_dynamic_cursor_soup:// filter-shaped SQL text gets no reuse from the statement cache but// does get generic-plan flips..persistent(false)- .fetch_all(db)+ .fetch_all(&mut *connection)+ .instrument(query_span.clone()).await?;-- let items = rows- .into_iter()- .map(|row| {- Ok(ItemGroupingInfo {- item: row.item.into_soup_item()?,- key: row.group.group_key,- total_group_count: usize::try_from(row.group.group_total_count)- .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,- index_in_group: usize::try_from(row.group.row_in_group)- .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,+ query_span.record("rows", rows.len());+ drop(query_span);+ drop(connection);++ let items = tracing::info_span!("soup.rows.convert", rows = rows.len()).in_scope(|| {+ rows.into_iter()+ .map(|row| {+ Ok(ItemGroupingInfo {+ item: row.item.into_soup_item()?,+ key: row.group.group_key,+ total_group_count: usize::try_from(row.group.group_total_count)+ .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,+ index_in_group: usize::try_from(row.group.row_in_group)+ .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,+ })})- })- .collect::<Result<Vec<_>, sqlx::Error>>()?;+ .collect::<Result<Vec<_>, sqlx::Error>>()+ })?;Ok(items.into_iter())}diff --git a/docker/docker-compose-databases.yml b/docker/docker-compose-databases.ymlindex 4120625274..664d2b8130 100644--- a/docker/docker-compose-databases.yml+++ b/docker/docker-compose-databases.yml@@ -9,7 +9,9 @@ services:# The default 100 max_connections is exhausted when every service's pool# fills under load (e.g. a large email backfill): workers then die on# "pool timed out" / "too many clients". Local-only headroom.- command: postgres -c max_connections=500+ # The migration installs pg_stat_statements; preload it so local query+ # profiling can separate database execution from application timings.+ command: postgres -c max_connections=500 -c shared_preload_libraries=pg_stat_statementshealthcheck:test: ["CMD-SHELL", "pg_isready -U user"]interval: 2sdiff --git a/services/document_storage_service/src/api/graphql_soup.rs b/services/document_storage_service/src/api/graphql_soup.rsindex 4802990e33..87a97a2826 100644--- a/services/document_storage_service/src/api/graphql_soup.rs+++ b/services/document_storage_service/src/api/graphql_soup.rs@@ -5,6 +5,7 @@ use async_graphql::{Data, http::GraphiQLSource};use async_graphql_axum::{GraphQLRequest, GraphQLResponse};use axum::{Router,+ body::HttpBody,extract::{OriginalUri, State},http::{StatusCode, request::Parts},response::{Html, IntoResponse, Response},@@ -56,7 +57,7 @@ async fn graphql_handler(>,request_parts: Parts,request: GraphQLRequest,-) -> GraphQLResponse {+) -> Response {let acting_user = auth.authorization.as_ref()@@ -68,11 +69,20 @@ async fn graphql_handler(acting_user.and_then(|user| user.user_context.organization_id.map(i64::from)),acting_user.and_then(opaque_viewer_id),);- state+ let result = state.graphql_soup_schema.execute(request.data(GraphqlRequestParts::new(request_parts)))- .await- .into()+ .await;+ graphql_response(result)+}++#[tracing::instrument(skip_all, name = "graphql.serialize", fields(response_bytes = tracing::field::Empty))]+fn graphql_response(result: async_graphql::Response) -> Response {+ let response = GraphQLResponse::from(result).into_response();+ if let Some(bytes) = response.body().size_hint().exact() {+ tracing::Span::current().record("response_bytes", bytes);+ }+ response}async fn subscription_handler(@@ -257,3 +267,6 @@ fn insert_graphql_context_data(state.activity_reader.clone(),));}++#[cfg(test)]+mod test;diff --git a/services/document_storage_service/src/api/graphql_soup/test.rs b/services/document_storage_service/src/api/graphql_soup/test.rsnew file mode 100644index 0000000000..6cabbeb588--- /dev/null+++ b/services/document_storage_service/src/api/graphql_soup/test.rs@@ -0,0 +1,38 @@+use super::graphql_response;+use axum::{+ body::{HttpBody, to_bytes},+ http::{HeaderValue, StatusCode},+};++#[tokio::test]+async fn serialization_preserves_data_headers_and_body_length() {+ let mut result = async_graphql::Response::new(async_graphql::value!({ "rows": [1, 2] }));+ result+ .http_headers+ .insert("x-profile-test", HeaderValue::from_static("preserved"));+ let response = graphql_response(result);+ assert_eq!(response.status(), StatusCode::OK);+ assert_eq!(+ response.headers()["content-type"],+ "application/graphql-response+json"+ );+ assert_eq!(response.headers()["x-profile-test"], "preserved");+ let bytes = response.body().size_hint().exact().unwrap();+ let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();+ assert_eq!(bytes, body.len() as u64);+ assert_eq!(body.as_ref(), br#"{"data":{"rows":[1,2]}}"#);+}++#[tokio::test]+async fn serialization_preserves_graphql_errors() {+ let result = async_graphql::Response::from_errors(vec![async_graphql::ServerError::new(+ "refused", None,+ )]);+ let response = graphql_response(result);+ assert_eq!(response.status(), StatusCode::OK);+ let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();+ assert_eq!(+ body.as_ref(),+ br#"{"data":null,"errors":[{"message":"refused"}]}"#+ );+}