Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(databases): trace table-open phases and add a local profiling harness (#7496)

Wolf Mermelstein97892c20d29b
diff --git a/apps/web/scripts/database-profile/README.md b/apps/web/scripts/database-profile/README.md
new file mode 100644
index 0000000000..3392aa6e08
--- /dev/null
+++ b/apps/web/scripts/database-profile/README.md
@@ -0,0 +1,94 @@
+# Database profiling
+
+Run the isolated stack with browser telemetry enabled, following
+[Running locally](../../../../docs/development/running-locally.md). Keep the instance running
+throughout the audit; `run_local` startup and its quit command rebuild/remove
+instance data. Use the `r` hotkey to rebuild services without replacing data.
+
+```sh
+nix develop --command just run_local --no-doppler --instance database-profiling-audit --port-base 21000
+```
+
+Use a private Playwright browser to sign in and save its `storageState` outside
+the repository. That file contains credentials: do not commit it or attach it to
+a PR. Create dedicated synthetic databases through the app's supported APIs.
+Keep existing fixtures. Test 1k, 5k, 10k, 25k, 50k and 100k **stored** rows, with
+both unfiltered and selective views. Record columns, types, relationships,
+filters, build SHA and browser version alongside each run.
+
+From `apps/web`, measure without changing data:
+
+```sh
+bun scripts/database-profile/measure.mjs \
+ --url https://localhost:21009/app/database/UUID \
+ --state /tmp/database-profile-auth.json \
+ --table Records --rows 1000 --entries 1000 \
+ --output /tmp/database-profile-1000 --repeats 3 \
+ --chrome /path/to/google-chrome
+```
+
+Use the actual proxy URL printed by `run_local`. The harness accepts loopback
+URLs only, opens its own browser, and refuses to overwrite an output directory.
+Each repetition starts with a fresh context containing only the saved login
+state, then measures a warm reload in that same context. `--rows` is the number
+of saved rows expected in the selected view; the blank insertion row is excluded.
+`--entries` is the fixture's total stored row count. The engine currently caps
+reads at 20,000 rows, so a 100k database does **not** imply 100k rendered rows.
+Confirm the trace's `database_sql.truncated` attribute when interpreting results.
+
+The JSON results include first complete grid and subsequent frame opportunity,
+long tasks, CDP script/layout/task counters, heap size, HTTP timing/bytes, and
+traceparents. The frame measurement is a two-frame rendering opportunity, not
+an assertion that the browser painted. The 100ms row polling adds up to 100ms
+measurement uncertainty; use the internal spans to attribute work precisely.
+Requests completing within 1.5 seconds after that frame are included to expose
+background refreshes; this is not a measurement of network quiescence.
+
+Run `--profile` separately to save Chrome CPU profiles. Compare ordinary runs
+first: profiling adds overhead. Use at least three repetitions and report
+medians and ranges, not the fastest run. Vite measurements include development
+overhead; repeat production-bundle measurements before claiming production
+latency. Exported profiles may contain application source URLs and fixture IDs.
+
+## Trace boundaries
+
+| Span | What it measures |
+| --- | --- |
+| `database_sql.query` | Catalog through publishing, with reason, policy, target, row count, truncation and outcome |
+| `database_sql.catalog` | WASM load if necessary and catalog construction |
+| `database_sql.engine.open` / `.start` | Engine opening/compilation and initial planning |
+| `database_sql.fetch` / `.fold` | Each existing 500-row source request and the engine consuming its answer |
+| `database_sql.rows.decode` | GraphQL property values converted into engine cells, once per page |
+| `database_sql.compare` | Equality comparison of the new catalog/result with the prior answer |
+| `database_sql.publish` | Signal updates and synchronous downstream reactive work |
+| `database_sql.frame` | Time after a changed result is published until a rendering opportunity; cancelled/hidden/timeout outcomes are not ready frames |
+| `soup.pool.acquire` | Connection acquisition, with pool size and idle count at entry |
+| `soup.sql.fetch_decode` | Query construction, SQL round trip, row decoding and flat-row conversion, excluding connection acquisition |
+| `soup.rows.convert` | Grouped-row conversion after the SQL read |
+| `graphql.serialize` | GraphQL response serialization and response construction, with uncompressed response bytes |
+
+`graphql.execute` includes serialization. HTTP transfer, compression and proxy
+time are outside `graphql.serialize`. Fetch/decode is not server-only SQL time:
+use PostgreSQL execution plans, `pg_stat_statements`, and lock/I/O diagnostics
+to separate server execution from transfer and client decoding. Cache worker
+spans already measure cache reads, locks and queueing. No per-cell spans or
+per-row conversion spans are emitted by these additions.
+
+New local stacks preload `pg_stat_statements` (the existing migration installs
+the extension). Verify `SHOW shared_preload_libraries`, then take snapshots of
+`pg_stat_statements` before and after each measured workload, restricted to the
+current database’s `dbid` from `pg_database`. Compare deltas in
+`calls`, `total_exec_time`, `rows`, `shared_blks_hit` and `shared_blks_read` by
+`dbid`, `userid`, `queryid` and `toplevel`; do not reset shared statistics. Statistics span all requests to this
+database, so correlate the measured window and query shape with the trace.
+Use `EXPLAIN (ANALYZE, BUFFERS)` only for the audit's read queries on synthetic
+fixtures. Existing running stacks need a PostgreSQL restart to activate preload;
+do not restart another session's stack or rerun `run_local` over retained data.
+
+Use a request's traceparent to retrieve its complete trace from Tempo. Verify
+the browser flag/exporter and server OTEL filter before an audit; do not infer
+that absent spans mean zero cost. Correlate rows, bytes, page count and cache
+policy across browser, GraphQL, authorization, properties and SQL. Validate
+edits, added rows, picker latency, table/view switching and scrolling separately
+on synthetic fixtures after every optimization; this read-only harness covers
+opening and reloading only. Preserve the 500-row limits and grid behavior.
diff --git a/apps/web/scripts/database-profile/measure.mjs b/apps/web/scripts/database-profile/measure.mjs
new file mode 100644
index 0000000000..165d048899
--- /dev/null
+++ b/apps/web/scripts/database-profile/measure.mjs
@@ -0,0 +1,215 @@
+/** Read-only browser benchmark. Run with --help for arguments. */
+import { mkdir, writeFile } from 'node:fs/promises';
+import { resolve } from 'node:path';
+import { parseArgs } from 'node:util';
+import { chromium } from 'playwright';
+
+const { values } = parseArgs({
+ options: {
+ url: { type: 'string' },
+ state: { type: 'string' },
+ table: { type: 'string' },
+ rows: { type: 'string' },
+ entries: { type: 'string' },
+ repeats: { type: 'string', default: '3' },
+ output: { type: 'string' },
+ chrome: { type: 'string' },
+ profile: { type: 'boolean', default: false },
+ help: { type: 'boolean', default: false },
+ },
+});
+if (values.help) {
+ console.log(
+ 'bun scripts/database-profile/measure.mjs --url <database URL> --state <Playwright auth JSON> --rows <expected loaded rows> --entries <stored rows> --output <new directory> [--table <name>] [--repeats 3] [--chrome <executable>] [--profile]'
+ );
+ process.exit(0);
+}
+if (
+ !values.url ||
+ !values.state ||
+ !values.output ||
+ !values.rows ||
+ !values.entries
+)
+ throw new Error('Required: --url, --state, --rows, --entries, --output');
+const url = new URL(values.url);
+if (!['localhost', '127.0.0.1', '[::1]'].includes(url.hostname))
+ throw new Error('Use an isolated local stack on localhost.');
+const expectedRows = Number(values.rows);
+const entries = Number(values.entries);
+const repeats = Number(values.repeats);
+if (
+ ![expectedRows, entries, repeats].every(
+ (n) => Number.isSafeInteger(n) && n > 0
+ ) ||
+ entries < expectedRows
+)
+ throw new Error(
+ 'Row counts and repeats must be positive integers; entries must cover loaded rows.'
+ );
+const output = resolve(values.output);
+await mkdir(output); // Refuse to overwrite a previous run.
+const browser = await chromium.launch({
+ executablePath: values.chrome,
+ headless: true,
+});
+await writeFile(
+ `${output}/environment.json`,
+ JSON.stringify(
+ {
+ browser: browser.version(),
+ startedAt: new Date().toISOString(),
+ databasePath: url.pathname,
+ entries,
+ expectedRows,
+ profile: values.profile,
+ },
+ null,
+ 2
+ )
+);
+const results = [];
+try {
+ for (let repeat = 1; repeat <= repeats; repeat++) {
+ const context = await browser.newContext({
+ storageState: values.state,
+ ignoreHTTPSErrors: true,
+ viewport: { width: 1600, height: 1000 },
+ });
+ await context.addInitScript(() => {
+ window.__databaseProfile = { longTasks: [] };
+ new PerformanceObserver((list) => {
+ for (const entry of list.getEntries())
+ window.__databaseProfile.longTasks.push({
+ start: entry.startTime,
+ duration: entry.duration,
+ });
+ }).observe({ type: 'longtask', buffered: true });
+ });
+ const page = await context.newPage();
+ const session = await context.newCDPSession(page);
+ await session.send('Performance.enable');
+ for (const temperature of ['cold', 'warm']) {
+ const requests = [];
+ const pending = [];
+ const finished = (request) => {
+ const path = new URL(request.url()).pathname;
+ if (
+ path !== '/dss/items/soup/graphql' &&
+ path !== '/dss/databases' &&
+ !path.startsWith('/dss/databases/')
+ )
+ return;
+ pending.push(
+ (async () => {
+ const response = await request.response();
+ requests.push({
+ path,
+ status: response?.status(),
+ traceparent: request.headers().traceparent,
+ timing: request.timing(),
+ sizes: await request.sizes().catch(() => undefined),
+ });
+ })()
+ );
+ };
+ page.on('requestfinished', finished);
+ if (values.profile) {
+ await session.send('Profiler.enable');
+ await session.send('Profiler.start');
+ }
+ const started = performance.now();
+ await page.goto(values.url, {
+ waitUntil: 'domcontentloaded',
+ timeout: 180_000,
+ });
+ if (values.table)
+ await page
+ .locator('[aria-label="Database tables"]')
+ .getByText(values.table, { exact: true })
+ .click();
+ await page.waitForFunction(
+ (count) => {
+ const grid = document.querySelector('[role="grid"]');
+ return (
+ grid &&
+ grid.querySelectorAll(
+ '[data-grid-row-id]:not([data-grid-row-id^="draft:"])'
+ ).length >= count
+ );
+ },
+ expectedRows,
+ { timeout: 180_000, polling: 100 }
+ );
+ const dataReadyMs = performance.now() - started;
+ const snapshot = await page.evaluate(async () => {
+ if (document.hidden)
+ throw new Error('The benchmark tab must be visible.');
+ await new Promise((resolve) =>
+ requestAnimationFrame(() => requestAnimationFrame(resolve))
+ );
+ const grid = document.querySelector('[role="grid"]');
+ return {
+ rows: grid.querySelectorAll(
+ '[data-grid-row-id]:not([data-grid-row-id^="draft:"])'
+ ).length,
+ cells: grid.querySelectorAll('[role="gridcell"]').length,
+ visibility: document.visibilityState,
+ longTasks: window.__databaseProfile.longTasks,
+ };
+ });
+ const frameReadyMs = performance.now() - started;
+ const after = await session.send('Performance.getMetrics');
+ // CDP navigation counters reset on document navigation.
+ const metrics = Object.fromEntries(
+ after.metrics.map(({ name, value }) => [name, value])
+ );
+ if (values.profile) {
+ const { profile } = await session.send('Profiler.stop');
+ await writeFile(
+ `${output}/${repeat}-${temperature}.cpuprofile`,
+ JSON.stringify(profile)
+ );
+ }
+ // Capture refresh requests too, separately from the first rendering opportunity.
+ await page.waitForTimeout(1500);
+ page.off('requestfinished', finished);
+ await Promise.all(pending);
+ const result = {
+ repeat,
+ temperature,
+ entries,
+ expectedRows,
+ dataReadyMs,
+ frameReadyMs,
+ ...snapshot,
+ metrics,
+ requests,
+ };
+ results.push(result);
+ await writeFile(
+ `${output}/results.json`,
+ JSON.stringify(results, null, 2)
+ );
+ console.log(
+ JSON.stringify({
+ repeat,
+ temperature,
+ entries,
+ loaded: snapshot.rows,
+ dataReadyMs,
+ frameReadyMs,
+ })
+ );
+ }
+ await context.close();
+ }
+} catch (error) {
+ await writeFile(
+ `${output}/failure.json`,
+ JSON.stringify({ error: String(error), completed: results.length }, null, 2)
+ );
+ throw error;
+} finally {
+ await browser.close();
+}
diff --git a/apps/web/src/lib/core/database-sql/driver.ts b/apps/web/src/lib/core/database-sql/driver.ts
index c6d25a7517..7c167985c3 100644
--- a/apps/web/src/lib/core/database-sql/driver.ts
+++ b/apps/web/src/lib/core/database-sql/driver.ts
@@ -147,7 +147,7 @@ async function steps(
source: RowSource,
trace: DatabaseSqlRunTrace
): Promise<Result<Outcome, DatabaseSqlFailure>> {
- let step = feed(() => query.start());
+ let step = trace.start(() => feed(() => query.start()));
while (step.isOk()) {
const current = step.value;
if (current.step === 'done') {
@@ -164,7 +164,7 @@ function drive(
source: RowSource,
trace: DatabaseSqlRunTrace
): ResultAsync<Outcome, DatabaseSqlFailure> {
- return ResultAsync.fromPromise(opened(), engineFailure).andThen(
+ return ResultAsync.fromPromise(trace.open(opened), engineFailure).andThen(
(query) =>
new ResultAsync(steps(query, source, trace).finally(() => query.free()))
);
diff --git a/apps/web/src/lib/core/database-sql/profile.test.ts b/apps/web/src/lib/core/database-sql/profile.test.ts
new file mode 100644
index 0000000000..7b9f2ce455
--- /dev/null
+++ b/apps/web/src/lib/core/database-sql/profile.test.ts
@@ -0,0 +1,89 @@
+import type { Span } from '@macro-inc/observability';
+import { afterEach, expect, it, vi } from 'vitest';
+import { profileDatabasePhase, traceDatabaseFrame } from './profile';
+
+function span() {
+ return {
+ run: <T>(operation: () => T) => operation(),
+ setAttr: vi.fn(),
+ end: vi.fn(),
+ } as unknown as Span;
+}
+
+afterEach(() => vi.restoreAllMocks());
+
+it('ends a synchronous phase even when it throws', () => {
+ const phase = span();
+ expect(() =>
+ profileDatabasePhase(phase, () => {
+ throw new Error('failed');
+ })
+ ).toThrow('failed');
+ expect(phase.end).toHaveBeenCalledTimes(1);
+});
+
+it('waits for two frames and records a frame opportunity, not a paint timestamp', () => {
+ const callbacks: FrameRequestCallback[] = [];
+ vi.spyOn(window, 'requestAnimationFrame').mockImplementation((callback) =>
+ callbacks.push(callback)
+ );
+ vi.spyOn(window, 'cancelAnimationFrame').mockImplementation(() => {});
+ const phase = span();
+ const cancel = traceDatabaseFrame(phase);
+ callbacks[0](0);
+ expect(phase.end).not.toHaveBeenCalled();
+ callbacks[1](16);
+ expect(phase.setAttr).toHaveBeenCalledWith(
+ 'database_sql.frame_status',
+ 'ready'
+ );
+ cancel();
+ expect(phase.end).toHaveBeenCalledTimes(1);
+});
+
+it('cancels a superseded result before recording a ready frame', () => {
+ vi.spyOn(window, 'requestAnimationFrame').mockReturnValue(7);
+ const cancelled = vi
+ .spyOn(window, 'cancelAnimationFrame')
+ .mockImplementation(() => {});
+ const phase = span();
+ traceDatabaseFrame(phase)();
+ expect(cancelled).toHaveBeenCalledWith(7);
+ expect(phase.setAttr).toHaveBeenCalledWith(
+ 'database_sql.frame_status',
+ 'cancelled'
+ );
+ expect(phase.end).toHaveBeenCalledTimes(1);
+});
+
+it('does not call a hidden document a ready frame', () => {
+ vi.spyOn(document, 'hidden', 'get').mockReturnValue(true);
+ const request = vi.spyOn(window, 'requestAnimationFrame');
+ const phase = span();
+ traceDatabaseFrame(phase)();
+ expect(request).not.toHaveBeenCalled();
+ expect(phase.setAttr).toHaveBeenCalledWith(
+ 'database_sql.frame_status',
+ 'hidden'
+ );
+ expect(phase.end).toHaveBeenCalledTimes(1);
+});
+
+it('ends a frame span if animation frames never arrive', () => {
+ vi.useFakeTimers();
+ try {
+ vi.spyOn(window, 'requestAnimationFrame').mockReturnValue(7);
+ vi.spyOn(window, 'cancelAnimationFrame').mockImplementation(() => {});
+ const phase = span();
+ const cancel = traceDatabaseFrame(phase);
+ vi.advanceTimersByTime(10_000);
+ expect(phase.setAttr).toHaveBeenCalledWith(
+ 'database_sql.frame_status',
+ 'timeout'
+ );
+ cancel();
+ expect(phase.end).toHaveBeenCalledTimes(1);
+ } finally {
+ vi.useRealTimers();
+ }
+});
diff --git a/apps/web/src/lib/core/database-sql/profile.ts b/apps/web/src/lib/core/database-sql/profile.ts
new file mode 100644
index 0000000000..60008d7101
--- /dev/null
+++ b/apps/web/src/lib/core/database-sql/profile.ts
@@ -0,0 +1,36 @@
+import type { Span } from '@macro-inc/observability';
+
+/** Synchronous work, including Solid's downstream reactive updates. */
+export function profileDatabasePhase<T>(span: Span, operation: () => T): T {
+ try {
+ return span.run(operation);
+ } finally {
+ span.end();
+ }
+}
+
+/** Two animation frames provide a rendering opportunity, not proof of paint. */
+export function traceDatabaseFrame(span: Span): () => void {
+ let ended = false;
+ let frame: number | undefined;
+ const finish = (status: 'ready' | 'cancelled' | 'hidden' | 'timeout') => {
+ if (ended) return;
+ ended = true;
+ if (frame !== undefined) cancelAnimationFrame(frame);
+ clearTimeout(timeout);
+ document.removeEventListener('visibilitychange', visibilityChanged);
+ span.setAttr('database_sql.frame_status', status);
+ span.end();
+ };
+ const visibilityChanged = () => {
+ if (document.hidden) finish('hidden');
+ };
+ const timeout = setTimeout(() => finish('timeout'), 10_000);
+ document.addEventListener('visibilitychange', visibilityChanged);
+ if (document.hidden) finish('hidden');
+ else
+ frame = requestAnimationFrame(() => {
+ frame = requestAnimationFrame(() => finish('ready'));
+ });
+ return () => finish('cancelled');
+}
diff --git a/apps/web/src/lib/core/database-sql/trace.test.ts b/apps/web/src/lib/core/database-sql/trace.test.ts
index b54ed8a1bb..d32f65fd4c 100644
--- a/apps/web/src/lib/core/database-sql/trace.test.ts
+++ b/apps/web/src/lib/core/database-sql/trace.test.ts
@@ -1,4 +1,4 @@
-import { errAsync, ok, okAsync } from 'neverthrow';
+import { errAsync, ok, okAsync, ResultAsync } from 'neverthrow';
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { Catalog, Outcome } from './generated/types';
import { traceDatabaseSqlRun } from './trace';
@@ -254,3 +254,54 @@ describe('database SQL run traces', () => {
);
});
});
+
+it('keeps engine and decoding phases under explicit parents across async work', async () => {
+ vi.spyOn(console, 'groupCollapsed').mockImplementation(() => {});
+ vi.spyOn(console, 'log').mockImplementation(() => {});
+ vi.spyOn(console, 'groupEnd').mockImplementation(() => {});
+ await traceDatabaseSqlRun(
+ { kind: 'sql', sql: 'SELECT 1' },
+ catalog,
+ (trace) =>
+ new ResultAsync(
+ (async () => {
+ await trace.open(async () => 1);
+ trace.start(() => 2);
+ await trace.fetch(
+ {
+ step: 'fetch',
+ id: 0,
+ query: {
+ type: 'soup',
+ table: 'guests',
+ propf: null,
+ keyHint: null,
+ },
+ needs: [],
+ cursor: null,
+ limit: 500,
+ },
+ (step) => okAsync(step.decode(2, () => [1, 2])),
+ (rows) => rows.length
+ );
+ return ok(outcome);
+ })()
+ )
+ );
+ expect(recorded.roots).toHaveLength(1);
+ const children = recorded.roots[0].children;
+ expect(children.map((span) => span.name)).toEqual([
+ 'database_sql.engine.open',
+ 'database_sql.engine.start',
+ 'database_sql.fetch',
+ ]);
+ expect(children.every((span) => span.ended)).toBe(true);
+ expect(children[2].children).toEqual([
+ {
+ name: 'database_sql.rows.decode',
+ attributes: { 'database_sql.rows': 2 },
+ children: [],
+ ended: true,
+ },
+ ]);
+});
diff --git a/apps/web/src/lib/core/database-sql/trace.ts b/apps/web/src/lib/core/database-sql/trace.ts
index c3ef05fa14..a51f27ca85 100644
--- a/apps/web/src/lib/core/database-sql/trace.ts
+++ b/apps/web/src/lib/core/database-sql/trace.ts
@@ -14,6 +14,8 @@ import type {
RunError,
} from './generated/types';
+import { profileDatabasePhase } from './profile';
+
const PREFIX = '[database-sql]';
/** Long statements are cut to this many characters on the run span. */
const SQL_ATTRIBUTE_LENGTH = 4000;
@@ -46,6 +48,8 @@ export type DatabaseSqlReadContext = {
/** What a row source tells a step about the GraphQL request it sends. */
export interface DatabaseSqlStepTrace {
+ /** Convert a page under its fetch span, independent of ambient async context. */
+ decode<Value>(rows: number, operation: () => Value): Value;
request(document: string, variables: unknown): void;
/** W3C trace headers that put the request inside the step's span. */
headers(): Record<string, string>;
@@ -72,6 +76,9 @@ export type DatabaseSqlStepRecord = {
/** What the driver asks of a run's trace while it drives the engine. */
export interface DatabaseSqlRunTrace {
+ /** Compile/open and start under this run even after an async boundary. */
+ open<Value>(operation: () => Promise<Value>): Promise<Value>;
+ start<Value>(operation: () => Value): Value;
/** Read a step's rows inside its span; `count` says how many came back. */
fetch<Value, Failure>(
request: DatabaseSqlFetch,
@@ -214,6 +221,20 @@ export function traceDatabaseSqlRun<
const steps: DatabaseSqlStepRecord[] = [];
const pages = new Map<string, number>();
const trace: DatabaseSqlRunTrace = {
+ async open(operation) {
+ const span = runSpan.span('database_sql.engine.open');
+ try {
+ return await span.run(operation);
+ } finally {
+ span.end();
+ }
+ },
+ start(operation) {
+ return profileDatabasePhase(
+ runSpan.span('database_sql.engine.start'),
+ operation
+ );
+ },
fetch(request, read, count) {
const kind = fetchKind(request);
const table = fetchTable(request);
@@ -237,6 +258,11 @@ export function traceDatabaseSqlRun<
span.setAttr('database_sql.page', page);
span.setAttr('database_sql.hint_count', step.hints);
const stepTrace: DatabaseSqlStepTrace = {
+ decode(rows, operation) {
+ const decodeSpan = span.span('database_sql.rows.decode');
+ decodeSpan.setAttr('database_sql.rows', rows);
+ return profileDatabasePhase(decodeSpan, operation);
+ },
request: (document, variables) =>
step.requests.push({ document, variables }),
headers: () => {
diff --git a/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts b/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts
index b610fd916a..70689f258e 100644
--- a/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts
+++ b/apps/web/src/lib/queries/database-sql/create-database-sql-query.ts
@@ -19,6 +19,10 @@ import type {
Outcome,
Schema,
} from '@core/database-sql/generated/types';
+import {
+ profileDatabasePhase,
+ traceDatabaseFrame,
+} from '@core/database-sql/profile';
import type {
DatabaseSqlReadContext,
DatabaseSqlReadReason,
@@ -26,6 +30,7 @@ import type {
import { buildDatabaseSqlCatalog } from '@core/database-sql/wasm-module';
import { idToDisplayName, idToEmail } from '@core/user/util';
import type { CacheHost } from '@graphql-cache/host/types';
+import { Telemetry } from '@macro-inc/observability';
import { queryClient } from '@queries/client';
import { contactsQueryOptions } from '@queries/contacts/contacts';
import { subscribeToVisibleCacheChanges } from '@queries/subscribe-to-visible-cache-changes';
@@ -168,6 +173,7 @@ export function createDatabaseSqlQuery(
const [error, setError] = createSignal<DatabaseSqlFailure>();
const [loading, setLoading] = createSignal(false);
let latest = 0;
+ let cancelFrame: (() => void) | undefined;
// The cache may not hold what an in-flight network read will bring, so a
// cache change waits for it instead of answering from older rows.
let networkRead: ResultAsync<DatabaseSqlRun, DatabaseSqlFailure> | undefined;
@@ -183,40 +189,77 @@ export function createDatabaseSqlQuery(
options: { reportFailure?: boolean; keepLoading?: boolean } = {}
): ResultAsync<DatabaseSqlRun, DatabaseSqlFailure> => {
const generation = ++latest;
+ const querySpan = Telemetry.span('database_sql.query');
+ querySpan.setAttr('database_sql.read_reason', reason);
+ querySpan.setAttr('database_sql.request_policy', requestPolicy);
+ if (current.scope)
+ querySpan.setAttr('database_sql.scope_id', current.scope);
+ if (current.view)
+ querySpan.setAttr('database_sql.table_id', current.view.tableId);
+ const catalogSpan = querySpan.span('database_sql.catalog');
const host = capabilities.cacheHost();
setLoading(true);
- const answered = statementCatalog(current, capabilities)
+ const answered = catalogSpan
+ .run(() => statementCatalog(current, capabilities))
+ .andTee(() => catalogSpan.end())
+ .orTee(() => catalogSpan.end())
.andThen((built) =>
- runStatement(
- built,
- current,
- createGraphqlRowSource({
- client: capabilities.client(),
- catalog: built,
- requestPolicy,
- cache: host,
- people: capabilities.people,
- membership: host ? { host, baselines, reconcile } : undefined,
- }),
- capabilities,
- {
- reason,
- scope: current.scope,
- requestPolicy,
- reportFailure: options.reportFailure,
- }
- ).map((answer): DatabaseSqlRun => {
- if (generation !== latest) return { landed: false };
- // A cache change that left the answer alone keeps the same outcome.
- batch(() => {
- if (JSON.stringify(untrack(catalog)) !== JSON.stringify(built))
- setCatalog(built);
- if (JSON.stringify(untrack(outcome)) !== JSON.stringify(answer))
- setOutcome(answer);
- setError(undefined);
- });
- return { landed: true };
- })
+ querySpan
+ .run(() =>
+ runStatement(
+ built,
+ current,
+ createGraphqlRowSource({
+ client: capabilities.client(),
+ catalog: built,
+ requestPolicy,
+ cache: host,
+ people: capabilities.people,
+ membership: host ? { host, baselines, reconcile } : undefined,
+ }),
+ capabilities,
+ {
+ reason,
+ scope: current.scope,
+ requestPolicy,
+ reportFailure: options.reportFailure,
+ }
+ )
+ )
+ .map((answer): DatabaseSqlRun => {
+ if (generation !== latest) return { landed: false };
+ // A cache change that left the answer alone keeps the same outcome.
+ querySpan.setAttr('database_sql.row_count', answer.rows.length);
+ querySpan.setAttr('database_sql.truncated', answer.truncated);
+ const changed = profileDatabasePhase(
+ querySpan.span('database_sql.compare'),
+ () => ({
+ catalog:
+ JSON.stringify(untrack(catalog)) !== JSON.stringify(built),
+ outcome:
+ JSON.stringify(untrack(outcome)) !== JSON.stringify(answer),
+ })
+ );
+ querySpan.setAttr(
+ 'database_sql.result_changed',
+ changed.catalog || changed.outcome
+ );
+ const publishSpan = querySpan.span('database_sql.publish');
+ profileDatabasePhase(publishSpan, () =>
+ batch(() => {
+ if (changed.catalog) setCatalog(built);
+ if (changed.outcome) setOutcome(answer);
+ setError(undefined);
+ })
+ );
+ if (changed.catalog || changed.outcome) {
+ cancelFrame?.();
+ cancelFrame = traceDatabaseFrame(
+ querySpan.span('database_sql.frame')
+ );
+ }
+ return { landed: true };
+ })
)
.orElse((failure) => {
if (generation !== latest) return okAsync({ landed: false });
@@ -224,9 +267,21 @@ export function createDatabaseSqlQuery(
return errAsync(failure);
});
const settle = async () => {
- const result = await answered;
- if (generation === latest && !options.keepLoading) setLoading(false);
- return result;
+ try {
+ const result = await answered;
+ if (generation === latest && !options.keepLoading) setLoading(false);
+ querySpan.setAttr(
+ 'database_sql.outcome',
+ result.isErr()
+ ? 'error'
+ : result.value.landed
+ ? 'published'
+ : 'superseded'
+ );
+ return result;
+ } finally {
+ querySpan.end();
+ }
};
return new ResultAsync(settle());
};
@@ -267,6 +322,7 @@ export function createDatabaseSqlQuery(
createEffect(
on(statement, (current, previous) => {
+ cancelFrame?.();
baselines = new Map();
latest += 1;
setError(undefined);
@@ -305,6 +361,7 @@ export function createDatabaseSqlQuery(
});
onCleanup(() => {
+ cancelFrame?.();
latest += 1;
});
diff --git a/apps/web/src/lib/queries/database-sql/graphql-source.test.ts b/apps/web/src/lib/queries/database-sql/graphql-source.test.ts
index f4fc676fb8..84c82fe109 100644
--- a/apps/web/src/lib/queries/database-sql/graphql-source.test.ts
+++ b/apps/web/src/lib/queries/database-sql/graphql-source.test.ts
@@ -365,7 +365,11 @@ describe('the GraphQL row source', () => {
[],
null,
500,
- { request: () => {}, headers: () => ({}) }
+ {
+ decode: (_rows, operation) => operation(),
+ request: () => {},
+ headers: () => ({}),
+ }
);
expect(page._unsafeUnwrap().rows).toEqual([
diff --git a/apps/web/src/lib/queries/database-sql/graphql-source.ts b/apps/web/src/lib/queries/database-sql/graphql-source.ts
index 5f92de45a1..d57f693333 100644
--- a/apps/web/src/lib/queries/database-sql/graphql-source.ts
+++ b/apps/web/src/lib/queries/database-sql/graphql-source.ts
@@ -376,7 +376,7 @@ function soupPage(
const evidence = input.initial ? JSON.stringify(input) : undefined;
const local: ResultAsync<Page | undefined, DatabaseSqlFetchFailure> =
evidence && membership?.reconcile
- ? reconciledPage(input, evidence, membership)
+ ? reconciledPage(input, evidence, membership, step)
: okAsync(undefined);
return local.andThen((page) =>
page
@@ -397,7 +397,10 @@ function soupPage(
complete: nextCursor === null,
});
}
- return tableRows(items).map((rows) => ({ rows, next: nextCursor }));
+ return tableRows(items, step).map((rows) => ({
+ rows,
+ next: nextCursor,
+ }));
})
);
}
@@ -416,7 +419,8 @@ function isNetworkRead(requestPolicy: RequestPolicy): boolean {
function reconciledPage(
input: SoupInput,
evidence: string,
- { host, baselines }: LocalMembership
+ { host, baselines }: LocalMembership,
+ step: DatabaseSqlStepTrace
): ResultAsync<Page | undefined, DatabaseSqlFetchFailure> {
const initial = input.initial;
const baseline = baselines.get(evidence);
@@ -448,7 +452,8 @@ function reconciledPage(
thrownFetchFailure
).andThen(({ records }) =>
tableRows(
- materializeReconciledSoup(result.keys, records, baseline.items)
+ materializeReconciledSoup(result.keys, records, baseline.items),
+ step
).map((rows) => ({ rows, next: null }))
);
}
@@ -456,13 +461,16 @@ function reconciledPage(
}
function tableRows(
- items: readonly DatabaseRowFieldsFragment[]
+ items: readonly DatabaseRowFieldsFragment[],
+ step: DatabaseSqlStepTrace
): Result<Row[], DatabaseSqlFetchFailure> {
- return Result.combine(
- items.map((item) =>
- item.__typename === 'GraphqlSoupDatabaseRow'
- ? ok({ id: item.id, position: item.position, cells: rowCells(item) })
- : err(fetchFailure(`a table query returned a ${item.__typename}`))
+ return step.decode(items.length, () =>
+ Result.combine(
+ items.map((item) =>
+ item.__typename === 'GraphqlSoupDatabaseRow'
+ ? ok({ id: item.id, position: item.position, cells: rowCells(item) })
+ : err(fetchFailure(`a table query returned a ${item.__typename}`))
+ )
)
);
}
diff --git a/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs b/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs
index cd2a983f22..021cd20290 100644
--- a/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs
+++ b/crates/soup/src/outbound/pg_soup_repo/expanded/dynamic.rs
@@ -48,6 +48,7 @@ use models_soup::{
use recursion::CollapsibleExt;
use sqlx::{PgPool, Postgres, QueryBuilder, Row, postgres::PgRow, prelude::FromRow};
use system_properties::{StatusOption, SystemPropertyKey};
+use tracing::Instrument;
use uuid::Uuid;
use crate::domain::models::{
@@ -1832,7 +1833,6 @@ impl SoupRow {
}
}
- #[tracing::instrument(err)]
fn into_projection_hydration(self) -> Result<SoupProjectionHydration, sqlx::Error> {
let document_server_facts = self.document_server_facts();
Ok(SoupProjectionHydration {
@@ -1841,7 +1841,6 @@ impl SoupRow {
})
}
- #[tracing::instrument(err)]
fn into_soup_item(self) -> Result<SoupItem<()>, sqlx::Error> {
Ok(match self {
SoupRow::Document(DocumentRow {
@@ -2025,6 +2024,15 @@ async fn expanded_dynamic_cursor_soup_hydrated(
for ids in document_ids.into_binds() {
query = query.bind(ids);
}
+ let mut connection = db
+ .acquire()
+ .instrument(tracing::info_span!(
+ "soup.pool.acquire",
+ pool_size = db.size(),
+ pool_idle = db.num_idle()
+ ))
+ .await?;
+ let query_span = tracing::info_span!("soup.sql.fetch_decode", rows = tracing::field::Empty);
let items = query
// Unnamed statement: the SQL text varies per filter shape and per
// interpolated literal (dates change daily), so a cached prepared
@@ -2034,8 +2042,11 @@ async fn expanded_dynamic_cursor_soup_hydrated(
// time costs ~1ms and keeps the plan stable.
.persistent(false)
.try_map(|row| SoupRow::from_row(&row)?.into_projection_hydration())
- .fetch_all(db)
+ .fetch_all(&mut *connection)
+ .instrument(query_span.clone())
.await?;
+ query_span.record("rows", items.len());
+ drop(query_span);
Ok(items)
}
@@ -2454,27 +2465,41 @@ pub async fn expanded_dynamic_cursor_soup_grouped(
query = query.bind(ids);
}
+ let mut connection = db
+ .acquire()
+ .instrument(tracing::info_span!(
+ "soup.pool.acquire",
+ pool_size = db.size(),
+ pool_idle = db.num_idle()
+ ))
+ .await?;
+ let query_span = tracing::info_span!("soup.sql.fetch_decode", rows = tracing::field::Empty);
let rows: Vec<GroupedSoupRow> = query
// Unnamed statement — same rationale as expanded_dynamic_cursor_soup:
// filter-shaped SQL text gets no reuse from the statement cache but
// does get generic-plan flips.
.persistent(false)
- .fetch_all(db)
+ .fetch_all(&mut *connection)
+ .instrument(query_span.clone())
.await?;
-
- let items = rows
- .into_iter()
- .map(|row| {
- Ok(ItemGroupingInfo {
- item: row.item.into_soup_item()?,
- key: row.group.group_key,
- total_group_count: usize::try_from(row.group.group_total_count)
- .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,
- index_in_group: usize::try_from(row.group.row_in_group)
- .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,
+ query_span.record("rows", rows.len());
+ drop(query_span);
+ drop(connection);
+
+ let items = tracing::info_span!("soup.rows.convert", rows = rows.len()).in_scope(|| {
+ rows.into_iter()
+ .map(|row| {
+ Ok(ItemGroupingInfo {
+ item: row.item.into_soup_item()?,
+ key: row.group.group_key,
+ total_group_count: usize::try_from(row.group.group_total_count)
+ .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,
+ index_in_group: usize::try_from(row.group.row_in_group)
+ .map_err(|error| sqlx::Error::Decode(Box::new(error)))?,
+ })
})
- })
- .collect::<Result<Vec<_>, sqlx::Error>>()?;
+ .collect::<Result<Vec<_>, sqlx::Error>>()
+ })?;
Ok(items.into_iter())
}
diff --git a/docker/docker-compose-databases.yml b/docker/docker-compose-databases.yml
index 4120625274..664d2b8130 100644
--- a/docker/docker-compose-databases.yml
+++ b/docker/docker-compose-databases.yml
@@ -9,7 +9,9 @@ services:
# The default 100 max_connections is exhausted when every service's pool
# fills under load (e.g. a large email backfill): workers then die on
# "pool timed out" / "too many clients". Local-only headroom.
- command: postgres -c max_connections=500
+ # The migration installs pg_stat_statements; preload it so local query
+ # profiling can separate database execution from application timings.
+ command: postgres -c max_connections=500 -c shared_preload_libraries=pg_stat_statements
healthcheck:
test: ["CMD-SHELL", "pg_isready -U user"]
interval: 2s
diff --git a/services/document_storage_service/src/api/graphql_soup.rs b/services/document_storage_service/src/api/graphql_soup.rs
index 4802990e33..87a97a2826 100644
--- a/services/document_storage_service/src/api/graphql_soup.rs
+++ b/services/document_storage_service/src/api/graphql_soup.rs
@@ -5,6 +5,7 @@ use async_graphql::{Data, http::GraphiQLSource};
use async_graphql_axum::{GraphQLRequest, GraphQLResponse};
use axum::{
Router,
+ body::HttpBody,
extract::{OriginalUri, State},
http::{StatusCode, request::Parts},
response::{Html, IntoResponse, Response},
@@ -56,7 +57,7 @@ async fn graphql_handler(
>,
request_parts: Parts,
request: GraphQLRequest,
-) -> GraphQLResponse {
+) -> Response {
let acting_user = auth
.authorization
.as_ref()
@@ -68,11 +69,20 @@ async fn graphql_handler(
acting_user.and_then(|user| user.user_context.organization_id.map(i64::from)),
acting_user.and_then(opaque_viewer_id),
);
- state
+ let result = state
.graphql_soup_schema
.execute(request.data(GraphqlRequestParts::new(request_parts)))
- .await
- .into()
+ .await;
+ graphql_response(result)
+}
+
+#[tracing::instrument(skip_all, name = "graphql.serialize", fields(response_bytes = tracing::field::Empty))]
+fn graphql_response(result: async_graphql::Response) -> Response {
+ let response = GraphQLResponse::from(result).into_response();
+ if let Some(bytes) = response.body().size_hint().exact() {
+ tracing::Span::current().record("response_bytes", bytes);
+ }
+ response
}
async fn subscription_handler(
@@ -257,3 +267,6 @@ fn insert_graphql_context_data(
state.activity_reader.clone(),
));
}
+
+#[cfg(test)]
+mod test;
diff --git a/services/document_storage_service/src/api/graphql_soup/test.rs b/services/document_storage_service/src/api/graphql_soup/test.rs
new file mode 100644
index 0000000000..6cabbeb588
--- /dev/null
+++ b/services/document_storage_service/src/api/graphql_soup/test.rs
@@ -0,0 +1,38 @@
+use super::graphql_response;
+use axum::{
+ body::{HttpBody, to_bytes},
+ http::{HeaderValue, StatusCode},
+};
+
+#[tokio::test]
+async fn serialization_preserves_data_headers_and_body_length() {
+ let mut result = async_graphql::Response::new(async_graphql::value!({ "rows": [1, 2] }));
+ result
+ .http_headers
+ .insert("x-profile-test", HeaderValue::from_static("preserved"));
+ let response = graphql_response(result);
+ assert_eq!(response.status(), StatusCode::OK);
+ assert_eq!(
+ response.headers()["content-type"],
+ "application/graphql-response+json"
+ );
+ assert_eq!(response.headers()["x-profile-test"], "preserved");
+ let bytes = response.body().size_hint().exact().unwrap();
+ let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
+ assert_eq!(bytes, body.len() as u64);
+ assert_eq!(body.as_ref(), br#"{"data":{"rows":[1,2]}}"#);
+}
+
+#[tokio::test]
+async fn serialization_preserves_graphql_errors() {
+ let result = async_graphql::Response::from_errors(vec![async_graphql::ServerError::new(
+ "refused", None,
+ )]);
+ let response = graphql_response(result);
+ assert_eq!(response.status(), StatusCode::OK);
+ let body = to_bytes(response.into_body(), usize::MAX).await.unwrap();
+ assert_eq!(
+ body.as_ref(),
+ br#"{"data":null,"errors":[{"message":"refused"}]}"#
+ );
+}