Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(graphql-cache): trace the mutation queue lifecycle (#8616)

teoa6d54ad0d1dc
diff --git a/.agents/skills/instrumentation/SKILL.md b/.agents/skills/instrumentation/SKILL.md
index e500229aee..f67963a322 100644
--- a/.agents/skills/instrumentation/SKILL.md
+++ b/.agents/skills/instrumentation/SKILL.md
@@ -86,6 +86,22 @@ with outcome fields (`edit.mode`, `edit.ops_total`, `edit.blocked`,
`edit.flush`. On the agent side, `crates/agent_session/src/domain/session/telemetry.rs`
emits `invoke_agent` and `execute_tool` and propagates into the ACP agent.
+**GraphQL mutation queue (web → DSS `graphql.execute` → resolver).**
+`apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts` opens a
+`graphql.mutation_queue.attempt` span per claimed attempt that actually sends
+(`mutation_queue.item_id`, operation name, attempt count, queue depth, oldest
+item age, auth/online/websocket state, `mutation_queue.outcome`,
+`mutation_queue.failure`, `graphql.error.code`). Ages come from the durable
+enqueue time (`enqueuedAtMs`), so they survive reloads; when it is missing or
+skewed, `mutation_queue.age_known` is false and the age is omitted rather than
+zero. The runner sends inside it, so
+`dssGraphqlFetch` injects `traceparent` and the DSS `graphql.execute` span and
+the instrumented resolvers (`crates/graphql_entity_mutation`,
+`graphql_properties`, `graphql_notification`) join the same trace. A loop that
+can repeat forever reports counters instead of lines: each item logs a failure
+class once, and `graphql.mutation_queue.summary` flushes counters every five
+minutes only when something non-routine happened.
+
## Verify
Run the flow on the local stack and find its trace (see the `live-debug`
diff --git a/Cargo.lock b/Cargo.lock
index df1a3d187e..b08f7a59f2 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -5523,9 +5523,9 @@ dependencies = [
[[package]]
name = "ctutils"
-version = "0.4.2"
+version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"
+checksum = "38d28fb6aa7dd680b3ffd1a3e64e115c177bdba63c0ae7f9cd48129ff05c4fa0"
dependencies = [
"cmov",
"subtle",
@@ -8956,6 +8956,8 @@ dependencies = [
"model-entity",
"models_permissions",
"tokio",
+ "tracing",
+ "tracing-subscriber",
"uuid",
"workspace-hack",
]
@@ -19652,9 +19654,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
[[package]]
name = "tokio"
-version = "1.52.3"
+version = "1.54.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe"
+checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3"
dependencies = [
"bytes",
"libc",
diff --git a/apps/web/AGENTS.md b/apps/web/AGENTS.md
index 9699087574..8c1a9baac7 100644
--- a/apps/web/AGENTS.md
+++ b/apps/web/AGENTS.md
@@ -184,6 +184,7 @@ ownership. Cancel, logout, and the ten-minute timeout retire pending callbacks.
- Diagnose browser call media with `call.media.*` spans and the `meeting.setup.id`
on `meeting.join`; keep provider messages, device labels/IDs, and call-link
tokens out of telemetry. See [call diagnostics](../../docs/agent-guide/calls.md#browser-call-diagnostics).
+- Diagnose the GraphQL mutation queue with `graphql.mutation_queue.*` spans and logs keyed by `mutation_queue.item_id`; route queue telemetry through `mutation-queue-telemetry.ts`, which dedupes per item and failure class and turns repeats into the periodic summary. Never add per-attempt `console.*` lines or record variables, draft content, or emails.
- Keep desktop Gmail and Outlook inbox consent in the system browser with a
`macro:///inbox-link-callback` return; webview origins are not desktop callbacks.
- Keep send reservations, leases, and cancellation in the email journal; the general GraphQL queue handles draft autosaves only.
diff --git a/apps/web/src/components/app/SessionExpiredRedirect.tsx b/apps/web/src/components/app/SessionExpiredRedirect.tsx
index de362c15f4..3a3117f5b8 100644
--- a/apps/web/src/components/app/SessionExpiredRedirect.tsx
+++ b/apps/web/src/components/app/SessionExpiredRedirect.tsx
@@ -33,7 +33,7 @@ export function SessionExpiredRedirect() {
);
if (!stillUnauthorized) return false;
}
- await clearLocalAuthSession().catch((error) => {
+ await clearLocalAuthSession('session-expired').catch((error) => {
console.error('Failed to clear local auth session', error);
});
return true;
diff --git a/apps/web/src/lib/core/auth/logout.test.ts b/apps/web/src/lib/core/auth/logout.test.ts
index 486efcd083..be3dea598b 100644
--- a/apps/web/src/lib/core/auth/logout.test.ts
+++ b/apps/web/src/lib/core/auth/logout.test.ts
@@ -48,6 +48,7 @@ vi.mock('@graphql-cache/lifecycle', () => ({
clearRegisteredCaches: vi.fn(async () => {}),
}));
vi.mock('@service-storage/graphql-soup', () => ({
+ recordGraphqlMutationQueueCleared: vi.fn(async () => {}),
setGraphqlSoupSessionActive: vi.fn(),
}));
vi.mock('@queries/auth/user-info', () => ({
diff --git a/apps/web/src/lib/core/auth/logout.ts b/apps/web/src/lib/core/auth/logout.ts
index 27eacb444d..033dd22768 100644
--- a/apps/web/src/lib/core/auth/logout.ts
+++ b/apps/web/src/lib/core/auth/logout.ts
@@ -26,7 +26,10 @@ import { resetGraphqlSoupDoneSession } from '@queries/soup/graphql/done-session'
import { clearDocumentQueryCache } from '@queries/storage/document-cache';
import { clearOfflineDocumentContexts } from '@queries/storage/documentLoad/offline-context-runtime';
import { authServiceClient } from '@service-auth/client';
-import { setGraphqlSoupSessionActive } from '@service-storage/graphql-soup';
+import {
+ recordGraphqlMutationQueueCleared,
+ setGraphqlSoupSessionActive,
+} from '@service-storage/graphql-soup';
import { raceTimeout } from '@solid-primitives/promise';
import { createCallback } from '@solid-primitives/rootless';
import { useNavigate } from '@solidjs/router';
@@ -52,7 +55,9 @@ const unauthenticatedUserInfo: UserInfoData = {
createdAt: undefined,
};
-export async function clearLocalAuthSession() {
+export async function clearLocalAuthSession(
+ cause: 'logout' | 'session-expired' = 'logout'
+) {
await clearNativeDiagnostics();
clearPipedreamCallbacks();
document.cookie =
@@ -64,6 +69,8 @@ export async function clearLocalAuthSession() {
clearTeamCalendarQueries();
// Start every wipe before waiting, including stores not hydrated this session.
const results = await Promise.allSettled([
+ // Reads the queue before the cache wipe below deletes it.
+ recordGraphqlMutationQueueCleared(cause),
Promise.resolve().then(clearComposerStorage),
documentContextsCleared,
clearRegisteredCaches(),
diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts
index 9d293d0345..55a28ccd1b 100644
--- a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts
@@ -1,5 +1,6 @@
import { SaveEmailDraftDocument } from '@service-storage/graphql/generated/graphql';
import {
+ CombinedError,
createClient,
createRequest,
makeOperation,
@@ -20,8 +21,14 @@ import {
import {
createMutationQueueRunner,
type MutationQueueRunner,
+ type MutationQueueRunnerDeps,
queueAttemptOf,
+ withQueueRequestTimeout,
} from './mutation-queue-runner';
+import {
+ type MutationAttemptTelemetry,
+ NOOP_MUTATION_QUEUE_TELEMETRY,
+} from './mutation-queue-telemetry';
beforeEach(() => vi.useFakeTimers());
afterEach(() => {
@@ -68,7 +75,7 @@ function enqueue(
: { kind: 'not-runnable' },
};
}
-function fixture() {
+function fixture(options: MutationQueueRunnerDeps['options'] = {}) {
const forwarded: Operation[] = [];
const commit = vi.fn<CacheHost['commitOptimisticWrite']>(
async () =>
@@ -83,6 +90,15 @@ function fixture() {
);
const host = {
commitOptimisticWrite: commit,
+ deferOptimisticWrite: vi.fn(async () => ({ kind: 'deferred' })),
+ rollbackOptimisticWrite: vi.fn(async () => ({
+ kind: 'rolled-back',
+ revision: INITIAL_CACHE_REVISION,
+ revisionAdvanced: true,
+ changed: [],
+ affectedOps: [],
+ reset: false,
+ })),
currentRevision: async () => INITIAL_CACHE_REVISION,
claimNextMutation: async () => undefined,
} as unknown as CacheHost;
@@ -103,7 +119,7 @@ function fixture() {
),
],
}),
- options: {},
+ options,
owner: 'owner',
forward: (op) => forwarded.push(op),
writeThrough: (result) =>
@@ -235,3 +251,111 @@ it('preserves a pump claim when its unclaimed enqueue acknowledgement arrives af
expect.anything()
);
});
+
+function recordingTelemetry() {
+ const attempts: Array<{ sent: string[]; finished: unknown[][] }> = [];
+ const run = vi.fn((send: () => unknown) => send());
+ const telemetry = {
+ ...NOOP_MUTATION_QUEUE_TELEMETRY,
+ attempt: () => {
+ const record = { sent: [] as string[], finished: [] as unknown[][] };
+ attempts.push(record);
+ return {
+ run: <T>(send: () => T) => run(send) as T,
+ sent: (mode) => {
+ record.sent.push(mode);
+ },
+ finish: (...args) => {
+ record.finished.push(args);
+ },
+ } satisfies MutationAttemptTelemetry;
+ },
+ };
+ return { telemetry, attempts, run };
+}
+
+const savedDraft = { saveEmailDraft: { draftId: 'draft', subject: 'Draft' } };
+
+it('ends a live attempt once, as committed', async () => {
+ const { telemetry, attempts } = recordingTelemetry();
+ const { runner, forwarded } = fixture({ mutationQueueTelemetry: telemetry });
+ await runner.admit(operation(), enqueue(operation(), '1'), runner.epoch());
+ await runner.settle(
+ { operation: forwarded[0], data: savedDraft, stale: false, hasNext: false },
+ queueAttemptOf(forwarded[0])!
+ );
+ expect(attempts).toEqual([{ sent: ['live'], finished: [['committed', {}]] }]);
+});
+
+it('sends a queued request inside its attempt span', async () => {
+ const { telemetry, run } = recordingTelemetry();
+ const { runner, forwarded } = fixture({ mutationQueueTelemetry: telemetry });
+ await runner.admit(operation(), enqueue(operation(), '1'), runner.epoch());
+ const transport = vi.fn(async () => new Response('{}', { status: 200 }));
+ const op = makeOperation(forwarded[0].kind, forwarded[0], {
+ ...forwarded[0].context,
+ fetch: transport,
+ });
+ await withQueueRequestTimeout(op).context.fetch!('http://test', {});
+ expect(run).toHaveBeenCalledOnce();
+ expect(transport).toHaveBeenCalledOnce();
+});
+
+it('reports a retried server failure with its backoff', async () => {
+ const { telemetry, attempts } = recordingTelemetry();
+ const { runner, forwarded } = fixture({
+ mutationQueueTelemetry: telemetry,
+ shouldRetryMutation: () => true,
+ });
+ await runner.admit(operation(), enqueue(operation(), '2'), runner.epoch());
+ const attempt = queueAttemptOf(forwarded[0])!;
+ attempt.response = new Response(null, { status: 503 });
+ await runner.settle(
+ { operation: forwarded[0], data: savedDraft, stale: false, hasNext: false },
+ attempt
+ );
+ expect(attempts[0]?.finished).toEqual([
+ [
+ 'retry-scheduled',
+ {
+ failure: 'server',
+ errorCode: undefined,
+ httpStatus: 503,
+ retryDelayMs: 2_000,
+ },
+ ],
+ ]);
+});
+
+it('reports retry exhaustion apart from a permanent rollback', async () => {
+ const { telemetry, attempts } = recordingTelemetry();
+ const { runner, forwarded } = fixture({
+ mutationQueueTelemetry: telemetry,
+ shouldRetryMutation: () => true,
+ });
+ const exhausted = enqueue(operation(), '10');
+ if (exhausted.initialClaim.kind !== 'claimed')
+ throw new Error('Missing claim');
+ exhausted.initialClaim.mutation.serverFailureCount = 9;
+ await runner.admit(operation(), exhausted, runner.epoch());
+ await runner.settle(
+ {
+ operation: forwarded[0],
+ data: undefined,
+ error: new CombinedError({
+ graphQLErrors: [
+ { message: 'boom', extensions: { code: 'INTERNAL' }, path: ['x'] },
+ ],
+ }),
+ stale: false,
+ hasNext: false,
+ },
+ queueAttemptOf(forwarded[0])!
+ );
+ expect(attempts[0]?.finished).toEqual([
+ [
+ 'retry-exhausted',
+ { failure: 'server', errorCode: 'INTERNAL', httpStatus: undefined },
+ ],
+ ]);
+});
diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts
index f2ed4c9261..363bfc38ad 100644
--- a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts
@@ -48,6 +48,15 @@ import type {
MutationClaim,
QueryRevalidationWire,
} from '../protocol';
+import {
+ failureDetails,
+ type MutationAttemptDetails,
+ type MutationAttemptOutcome,
+ type MutationAttemptTelemetry,
+ NOOP_MUTATION_ATTEMPT_TELEMETRY,
+ NOOP_MUTATION_QUEUE_TELEMETRY,
+ recoverySkipReason,
+} from './mutation-queue-telemetry';
import type { NormalizedCacheExchangeOptions } from './normalized-cache-exchange';
import {
notifyOptimisticMutationEnqueued,
@@ -104,6 +113,7 @@ export type QueueAttemptContext = AttemptToken & {
serverFailureCount: number;
/** Transport-only metadata; urql drops the response on error-free payloads. */
response?: Response;
+ telemetry?: MutationAttemptTelemetry;
};
function claimOf(token: AttemptToken): MutationClaim {
@@ -164,12 +174,14 @@ export function withQueueRequestTimeout(op: Operation): Operation {
...op.context,
fetch: async (input, init) => {
const timeoutSignal = AbortSignal.timeout(QUEUE_REQUEST_TIMEOUT_MS);
- const response = await operationFetch(input, {
- ...init,
- signal: init?.signal
- ? AbortSignal.any([init.signal, timeoutSignal])
- : timeoutSignal,
- });
+ const send = () =>
+ operationFetch(input, {
+ ...init,
+ signal: init?.signal
+ ? AbortSignal.any([init.signal, timeoutSignal])
+ : timeoutSignal,
+ });
+ const response = await (attempt?.telemetry?.run(send) ?? send());
if (attempt) attempt.response = response;
return response;
},
@@ -274,6 +286,7 @@ export type MutationQueueRunnerDeps = {
| 'onMutationAttemptResult'
| 'shouldRetryMutation'
| 'onCacheError'
+ | 'mutationQueueTelemetry'
>;
/** Lease owner for this exchange's claims. */
owner: string;
@@ -336,6 +349,8 @@ export function createMutationQueueRunner(
deps: MutationQueueRunnerDeps
): MutationQueueRunner {
const { host, client, options, owner, forward, writeThrough } = deps;
+ const telemetry =
+ options.mutationQueueTelemetry ?? NOOP_MUTATION_QUEUE_TELEMETRY;
const liveQueuedOps = new Map<string, LiveQueuedOperation>();
let storageGeneration = 0;
let activeAttempt: AttemptToken | undefined;
@@ -351,7 +366,6 @@ export function createMutationQueueRunner(
let head: HeadState = IDLE;
// Pump state: wakeups and the retry hint are independent of the head.
let queuePreparation: Promise<void> | undefined;
- const reportedRecoveryFailures = new Set<string>();
let drainRunning = false;
let drainRequested = false;
let deferredUntil: number | undefined;
@@ -434,6 +448,26 @@ export function createMutationQueueRunner(
if (!isCurrent(epoch) || !isNewClaim(claimed)) return;
latestClaim = claimed;
activeAttempt = attempt;
+ const attemptTelemetry = telemetry.attempt(claimed);
+ attempt.telemetry = attemptTelemetry;
+ // Settlement ends a sent attempt; anything that returns before sending
+ // without its own outcome was retired by a storage reset.
+ let sent = false;
+ try {
+ await routeClaimedAttempt(claimed, attempt, attemptTelemetry, () => {
+ sent = true;
+ });
+ } finally {
+ if (!sent) attemptTelemetry.finish('detached');
+ }
+ }
+
+ async function routeClaimedAttempt(
+ claimed: ClaimedMutation,
+ attempt: QueueAttemptContext,
+ attemptTelemetry: MutationAttemptTelemetry,
+ markSent: () => void
+ ): Promise<void> {
const confirmed = confirmedOf(head);
if (confirmed?.transactionId === claimed.transactionId) {
// claim → settling: re-apply the server's response before admission
@@ -459,6 +493,7 @@ export function createMutationQueueRunner(
: true;
if (admission === 'defer') {
await deferAdmission(claimed, attempt);
+ attemptTelemetry.finish('deferred');
return;
}
if (!admission) {
@@ -471,12 +506,13 @@ export function createMutationQueueRunner(
'LOCAL_SUPERSEDED'
);
if (!transition(attempt, IDLE)) return;
+ attemptTelemetry.finish('discarded', { errorCode: 'LOCAL_SUPERSEDED' });
resolveLiveOperationsAsQueued();
scheduleDrain();
return;
}
} catch {
- await failPreparation(claimed, attempt);
+ await failPreparation(claimed, attempt, attemptTelemetry);
return;
}
// A superseded create can already exist on the server. Replay it to
@@ -498,6 +534,7 @@ export function createMutationQueueRunner(
if (discarded.kind !== 'discarded-superseded') {
throw new Error('superseded mutation was unexpectedly deferred');
}
+ attemptTelemetry.finish('superseded');
const live = liveQueuedOps.get(claimed.transactionId);
if (live) {
liveQueuedOps.delete(claimed.transactionId);
@@ -513,11 +550,14 @@ export function createMutationQueueRunner(
if (live) {
liveQueuedOps.delete(claimed.transactionId);
live.resolveRoute(undefined);
+ markSent();
+ attemptTelemetry.sent('live');
forward(
withQueueRequestTimeout(withQueueAttempt(live.operation, attempt))
);
} else {
let replayOperation: Operation | undefined;
+ attemptTelemetry.sent('replay');
try {
const replay = client.mutation(
replayDocument(claimed.query, claimed.operationName),
@@ -548,6 +588,12 @@ export function createMutationQueueRunner(
: undefined
);
if (!isCurrent(attempt)) return;
+ attemptTelemetry.finish(
+ rolledBack.kind === 'rolled-back' ? 'rolled-back' : 'superseded',
+ error instanceof CombinedError
+ ? failureDetails(error, error.response ?? attempt.response)
+ : { failure: 'local' }
+ );
if (rolledBack.kind === 'rolled-back') {
fireAndForget(
deps.revalidate(rolledBack.revalidations ?? [], replayOperation),
@@ -589,7 +635,8 @@ export function createMutationQueueRunner(
/** preparing → settling → idle for a head whose recovery state failed. */
async function failPreparation(
claimed: ClaimedMutation,
- attempt: QueueAttemptContext
+ attempt: QueueAttemptContext,
+ attemptTelemetry: MutationAttemptTelemetry
): Promise<void> {
queuePreparation = undefined;
try {
@@ -631,6 +678,10 @@ export function createMutationQueueRunner(
'LOCAL_RECOVERY_FAILED'
);
if (!isCurrent(attempt)) return;
+ attemptTelemetry.finish('recovery-failed', {
+ failure: 'local',
+ errorCode: 'LOCAL_RECOVERY_FAILED',
+ });
liveQueuedOps.delete(claimed.transactionId);
live?.resolveRoute(
withOptimisticMutationDisposition(result, {
@@ -651,6 +702,7 @@ export function createMutationQueueRunner(
// A retired host stays disabled for good: stop polling instead of failing
// every 30 seconds against the no-op host.
if (host.disabled) {
+ telemetry.skipped('host-disabled');
resolveLiveOperationsAsQueued();
return;
}
@@ -698,6 +750,7 @@ export function createMutationQueueRunner(
await routeClaimedMutation(claimed, epoch);
} catch {
+ telemetry.skipped('claim-failed');
// Enqueue already succeeded, so callers must observe these as
// queued even if the runner cannot currently inspect the head.
resolveLiveOperationsAsQueued();
@@ -719,14 +772,7 @@ export function createMutationQueueRunner(
await queuePreparation;
} catch (error) {
queuePreparation = undefined;
- // A stuck tab retries every poll; one warning per cause is enough.
- const cause = error instanceof Error ? error.message : String(error);
- if (reportedRecoveryFailures.has(cause)) return;
- reportedRecoveryFailures.add(cause);
- console.warn(
- '[graphql-cache] Unable to prepare mutation recovery',
- error
- );
+ telemetry.skipped(recoverySkipReason(error));
}
}
@@ -835,7 +881,12 @@ export function createMutationQueueRunner(
: 'permanently-failed',
transactionId: attempt.transactionId,
});
- if (!transition(attempt, receiveResult)) return detachedResult();
+ const attemptTelemetry =
+ attempt.telemetry ?? NOOP_MUTATION_ATTEMPT_TELEMETRY;
+ if (!transition(attempt, receiveResult)) {
+ attemptTelemetry.finish('detached');
+ return detachedResult();
+ }
const claim = claimOf(attempt);
let retryAt: number | undefined;
let settled = false;
@@ -845,6 +896,8 @@ export function createMutationQueueRunner(
| 'queued'
| 'superseded'
| 'permanently-failed' = 'queued';
+ let outcome: MutationAttemptOutcome = 'detached';
+ let details: MutationAttemptDetails = {};
try {
const response = result.error?.response ?? attempt.response;
const httpServerFailure = (response?.status ?? 0) >= 500;
@@ -861,6 +914,7 @@ export function createMutationQueueRunner(
};
}
if (result.error || result.data == null) {
+ details = failureDetails(result.error, response);
let retry = false;
if (result.error && options.shouldRetryMutation) {
try {
@@ -874,11 +928,11 @@ export function createMutationQueueRunner(
// server responses, never connection failures or timeouts.
const serverFailure =
httpServerFailure || (result.error?.graphQLErrors.length ?? 0) > 0;
- if (
+ const exhausted =
retry &&
serverFailure &&
- attempt.serverFailureCount + 1 >= MAX_MUTATION_SERVER_FAILURES
- ) {
+ attempt.serverFailureCount + 1 >= MAX_MUTATION_SERVER_FAILURES;
+ if (exhausted) {
retry = false;
result = {
...result,
@@ -896,7 +950,8 @@ export function createMutationQueueRunner(
await recordAttemptResult(attempt, result, retry);
if (!isCurrent(attempt)) return detachedResult();
if (retry) {
- retryAt = Date.now() + retryDelayMs(attempt.attemptCount);
+ details.retryDelayMs = retryDelayMs(attempt.attemptCount);
+ retryAt = Date.now() + details.retryDelayMs;
const deferred = await host.deferOptimisticWrite(
attempt.transactionId,
claim,
@@ -910,8 +965,10 @@ export function createMutationQueueRunner(
retryAt = undefined;
replacementTransactionId = deferred.replacementTransactionId;
disposition = 'superseded';
+ outcome = 'superseded';
} else {
disposition = 'queued';
+ outcome = 'retry-scheduled';
}
} else {
const rolledBack = await host.rollbackOptimisticWrite(
@@ -925,8 +982,10 @@ export function createMutationQueueRunner(
if (rolledBack.kind === 'discarded-superseded') {
replacementTransactionId = rolledBack.replacementTransactionId;
disposition = 'superseded';
+ outcome = 'superseded';
} else {
disposition = 'permanently-failed';
+ outcome = exhausted ? 'retry-exhausted' : 'rolled-back';
fireAndForget(
deps.revalidate(rolledBack.revalidations ?? [], op),
'graphql-cache-mutation-queue'
@@ -971,6 +1030,8 @@ export function createMutationQueueRunner(
disposition = replacementTransactionId
? 'superseded'
: 'permanently-failed';
+ outcome = replacementTransactionId ? 'superseded' : 'rolled-back';
+ details = { failure: 'local', errorCode: 'LOCAL_COMMIT_FAILED' };
if (!replacementTransactionId) {
fireAndForget(
deps.revalidate(committed.revalidations ?? [], op),
@@ -980,9 +1041,11 @@ export function createMutationQueueRunner(
} else if (committed.kind === 'committed-superseded') {
replacementTransactionId = committed.replacementTransactionId;
disposition = 'superseded';
+ outcome = 'superseded';
await deps.deleteReportedRecords(result);
} else {
disposition = 'committed';
+ outcome = 'committed';
await deps.deleteReportedRecords(result);
if (!isCurrent(attempt)) return detachedResult();
const replay = deps.replayExplicitEffects(op, result.data, () =>
@@ -1004,6 +1067,8 @@ export function createMutationQueueRunner(
// A lost acknowledgement may hide a completed transaction.
// Preserve optimism and let a fresh durable claim decide.
disposition = 'queued';
+ outcome = 'ack-lost';
+ details = { ...details, retryDelayMs: QUEUE_LEASE_MS };
}
} finally {
if (transition(attempt, release)) {
@@ -1013,6 +1078,7 @@ export function createMutationQueueRunner(
retryAt === undefined ? 0 : Math.max(0, retryAt - Date.now())
);
}
+ attemptTelemetry.finish(outcome, details);
}
return withOptimisticMutationDisposition(
result,
@@ -1036,6 +1102,7 @@ export function createMutationQueueRunner(
latestClaim = undefined;
head = IDLE;
deferredUntil = undefined;
+ telemetry.storageReset();
resolveLiveOperationsAsQueued();
}
@@ -1044,7 +1111,10 @@ export function createMutationQueueRunner(
isCurrent,
admit,
settle,
- start: () => scheduleDrain(),
+ start: () => {
+ telemetry.loaded(async () => host.inspectMutations?.());
+ scheduleDrain();
+ },
wake: wakeDrain,
resetStorage,
};
diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.test.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.test.ts
new file mode 100644
index 0000000000..643e652dcd
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.test.ts
@@ -0,0 +1,590 @@
+import { CombinedError } from '@urql/core';
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import type { ClaimedMutation } from '../protocol';
+import {
+ classifyMutationFailure,
+ createMutationQueueTelemetry,
+ errorCodeAttribute,
+ MUTATION_QUEUE_SUMMARY_INTERVAL_MS,
+ type MutationQueueEnvironment,
+ queueItemAge,
+ recoverySkipReason,
+ reportMutationQueueCleared,
+} from './mutation-queue-telemetry';
+
+const telemetry = vi.hoisted(() => {
+ type Recorded = {
+ name: string;
+ attributes: Record<string, unknown>;
+ events: string[];
+ errors: unknown[];
+ ended: boolean;
+ };
+ const spans: Recorded[] = [];
+ const makeSpan = (name: string) => {
+ const recorded: Recorded = {
+ name,
+ attributes: {},
+ events: [],
+ errors: [],
+ ended: false,
+ };
+ spans.push(recorded);
+ return {
+ recorded,
+ setAttr: (key: string, value: unknown) => {
+ recorded.attributes[key] = value;
+ },
+ event: (event: string) => recorded.events.push(event),
+ error: (error: unknown) => recorded.errors.push(error),
+ run: <T>(work: () => T) => work(),
+ end: () => {
+ recorded.ended = true;
+ },
+ };
+ };
+ return {
+ spans,
+ warn: vi.fn(),
+ info: vi.fn(),
+ span: vi.fn(
+ (name: string, work?: (span: ReturnType<typeof makeSpan>) => unknown) => {
+ const span = makeSpan(name);
+ if (!work) return span;
+ return Promise.resolve(work(span)).finally(span.end);
+ }
+ ),
+ };
+});
+
+vi.mock('@macro-inc/observability', () => ({
+ Telemetry: {
+ span: telemetry.span,
+ warn: telemetry.warn,
+ info: telemetry.info,
+ },
+}));
+
+const environment: MutationQueueEnvironment = {
+ authenticated: true,
+ online: true,
+ websocket: 'connected',
+};
+
+function claimed(overrides: Partial<ClaimedMutation> = {}): ClaimedMutation {
+ return {
+ transactionId: '7',
+ uuid: 'item-1',
+ superseded: false,
+ requiresConfirmation: false,
+ leaseGeneration: '1',
+ query: 'mutation SaveEmailDraft { saveEmailDraft }',
+ operationName: 'SaveEmailDraft',
+ variables: { input: { body: 'private draft body', to: '[email protected]' } },
+ attemptCount: 1,
+ serverFailureCount: 0,
+ ...overrides,
+ };
+}
+
+function graphQLError(code: unknown, path: string[] = ['field']) {
+ return new CombinedError({
+ graphQLErrors: [{ message: 'failed', extensions: { code }, path }],
+ });
+}
+
+let now = 0;
+function create() {
+ return createMutationQueueTelemetry({
+ environment: () => environment,
+ now: () => now,
+ });
+}
+
+beforeEach(() => {
+ vi.useFakeTimers();
+ now = 1_000;
+ telemetry.spans.length = 0;
+ telemetry.warn.mockClear();
+ telemetry.info.mockClear();
+});
+afterEach(() => {
+ vi.clearAllTimers();
+ vi.useRealTimers();
+});
+
+describe('classifyMutationFailure', () => {
+ it.each([
+ ['HTTP 401', undefined, 401, 'auth'],
+ ['HTTP 409', undefined, 409, 'conflict'],
+ ['HTTP 502', undefined, 502, 'server'],
+ ['HTTP 422', undefined, 422, 'validation'],
+ ['local code', graphQLError('LOCAL_RECOVERY_FAILED'), undefined, 'local'],
+ ['UNAUTHENTICATED', graphQLError('UNAUTHENTICATED'), undefined, 'auth'],
+ ['FORBIDDEN', graphQLError('FORBIDDEN'), undefined, 'auth'],
+ ['CONFLICT', graphQLError('CONFLICT'), undefined, 'conflict'],
+ ['DRAFT_ALREADY_SENT', graphQLError('DRAFT_ALREADY_SENT'), 200, 'conflict'],
+ ['INTERNAL', graphQLError('INTERNAL_SERVER_ERROR'), 200, 'server'],
+ ['INVALID_INPUT', graphQLError('INVALID_INPUT'), 200, 'validation'],
+ ['BAD_USER_INPUT', graphQLError('BAD_USER_INPUT'), 200, 'validation'],
+ ['parse error', graphQLError(undefined, []), 200, 'validation'],
+ ['NOT_FOUND', graphQLError('NOT_FOUND'), 200, 'graphql'],
+ [
+ 'timeout',
+ new CombinedError({
+ networkError: Object.assign(new Error('signal timed out'), {
+ name: 'TimeoutError',
+ }),
+ }),
+ undefined,
+ 'timeout',
+ ],
+ [
+ 'offline',
+ new CombinedError({ networkError: new TypeError('Failed to fetch') }),
+ undefined,
+ 'network',
+ ],
+ ['empty payload', undefined, 200, 'no-data'],
+ ] as const)('%s', (_label, error, status, expected) => {
+ expect(classifyMutationFailure(error, status)).toBe(expected);
+ });
+
+ it('treats the server auth message as auth without a code', () => {
+ const error = new CombinedError({
+ graphQLErrors: [{ message: 'authentication required', path: ['x'] }],
+ });
+ expect(classifyMutationFailure(error, 200)).toBe('auth');
+ });
+});
+
+describe('errorCodeAttribute', () => {
+ it('keeps identifier codes and replaces anything that could carry text', () => {
+ expect(errorCodeAttribute('MUTATION_RETRY_EXHAUSTED')).toBe(
+ 'MUTATION_RETRY_EXHAUSTED'
+ );
+ expect(errorCodeAttribute('draft for [email protected]')).toBe('UNRECOGNIZED');
+ expect(errorCodeAttribute(42)).toBe('UNRECOGNIZED');
+ expect(errorCodeAttribute(undefined)).toBeUndefined();
+ });
+});
+
+describe('recoverySkipReason', () => {
+ it.each([
+ ['Sign in to save drafts on this device.', 'recovery-signed-out'],
+ [
+ 'This draft session is no longer active. Reload Macro.',
+ 'recovery-session-inactive',
+ ],
+ ['Queue inspection is unavailable', 'recovery-inspection-unavailable'],
+ ['Mutation recovery timed out', 'recovery-timeout'],
+ ['IndexedDB unavailable', 'recovery-failed'],
+ ] as const)('%s', (message, reason) => {
+ expect(recoverySkipReason(new Error(message))).toBe(reason);
+ });
+});
+
+describe('createMutationQueueTelemetry', () => {
+ it('records one attempt span with ids, counts, and outcome only', () => {
+ const queue = create();
+ queue
+ .enqueue({ uuid: 'item-1', operationName: 'SaveEmailDraft' })
+ .persisted({
+ transactionId: '7',
+ replaced: false,
+ claimed: true,
+ linkPatchesDegraded: false,
+ });
+ now = 1_200;
+ const attempt = queue.attempt(claimed({ enqueuedAtMs: 1_000 }));
+ attempt.sent('live');
+ now = 1_500;
+ attempt.finish('committed');
+ attempt.finish('rolled-back');
+
+ const span = telemetry.spans.find(
+ (span) => span.name === 'graphql.mutation_queue.attempt'
+ );
+ expect(span?.ended).toBe(true);
+ expect(span?.attributes).toMatchObject({
+ 'mutation_queue.item_id': 'item-1',
+ 'mutation_queue.transaction_id': '7',
+ 'graphql.operation.name': 'SaveEmailDraft',
+ 'mutation_queue.attempt_count': 1,
+ 'auth.state': 'signed-in',
+ 'network.online': true,
+ 'graphql.websocket.state': 'connected',
+ 'mutation_queue.pending_items': 1,
+ 'mutation_queue.recovered': false,
+ 'mutation_queue.send_mode': 'live',
+ 'mutation_queue.outcome': 'committed',
+ 'mutation_queue.attempt_ms': 300,
+ 'mutation_queue.end_to_end_ms': 500,
+ 'mutation_queue.item_age_ms': 500,
+ 'mutation_queue.age_known': true,
+ });
+ const exported = JSON.stringify(telemetry.spans);
+ expect(exported).not.toContain('private draft body');
+ expect(exported).not.toContain('[email protected]');
+ expect(telemetry.warn).not.toHaveBeenCalled();
+ });
+
+ it('replaces an operation name that is not an identifier', () => {
+ create()
+ .attempt(claimed({ operationName: 'mutation { secret }' }))
+ .finish('committed');
+ expect(telemetry.spans[0]?.attributes['graphql.operation.name']).toBe(
+ 'unknown'
+ );
+ });
+
+ it('logs the first retry per item and failure class, then only counts', () => {
+ const queue = create();
+ for (let attemptCount = 1; attemptCount <= 5; attemptCount++) {
+ queue
+ .attempt(claimed({ attemptCount }))
+ .finish('retry-scheduled', { failure: 'network', retryDelayMs: 1_000 });
+ }
+ expect(telemetry.warn).toHaveBeenCalledOnce();
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.retry_scheduled',
+ expect.objectContaining({
+ 'mutation_queue.item_id': 'item-1',
+ 'mutation_queue.failure': 'network',
+ })
+ );
+
+ queue
+ .attempt(claimed({ attemptCount: 6 }))
+ .finish('retry-scheduled', { failure: 'server', httpStatus: 503 });
+ queue.attempt(claimed({ uuid: 'item-2' })).finish('retry-scheduled', {
+ failure: 'network',
+ });
+ expect(telemetry.warn).toHaveBeenCalledTimes(3);
+
+ queue.attempt(claimed({ attemptCount: 7 })).finish('committed');
+ expect(telemetry.info).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.committed_after_failure',
+ expect.objectContaining({ 'mutation_queue.item_id': 'item-1' })
+ );
+ });
+
+ it('summarizes repeated failures once per interval', async () => {
+ const queue = create();
+ for (let attemptCount = 1; attemptCount <= 3; attemptCount++) {
+ queue.attempt(claimed({ attemptCount })).finish('retry-scheduled', {
+ failure: 'auth',
+ httpStatus: 401,
+ });
+ }
+ telemetry.warn.mockClear();
+ now += MUTATION_QUEUE_SUMMARY_INTERVAL_MS;
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+
+ expect(telemetry.warn).toHaveBeenCalledOnce();
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.summary',
+ expect.objectContaining({
+ 'mutation_queue.count.attempt.retry-scheduled': 3,
+ 'mutation_queue.count.failure.auth': 3,
+ 'mutation_queue.pending_items': 1,
+ 'mutation_queue.max_attempt_count': 3,
+ })
+ );
+ });
+
+ it('stays quiet when an interval only had routine successes', async () => {
+ const queue = create();
+ queue.attempt(claimed()).finish('committed');
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ expect(telemetry.warn).not.toHaveBeenCalled();
+ });
+
+ it('logs a skip reason once and backs off summaries while it repeats', async () => {
+ const queue = create();
+ const summaries = () =>
+ telemetry.warn.mock.calls.filter(
+ ([message]) => message === 'graphql.mutation_queue.summary'
+ );
+ queue.skipped('recovery-session-inactive');
+ queue.skipped('recovery-session-inactive');
+ expect(telemetry.warn).toHaveBeenCalledOnce();
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.skipped',
+ expect.objectContaining({
+ 'mutation_queue.skip_reason': 'recovery-session-inactive',
+ })
+ );
+
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ expect(summaries()).toHaveLength(1);
+ expect(summaries()[0]?.[1]).toMatchObject({
+ 'mutation_queue.count.skip.recovery-session-inactive': 2,
+ 'mutation_queue.summary_interval_ms': MUTATION_QUEUE_SUMMARY_INTERVAL_MS,
+ });
+
+ queue.skipped('recovery-session-inactive');
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ expect(summaries()).toHaveLength(1);
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ expect(summaries()).toHaveLength(2);
+ expect(summaries()[1]?.[1]).toMatchObject({
+ 'mutation_queue.summary_interval_ms':
+ 2 * MUTATION_QUEUE_SUMMARY_INTERVAL_MS,
+ });
+ });
+
+ it('counts a deferred admission without opening a span', () => {
+ create().attempt(claimed()).finish('deferred');
+ expect(telemetry.spans).toHaveLength(0);
+ });
+
+ it('reports a pending item that stays queued past a minute', async () => {
+ const queue = create();
+ queue.attempt(claimed()).finish('deferred');
+ now += MUTATION_QUEUE_SUMMARY_INTERVAL_MS;
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ const [, summary] = telemetry.warn.mock.calls[0] ?? [];
+ expect(summary).toMatchObject({
+ 'mutation_queue.count.attempt.deferred': 1,
+ 'mutation_queue.oldest_observed_wait_ms':
+ MUTATION_QUEUE_SUMMARY_INTERVAL_MS,
+ 'mutation_queue.age_unknown_items': 1,
+ });
+ expect(summary).not.toHaveProperty('mutation_queue.oldest_item_age_ms');
+ });
+
+ it('measures age from the durable enqueue time across a reload', async () => {
+ const queue = create();
+ now = 600_000;
+ const attempt = queue.attempt(
+ claimed({ enqueuedAtMs: 100_000, attemptCount: 4 })
+ );
+ attempt.finish('deferred');
+ expect(telemetry.spans).toHaveLength(0);
+ queue
+ .attempt(claimed({ enqueuedAtMs: 100_000, attemptCount: 5 }))
+ .finish('committed');
+ expect(telemetry.spans[0]?.attributes).toMatchObject({
+ 'mutation_queue.recovered': true,
+ 'mutation_queue.observed_wait_ms': 0,
+ 'mutation_queue.item_age_ms': 500_000,
+ 'mutation_queue.end_to_end_ms': 500_000,
+ 'mutation_queue.age_known': true,
+ });
+ });
+
+ it('summarizes an old recovered item even without an observed wait', async () => {
+ const queue = create();
+ now = 600_000;
+ queue.attempt(claimed({ enqueuedAtMs: 100_000 })).finish('deferred');
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.summary',
+ expect.objectContaining({
+ 'mutation_queue.oldest_item_age_ms': 500_000,
+ 'mutation_queue.oldest_observed_wait_ms': 0,
+ 'mutation_queue.age_unknown_items': 0,
+ })
+ );
+ });
+
+ it('prefers the stored queue sample in a summary', async () => {
+ const queue = create();
+ now = 600_000;
+ queue.loaded(async () => [
+ { enqueuedAtMs: 200_000 },
+ { enqueuedAtMs: 590_000 },
+ {},
+ ]);
+ queue
+ .attempt(claimed({ enqueuedAtMs: 590_000 }))
+ .finish('retry-scheduled', { failure: 'network' });
+ telemetry.warn.mockClear();
+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.summary',
+ expect.objectContaining({
+ 'mutation_queue.depth': 3,
+ 'mutation_queue.pending_items': 1,
+ 'mutation_queue.oldest_item_age_ms': 400_000,
+ 'mutation_queue.age_unknown_items': 1,
+ })
+ );
+ });
+
+ it.each([
+ ['missing', undefined, 'missing'],
+ ['zero', 0, 'invalid'],
+ ['fractional', 1.5, 'invalid'],
+ ['in the future', 2_000, 'clock-skew'],
+ ['negative', -400 * 24 * 60 * 60_000, 'invalid'],
+ ] as const)(
+ 'reports a %s enqueue time as an unknown age',
+ (_label, enqueuedAtMs, reason) => {
+ create()
+ .attempt(claimed({ enqueuedAtMs }))
+ .finish('retry-exhausted', { failure: 'server', httpStatus: 503 });
+ const attributes = telemetry.spans[0]?.attributes;
+ expect(attributes).toMatchObject({
+ 'mutation_queue.age_known': false,
+ 'mutation_queue.age_unknown_reason': reason,
+ });
+ expect(attributes).not.toHaveProperty('mutation_queue.item_age_ms');
+ expect(attributes).not.toHaveProperty('mutation_queue.end_to_end_ms');
+ }
+ );
+
+ it('flags an age past a year as clock skew', () => {
+ now = 400 * 24 * 60 * 60_000;
+ create()
+ .attempt(claimed({ enqueuedAtMs: 1 }))
+ .finish('committed');
+ expect(telemetry.spans[0]?.attributes).toMatchObject({
+ 'mutation_queue.age_known': false,
+ 'mutation_queue.age_unknown_reason': 'clock-skew',
+ });
+ });
+
+ it('logs how long an item waited before the queue gave up', () => {
+ now = 90_000;
+ create()
+ .attempt(claimed({ enqueuedAtMs: 10_000, serverFailureCount: 5 }))
+ .finish('retry-exhausted', { failure: 'server', httpStatus: 503 });
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.retry_exhausted',
+ expect.objectContaining({
+ 'mutation_queue.item_age_ms': 80_000,
+ 'mutation_queue.end_to_end_ms': 80_000,
+ 'mutation_queue.age_known': true,
+ })
+ );
+ });
+
+ it('tracks a replacement as unknown until a claim reports its time', () => {
+ const queue = create();
+ queue
+ .enqueue({ uuid: 'item-1', operationName: 'SaveEmailDraft' })
+ .persisted({
+ transactionId: '8',
+ replaced: true,
+ claimed: false,
+ linkPatchesDegraded: false,
+ });
+ expect(
+ telemetry.spans.find(
+ (span) => span.name === 'graphql.mutation_queue.enqueue'
+ )?.attributes
+ ).toMatchObject({ 'mutation_queue.age_unknown_items': 1 });
+ });
+
+ it('logs an enqueue failure once per reason', () => {
+ const queue = create();
+ queue.enqueue({ operationName: 'SaveEmailDraft' }).failed('uncertain');
+ queue.enqueue({ operationName: 'SaveEmailDraft' }).failed('uncertain');
+ expect(telemetry.warn).toHaveBeenCalledOnce();
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.enqueue_failed',
+ expect.objectContaining({ 'mutation_queue.enqueue_failure': 'uncertain' })
+ );
+ expect(
+ telemetry.spans.every((span) => span.ended && span.errors.length === 1)
+ ).toBe(true);
+ });
+
+ it('samples the persisted queue on load', async () => {
+ now = 50_000;
+ create().loaded(async () => [
+ { enqueuedAtMs: 20_000 },
+ { enqueuedAtMs: 45_000 },
+ {},
+ { enqueuedAtMs: 60_000 },
+ ]);
+ await vi.advanceTimersByTimeAsync(0);
+ const loaded = {
+ 'mutation_queue.depth': 4,
+ 'mutation_queue.oldest_item_age_ms': 30_000,
+ 'mutation_queue.age_unknown_items': 2,
+ };
+ expect(telemetry.spans[0]).toMatchObject({
+ name: 'graphql.mutation_queue.loaded',
+ attributes: loaded,
+ ended: true,
+ });
+ expect(telemetry.info).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.loaded',
+ expect.objectContaining(loaded)
+ );
+ });
+
+ it('omits the oldest age on load when no age is known', async () => {
+ create().loaded(async () => [{}]);
+ await vi.advanceTimersByTimeAsync(0);
+ expect(telemetry.spans[0]?.attributes).toMatchObject({
+ 'mutation_queue.depth': 1,
+ 'mutation_queue.age_unknown_items': 1,
+ });
+ expect(telemetry.spans[0]?.attributes).not.toHaveProperty(
+ 'mutation_queue.oldest_item_age_ms'
+ );
+ });
+});
+
+describe('queueItemAge', () => {
+ it.each([
+ [undefined, { known: false, reason: 'missing' }],
+ [null, { known: false, reason: 'missing' }],
+ ['100', { known: false, reason: 'invalid' }],
+ [-5, { known: false, reason: 'invalid' }],
+ [Number.MAX_SAFE_INTEGER + 2, { known: false, reason: 'invalid' }],
+ [1_001, { known: false, reason: 'clock-skew' }],
+ [1_000, { known: true, ageMs: 0 }],
+ [400, { known: true, ageMs: 600 }],
+ ] as const)('%s', (enqueuedAtMs, expected) => {
+ expect(queueItemAge(enqueuedAtMs, 1_000)).toEqual(expected);
+ });
+});
+
+describe('reportMutationQueueCleared', () => {
+ it('lists each lost item with its wait, oldest first', () => {
+ reportMutationQueueCleared(
+ [
+ { operationName: 'SaveEmailDraft', enqueuedAtMs: 9_000 },
+ { operationName: 'mutation { secret }' },
+ { operationName: 'SetEntityProperty', enqueuedAtMs: 1_000 },
+ { operationName: 'SaveEmailDraft', enqueuedAtMs: 20_000 },
+ ],
+ 'logout',
+ 10_000
+ );
+ expect(telemetry.warn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.cleared',
+ {
+ 'mutation_queue.clear_cause': 'logout',
+ 'mutation_queue.depth': 4,
+ 'mutation_queue.operation_names':
+ 'SaveEmailDraft,SetEntityProperty,unknown',
+ 'mutation_queue.items':
+ 'SetEntityProperty:9000,SaveEmailDraft:1000,unknown:unknown,SaveEmailDraft:unknown',
+ 'mutation_queue.oldest_item_age_ms': 9_000,
+ 'mutation_queue.age_unknown_items': 2,
+ }
+ );
+ });
+
+ it('caps the item list', () => {
+ reportMutationQueueCleared(
+ Array.from({ length: 30 }, (_, index) => ({
+ operationName: 'SaveEmailDraft',
+ enqueuedAtMs: 1_000 + index,
+ })),
+ 'session-expired',
+ 10_000
+ );
+ const [, attributes] = telemetry.warn.mock.calls[0] ?? [];
+ expect(
+ String(attributes?.['mutation_queue.items']).split(',')
+ ).toHaveLength(20);
+ expect(attributes?.['mutation_queue.depth']).toBe(30);
+ });
+});
diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts
new file mode 100644
index 0000000000..71e7782d4e
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts
@@ -0,0 +1,726 @@
+/**
+ * Telemetry for the durable GraphQL mutation queue.
+ *
+ * Each claimed attempt is one `graphql.mutation_queue.attempt` span. Its fetch
+ * runs inside the span, so `graphql.transport` and the server's request and
+ * `graphql.execute` spans join the attempt's trace through `traceparent`.
+ *
+ * Logs are bounded. An item logs the first occurrence of each outcome and
+ * failure class, and the queue logs the first occurrence of each skip reason.
+ * Repeats only increment counters, which a summary reports at most once per
+ * interval while something other than routine success happened.
+ *
+ * Attributes are opaque ids, counts, durations, and fixed classes. Variables,
+ * responses, and error messages never leave this module.
+ *
+ * Item ages come from the durable enqueue time, so they survive reloads. When
+ * that time is missing or implausible, `mutation_queue.age_known` is false
+ * and the age is omitted rather than reported as zero.
+ */
+
+import { fireAndForget } from '@core/util/fireAndForget';
+import { type Span, Telemetry } from '@macro-inc/observability';
+import type { CombinedError } from '@urql/core';
+import type { ClaimedMutation } from '../protocol';
+
+export type MutationQueueEnvironment = {
+ authenticated: boolean;
+ online: boolean;
+ websocket: 'connecting' | 'connected' | 'disconnected' | 'none';
+};
+
+export type MutationFailureClass =
+ | 'network'
+ | 'timeout'
+ | 'auth'
+ | 'server'
+ | 'conflict'
+ | 'validation'
+ | 'graphql'
+ | 'no-data'
+ | 'local';
+
+export type MutationAttemptOutcome =
+ /** The server confirmed the mutation and the cache committed it. */
+ | 'committed'
+ /** A retryable failure; the head stays queued with a backoff. */
+ | 'retry-scheduled'
+ /** A permanent failure; the optimistic state was rolled back. */
+ | 'rolled-back'
+ /** A newer write for the same record replaced this one. */
+ | 'superseded'
+ /** Too many server failures; the queue gave up and rolled back. */
+ | 'retry-exhausted'
+ /** Settlement failed after sending; the lease backoff decides later. */
+ | 'ack-lost'
+ /** A storage reset retired the attempt before it settled. */
+ | 'detached'
+ /** Admission could not judge the intent yet; it is asked again shortly. */
+ | 'deferred'
+ /** Admission discarded obsolete local intent without sending it. */
+ | 'discarded'
+ /** Startup recovery for the head failed; the head was rolled back. */
+ | 'recovery-failed';
+
+export type MutationQueueSkipReason =
+ | 'host-disabled'
+ | 'claim-failed'
+ | 'recovery-signed-out'
+ | 'recovery-session-inactive'
+ | 'recovery-inspection-unavailable'
+ | 'recovery-timeout'
+ | 'recovery-failed';
+
+export type MutationEnqueueFailure =
+ /** The queue may hold the write; the caller cannot safely retry. */
+ | 'uncertain'
+ | 'storage-changed'
+ /** Another context holds the database; the mutation went straight out. */
+ | 'owner-lock'
+ /** The engine refused the write; the mutation went straight out. */
+ | 'rejected';
+
+export type MutationAttemptDetails = {
+ failure?: MutationFailureClass;
+ /** Sanitized before export; any non-identifier becomes `UNRECOGNIZED`. */
+ errorCode?: unknown;
+ httpStatus?: number;
+ retryDelayMs?: number;
+};
+
+export interface MutationAttemptTelemetry {
+ /** Runs the network send so its spans join this attempt's trace. */
+ run<T>(send: () => T): T;
+ sent(mode: 'live' | 'replay'): void;
+ /** Ends the attempt; later calls are ignored. */
+ finish(
+ outcome: MutationAttemptOutcome,
+ details?: MutationAttemptDetails
+ ): void;
+}
+
+export interface MutationEnqueueTelemetry {
+ persisted(result: {
+ transactionId: string;
+ /** Replaced an unsent write for the same record. */
+ replaced: boolean;
+ /** Claimed as the queue head in the same call. */
+ claimed: boolean;
+ /** Persisted without its list patches after the engine rejected them. */
+ linkPatchesDegraded: boolean;
+ }): void;
+ failed(reason: MutationEnqueueFailure): void;
+}
+
+export type QueuedItemTiming = { enqueuedAtMs?: number };
+
+export interface MutationQueueTelemetry {
+ /**
+ * Samples the persisted queue once when the runner starts; summaries
+ * sample it again through the same callback.
+ */
+ loaded(
+ inspect: () => Promise<ReadonlyArray<QueuedItemTiming> | undefined>
+ ): void;
+ enqueue(item: {
+ uuid?: string;
+ operationName?: string;
+ }): MutationEnqueueTelemetry;
+ attempt(mutation: ClaimedMutation): MutationAttemptTelemetry;
+ skipped(reason: MutationQueueSkipReason): void;
+ storageReset(): void;
+}
+
+export const NOOP_MUTATION_ATTEMPT_TELEMETRY: MutationAttemptTelemetry = {
+ run: (send) => send(),
+ sent: () => undefined,
+ finish: () => undefined,
+};
+
+const NOOP_ENQUEUE: MutationEnqueueTelemetry = {
+ persisted: () => undefined,
+ failed: () => undefined,
+};
+
+export const NOOP_MUTATION_QUEUE_TELEMETRY: MutationQueueTelemetry = {
+ loaded: () => undefined,
+ enqueue: () => NOOP_ENQUEUE,
+ attempt: () => NOOP_MUTATION_ATTEMPT_TELEMETRY,
+ skipped: () => undefined,
+ storageReset: () => undefined,
+};
+
+export const MUTATION_QUEUE_SUMMARY_INTERVAL_MS = 5 * 60_000;
+/** A tab that only keeps skipping backs its summaries off to this interval. */
+const MAX_SUMMARY_INTERVAL_MS = 60 * 60_000;
+/** A pending item older than this makes the summary worth reporting. */
+const STUCK_ITEM_AGE_MS = 60_000;
+/** Bounds per-item log keys for a tab that never settles its items. */
+const MAX_LOGGED_KEYS = 1_000;
+/** A longer wait means the clock was wrong at enqueue, not a real wait. */
+const MAX_PLAUSIBLE_AGE_MS = 365 * 24 * 60 * 60_000;
+const MAX_CLEARED_ITEMS = 20;
+
+export type QueueItemAge =
+ | { known: true; ageMs: number }
+ | { known: false; reason: 'missing' | 'invalid' | 'clock-skew' };
+
+/**
+ * Age of a durable enqueue time. Older hosts omit the time and a wall clock
+ * can move backwards, so an age is either trustworthy or unknown, never a
+ * stand-in zero.
+ */
+export function queueItemAge(
+ enqueuedAtMs: unknown,
+ nowMs: number
+): QueueItemAge {
+ if (enqueuedAtMs === undefined || enqueuedAtMs === null)
+ return { known: false, reason: 'missing' };
+ if (
+ typeof enqueuedAtMs !== 'number' ||
+ !Number.isSafeInteger(enqueuedAtMs) ||
+ enqueuedAtMs <= 0
+ )
+ return { known: false, reason: 'invalid' };
+ const ageMs = nowMs - enqueuedAtMs;
+ if (ageMs < 0 || ageMs > MAX_PLAUSIBLE_AGE_MS)
+ return { known: false, reason: 'clock-skew' };
+ return { known: true, ageMs };
+}
+
+function ageAttributes(
+ age: QueueItemAge,
+ key: string
+): Record<string, string | number | boolean> {
+ return age.known
+ ? { [key]: age.ageMs, 'mutation_queue.age_known': true }
+ : {
+ 'mutation_queue.age_known': false,
+ 'mutation_queue.age_unknown_reason': age.reason,
+ };
+}
+
+/** Oldest known age and the count of items whose age is unknown. */
+function oldestAge(
+ enqueueTimes: Iterable<unknown>,
+ nowMs: number
+): Record<string, number> {
+ let oldest: number | undefined;
+ let unknown = 0;
+ for (const enqueuedAtMs of enqueueTimes) {
+ const age = queueItemAge(enqueuedAtMs, nowMs);
+ if (age.known) oldest = Math.max(oldest ?? 0, age.ageMs);
+ else unknown += 1;
+ }
+ const attributes: Record<string, number> = {
+ 'mutation_queue.age_unknown_items': unknown,
+ };
+ if (oldest !== undefined)
+ attributes['mutation_queue.oldest_item_age_ms'] = oldest;
+ return attributes;
+}
+
+const TERMINAL_OUTCOMES: ReadonlySet<MutationAttemptOutcome> = new Set([
+ 'committed',
+ 'rolled-back',
+ 'superseded',
+ 'retry-exhausted',
+ 'discarded',
+ 'recovery-failed',
+]);
+
+/** Outcomes that log their first occurrence per item and failure class. */
+const LOGGED_OUTCOMES: ReadonlySet<MutationAttemptOutcome> = new Set([
+ 'retry-scheduled',
+ 'rolled-back',
+ 'retry-exhausted',
+ 'ack-lost',
+ 'recovery-failed',
+]);
+
+/** Counters a summary does not report on their own. */
+const ROUTINE_COUNTERS: ReadonlySet<string> = new Set([
+ 'enqueue.persisted',
+ 'attempt.started',
+ 'attempt.committed',
+ 'attempt.deferred',
+ 'attempt.discarded',
+ 'attempt.superseded',
+]);
+
+const OPERATION_NAME = /^[A-Za-z_][A-Za-z0-9_]{0,99}$/;
+const ERROR_CODE = /^[A-Za-z][A-Za-z0-9_]{0,63}$/;
+
+function operationNameAttribute(name: string | undefined): string {
+ return name && OPERATION_NAME.test(name) ? name : 'unknown';
+}
+
+/** Server codes are identifiers; anything else may carry input text. */
+export function errorCodeAttribute(code: unknown): string | undefined {
+ if (code === undefined || code === null) return undefined;
+ return typeof code === 'string' && ERROR_CODE.test(code)
+ ? code
+ : 'UNRECOGNIZED';
+}
+
+function graphQLErrorCode(error: CombinedError | undefined): unknown {
+ return error?.graphQLErrors[0]?.extensions.code;
+}
+
+function detailAttributes(
+ details: MutationAttemptDetails,
+ errorCode: string | undefined
+): Record<string, string | number> {
+ const attributes: Record<string, string | number> = {};
+ if (details.failure) attributes['mutation_queue.failure'] = details.failure;
+ if (errorCode) attributes['graphql.error.code'] = errorCode;
+ if (details.httpStatus)
+ attributes['http.response.status_code'] = details.httpStatus;
+ if (details.retryDelayMs !== undefined)
+ attributes['mutation_queue.retry_delay_ms'] = details.retryDelayMs;
+ return attributes;
+}
+
+/** Classifies a failed attempt without retaining its message. */
+export function classifyMutationFailure(
+ error: CombinedError | undefined,
+ httpStatus: number | undefined
+): MutationFailureClass {
+ if (httpStatus === 401 || httpStatus === 403) return 'auth';
+ if (httpStatus === 409) return 'conflict';
+ if (httpStatus !== undefined && httpStatus >= 500) return 'server';
+ const graphQLError = error?.graphQLErrors[0];
+ if (graphQLError) {
+ const code = graphQLErrorCode(error);
+ const upper = typeof code === 'string' ? code.toUpperCase() : '';
+ if (upper.startsWith('LOCAL_')) return 'local';
+ if (
+ ['UNAUTHENTICATED', 'UNAUTHORIZED', 'FORBIDDEN'].includes(upper) ||
+ graphQLError.message === 'authentication required'
+ )
+ return 'auth';
+ if (upper.includes('CONFLICT') || upper.endsWith('ALREADY_SENT'))
+ return 'conflict';
+ if (upper.startsWith('INTERNAL')) return 'server';
+ if (
+ upper === 'BAD_USER_INPUT' ||
+ upper.startsWith('INVALID') ||
+ upper.startsWith('GRAPHQL_') ||
+ // Parse and validation errors have neither a code nor a field path.
+ (!upper && !graphQLError.path?.length)
+ )
+ return 'validation';
+ return 'graphql';
+ }
+ if (httpStatus !== undefined && httpStatus >= 400) return 'validation';
+ const networkError = error?.networkError;
+ if (networkError) {
+ return networkError.name === 'TimeoutError' ||
+ networkError.name === 'AbortError' ||
+ /timed? ?out/i.test(networkError.message)
+ ? 'timeout'
+ : 'network';
+ }
+ return 'no-data';
+}
+
+/** Failure attributes for an attempt that did not return usable data. */
+export function failureDetails(
+ error: CombinedError | undefined,
+ response: Response | undefined
+): MutationAttemptDetails {
+ return {
+ failure: classifyMutationFailure(error, response?.status),
+ errorCode: error?.networkError ? undefined : graphQLErrorCode(error),
+ httpStatus: response?.status,
+ };
+}
+
+/** Maps a recovery failure to a fixed reason; its message is not exported. */
+export function recoverySkipReason(error: unknown): MutationQueueSkipReason {
+ const message = error instanceof Error ? error.message : String(error);
+ if (message.includes('Sign in to save drafts')) return 'recovery-signed-out';
+ if (message.includes('no longer active')) return 'recovery-session-inactive';
+ if (message.includes('Queue inspection is unavailable'))
+ return 'recovery-inspection-unavailable';
+ if (message.includes('timed out')) return 'recovery-timeout';
+ return 'recovery-failed';
+}
+
+export type MutationQueueClearCause = 'logout' | 'session-expired';
+
+/**
+ * Reports queued mutations a local sign-out is about to delete, with how long
+ * each waited: `mutation_queue.items` lists `operation:age_ms`, oldest first,
+ * with `unknown` for an item whose age is unknown.
+ */
+export function reportMutationQueueCleared(
+ queue: ReadonlyArray<{ operationName?: string } & QueuedItemTiming>,
+ cause: MutationQueueClearCause,
+ nowMs = Date.now()
+): void {
+ if (queue.length === 0) return;
+ const operationNames = [
+ ...new Set(queue.map((item) => operationNameAttribute(item.operationName))),
+ ].sort();
+ const items = queue
+ .map((item) => ({
+ operationName: operationNameAttribute(item.operationName),
+ age: queueItemAge(item.enqueuedAtMs, nowMs),
+ }))
+ .sort(
+ (left, right) =>
+ (right.age.known ? right.age.ageMs : -1) -
+ (left.age.known ? left.age.ageMs : -1)
+ )
+ .slice(0, MAX_CLEARED_ITEMS)
+ .map(
+ ({ operationName, age }) =>
+ `${operationName}:${age.known ? age.ageMs : 'unknown'}`
+ );
+ Telemetry.warn('graphql.mutation_queue.cleared', {
+ 'mutation_queue.clear_cause': cause,
+ 'mutation_queue.depth': queue.length,
+ 'mutation_queue.operation_names': operationNames
+ .slice(0, MAX_CLEARED_ITEMS)
+ .join(','),
+ 'mutation_queue.items': items.join(','),
+ ...oldestAge(
+ queue.map((item) => item.enqueuedAtMs),
+ nowMs
+ ),
+ });
+}
+
+type TrackedItem = {
+ firstSeenAt: number;
+ /** Claimed from storage written before this runner started. */
+ recovered: boolean;
+ /** Durable enqueue time from the host, or this tab's enqueue time. */
+ enqueuedAtMs?: number;
+};
+
+export function createMutationQueueTelemetry({
+ environment,
+ now = Date.now,
+ summaryIntervalMs = MUTATION_QUEUE_SUMMARY_INTERVAL_MS,
+}: {
+ environment: () => MutationQueueEnvironment;
+ now?: () => number;
+ summaryIntervalMs?: number;
+}): MutationQueueTelemetry {
+ const items = new Map<string, TrackedItem>();
+ const loggedKeys = new Set<string>();
+ const counters = new Map<string, number>();
+ let maxAttemptCount = 0;
+ let summaryScheduled = false;
+ let nextSummaryIntervalMs = summaryIntervalMs;
+ let inspectQueue:
+ | (() => Promise<ReadonlyArray<QueuedItemTiming> | undefined>)
+ | undefined;
+
+ function environmentAttributes(): Record<string, string | boolean> {
+ const current = environment();
+ return {
+ 'auth.state': current.authenticated ? 'signed-in' : 'signed-out',
+ 'network.online': current.online,
+ 'graphql.websocket.state': current.websocket,
+ };
+ }
+
+ /**
+ * Items this tab enqueued or claimed. `oldest_item_age_ms` comes from
+ * durable enqueue times; `oldest_observed_wait_ms` only covers this tab.
+ */
+ function queueAttributes(): Record<string, number> {
+ const at = now();
+ let firstSeen: number | undefined;
+ for (const item of items.values())
+ firstSeen = Math.min(firstSeen ?? item.firstSeenAt, item.firstSeenAt);
+ return {
+ 'mutation_queue.pending_items': items.size,
+ 'mutation_queue.oldest_observed_wait_ms':
+ firstSeen === undefined ? 0 : Math.max(0, at - firstSeen),
+ ...oldestAge(
+ [...items.values()].map((item) => item.enqueuedAtMs),
+ at
+ ),
+ };
+ }
+
+ function count(key: string): void {
+ counters.set(key, (counters.get(key) ?? 0) + 1);
+ if (summaryScheduled) return;
+ summaryScheduled = true;
+ setTimeout(() => {
+ summaryScheduled = false;
+ fireAndForget(reportSummary(), 'graphql-cache-mutation-queue');
+ }, nextSummaryIntervalMs);
+ }
+
+ /** Returns true the first time `key` is seen. */
+ function firstOccurrence(key: string): boolean {
+ if (loggedKeys.has(key)) return false;
+ if (loggedKeys.size >= MAX_LOGGED_KEYS) loggedKeys.clear();
+ loggedKeys.add(key);
+ return true;
+ }
+
+ function forgetItem(uuid: string): void {
+ items.delete(uuid);
+ for (const key of loggedKeys)
+ if (key.startsWith(`${uuid}|`)) loggedKeys.delete(key);
+ }
+
+ /** Depth and ages across every persisted item, from storage. */
+ async function sampleQueue(): Promise<Record<string, number> | undefined> {
+ try {
+ const queue = await inspectQueue?.();
+ if (!queue) return undefined;
+ return {
+ 'mutation_queue.depth': queue.length,
+ ...oldestAge(
+ queue.map((item) => item.enqueuedAtMs),
+ now()
+ ),
+ };
+ } catch {
+ return undefined;
+ }
+ }
+
+ async function reportSummary(): Promise<void> {
+ const queue = queueAttributes();
+ const stuck =
+ Math.max(
+ queue['mutation_queue.oldest_item_age_ms'] ?? 0,
+ queue['mutation_queue.oldest_observed_wait_ms'] ?? 0
+ ) >= STUCK_ITEM_AGE_MS;
+ const notable = [...counters.keys()].some(
+ (key) => !ROUTINE_COUNTERS.has(key)
+ );
+ const snapshot = new Map(counters);
+ counters.clear();
+ const intervalMs = nextSummaryIntervalMs;
+ // A signed-out or retired tab skips every poll for as long as it stays
+ // open; only a change in queue activity resets the interval.
+ nextSummaryIntervalMs = [...snapshot.keys()].every((key) =>
+ key.startsWith('skip.')
+ )
+ ? Math.min(intervalMs * 2, MAX_SUMMARY_INTERVAL_MS)
+ : summaryIntervalMs;
+ if (!notable && !stuck) return;
+ const stored = await sampleQueue();
+ // Storage covers items this tab never saw, so its ages replace the tab's.
+ const {
+ 'mutation_queue.oldest_item_age_ms': _trackedOldest,
+ 'mutation_queue.age_unknown_items': _trackedUnknown,
+ ...observed
+ } = queue;
+ const attributes: Record<string, string | number | boolean> = {
+ ...environmentAttributes(),
+ ...(stored ? { ...observed, ...stored } : queue),
+ 'mutation_queue.max_attempt_count': maxAttemptCount,
+ 'mutation_queue.summary_interval_ms': intervalMs,
+ };
+ for (const [key, value] of snapshot)
+ attributes[`mutation_queue.count.${key}`] = value;
+ Telemetry.warn('graphql.mutation_queue.summary', attributes);
+ }
+
+ function attempt(mutation: ClaimedMutation): MutationAttemptTelemetry {
+ const startedAt = now();
+ let item = items.get(mutation.uuid);
+ if (!item) {
+ item = { firstSeenAt: startedAt, recovered: true };
+ items.set(mutation.uuid, item);
+ }
+ if (mutation.enqueuedAtMs !== undefined)
+ item.enqueuedAtMs = mutation.enqueuedAtMs;
+ const tracked = item;
+ const waited = (at: number) => ({
+ 'mutation_queue.observed_wait_ms': at - tracked.firstSeenAt,
+ ...ageAttributes(
+ queueItemAge(tracked.enqueuedAtMs, at),
+ 'mutation_queue.item_age_ms'
+ ),
+ });
+ maxAttemptCount = Math.max(maxAttemptCount, mutation.attemptCount);
+ const operationName = operationNameAttribute(mutation.operationName);
+ const identity = {
+ 'mutation_queue.item_id': mutation.uuid,
+ 'mutation_queue.transaction_id': mutation.transactionId,
+ 'graphql.operation.name': operationName,
+ 'mutation_queue.attempt_count': mutation.attemptCount,
+ 'mutation_queue.server_failure_count': mutation.serverFailureCount,
+ };
+ const claimAttributes = {
+ ...identity,
+ ...environmentAttributes(),
+ ...queueAttributes(),
+ 'mutation_queue.superseded': mutation.superseded,
+ 'mutation_queue.recovered': tracked.recovered,
+ ...waited(startedAt),
+ };
+ // Opened on first use: a head whose admission keeps deferring is claimed
+ // every second and should only count, not emit a span per claim.
+ let openedSpan: Span | undefined;
+ const span = (): Span => {
+ if (openedSpan) return openedSpan;
+ openedSpan = Telemetry.span('graphql.mutation_queue.attempt');
+ for (const [key, value] of Object.entries(claimAttributes))
+ openedSpan.setAttr(key, value);
+ return openedSpan;
+ };
+ count('attempt.started');
+ let finished = false;
+ return {
+ run: (send) => span().run(send),
+ sent(mode) {
+ span().setAttr('mutation_queue.send_mode', mode);
+ span().setAttr('mutation_queue.prepare_ms', now() - startedAt);
+ span().event('sent');
+ },
+ finish(outcome, details = {}) {
+ if (finished) return;
+ finished = true;
+ if (outcome === 'deferred' && !openedSpan) {
+ count('attempt.deferred');
+ return;
+ }
+ const finishedAt = now();
+ const errorCode = errorCodeAttribute(details.errorCode);
+ const age = queueItemAge(tracked.enqueuedAtMs, finishedAt);
+ const attributes: Record<string, string | number | boolean> = {
+ ...identity,
+ 'mutation_queue.outcome': outcome,
+ 'mutation_queue.attempt_ms': finishedAt - startedAt,
+ ...waited(finishedAt),
+ ...detailAttributes(details, errorCode),
+ };
+ if (TERMINAL_OUTCOMES.has(outcome) && age.known)
+ attributes['mutation_queue.end_to_end_ms'] = age.ageMs;
+ const attemptSpan = span();
+ for (const [key, value] of Object.entries(attributes))
+ attemptSpan.setAttr(key, value);
+ if (details.failure)
+ attemptSpan.error(`mutation ${outcome}: ${details.failure}`);
+ count(`attempt.${outcome}`);
+ if (details.failure) count(`failure.${details.failure}`);
+
+ const hadLoggedFailure = loggedKeys.has(`${mutation.uuid}|failed`);
+ if (
+ LOGGED_OUTCOMES.has(outcome) &&
+ firstOccurrence(
+ `${mutation.uuid}|${outcome}|${details.failure ?? ''}|${errorCode ?? ''}`
+ )
+ ) {
+ loggedKeys.add(`${mutation.uuid}|failed`);
+ attemptSpan.run(() =>
+ Telemetry.warn(
+ `graphql.mutation_queue.${outcome.replaceAll('-', '_')}`,
+ {
+ ...attributes,
+ ...environmentAttributes(),
+ 'mutation_queue.recovered': tracked.recovered,
+ }
+ )
+ );
+ } else if (outcome === 'committed' && hadLoggedFailure) {
+ attemptSpan.run(() =>
+ Telemetry.info('graphql.mutation_queue.committed_after_failure', {
+ ...attributes,
+ ...environmentAttributes(),
+ })
+ );
+ }
+ attemptSpan.end();
+ if (TERMINAL_OUTCOMES.has(outcome)) forgetItem(mutation.uuid);
+ },
+ };
+ }
+
+ return {
+ loaded(inspect) {
+ inspectQueue = inspect;
+ fireAndForget(
+ Telemetry.span('graphql.mutation_queue.loaded', async (span) => {
+ const sampled = await sampleQueue();
+ span.setAttr('mutation_queue.depth_available', sampled !== undefined);
+ if (sampled === undefined) return;
+ for (const [key, value] of Object.entries(sampled))
+ span.setAttr(key, value);
+ if ((sampled['mutation_queue.depth'] ?? 0) > 0)
+ Telemetry.info('graphql.mutation_queue.loaded', {
+ ...environmentAttributes(),
+ ...sampled,
+ });
+ }),
+ 'graphql-cache-mutation-queue'
+ );
+ },
+ enqueue({ uuid, operationName }) {
+ const startedAt = now();
+ const span = Telemetry.span('graphql.mutation_queue.enqueue');
+ span.setAttr(
+ 'graphql.operation.name',
+ operationNameAttribute(operationName)
+ );
+ if (uuid) span.setAttr('mutation_queue.item_id', uuid);
+ return {
+ persisted({ transactionId, replaced, claimed, linkPatchesDegraded }) {
+ // A replacement inherits a durable time this tab learns at claim.
+ if (uuid && !items.has(uuid))
+ items.set(uuid, {
+ firstSeenAt: startedAt,
+ recovered: false,
+ enqueuedAtMs: replaced ? undefined : startedAt,
+ });
+ span.setAttr('mutation_queue.transaction_id', transactionId);
+ span.setAttr('mutation_queue.replaced_pending', replaced);
+ span.setAttr('mutation_queue.claimed_on_enqueue', claimed);
+ span.setAttr(
+ 'mutation_queue.link_patches_degraded',
+ linkPatchesDegraded
+ );
+ if (linkPatchesDegraded) count('enqueue.link-patches-degraded');
+ span.setAttr('mutation_queue.enqueue_ms', now() - startedAt);
+ for (const [key, value] of Object.entries(queueAttributes()))
+ span.setAttr(key, value);
+ count('enqueue.persisted');
+ span.end();
+ },
+ failed(reason) {
+ span.setAttr('mutation_queue.enqueue_failure', reason);
+ span.setAttr('mutation_queue.enqueue_ms', now() - startedAt);
+ span.error(`mutation enqueue failed: ${reason}`);
+ count(`enqueue.failed.${reason}`);
+ if (firstOccurrence(`enqueue|${reason}`))
+ span.run(() =>
+ Telemetry.warn('graphql.mutation_queue.enqueue_failed', {
+ ...environmentAttributes(),
+ 'graphql.operation.name': operationNameAttribute(operationName),
+ 'mutation_queue.enqueue_failure': reason,
+ })
+ );
+ span.end();
+ },
+ };
+ },
+ attempt,
+ skipped(reason) {
+ count(`skip.${reason}`);
+ if (firstOccurrence(`skip|${reason}`))
+ Telemetry.warn('graphql.mutation_queue.skipped', {
+ ...environmentAttributes(),
+ ...queueAttributes(),
+ 'mutation_queue.skip_reason': reason,
+ });
+ },
+ storageReset() {
+ count('storage-reset');
+ items.clear();
+ loggedKeys.clear();
+ },
+ };
+}
diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts
index 5132e0e40f..325f06225f 100644
--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts
@@ -55,6 +55,7 @@ import {
type WriteResult,
} from '../protocol';
import { entityFromArgument } from './entity-resolvers';
+import { NOOP_MUTATION_QUEUE_TELEMETRY } from './mutation-queue-telemetry';
import {
HYDRATE_ONLY_CONTEXT_KEY,
type NormalizedCacheExchangeOptions,
@@ -3107,7 +3108,7 @@ describe('normalizedCacheExchange', () => {
'replays unrelated mutations when recovery initialization %s',
async (failure) => {
vi.useFakeTimers();
- const warning = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const skipped = vi.fn();
try {
host.seedQueued({
uuid: 'unrelated',
@@ -3122,29 +3123,41 @@ describe('normalizedCacheExchange', () => {
throw new Error('IndexedDB unavailable');
await new Promise<void>(() => {});
},
+ mutationQueueTelemetry: {
+ ...NOOP_MUTATION_QUEUE_TELEMETRY,
+ skipped,
+ },
});
await vi.advanceTimersByTimeAsync(4_010);
expect(forwarded).toHaveLength(1);
expect(host.commits).toMatchObject([{ transactionId: '1' }]);
- expect(warning).toHaveBeenCalled();
+ expect(skipped).toHaveBeenCalledWith(
+ failure === 'rejects' ? 'recovery-failed' : 'recovery-timeout'
+ );
} finally {
- warning.mockRestore();
vi.useRealTimers();
}
}
);
- it('warns once per recovery failure while the queue keeps polling', async () => {
+ it('reports each recovery failure while the queue keeps polling', async () => {
vi.useFakeTimers();
const warning = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const skipped = vi.fn();
try {
const prepareMutationQueue = vi.fn(async () => {
throw new Error('Queue inspection is unavailable');
});
- harness(host, undefined, { prepareMutationQueue });
+ harness(host, undefined, {
+ prepareMutationQueue,
+ mutationQueueTelemetry: { ...NOOP_MUTATION_QUEUE_TELEMETRY, skipped },
+ });
await vi.advanceTimersByTimeAsync(90_010);
expect(prepareMutationQueue).toHaveBeenCalledTimes(4);
- expect(warning).toHaveBeenCalledOnce();
+ expect(skipped.mock.calls).toEqual(
+ Array(4).fill(['recovery-inspection-unavailable'])
+ );
+ expect(warning).not.toHaveBeenCalled();
} finally {
warning.mockRestore();
vi.useRealTimers();
diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts
index 51fc85772c..89b1b07749 100644
--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts
@@ -96,6 +96,10 @@ import {
replayDocument,
withQueueRequestTimeout,
} from './mutation-queue-runner';
+import {
+ type MutationQueueTelemetry,
+ NOOP_MUTATION_QUEUE_TELEMETRY,
+} from './mutation-queue-telemetry';
import type { QueryRevalidation } from './optimistic';
import {
normalizedEntityKey,
@@ -419,6 +423,8 @@ export interface NormalizedCacheExchangeOptions {
* a later background attempt. Defaults to `false`.
*/
shouldRetryMutation?: (error: CombinedError) => boolean | Promise<boolean>;
+ /** Spans, bounded logs, and counters for the durable mutation queue. */
+ mutationQueueTelemetry?: MutationQueueTelemetry;
}
export function normalizedCacheExchange(
@@ -847,6 +853,8 @@ export function normalizedCacheExchange(
};
const queueOwner = `exchange:${host.clientId}`;
+ const queueTelemetry =
+ options.mutationQueueTelemetry ?? NOOP_MUTATION_QUEUE_TELEMETRY;
const mutationQueue = createMutationQueueRunner({
host,
client,
@@ -1067,18 +1075,26 @@ export function normalizedCacheExchange(
nowMs: now,
leaseExpiresAtMs: now + QUEUE_LEASE_MS,
};
+ const enqueueTelemetry = queueTelemetry.enqueue({
+ uuid: args.uuid,
+ operationName: args.operationName,
+ });
let enqueue: EnqueueOptimisticMutationResult;
+ let linkPatchesDegraded = false;
try {
enqueue = await host.enqueueOptimisticMutation(args, claim);
} catch (error) {
- if (!mutationQueue.isCurrent(epoch))
+ if (!mutationQueue.isCurrent(epoch)) {
+ enqueueTelemetry.failed('storage-changed');
return uncertainEnqueueResult(
op,
new Error('cache storage changed during enqueue', {
cause: error,
})
);
+ }
if (isAdmittedEnqueueUncertainError(error)) {
+ enqueueTelemetry.failed('uncertain');
options.onCacheError?.(error, op);
// The old-scope queue may already contain the side effect. It is
// unsafe to forward or retry without a coordinator fence.
@@ -1087,6 +1103,7 @@ export function normalizedCacheExchange(
// Another context holds the database, so the enqueue never reached
// an engine. Send it as a disabled cache would.
if (isOwnerLockUnavailableError(error)) {
+ enqueueTelemetry.failed('owner-lock');
enqueueForward(op);
return undefined;
}
@@ -1094,10 +1111,12 @@ export function normalizedCacheExchange(
// not expose a partial relation move: retain entity optimism and
// the post-success revalidation descriptors instead.
if (args.linkPatches.length === 0) {
+ enqueueTelemetry.failed('rejected');
options.onCacheError?.(error, op);
enqueueForward(op);
return undefined;
}
+ linkPatchesDegraded = true;
// Deliberately loud: this fallback is durable but invisible until
// reconnect — the mutation's optimistic list membership silently
// becomes eventual. A systematic rejection (schema drift, a patch
@@ -1131,27 +1150,38 @@ export function normalizedCacheExchange(
claim
);
} catch (fallbackError) {
- if (!mutationQueue.isCurrent(epoch))
+ if (!mutationQueue.isCurrent(epoch)) {
+ enqueueTelemetry.failed('storage-changed');
return uncertainEnqueueResult(
op,
new Error('cache storage changed during enqueue', {
cause: fallbackError,
})
);
+ }
options.onCacheError?.(fallbackError, op);
if (isAdmittedEnqueueUncertainError(fallbackError)) {
+ enqueueTelemetry.failed('uncertain');
return uncertainEnqueueResult(op, fallbackError);
}
+ enqueueTelemetry.failed('rejected');
enqueueForward(op);
return undefined;
}
}
if (!mutationQueue.isCurrent(epoch)) {
+ enqueueTelemetry.failed('storage-changed');
return uncertainEnqueueResult(
op,
new Error('cache storage changed during enqueue')
);
}
+ enqueueTelemetry.persisted({
+ transactionId: enqueue.transactionId,
+ replaced: enqueue.upsertKind.kind === 'replaced-pending',
+ claimed: enqueue.initialClaim.kind === 'claimed',
+ linkPatchesDegraded,
+ });
return await mutationQueue.admit(op, enqueue, epoch);
}
diff --git a/apps/web/src/lib/graphql-cache/protocol.ts b/apps/web/src/lib/graphql-cache/protocol.ts
index 6a00ae03e0..4686ae96f8 100644
--- a/apps/web/src/lib/graphql-cache/protocol.ts
+++ b/apps/web/src/lib/graphql-cache/protocol.ts
@@ -656,6 +656,7 @@ export type MutationInspection = Pick<
| 'operationName'
| 'variables'
| 'clientMetadata'
+ | 'enqueuedAtMs'
> & { optimisticData: unknown };
/** Claimed strict queue head, ready to be forwarded through urql. */
@@ -675,6 +676,12 @@ export type ClaimedMutation = {
attemptCount: number;
/** Retryable server failures; transport failures do not consume this budget. */
serverFailureCount: number;
+ /**
+ * Durable first-enqueue wall-clock time in epoch ms, kept when a write
+ * replaces an unsent one for the same UUID. Native claims and hosts built
+ * before this field omit it, so it is unknown rather than zero.
+ */
+ enqueuedAtMs?: number;
};
/** Outcome of the strict-head claim attempted immediately after enqueue. */
diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts
index d5aa111ec9..383b43879e 100644
--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts
+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts
@@ -357,6 +357,8 @@ const mocks = vi.hoisted(() => {
platformFetch,
toastFailure,
telemetryError: vi.fn(),
+ telemetryWarn: vi.fn(),
+ traceparent: '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01',
normalizedCacheExchange: vi.fn(
(host: unknown, _options?: NormalizedCacheExchangeOptions) => ({
kind: 'cache',
@@ -440,10 +442,14 @@ vi.mock('@graphql-cache/exchange/normalized-cache-exchange', () => ({
vi.mock('@macro-inc/observability', () => ({
Telemetry: {
error: mocks.telemetryError,
+ warn: mocks.telemetryWarn,
span: (_name: string, work: (span: unknown) => Promise<unknown>) =>
work({
event: vi.fn(),
setAttr: vi.fn(),
+ injectTraceHeaders: (headers: Record<string, string>) => {
+ headers.traceparent = mocks.traceparent;
+ },
}),
},
}));
@@ -808,6 +814,72 @@ describe('GraphQL Soup browser cache session gate', () => {
expect(payload.query).toContain('cacheProjection');
expect(payload.query).toContain('@include');
expect(payload.variables).toEqual({ includeId: true });
+ expect(new Headers(transport.headers).get('traceparent')).toBe(
+ mocks.traceparent
+ );
+ });
+
+ it('reports the queued mutations a sign-out deletes', async () => {
+ const now = Date.now();
+ mocks.host.inspectMutations.mockResolvedValueOnce([
+ {
+ operationName: 'SaveEmailDraft',
+ variables: { body: 'secret' },
+ enqueuedAtMs: now - 5_000,
+ },
+ { operationName: 'SaveEmailDraft', variables: { body: 'secret' } },
+ {
+ operationName: 'SetEntityProperty',
+ variables: {},
+ enqueuedAtMs: now - 90_000,
+ },
+ ] as never);
+ const soup = await import('./graphql-soup');
+ soup.getGraphqlSoupClient();
+
+ await soup.recordGraphqlMutationQueueCleared('session-expired');
+
+ expect(mocks.telemetryWarn).toHaveBeenCalledWith(
+ 'graphql.mutation_queue.cleared',
+ {
+ 'mutation_queue.clear_cause': 'session-expired',
+ 'mutation_queue.depth': 3,
+ 'mutation_queue.operation_names': 'SaveEmailDraft,SetEntityProperty',
+ 'mutation_queue.items': expect.stringMatching(
+ /^SetEntityProperty:9\d{4},SaveEmailDraft:\d+,SaveEmailDraft:unknown$/
+ ),
+ 'mutation_queue.oldest_item_age_ms': expect.any(Number),
+ 'mutation_queue.age_unknown_items': 1,
+ }
+ );
+ const exported = JSON.stringify(mocks.telemetryWarn.mock.calls);
+ expect(exported).not.toContain('secret');
+ });
+
+ it('never rejects or stalls sign-out when queue inspection fails', async () => {
+ const soup = await import('./graphql-soup');
+ soup.getGraphqlSoupClient();
+
+ mocks.host.inspectMutations.mockImplementationOnce(() => {
+ throw new Error('worker gone');
+ });
+ await expect(
+ soup.recordGraphqlMutationQueueCleared('logout')
+ ).resolves.toBeUndefined();
+
+ vi.useFakeTimers();
+ try {
+ mocks.host.inspectMutations.mockReturnValueOnce(new Promise(() => {}));
+ const pending = soup.recordGraphqlMutationQueueCleared('logout');
+ await vi.advanceTimersByTimeAsync(1000);
+ await expect(pending).resolves.toBeUndefined();
+ } finally {
+ vi.useRealTimers();
+ }
+ expect(mocks.telemetryWarn).not.toHaveBeenCalledWith(
+ 'graphql.mutation_queue.cleared',
+ expect.anything()
+ );
});
it.each([
diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
index 2307c3d5a6..50c3a0fdfa 100644
--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
@@ -11,6 +11,12 @@ import { fireAndForget } from '@core/util/fireAndForget';
import { isTauri } from '@core/util/platform';
import { platformFetch } from '@core/util/platformFetch';
import { reloadForNewerBuild } from '@core/util/reloadForNewerBuild';
+import {
+ createMutationQueueTelemetry,
+ type MutationQueueClearCause,
+ type MutationQueueEnvironment,
+ reportMutationQueueCleared,
+} from '@graphql-cache/exchange/mutation-queue-telemetry';
import { networkRevalidationExchange } from '@graphql-cache/exchange/network-revalidation-exchange';
import {
HYDRATE_ONLY_CONTEXT_KEY,
@@ -48,6 +54,7 @@ import type { GithubPrMentionLocation } from '@service-notification/generated/sc
import type { GithubPrReviewState } from '@service-notification/generated/schemas/githubPrReviewState';
import type { NotifEvent } from '@service-notification/generated/schemas/notifEvent';
import type { NotificationDocumentSubType } from '@service-notification/generated/schemas/notificationDocumentSubType';
+import { raceTimeout } from '@solid-primitives/promise';
import {
type AnyVariables,
type Client,
@@ -106,6 +113,7 @@ import {
} from './graphql-soup-websocket';
const dssHost = SERVER_HOSTS['document-storage-service'];
+const QUEUE_CLEAR_INSPECTION_TIMEOUT_MS = 1000;
function mergeHeaders(...headers: Array<HeadersInit | undefined>): Headers {
const result = new Headers();
@@ -313,7 +321,15 @@ export async function dssGraphqlFetch(
return Telemetry.span('graphql.transport', async (span) => {
const started = performance.now();
try {
- const transportInit = graphqlSoupTransportRequest(init);
+ const traceHeaders: Record<string, string> = {};
+ span.injectTraceHeaders(traceHeaders);
+ const requestInit = graphqlSoupTransportRequest(init);
+ // Links this request, and a queued mutation attempt above it, to the
+ // server's request and `graphql.execute` spans.
+ const transportInit: RequestInit = {
+ ...requestInit,
+ headers: mergeHeaders(requestInit?.headers, traceHeaders),
+ };
span.event('request_dispatch');
const response = await authorizedDssGraphqlFetch(input, transportInit);
span.setAttr('graphql.response_headers_ms', performance.now() - started);
@@ -381,7 +397,9 @@ export function subscribeGraphqlSoupReconnected(
}
function createGraphqlSoupWebSocketClient(
- onConnected: () => void
+ onConnected: () => void,
+ onState: (state: MutationQueueEnvironment['websocket']) => void = () =>
+ undefined
): GraphqlWsClient {
let connections = 0;
const resolveWebSocketUrl = createGraphqlSoupWebSocketUrlResolver({
@@ -401,7 +419,10 @@ function createGraphqlSoupWebSocketClient(
retryAttempts: SOUP_GRAPHQL_WEBSOCKET_RETRY_ATTEMPTS,
retryWait: waitToRetryGraphqlSoupWebSocket,
on: {
+ connecting: () => onState('connecting'),
+ closed: () => onState('disconnected'),
connected: () => {
+ onState('connected');
onConnected();
connections += 1;
if (connections === 1) return;
@@ -597,6 +618,26 @@ function fallbackAfterInitializationFailure(error: unknown): void {
setCacheAvailability((version) => version + 1);
}
+/**
+ * Reports the queued mutations a local sign-out deletes. Start it before the
+ * cache wipe; it never delays or fails the wipe.
+ */
+export async function recordGraphqlMutationQueueCleared(
+ cause: MutationQueueClearCause
+): Promise<void> {
+ const host = cachedCacheHost;
+ if (!host?.inspectMutations || host.disabled) return;
+ try {
+ const queue = await raceTimeout(
+ host.inspectMutations(),
+ QUEUE_CLEAR_INSPECTION_TIMEOUT_MS
+ );
+ if (queue) reportMutationQueueCleared(queue, cause);
+ } catch {
+ // A rejected entry makes sign-out rotate the cache scope.
+ }
+}
+
/** Returns the persistent normalized-cache host after client initialization. */
export function getGraphqlCacheHost(): CacheHost | undefined {
cacheAvailability();
@@ -720,10 +761,21 @@ export function getGraphqlSoupClient(): Client {
onSuperseded: () => reloadForNewerBuild(),
})
);
+ let websocketState: MutationQueueEnvironment['websocket'] = 'none';
const graphqlWsClient = createGraphqlSoupWebSocketClient(
- subscriptionsLifecycle.connected
+ subscriptionsLifecycle.connected,
+ (state) => {
+ websocketState = state;
+ }
);
websocketClient = graphqlWsClient;
+ const mutationQueueTelemetry = createMutationQueueTelemetry({
+ environment: () => ({
+ authenticated: untrack(sessionActive),
+ online: navigator.onLine,
+ websocket: websocketState,
+ }),
+ });
const client = createClient({
url: `${dssHost}/items/soup/graphql`,
// See graphqlSoupClient: GET serves GraphiQL on this path.
@@ -756,6 +808,7 @@ export function getGraphqlSoupClient(): Client {
?.id,
// Preserve optimistic intent only while transport failures retry.
shouldRetryMutation: shouldRetryGraphqlMutation,
+ mutationQueueTelemetry,
delegateRevalidation: delegateChannelNotificationRefresh,
}),
graphqlSoupSubscriptionExchange(graphqlWsClient),
diff --git a/crates/client/cache-core/AGENTS.md b/crates/client/cache-core/AGENTS.md
index 94c4729ce8..8f148654c8 100644
--- a/crates/client/cache-core/AGENTS.md
+++ b/crates/client/cache-core/AGENTS.md
@@ -60,6 +60,10 @@ GraphQL response shape for callers. A concrete `Storage` impl (e.g.
- Opaque JSON scalars must preserve floating-point values on reparse — `serde_json`
is pulled in with the `float_roundtrip` feature; don't drop it.
+- `StoredMutation.created_at_ms` is the first-enqueue time hosts report as
+ `enqueuedAtMs`: a row replacing an unsent row for the same UUID inherits the
+ earlier time. Lease and backoff checks take `now_ms` separately; never read
+ `created_at_ms` as the current time.
- Patches structural query changes in place instead of resending full results
(#7825); a new query shape change needs a patch path, not just a resend.
diff --git a/crates/client/cache-core/src/engine.rs b/crates/client/cache-core/src/engine.rs
index d718014456..e3c3e6e737 100644
--- a/crates/client/cache-core/src/engine.rs
+++ b/crates/client/cache-core/src/engine.rs
@@ -1866,12 +1866,19 @@ impl<S: Storage> Engine<S> {
queued.mutation.attempt_count > 0,
&queued.optimistic.optimistic_data_json,
),
+ queued.mutation.created_at_ms,
)
});
let expected_kind = match collision {
None => MutationUpsertKind::Inserted,
- Some((active_id, true)) => MutationUpsertKind::AppendedAfterActive { active_id },
- Some((removed_id, false)) => MutationUpsertKind::ReplacedPending { removed_id },
+ Some((active_id, true, _)) => MutationUpsertKind::AppendedAfterActive { active_id },
+ Some((removed_id, false, _)) => MutationUpsertKind::ReplacedPending { removed_id },
+ };
+ // The removed row never reached the server, so its wait carries over;
+ // an active row stays queued and keeps reporting its own.
+ let enqueued_at_ms = match collision {
+ Some((_, false, replaced_at_ms)) => replaced_at_ms.min(created_at_ms),
+ _ => created_at_ms,
};
let patches = deduplicate_patches(&self.schema, link_patches)?;
@@ -1908,7 +1915,7 @@ impl<S: Storage> Engine<S> {
variables_json: canonical_json(&Json::Object(variables.clone())),
identity,
},
- created_at_ms,
+ enqueued_at_ms,
),
optimistic: PersistedOptimisticLayer {
optimistic_data_json: encode_optimistic_source(&source),
diff --git a/crates/client/cache-core/src/queue.rs b/crates/client/cache-core/src/queue.rs
index ee94008a6d..e4f7b01517 100644
--- a/crates/client/cache-core/src/queue.rs
+++ b/crates/client/cache-core/src/queue.rs
@@ -50,11 +50,23 @@ pub struct StoredMutation {
pub lease_expires_at_ms: Option<i64>,
/// Last retryable network error, for diagnostics.
pub last_error: Option<String>,
- /// Wall-clock enqueue time, for diagnostics.
+ /// Wall-clock time the write was first enqueued, for diagnostics. A row
+ /// replacing an unsent row for the same UUID keeps the earlier time.
pub created_at_ms: i64,
}
+/// Largest integer a JavaScript number represents exactly.
+const MAX_SAFE_JS_INTEGER: i64 = (1 << 53) - 1;
+
impl StoredMutation {
+ /// Enqueue time for JavaScript callers, or `None` when a number cannot
+ /// carry it exactly.
+ pub fn enqueued_at_ms(&self) -> Option<i64> {
+ (0..=MAX_SAFE_JS_INTEGER)
+ .contains(&self.created_at_ms)
+ .then_some(self.created_at_ms)
+ }
+
/// Creates an unclaimed queue row for `request`.
pub fn new(request: MutationRequest, created_at_ms: i64) -> Self {
Self {
@@ -333,6 +345,9 @@ pub struct MutationInspection {
pub client_metadata: Option<Json>,
/// Original optimistic response for migrating durable local intent.
pub optimistic_data: Json,
+ /// Wall-clock first-enqueue time in epoch milliseconds.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub enqueued_at_ms: Option<i64>,
}
impl TryFrom<QueuedMutation> for MutationInspection {
@@ -340,6 +355,7 @@ impl TryFrom<QueuedMutation> for MutationInspection {
fn try_from(queued: QueuedMutation) -> Result<Self, Self::Error> {
let source = decode_optimistic_source(&queued.optimistic.optimistic_data_json)?;
+ let enqueued_at_ms = queued.mutation.enqueued_at_ms();
let request = queued.mutation.request;
Ok(Self {
transaction_id: queued.id.to_string(),
@@ -351,6 +367,7 @@ impl TryFrom<QueuedMutation> for MutationInspection {
.map_err(|error| error.to_string())?,
client_metadata: source.client_metadata,
optimistic_data: source.mutation_data,
+ enqueued_at_ms,
})
}
}
diff --git a/crates/client/cache-core/tests/optimistic.rs b/crates/client/cache-core/tests/optimistic.rs
index 6ede1f794c..498db3fe97 100644
--- a/crates/client/cache-core/tests/optimistic.rs
+++ b/crates/client/cache-core/tests/optimistic.rs
@@ -1107,6 +1107,74 @@ fn pending_uuid_replacement_moves_to_tail_and_keeps_only_latest_layer() {
});
}
+#[test]
+fn pending_uuid_replacement_keeps_the_first_enqueue_time() {
+ block_on(async {
+ const X: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a01";
+ const Y: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a02";
+ let mut engine = engine_with_base("Status", "todo").await;
+ begin_value(&mut engine, X, "a", 100).await;
+ begin_value(&mut engine, X, "b", 250).await;
+ // A clock that moved backwards cannot make the replacement look newer.
+ begin_value(&mut engine, Y, "c", 500).await;
+ begin_value(&mut engine, Y, "d", 400).await;
+
+ let queued = engine.storage().load_mutation_queue().await.unwrap();
+ assert_eq!(
+ queued
+ .iter()
+ .map(|row| row.mutation.created_at_ms)
+ .collect::<Vec<_>>(),
+ [100, 400]
+ );
+ let inspected = engine.inspect_mutations().await.unwrap();
+ assert_eq!(
+ inspected
+ .iter()
+ .map(|item| item.enqueued_at_ms)
+ .collect::<Vec<_>>(),
+ [Some(100), Some(400)]
+ );
+ let wire = serde_json::to_value(&inspected[0]).unwrap();
+ assert_eq!(wire["enqueuedAtMs"], json!(100));
+ });
+}
+
+#[test]
+fn active_uuid_replacement_keeps_its_own_enqueue_time() {
+ block_on(async {
+ const X: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a03";
+ let mut engine = engine_with_base("Status", "todo").await;
+ begin_value(&mut engine, X, "a", 100).await;
+ let (_, _) = claim_head(&mut engine, "runner", 110).await;
+ begin_value(&mut engine, X, "b", 300).await;
+
+ let queued = engine.storage().load_mutation_queue().await.unwrap();
+ assert!(queued[0].superseded);
+ assert_eq!(
+ queued
+ .iter()
+ .map(|row| row.mutation.created_at_ms)
+ .collect::<Vec<_>>(),
+ [100, 300]
+ );
+ });
+}
+
+#[test]
+fn enqueue_time_outside_the_javascript_safe_range_is_unknown() {
+ block_on(async {
+ const X: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a04";
+ let mut engine = engine_with_base("Status", "todo").await;
+ begin_value(&mut engine, X, "a", 1 << 60).await;
+
+ let inspected = engine.inspect_mutations().await.unwrap();
+ assert_eq!(inspected[0].enqueued_at_ms, None);
+ let wire = serde_json::to_value(&inspected[0]).unwrap();
+ assert!(wire.get("enqueuedAtMs").is_none());
+ });
+}
+
#[test]
fn pending_replacement_restores_fields_omitted_by_the_new_intent() {
block_on(async {
diff --git a/crates/client/cache-wasm/src/shell.rs b/crates/client/cache-wasm/src/shell.rs
index a5ad061cee..948f6e8bc5 100644
--- a/crates/client/cache-wasm/src/shell.rs
+++ b/crates/client/cache-wasm/src/shell.rs
@@ -253,6 +253,8 @@ struct JsClaimedMutation {
identity: Option<String>,
attempt_count: u32,
server_failure_count: u32,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ enqueued_at_ms: Option<i64>,
}
impl TryFrom<ClaimedMutation> for JsClaimedMutation {
@@ -265,6 +267,7 @@ impl TryFrom<ClaimedMutation> for JsClaimedMutation {
)
.map_err(err_js)?
.client_metadata;
+ let enqueued_at_ms = claimed.queued.mutation.enqueued_at_ms();
let request = claimed.queued.mutation.request;
Ok(Self {
client_metadata,
@@ -279,6 +282,7 @@ impl TryFrom<ClaimedMutation> for JsClaimedMutation {
identity: request.identity,
attempt_count: claimed.queued.mutation.attempt_count,
server_failure_count: claimed.queued.mutation.server_failure_count,
+ enqueued_at_ms,
})
}
}
diff --git a/crates/graphql_entity_mutation/Cargo.toml b/crates/graphql_entity_mutation/Cargo.toml
index baf0216a40..052b5b32f4 100644
--- a/crates/graphql_entity_mutation/Cargo.toml
+++ b/crates/graphql_entity_mutation/Cargo.toml
@@ -12,9 +12,11 @@ graphql_permission = { path = "../graphql_permission" }
graphql_soup = { path = "../graphql_soup" }
model-entity = { path = "../model-entity" }
models_permissions = { path = "../models_permissions" }
+tracing = { workspace = true }
workspace-hack = { version = "0.1", path = "../workspace-hack" }
[dev-dependencies]
macro_user_id = { path = "../macro_user_id" }
tokio = { workspace = true }
+tracing-subscriber = { workspace = true }
uuid = { workspace = true }
diff --git a/crates/graphql_entity_mutation/src/mutations.rs b/crates/graphql_entity_mutation/src/mutations.rs
index 8a236d7197..f6e56d8467 100644
--- a/crates/graphql_entity_mutation/src/mutations.rs
+++ b/crates/graphql_entity_mutation/src/mutations.rs
@@ -313,7 +313,7 @@ fn validate_share_policy_inputs(
}
/// Stable machine-readable mutation error code.
-#[derive(Clone, Copy, Enum, Eq, PartialEq)]
+#[derive(Clone, Copy, Debug, Enum, Eq, PartialEq)]
pub enum GraphqlEntityMutationErrorCode {
/// Operation is unavailable for the requested entity kind.
UnsupportedOperation,
@@ -463,7 +463,21 @@ pub struct EntityMutationPayload<E> {
impl<E> EntityMutationPayload<E> {
/// Construct a batch payload from ordered domain mutation outcomes.
+ ///
+ /// Per-item failures are successful GraphQL responses, so the resolver
+ /// span records them for the batch.
fn new(results: Vec<MutateEntitiesResult>) -> Self {
+ let span = tracing::Span::current();
+ let mut failures = results.iter().filter_map(|result| result.as_ref().err());
+ if let Some(first) = failures.next() {
+ span.record("entity_mutation.failed_count", failures.count() + 1);
+ span.record(
+ "entity_mutation.error_code",
+ tracing::field::debug(GraphqlEntityMutationErrorCode::new(*first)),
+ );
+ } else {
+ span.record("entity_mutation.failed_count", 0);
+ }
EntityMutationPayload {
results,
edges: PhantomData,
@@ -499,6 +513,15 @@ fn mutation_actor(ctx: &Context<'_>) -> async_graphql::Result<EntityMutationActo
#[Object]
impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
/// Rename heterogeneous entities in one request.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = inputs.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn rename_entities(
&self,
ctx: &Context<'_>,
@@ -522,6 +545,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
}
/// Move heterogeneous entities to a project or to the root.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = inputs.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn move_entities(
&self,
ctx: &Context<'_>,
@@ -545,6 +577,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
}
/// Update link, explicit team, and channel share policies across supported entity kinds.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = inputs.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn update_entity_share_policies(
&self,
ctx: &Context<'_>,
@@ -569,6 +610,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
}
/// Soft-delete entities with a reversible trash lifecycle.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = entities.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn trash_entities(
&self,
ctx: &Context<'_>,
@@ -589,6 +639,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
}
/// Restore reversibly deleted entities.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = entities.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn restore_entities(
&self,
ctx: &Context<'_>,
@@ -609,6 +668,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
}
/// Irreversibly delete entities.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = entities.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn delete_entities_permanently(
&self,
ctx: &Context<'_>,
@@ -632,6 +700,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {
}
/// Duplicate supported entities.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(
+ entity_mutation.count = inputs.len(),
+ entity_mutation.failed_count = tracing::field::Empty,
+ entity_mutation.error_code = tracing::field::Empty,
+ )
+ )]
async fn duplicate_entities(
&self,
ctx: &Context<'_>,
diff --git a/crates/graphql_entity_mutation/src/mutations/test.rs b/crates/graphql_entity_mutation/src/mutations/test.rs
index 9d858483f0..a0d6a7503f 100644
--- a/crates/graphql_entity_mutation/src/mutations/test.rs
+++ b/crates/graphql_entity_mutation/src/mutations/test.rs
@@ -1,4 +1,4 @@
-use std::sync::Arc;
+use std::sync::{Arc, Mutex};
use async_graphql::{
Context, EmptySubscription, MaybeUndefined, Object, Request, Schema, SimpleObject, value,
@@ -11,6 +11,14 @@ use graphql_soup::SoupEntityEdges;
use macro_user_id::user_id::MacroUserIdStr;
use model_entity::{Entity, EntityType};
use models_permissions::share_permission::{LinkShare, access_level::AccessLevel};
+use tracing::{
+ field::{Field, Visit},
+ span::{Attributes, Id, Record},
+};
+use tracing_subscriber::{
+ Layer, Registry,
+ layer::{Context as LayerContext, SubscriberExt},
+};
/// Minimal composed Soup edge object used by the isolated mutation schema.
#[derive(Clone, SimpleObject)]
@@ -456,3 +464,78 @@ fn share_policy_validation_requires_access_levels_for_channel_grants() {
let error = super::validate_share_policy_inputs(&inputs).unwrap_err();
assert!(error.message.contains("accessLevel is required"));
}
+
+/// Captures every span field value, including values recorded after creation.
+#[derive(Clone, Default)]
+struct RecordedSpanFields(Arc<Mutex<Vec<(String, String)>>>);
+
+impl RecordedSpanFields {
+ fn get(&self, name: &str) -> Option<String> {
+ self.0
+ .lock()
+ .unwrap()
+ .iter()
+ .rev()
+ .find(|(field, _)| field == name)
+ .map(|(_, value)| value.clone())
+ }
+}
+
+impl Visit for RecordedSpanFields {
+ fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
+ self.0
+ .lock()
+ .unwrap()
+ .push((field.name().to_string(), format!("{value:?}")));
+ }
+}
+
+impl<S: tracing::Subscriber> Layer<S> for RecordedSpanFields {
+ fn on_new_span(&self, attrs: &Attributes<'_>, _id: &Id, _ctx: LayerContext<'_, S>) {
+ attrs.record(&mut self.clone());
+ }
+
+ fn on_record(&self, _id: &Id, values: &Record<'_>, _ctx: LayerContext<'_, S>) {
+ values.record(&mut self.clone());
+ }
+}
+
+#[tokio::test]
+async fn batch_span_records_per_item_failures() {
+ let fields = RecordedSpanFields::default();
+ let _guard = tracing::subscriber::set_default(Registry::default().with(fields.clone()));
+ let actor = EntityMutationActor {
+ user_id: MacroUserIdStr::parse_from_str("macro|[email protected]").unwrap(),
+ organization_id: Some(42),
+ };
+ let response = Schema::build(
+ QueryRoot,
+ crate::EntityMutationRoot::<UnavailableEntityMutationService, TestSoupEdges>::new(),
+ EmptySubscription,
+ )
+ .finish()
+ .execute(
+ Request::new(
+ r#"mutation {
+ trashEntities(entities: [
+ { type: DOCUMENT, id: "document-1" }
+ { type: DOCUMENT, id: "document-2" }
+ ]) { results { __typename } }
+ }"#,
+ )
+ .data(Arc::new(UnavailableEntityMutationService))
+ .data(actor),
+ )
+ .await;
+
+ assert!(response.errors.is_empty(), "{:?}", response.errors);
+ assert_eq!(fields.get("entity_mutation.count").as_deref(), Some("2"));
+ assert_eq!(
+ fields.get("entity_mutation.failed_count").as_deref(),
+ Some("2")
+ );
+ assert_eq!(
+ fields.get("entity_mutation.error_code").as_deref(),
+ Some("UnsupportedOperation")
+ );
+}
diff --git a/crates/graphql_notification/src/mutations.rs b/crates/graphql_notification/src/mutations.rs
index 8727daa014..cb5285167c 100644
--- a/crates/graphql_notification/src/mutations.rs
+++ b/crates/graphql_notification/src/mutations.rs
@@ -181,6 +181,11 @@ where
S: NotificationMutationService,
{
/// Update authenticated user-owned notifications and return authoritative rows.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(notification_count = input.notification_ids.len())
+ )]
async fn update_notifications(
&self,
ctx: &Context<'_>,
@@ -202,6 +207,11 @@ where
}
/// Update all notifications associated with the requested entities for the authenticated user.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(entity_count = input.entities.len())
+ )]
async fn update_notifications_for_entity(
&self,
ctx: &Context<'_>,
diff --git a/crates/graphql_properties/src/mutations.rs b/crates/graphql_properties/src/mutations.rs
index b57ccaf80b..525128d674 100644
--- a/crates/graphql_properties/src/mutations.rs
+++ b/crates/graphql_properties/src/mutations.rs
@@ -434,6 +434,7 @@ where
T: EntityPropertyWriter,
{
/// Remove a property assignment and identify its normalized cache record.
+ #[tracing::instrument(skip_all, err(Debug))]
async fn delete_entity_property(
&self,
ctx: &Context<'_>,
@@ -454,6 +455,7 @@ where
}
/// Set or attach one property on an entity.
+ #[tracing::instrument(skip_all, err(Debug))]
async fn set_entity_property(
&self,
ctx: &Context<'_>,
@@ -481,6 +483,11 @@ where
}
/// Add and remove options across one entity's multi-select properties.
+ #[tracing::instrument(
+ skip_all,
+ err(Debug),
+ fields(property_count = input.properties.len())
+ )]
async fn update_entity_property_options(
&self,
ctx: &Context<'_>,
diff --git a/deny.toml b/deny.toml
index 4c691e2a21..d4e74eb4ea 100644
--- a/deny.toml
+++ b/deny.toml
@@ -229,6 +229,7 @@ deny = [
"documents", # Dev logging only
"email", # Dev logging only
"genai_telemetry", # GenAI span tests
+ "graphql_entity_mutation", # Batch failure span field tests only
"lambda_runtime_api_client", # External crate allowed to use tracing-subscriber
"loom", # External concurrency testing dependency
"macro_entrypoint",
diff --git a/docs/agent-guide/email.md b/docs/agent-guide/email.md
index 0ea3b058b6..9b94f80896 100644
--- a/docs/agent-guide/email.md
+++ b/docs/agent-guide/email.md
@@ -547,7 +547,10 @@ server draft exists. If per-mutation recovery preparation fails or times out,
the mutation must fail and release the queue too, including legacy drafts that
have not been copied into recovery storage. Existing local working copies remain;
an unmigrated legacy edit can be lost if the queue held its only durable copy.
-Verify that an unrelated queued action still completes in both cases.
+Verify that an unrelated queued action still completes in both cases. With
+browser OTel enabled, the failure appears once per draft as
+`graphql.mutation_queue.recovery_failed` (or `graphql.mutation_queue.skipped` with a
+`mutation_queue.skip_reason`), never as a line per poll.
Repeat with two tabs, a save response arriving after a newer
edit, an attachment upload completing during another local save, and Discard
while an attachment snapshot is still being saved. A snapshot based on an older