Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(graphql-cache): trace the mutation queue lifecycle (#8616)
teoa6d54ad0d1dc
diff --git a/.agents/skills/instrumentation/SKILL.md b/.agents/skills/instrumentation/SKILL.mdindex e500229aee..f67963a322 100644--- a/.agents/skills/instrumentation/SKILL.md+++ b/.agents/skills/instrumentation/SKILL.md@@ -86,6 +86,22 @@ with outcome fields (`edit.mode`, `edit.ops_total`, `edit.blocked`,`edit.flush`. On the agent side, `crates/agent_session/src/domain/session/telemetry.rs`emits `invoke_agent` and `execute_tool` and propagates into the ACP agent.+**GraphQL mutation queue (web → DSS `graphql.execute` → resolver).**+`apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts` opens a+`graphql.mutation_queue.attempt` span per claimed attempt that actually sends+(`mutation_queue.item_id`, operation name, attempt count, queue depth, oldest+item age, auth/online/websocket state, `mutation_queue.outcome`,+`mutation_queue.failure`, `graphql.error.code`). Ages come from the durable+enqueue time (`enqueuedAtMs`), so they survive reloads; when it is missing or+skewed, `mutation_queue.age_known` is false and the age is omitted rather than+zero. The runner sends inside it, so+`dssGraphqlFetch` injects `traceparent` and the DSS `graphql.execute` span and+the instrumented resolvers (`crates/graphql_entity_mutation`,+`graphql_properties`, `graphql_notification`) join the same trace. A loop that+can repeat forever reports counters instead of lines: each item logs a failure+class once, and `graphql.mutation_queue.summary` flushes counters every five+minutes only when something non-routine happened.+## VerifyRun the flow on the local stack and find its trace (see the `live-debug`diff --git a/Cargo.lock b/Cargo.lockindex df1a3d187e..b08f7a59f2 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -5523,9 +5523,9 @@ dependencies = [[[package]]name = "ctutils"-version = "0.4.2"+version = "0.4.5"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e"+checksum = "38d28fb6aa7dd680b3ffd1a3e64e115c177bdba63c0ae7f9cd48129ff05c4fa0"dependencies = ["cmov","subtle",@@ -8956,6 +8956,8 @@ dependencies = ["model-entity","models_permissions","tokio",+ "tracing",+ "tracing-subscriber","uuid","workspace-hack",]@@ -19652,9 +19654,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"[[package]]name = "tokio"-version = "1.52.3"+version = "1.54.1"source = "registry+https://github.com/rust-lang/crates.io-index"-checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe"+checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3"dependencies = ["bytes","libc",diff --git a/apps/web/AGENTS.md b/apps/web/AGENTS.mdindex 9699087574..8c1a9baac7 100644--- a/apps/web/AGENTS.md+++ b/apps/web/AGENTS.md@@ -184,6 +184,7 @@ ownership. Cancel, logout, and the ten-minute timeout retire pending callbacks.- Diagnose browser call media with `call.media.*` spans and the `meeting.setup.id`on `meeting.join`; keep provider messages, device labels/IDs, and call-linktokens out of telemetry. See [call diagnostics](../../docs/agent-guide/calls.md#browser-call-diagnostics).+- Diagnose the GraphQL mutation queue with `graphql.mutation_queue.*` spans and logs keyed by `mutation_queue.item_id`; route queue telemetry through `mutation-queue-telemetry.ts`, which dedupes per item and failure class and turns repeats into the periodic summary. Never add per-attempt `console.*` lines or record variables, draft content, or emails.- Keep desktop Gmail and Outlook inbox consent in the system browser with a`macro:///inbox-link-callback` return; webview origins are not desktop callbacks.- Keep send reservations, leases, and cancellation in the email journal; the general GraphQL queue handles draft autosaves only.diff --git a/apps/web/src/components/app/SessionExpiredRedirect.tsx b/apps/web/src/components/app/SessionExpiredRedirect.tsxindex de362c15f4..3a3117f5b8 100644--- a/apps/web/src/components/app/SessionExpiredRedirect.tsx+++ b/apps/web/src/components/app/SessionExpiredRedirect.tsx@@ -33,7 +33,7 @@ export function SessionExpiredRedirect() {);if (!stillUnauthorized) return false;}- await clearLocalAuthSession().catch((error) => {+ await clearLocalAuthSession('session-expired').catch((error) => {console.error('Failed to clear local auth session', error);});return true;diff --git a/apps/web/src/lib/core/auth/logout.test.ts b/apps/web/src/lib/core/auth/logout.test.tsindex 486efcd083..be3dea598b 100644--- a/apps/web/src/lib/core/auth/logout.test.ts+++ b/apps/web/src/lib/core/auth/logout.test.ts@@ -48,6 +48,7 @@ vi.mock('@graphql-cache/lifecycle', () => ({clearRegisteredCaches: vi.fn(async () => {}),}));vi.mock('@service-storage/graphql-soup', () => ({+ recordGraphqlMutationQueueCleared: vi.fn(async () => {}),setGraphqlSoupSessionActive: vi.fn(),}));vi.mock('@queries/auth/user-info', () => ({diff --git a/apps/web/src/lib/core/auth/logout.ts b/apps/web/src/lib/core/auth/logout.tsindex 27eacb444d..033dd22768 100644--- a/apps/web/src/lib/core/auth/logout.ts+++ b/apps/web/src/lib/core/auth/logout.ts@@ -26,7 +26,10 @@ import { resetGraphqlSoupDoneSession } from '@queries/soup/graphql/done-session'import { clearDocumentQueryCache } from '@queries/storage/document-cache';import { clearOfflineDocumentContexts } from '@queries/storage/documentLoad/offline-context-runtime';import { authServiceClient } from '@service-auth/client';-import { setGraphqlSoupSessionActive } from '@service-storage/graphql-soup';+import {+ recordGraphqlMutationQueueCleared,+ setGraphqlSoupSessionActive,+} from '@service-storage/graphql-soup';import { raceTimeout } from '@solid-primitives/promise';import { createCallback } from '@solid-primitives/rootless';import { useNavigate } from '@solidjs/router';@@ -52,7 +55,9 @@ const unauthenticatedUserInfo: UserInfoData = {createdAt: undefined,};-export async function clearLocalAuthSession() {+export async function clearLocalAuthSession(+ cause: 'logout' | 'session-expired' = 'logout'+) {await clearNativeDiagnostics();clearPipedreamCallbacks();document.cookie =@@ -64,6 +69,8 @@ export async function clearLocalAuthSession() {clearTeamCalendarQueries();// Start every wipe before waiting, including stores not hydrated this session.const results = await Promise.allSettled([+ // Reads the queue before the cache wipe below deletes it.+ recordGraphqlMutationQueueCleared(cause),Promise.resolve().then(clearComposerStorage),documentContextsCleared,clearRegisteredCaches(),diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.tsindex 9d293d0345..55a28ccd1b 100644--- a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.test.ts@@ -1,5 +1,6 @@import { SaveEmailDraftDocument } from '@service-storage/graphql/generated/graphql';import {+ CombinedError,createClient,createRequest,makeOperation,@@ -20,8 +21,14 @@ import {import {createMutationQueueRunner,type MutationQueueRunner,+ type MutationQueueRunnerDeps,queueAttemptOf,+ withQueueRequestTimeout,} from './mutation-queue-runner';+import {+ type MutationAttemptTelemetry,+ NOOP_MUTATION_QUEUE_TELEMETRY,+} from './mutation-queue-telemetry';beforeEach(() => vi.useFakeTimers());afterEach(() => {@@ -68,7 +75,7 @@ function enqueue(: { kind: 'not-runnable' },};}-function fixture() {+function fixture(options: MutationQueueRunnerDeps['options'] = {}) {const forwarded: Operation[] = [];const commit = vi.fn<CacheHost['commitOptimisticWrite']>(async () =>@@ -83,6 +90,15 @@ function fixture() {);const host = {commitOptimisticWrite: commit,+ deferOptimisticWrite: vi.fn(async () => ({ kind: 'deferred' })),+ rollbackOptimisticWrite: vi.fn(async () => ({+ kind: 'rolled-back',+ revision: INITIAL_CACHE_REVISION,+ revisionAdvanced: true,+ changed: [],+ affectedOps: [],+ reset: false,+ })),currentRevision: async () => INITIAL_CACHE_REVISION,claimNextMutation: async () => undefined,} as unknown as CacheHost;@@ -103,7 +119,7 @@ function fixture() {),],}),- options: {},+ options,owner: 'owner',forward: (op) => forwarded.push(op),writeThrough: (result) =>@@ -235,3 +251,111 @@ it('preserves a pump claim when its unclaimed enqueue acknowledgement arrives afexpect.anything());});++function recordingTelemetry() {+ const attempts: Array<{ sent: string[]; finished: unknown[][] }> = [];+ const run = vi.fn((send: () => unknown) => send());+ const telemetry = {+ ...NOOP_MUTATION_QUEUE_TELEMETRY,+ attempt: () => {+ const record = { sent: [] as string[], finished: [] as unknown[][] };+ attempts.push(record);+ return {+ run: <T>(send: () => T) => run(send) as T,+ sent: (mode) => {+ record.sent.push(mode);+ },+ finish: (...args) => {+ record.finished.push(args);+ },+ } satisfies MutationAttemptTelemetry;+ },+ };+ return { telemetry, attempts, run };+}++const savedDraft = { saveEmailDraft: { draftId: 'draft', subject: 'Draft' } };++it('ends a live attempt once, as committed', async () => {+ const { telemetry, attempts } = recordingTelemetry();+ const { runner, forwarded } = fixture({ mutationQueueTelemetry: telemetry });+ await runner.admit(operation(), enqueue(operation(), '1'), runner.epoch());+ await runner.settle(+ { operation: forwarded[0], data: savedDraft, stale: false, hasNext: false },+ queueAttemptOf(forwarded[0])!+ );+ expect(attempts).toEqual([{ sent: ['live'], finished: [['committed', {}]] }]);+});++it('sends a queued request inside its attempt span', async () => {+ const { telemetry, run } = recordingTelemetry();+ const { runner, forwarded } = fixture({ mutationQueueTelemetry: telemetry });+ await runner.admit(operation(), enqueue(operation(), '1'), runner.epoch());+ const transport = vi.fn(async () => new Response('{}', { status: 200 }));+ const op = makeOperation(forwarded[0].kind, forwarded[0], {+ ...forwarded[0].context,+ fetch: transport,+ });+ await withQueueRequestTimeout(op).context.fetch!('http://test', {});+ expect(run).toHaveBeenCalledOnce();+ expect(transport).toHaveBeenCalledOnce();+});++it('reports a retried server failure with its backoff', async () => {+ const { telemetry, attempts } = recordingTelemetry();+ const { runner, forwarded } = fixture({+ mutationQueueTelemetry: telemetry,+ shouldRetryMutation: () => true,+ });+ await runner.admit(operation(), enqueue(operation(), '2'), runner.epoch());+ const attempt = queueAttemptOf(forwarded[0])!;+ attempt.response = new Response(null, { status: 503 });+ await runner.settle(+ { operation: forwarded[0], data: savedDraft, stale: false, hasNext: false },+ attempt+ );+ expect(attempts[0]?.finished).toEqual([+ [+ 'retry-scheduled',+ {+ failure: 'server',+ errorCode: undefined,+ httpStatus: 503,+ retryDelayMs: 2_000,+ },+ ],+ ]);+});++it('reports retry exhaustion apart from a permanent rollback', async () => {+ const { telemetry, attempts } = recordingTelemetry();+ const { runner, forwarded } = fixture({+ mutationQueueTelemetry: telemetry,+ shouldRetryMutation: () => true,+ });+ const exhausted = enqueue(operation(), '10');+ if (exhausted.initialClaim.kind !== 'claimed')+ throw new Error('Missing claim');+ exhausted.initialClaim.mutation.serverFailureCount = 9;+ await runner.admit(operation(), exhausted, runner.epoch());+ await runner.settle(+ {+ operation: forwarded[0],+ data: undefined,+ error: new CombinedError({+ graphQLErrors: [+ { message: 'boom', extensions: { code: 'INTERNAL' }, path: ['x'] },+ ],+ }),+ stale: false,+ hasNext: false,+ },+ queueAttemptOf(forwarded[0])!+ );+ expect(attempts[0]?.finished).toEqual([+ [+ 'retry-exhausted',+ { failure: 'server', errorCode: 'INTERNAL', httpStatus: undefined },+ ],+ ]);+});diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.tsindex f2ed4c9261..363bfc38ad 100644--- a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-runner.ts@@ -48,6 +48,15 @@ import type {MutationClaim,QueryRevalidationWire,} from '../protocol';+import {+ failureDetails,+ type MutationAttemptDetails,+ type MutationAttemptOutcome,+ type MutationAttemptTelemetry,+ NOOP_MUTATION_ATTEMPT_TELEMETRY,+ NOOP_MUTATION_QUEUE_TELEMETRY,+ recoverySkipReason,+} from './mutation-queue-telemetry';import type { NormalizedCacheExchangeOptions } from './normalized-cache-exchange';import {notifyOptimisticMutationEnqueued,@@ -104,6 +113,7 @@ export type QueueAttemptContext = AttemptToken & {serverFailureCount: number;/** Transport-only metadata; urql drops the response on error-free payloads. */response?: Response;+ telemetry?: MutationAttemptTelemetry;};function claimOf(token: AttemptToken): MutationClaim {@@ -164,12 +174,14 @@ export function withQueueRequestTimeout(op: Operation): Operation {...op.context,fetch: async (input, init) => {const timeoutSignal = AbortSignal.timeout(QUEUE_REQUEST_TIMEOUT_MS);- const response = await operationFetch(input, {- ...init,- signal: init?.signal- ? AbortSignal.any([init.signal, timeoutSignal])- : timeoutSignal,- });+ const send = () =>+ operationFetch(input, {+ ...init,+ signal: init?.signal+ ? AbortSignal.any([init.signal, timeoutSignal])+ : timeoutSignal,+ });+ const response = await (attempt?.telemetry?.run(send) ?? send());if (attempt) attempt.response = response;return response;},@@ -274,6 +286,7 @@ export type MutationQueueRunnerDeps = {| 'onMutationAttemptResult'| 'shouldRetryMutation'| 'onCacheError'+ | 'mutationQueueTelemetry'>;/** Lease owner for this exchange's claims. */owner: string;@@ -336,6 +349,8 @@ export function createMutationQueueRunner(deps: MutationQueueRunnerDeps): MutationQueueRunner {const { host, client, options, owner, forward, writeThrough } = deps;+ const telemetry =+ options.mutationQueueTelemetry ?? NOOP_MUTATION_QUEUE_TELEMETRY;const liveQueuedOps = new Map<string, LiveQueuedOperation>();let storageGeneration = 0;let activeAttempt: AttemptToken | undefined;@@ -351,7 +366,6 @@ export function createMutationQueueRunner(let head: HeadState = IDLE;// Pump state: wakeups and the retry hint are independent of the head.let queuePreparation: Promise<void> | undefined;- const reportedRecoveryFailures = new Set<string>();let drainRunning = false;let drainRequested = false;let deferredUntil: number | undefined;@@ -434,6 +448,26 @@ export function createMutationQueueRunner(if (!isCurrent(epoch) || !isNewClaim(claimed)) return;latestClaim = claimed;activeAttempt = attempt;+ const attemptTelemetry = telemetry.attempt(claimed);+ attempt.telemetry = attemptTelemetry;+ // Settlement ends a sent attempt; anything that returns before sending+ // without its own outcome was retired by a storage reset.+ let sent = false;+ try {+ await routeClaimedAttempt(claimed, attempt, attemptTelemetry, () => {+ sent = true;+ });+ } finally {+ if (!sent) attemptTelemetry.finish('detached');+ }+ }++ async function routeClaimedAttempt(+ claimed: ClaimedMutation,+ attempt: QueueAttemptContext,+ attemptTelemetry: MutationAttemptTelemetry,+ markSent: () => void+ ): Promise<void> {const confirmed = confirmedOf(head);if (confirmed?.transactionId === claimed.transactionId) {// claim → settling: re-apply the server's response before admission@@ -459,6 +493,7 @@ export function createMutationQueueRunner(: true;if (admission === 'defer') {await deferAdmission(claimed, attempt);+ attemptTelemetry.finish('deferred');return;}if (!admission) {@@ -471,12 +506,13 @@ export function createMutationQueueRunner('LOCAL_SUPERSEDED');if (!transition(attempt, IDLE)) return;+ attemptTelemetry.finish('discarded', { errorCode: 'LOCAL_SUPERSEDED' });resolveLiveOperationsAsQueued();scheduleDrain();return;}} catch {- await failPreparation(claimed, attempt);+ await failPreparation(claimed, attempt, attemptTelemetry);return;}// A superseded create can already exist on the server. Replay it to@@ -498,6 +534,7 @@ export function createMutationQueueRunner(if (discarded.kind !== 'discarded-superseded') {throw new Error('superseded mutation was unexpectedly deferred');}+ attemptTelemetry.finish('superseded');const live = liveQueuedOps.get(claimed.transactionId);if (live) {liveQueuedOps.delete(claimed.transactionId);@@ -513,11 +550,14 @@ export function createMutationQueueRunner(if (live) {liveQueuedOps.delete(claimed.transactionId);live.resolveRoute(undefined);+ markSent();+ attemptTelemetry.sent('live');forward(withQueueRequestTimeout(withQueueAttempt(live.operation, attempt)));} else {let replayOperation: Operation | undefined;+ attemptTelemetry.sent('replay');try {const replay = client.mutation(replayDocument(claimed.query, claimed.operationName),@@ -548,6 +588,12 @@ export function createMutationQueueRunner(: undefined);if (!isCurrent(attempt)) return;+ attemptTelemetry.finish(+ rolledBack.kind === 'rolled-back' ? 'rolled-back' : 'superseded',+ error instanceof CombinedError+ ? failureDetails(error, error.response ?? attempt.response)+ : { failure: 'local' }+ );if (rolledBack.kind === 'rolled-back') {fireAndForget(deps.revalidate(rolledBack.revalidations ?? [], replayOperation),@@ -589,7 +635,8 @@ export function createMutationQueueRunner(/** preparing → settling → idle for a head whose recovery state failed. */async function failPreparation(claimed: ClaimedMutation,- attempt: QueueAttemptContext+ attempt: QueueAttemptContext,+ attemptTelemetry: MutationAttemptTelemetry): Promise<void> {queuePreparation = undefined;try {@@ -631,6 +678,10 @@ export function createMutationQueueRunner('LOCAL_RECOVERY_FAILED');if (!isCurrent(attempt)) return;+ attemptTelemetry.finish('recovery-failed', {+ failure: 'local',+ errorCode: 'LOCAL_RECOVERY_FAILED',+ });liveQueuedOps.delete(claimed.transactionId);live?.resolveRoute(withOptimisticMutationDisposition(result, {@@ -651,6 +702,7 @@ export function createMutationQueueRunner(// A retired host stays disabled for good: stop polling instead of failing// every 30 seconds against the no-op host.if (host.disabled) {+ telemetry.skipped('host-disabled');resolveLiveOperationsAsQueued();return;}@@ -698,6 +750,7 @@ export function createMutationQueueRunner(await routeClaimedMutation(claimed, epoch);} catch {+ telemetry.skipped('claim-failed');// Enqueue already succeeded, so callers must observe these as// queued even if the runner cannot currently inspect the head.resolveLiveOperationsAsQueued();@@ -719,14 +772,7 @@ export function createMutationQueueRunner(await queuePreparation;} catch (error) {queuePreparation = undefined;- // A stuck tab retries every poll; one warning per cause is enough.- const cause = error instanceof Error ? error.message : String(error);- if (reportedRecoveryFailures.has(cause)) return;- reportedRecoveryFailures.add(cause);- console.warn(- '[graphql-cache] Unable to prepare mutation recovery',- error- );+ telemetry.skipped(recoverySkipReason(error));}}@@ -835,7 +881,12 @@ export function createMutationQueueRunner(: 'permanently-failed',transactionId: attempt.transactionId,});- if (!transition(attempt, receiveResult)) return detachedResult();+ const attemptTelemetry =+ attempt.telemetry ?? NOOP_MUTATION_ATTEMPT_TELEMETRY;+ if (!transition(attempt, receiveResult)) {+ attemptTelemetry.finish('detached');+ return detachedResult();+ }const claim = claimOf(attempt);let retryAt: number | undefined;let settled = false;@@ -845,6 +896,8 @@ export function createMutationQueueRunner(| 'queued'| 'superseded'| 'permanently-failed' = 'queued';+ let outcome: MutationAttemptOutcome = 'detached';+ let details: MutationAttemptDetails = {};try {const response = result.error?.response ?? attempt.response;const httpServerFailure = (response?.status ?? 0) >= 500;@@ -861,6 +914,7 @@ export function createMutationQueueRunner(};}if (result.error || result.data == null) {+ details = failureDetails(result.error, response);let retry = false;if (result.error && options.shouldRetryMutation) {try {@@ -874,11 +928,11 @@ export function createMutationQueueRunner(// server responses, never connection failures or timeouts.const serverFailure =httpServerFailure || (result.error?.graphQLErrors.length ?? 0) > 0;- if (+ const exhausted =retry &&serverFailure &&- attempt.serverFailureCount + 1 >= MAX_MUTATION_SERVER_FAILURES- ) {+ attempt.serverFailureCount + 1 >= MAX_MUTATION_SERVER_FAILURES;+ if (exhausted) {retry = false;result = {...result,@@ -896,7 +950,8 @@ export function createMutationQueueRunner(await recordAttemptResult(attempt, result, retry);if (!isCurrent(attempt)) return detachedResult();if (retry) {- retryAt = Date.now() + retryDelayMs(attempt.attemptCount);+ details.retryDelayMs = retryDelayMs(attempt.attemptCount);+ retryAt = Date.now() + details.retryDelayMs;const deferred = await host.deferOptimisticWrite(attempt.transactionId,claim,@@ -910,8 +965,10 @@ export function createMutationQueueRunner(retryAt = undefined;replacementTransactionId = deferred.replacementTransactionId;disposition = 'superseded';+ outcome = 'superseded';} else {disposition = 'queued';+ outcome = 'retry-scheduled';}} else {const rolledBack = await host.rollbackOptimisticWrite(@@ -925,8 +982,10 @@ export function createMutationQueueRunner(if (rolledBack.kind === 'discarded-superseded') {replacementTransactionId = rolledBack.replacementTransactionId;disposition = 'superseded';+ outcome = 'superseded';} else {disposition = 'permanently-failed';+ outcome = exhausted ? 'retry-exhausted' : 'rolled-back';fireAndForget(deps.revalidate(rolledBack.revalidations ?? [], op),'graphql-cache-mutation-queue'@@ -971,6 +1030,8 @@ export function createMutationQueueRunner(disposition = replacementTransactionId? 'superseded': 'permanently-failed';+ outcome = replacementTransactionId ? 'superseded' : 'rolled-back';+ details = { failure: 'local', errorCode: 'LOCAL_COMMIT_FAILED' };if (!replacementTransactionId) {fireAndForget(deps.revalidate(committed.revalidations ?? [], op),@@ -980,9 +1041,11 @@ export function createMutationQueueRunner(} else if (committed.kind === 'committed-superseded') {replacementTransactionId = committed.replacementTransactionId;disposition = 'superseded';+ outcome = 'superseded';await deps.deleteReportedRecords(result);} else {disposition = 'committed';+ outcome = 'committed';await deps.deleteReportedRecords(result);if (!isCurrent(attempt)) return detachedResult();const replay = deps.replayExplicitEffects(op, result.data, () =>@@ -1004,6 +1067,8 @@ export function createMutationQueueRunner(// A lost acknowledgement may hide a completed transaction.// Preserve optimism and let a fresh durable claim decide.disposition = 'queued';+ outcome = 'ack-lost';+ details = { ...details, retryDelayMs: QUEUE_LEASE_MS };}} finally {if (transition(attempt, release)) {@@ -1013,6 +1078,7 @@ export function createMutationQueueRunner(retryAt === undefined ? 0 : Math.max(0, retryAt - Date.now()));}+ attemptTelemetry.finish(outcome, details);}return withOptimisticMutationDisposition(result,@@ -1036,6 +1102,7 @@ export function createMutationQueueRunner(latestClaim = undefined;head = IDLE;deferredUntil = undefined;+ telemetry.storageReset();resolveLiveOperationsAsQueued();}@@ -1044,7 +1111,10 @@ export function createMutationQueueRunner(isCurrent,admit,settle,- start: () => scheduleDrain(),+ start: () => {+ telemetry.loaded(async () => host.inspectMutations?.());+ scheduleDrain();+ },wake: wakeDrain,resetStorage,};diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.test.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.test.tsnew file mode 100644index 0000000000..643e652dcd--- /dev/null+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.test.ts@@ -0,0 +1,590 @@+import { CombinedError } from '@urql/core';+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import type { ClaimedMutation } from '../protocol';+import {+ classifyMutationFailure,+ createMutationQueueTelemetry,+ errorCodeAttribute,+ MUTATION_QUEUE_SUMMARY_INTERVAL_MS,+ type MutationQueueEnvironment,+ queueItemAge,+ recoverySkipReason,+ reportMutationQueueCleared,+} from './mutation-queue-telemetry';++const telemetry = vi.hoisted(() => {+ type Recorded = {+ name: string;+ attributes: Record<string, unknown>;+ events: string[];+ errors: unknown[];+ ended: boolean;+ };+ const spans: Recorded[] = [];+ const makeSpan = (name: string) => {+ const recorded: Recorded = {+ name,+ attributes: {},+ events: [],+ errors: [],+ ended: false,+ };+ spans.push(recorded);+ return {+ recorded,+ setAttr: (key: string, value: unknown) => {+ recorded.attributes[key] = value;+ },+ event: (event: string) => recorded.events.push(event),+ error: (error: unknown) => recorded.errors.push(error),+ run: <T>(work: () => T) => work(),+ end: () => {+ recorded.ended = true;+ },+ };+ };+ return {+ spans,+ warn: vi.fn(),+ info: vi.fn(),+ span: vi.fn(+ (name: string, work?: (span: ReturnType<typeof makeSpan>) => unknown) => {+ const span = makeSpan(name);+ if (!work) return span;+ return Promise.resolve(work(span)).finally(span.end);+ }+ ),+ };+});++vi.mock('@macro-inc/observability', () => ({+ Telemetry: {+ span: telemetry.span,+ warn: telemetry.warn,+ info: telemetry.info,+ },+}));++const environment: MutationQueueEnvironment = {+ authenticated: true,+ online: true,+ websocket: 'connected',+};++function claimed(overrides: Partial<ClaimedMutation> = {}): ClaimedMutation {+ return {+ transactionId: '7',+ uuid: 'item-1',+ superseded: false,+ requiresConfirmation: false,+ leaseGeneration: '1',+ query: 'mutation SaveEmailDraft { saveEmailDraft }',+ operationName: 'SaveEmailDraft',+ variables: { input: { body: 'private draft body', to: '[email protected]' } },+ attemptCount: 1,+ serverFailureCount: 0,+ ...overrides,+ };+}++function graphQLError(code: unknown, path: string[] = ['field']) {+ return new CombinedError({+ graphQLErrors: [{ message: 'failed', extensions: { code }, path }],+ });+}++let now = 0;+function create() {+ return createMutationQueueTelemetry({+ environment: () => environment,+ now: () => now,+ });+}++beforeEach(() => {+ vi.useFakeTimers();+ now = 1_000;+ telemetry.spans.length = 0;+ telemetry.warn.mockClear();+ telemetry.info.mockClear();+});+afterEach(() => {+ vi.clearAllTimers();+ vi.useRealTimers();+});++describe('classifyMutationFailure', () => {+ it.each([+ ['HTTP 401', undefined, 401, 'auth'],+ ['HTTP 409', undefined, 409, 'conflict'],+ ['HTTP 502', undefined, 502, 'server'],+ ['HTTP 422', undefined, 422, 'validation'],+ ['local code', graphQLError('LOCAL_RECOVERY_FAILED'), undefined, 'local'],+ ['UNAUTHENTICATED', graphQLError('UNAUTHENTICATED'), undefined, 'auth'],+ ['FORBIDDEN', graphQLError('FORBIDDEN'), undefined, 'auth'],+ ['CONFLICT', graphQLError('CONFLICT'), undefined, 'conflict'],+ ['DRAFT_ALREADY_SENT', graphQLError('DRAFT_ALREADY_SENT'), 200, 'conflict'],+ ['INTERNAL', graphQLError('INTERNAL_SERVER_ERROR'), 200, 'server'],+ ['INVALID_INPUT', graphQLError('INVALID_INPUT'), 200, 'validation'],+ ['BAD_USER_INPUT', graphQLError('BAD_USER_INPUT'), 200, 'validation'],+ ['parse error', graphQLError(undefined, []), 200, 'validation'],+ ['NOT_FOUND', graphQLError('NOT_FOUND'), 200, 'graphql'],+ [+ 'timeout',+ new CombinedError({+ networkError: Object.assign(new Error('signal timed out'), {+ name: 'TimeoutError',+ }),+ }),+ undefined,+ 'timeout',+ ],+ [+ 'offline',+ new CombinedError({ networkError: new TypeError('Failed to fetch') }),+ undefined,+ 'network',+ ],+ ['empty payload', undefined, 200, 'no-data'],+ ] as const)('%s', (_label, error, status, expected) => {+ expect(classifyMutationFailure(error, status)).toBe(expected);+ });++ it('treats the server auth message as auth without a code', () => {+ const error = new CombinedError({+ graphQLErrors: [{ message: 'authentication required', path: ['x'] }],+ });+ expect(classifyMutationFailure(error, 200)).toBe('auth');+ });+});++describe('errorCodeAttribute', () => {+ it('keeps identifier codes and replaces anything that could carry text', () => {+ expect(errorCodeAttribute('MUTATION_RETRY_EXHAUSTED')).toBe(+ 'MUTATION_RETRY_EXHAUSTED'+ );+ expect(errorCodeAttribute('draft for [email protected]')).toBe('UNRECOGNIZED');+ expect(errorCodeAttribute(42)).toBe('UNRECOGNIZED');+ expect(errorCodeAttribute(undefined)).toBeUndefined();+ });+});++describe('recoverySkipReason', () => {+ it.each([+ ['Sign in to save drafts on this device.', 'recovery-signed-out'],+ [+ 'This draft session is no longer active. Reload Macro.',+ 'recovery-session-inactive',+ ],+ ['Queue inspection is unavailable', 'recovery-inspection-unavailable'],+ ['Mutation recovery timed out', 'recovery-timeout'],+ ['IndexedDB unavailable', 'recovery-failed'],+ ] as const)('%s', (message, reason) => {+ expect(recoverySkipReason(new Error(message))).toBe(reason);+ });+});++describe('createMutationQueueTelemetry', () => {+ it('records one attempt span with ids, counts, and outcome only', () => {+ const queue = create();+ queue+ .enqueue({ uuid: 'item-1', operationName: 'SaveEmailDraft' })+ .persisted({+ transactionId: '7',+ replaced: false,+ claimed: true,+ linkPatchesDegraded: false,+ });+ now = 1_200;+ const attempt = queue.attempt(claimed({ enqueuedAtMs: 1_000 }));+ attempt.sent('live');+ now = 1_500;+ attempt.finish('committed');+ attempt.finish('rolled-back');++ const span = telemetry.spans.find(+ (span) => span.name === 'graphql.mutation_queue.attempt'+ );+ expect(span?.ended).toBe(true);+ expect(span?.attributes).toMatchObject({+ 'mutation_queue.item_id': 'item-1',+ 'mutation_queue.transaction_id': '7',+ 'graphql.operation.name': 'SaveEmailDraft',+ 'mutation_queue.attempt_count': 1,+ 'auth.state': 'signed-in',+ 'network.online': true,+ 'graphql.websocket.state': 'connected',+ 'mutation_queue.pending_items': 1,+ 'mutation_queue.recovered': false,+ 'mutation_queue.send_mode': 'live',+ 'mutation_queue.outcome': 'committed',+ 'mutation_queue.attempt_ms': 300,+ 'mutation_queue.end_to_end_ms': 500,+ 'mutation_queue.item_age_ms': 500,+ 'mutation_queue.age_known': true,+ });+ const exported = JSON.stringify(telemetry.spans);+ expect(exported).not.toContain('private draft body');+ expect(exported).not.toContain('[email protected]');+ expect(telemetry.warn).not.toHaveBeenCalled();+ });++ it('replaces an operation name that is not an identifier', () => {+ create()+ .attempt(claimed({ operationName: 'mutation { secret }' }))+ .finish('committed');+ expect(telemetry.spans[0]?.attributes['graphql.operation.name']).toBe(+ 'unknown'+ );+ });++ it('logs the first retry per item and failure class, then only counts', () => {+ const queue = create();+ for (let attemptCount = 1; attemptCount <= 5; attemptCount++) {+ queue+ .attempt(claimed({ attemptCount }))+ .finish('retry-scheduled', { failure: 'network', retryDelayMs: 1_000 });+ }+ expect(telemetry.warn).toHaveBeenCalledOnce();+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.retry_scheduled',+ expect.objectContaining({+ 'mutation_queue.item_id': 'item-1',+ 'mutation_queue.failure': 'network',+ })+ );++ queue+ .attempt(claimed({ attemptCount: 6 }))+ .finish('retry-scheduled', { failure: 'server', httpStatus: 503 });+ queue.attempt(claimed({ uuid: 'item-2' })).finish('retry-scheduled', {+ failure: 'network',+ });+ expect(telemetry.warn).toHaveBeenCalledTimes(3);++ queue.attempt(claimed({ attemptCount: 7 })).finish('committed');+ expect(telemetry.info).toHaveBeenCalledWith(+ 'graphql.mutation_queue.committed_after_failure',+ expect.objectContaining({ 'mutation_queue.item_id': 'item-1' })+ );+ });++ it('summarizes repeated failures once per interval', async () => {+ const queue = create();+ for (let attemptCount = 1; attemptCount <= 3; attemptCount++) {+ queue.attempt(claimed({ attemptCount })).finish('retry-scheduled', {+ failure: 'auth',+ httpStatus: 401,+ });+ }+ telemetry.warn.mockClear();+ now += MUTATION_QUEUE_SUMMARY_INTERVAL_MS;+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);++ expect(telemetry.warn).toHaveBeenCalledOnce();+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.summary',+ expect.objectContaining({+ 'mutation_queue.count.attempt.retry-scheduled': 3,+ 'mutation_queue.count.failure.auth': 3,+ 'mutation_queue.pending_items': 1,+ 'mutation_queue.max_attempt_count': 3,+ })+ );+ });++ it('stays quiet when an interval only had routine successes', async () => {+ const queue = create();+ queue.attempt(claimed()).finish('committed');+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ expect(telemetry.warn).not.toHaveBeenCalled();+ });++ it('logs a skip reason once and backs off summaries while it repeats', async () => {+ const queue = create();+ const summaries = () =>+ telemetry.warn.mock.calls.filter(+ ([message]) => message === 'graphql.mutation_queue.summary'+ );+ queue.skipped('recovery-session-inactive');+ queue.skipped('recovery-session-inactive');+ expect(telemetry.warn).toHaveBeenCalledOnce();+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.skipped',+ expect.objectContaining({+ 'mutation_queue.skip_reason': 'recovery-session-inactive',+ })+ );++ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ expect(summaries()).toHaveLength(1);+ expect(summaries()[0]?.[1]).toMatchObject({+ 'mutation_queue.count.skip.recovery-session-inactive': 2,+ 'mutation_queue.summary_interval_ms': MUTATION_QUEUE_SUMMARY_INTERVAL_MS,+ });++ queue.skipped('recovery-session-inactive');+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ expect(summaries()).toHaveLength(1);+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ expect(summaries()).toHaveLength(2);+ expect(summaries()[1]?.[1]).toMatchObject({+ 'mutation_queue.summary_interval_ms':+ 2 * MUTATION_QUEUE_SUMMARY_INTERVAL_MS,+ });+ });++ it('counts a deferred admission without opening a span', () => {+ create().attempt(claimed()).finish('deferred');+ expect(telemetry.spans).toHaveLength(0);+ });++ it('reports a pending item that stays queued past a minute', async () => {+ const queue = create();+ queue.attempt(claimed()).finish('deferred');+ now += MUTATION_QUEUE_SUMMARY_INTERVAL_MS;+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ const [, summary] = telemetry.warn.mock.calls[0] ?? [];+ expect(summary).toMatchObject({+ 'mutation_queue.count.attempt.deferred': 1,+ 'mutation_queue.oldest_observed_wait_ms':+ MUTATION_QUEUE_SUMMARY_INTERVAL_MS,+ 'mutation_queue.age_unknown_items': 1,+ });+ expect(summary).not.toHaveProperty('mutation_queue.oldest_item_age_ms');+ });++ it('measures age from the durable enqueue time across a reload', async () => {+ const queue = create();+ now = 600_000;+ const attempt = queue.attempt(+ claimed({ enqueuedAtMs: 100_000, attemptCount: 4 })+ );+ attempt.finish('deferred');+ expect(telemetry.spans).toHaveLength(0);+ queue+ .attempt(claimed({ enqueuedAtMs: 100_000, attemptCount: 5 }))+ .finish('committed');+ expect(telemetry.spans[0]?.attributes).toMatchObject({+ 'mutation_queue.recovered': true,+ 'mutation_queue.observed_wait_ms': 0,+ 'mutation_queue.item_age_ms': 500_000,+ 'mutation_queue.end_to_end_ms': 500_000,+ 'mutation_queue.age_known': true,+ });+ });++ it('summarizes an old recovered item even without an observed wait', async () => {+ const queue = create();+ now = 600_000;+ queue.attempt(claimed({ enqueuedAtMs: 100_000 })).finish('deferred');+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.summary',+ expect.objectContaining({+ 'mutation_queue.oldest_item_age_ms': 500_000,+ 'mutation_queue.oldest_observed_wait_ms': 0,+ 'mutation_queue.age_unknown_items': 0,+ })+ );+ });++ it('prefers the stored queue sample in a summary', async () => {+ const queue = create();+ now = 600_000;+ queue.loaded(async () => [+ { enqueuedAtMs: 200_000 },+ { enqueuedAtMs: 590_000 },+ {},+ ]);+ queue+ .attempt(claimed({ enqueuedAtMs: 590_000 }))+ .finish('retry-scheduled', { failure: 'network' });+ telemetry.warn.mockClear();+ await vi.advanceTimersByTimeAsync(MUTATION_QUEUE_SUMMARY_INTERVAL_MS);+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.summary',+ expect.objectContaining({+ 'mutation_queue.depth': 3,+ 'mutation_queue.pending_items': 1,+ 'mutation_queue.oldest_item_age_ms': 400_000,+ 'mutation_queue.age_unknown_items': 1,+ })+ );+ });++ it.each([+ ['missing', undefined, 'missing'],+ ['zero', 0, 'invalid'],+ ['fractional', 1.5, 'invalid'],+ ['in the future', 2_000, 'clock-skew'],+ ['negative', -400 * 24 * 60 * 60_000, 'invalid'],+ ] as const)(+ 'reports a %s enqueue time as an unknown age',+ (_label, enqueuedAtMs, reason) => {+ create()+ .attempt(claimed({ enqueuedAtMs }))+ .finish('retry-exhausted', { failure: 'server', httpStatus: 503 });+ const attributes = telemetry.spans[0]?.attributes;+ expect(attributes).toMatchObject({+ 'mutation_queue.age_known': false,+ 'mutation_queue.age_unknown_reason': reason,+ });+ expect(attributes).not.toHaveProperty('mutation_queue.item_age_ms');+ expect(attributes).not.toHaveProperty('mutation_queue.end_to_end_ms');+ }+ );++ it('flags an age past a year as clock skew', () => {+ now = 400 * 24 * 60 * 60_000;+ create()+ .attempt(claimed({ enqueuedAtMs: 1 }))+ .finish('committed');+ expect(telemetry.spans[0]?.attributes).toMatchObject({+ 'mutation_queue.age_known': false,+ 'mutation_queue.age_unknown_reason': 'clock-skew',+ });+ });++ it('logs how long an item waited before the queue gave up', () => {+ now = 90_000;+ create()+ .attempt(claimed({ enqueuedAtMs: 10_000, serverFailureCount: 5 }))+ .finish('retry-exhausted', { failure: 'server', httpStatus: 503 });+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.retry_exhausted',+ expect.objectContaining({+ 'mutation_queue.item_age_ms': 80_000,+ 'mutation_queue.end_to_end_ms': 80_000,+ 'mutation_queue.age_known': true,+ })+ );+ });++ it('tracks a replacement as unknown until a claim reports its time', () => {+ const queue = create();+ queue+ .enqueue({ uuid: 'item-1', operationName: 'SaveEmailDraft' })+ .persisted({+ transactionId: '8',+ replaced: true,+ claimed: false,+ linkPatchesDegraded: false,+ });+ expect(+ telemetry.spans.find(+ (span) => span.name === 'graphql.mutation_queue.enqueue'+ )?.attributes+ ).toMatchObject({ 'mutation_queue.age_unknown_items': 1 });+ });++ it('logs an enqueue failure once per reason', () => {+ const queue = create();+ queue.enqueue({ operationName: 'SaveEmailDraft' }).failed('uncertain');+ queue.enqueue({ operationName: 'SaveEmailDraft' }).failed('uncertain');+ expect(telemetry.warn).toHaveBeenCalledOnce();+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.enqueue_failed',+ expect.objectContaining({ 'mutation_queue.enqueue_failure': 'uncertain' })+ );+ expect(+ telemetry.spans.every((span) => span.ended && span.errors.length === 1)+ ).toBe(true);+ });++ it('samples the persisted queue on load', async () => {+ now = 50_000;+ create().loaded(async () => [+ { enqueuedAtMs: 20_000 },+ { enqueuedAtMs: 45_000 },+ {},+ { enqueuedAtMs: 60_000 },+ ]);+ await vi.advanceTimersByTimeAsync(0);+ const loaded = {+ 'mutation_queue.depth': 4,+ 'mutation_queue.oldest_item_age_ms': 30_000,+ 'mutation_queue.age_unknown_items': 2,+ };+ expect(telemetry.spans[0]).toMatchObject({+ name: 'graphql.mutation_queue.loaded',+ attributes: loaded,+ ended: true,+ });+ expect(telemetry.info).toHaveBeenCalledWith(+ 'graphql.mutation_queue.loaded',+ expect.objectContaining(loaded)+ );+ });++ it('omits the oldest age on load when no age is known', async () => {+ create().loaded(async () => [{}]);+ await vi.advanceTimersByTimeAsync(0);+ expect(telemetry.spans[0]?.attributes).toMatchObject({+ 'mutation_queue.depth': 1,+ 'mutation_queue.age_unknown_items': 1,+ });+ expect(telemetry.spans[0]?.attributes).not.toHaveProperty(+ 'mutation_queue.oldest_item_age_ms'+ );+ });+});++describe('queueItemAge', () => {+ it.each([+ [undefined, { known: false, reason: 'missing' }],+ [null, { known: false, reason: 'missing' }],+ ['100', { known: false, reason: 'invalid' }],+ [-5, { known: false, reason: 'invalid' }],+ [Number.MAX_SAFE_INTEGER + 2, { known: false, reason: 'invalid' }],+ [1_001, { known: false, reason: 'clock-skew' }],+ [1_000, { known: true, ageMs: 0 }],+ [400, { known: true, ageMs: 600 }],+ ] as const)('%s', (enqueuedAtMs, expected) => {+ expect(queueItemAge(enqueuedAtMs, 1_000)).toEqual(expected);+ });+});++describe('reportMutationQueueCleared', () => {+ it('lists each lost item with its wait, oldest first', () => {+ reportMutationQueueCleared(+ [+ { operationName: 'SaveEmailDraft', enqueuedAtMs: 9_000 },+ { operationName: 'mutation { secret }' },+ { operationName: 'SetEntityProperty', enqueuedAtMs: 1_000 },+ { operationName: 'SaveEmailDraft', enqueuedAtMs: 20_000 },+ ],+ 'logout',+ 10_000+ );+ expect(telemetry.warn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.cleared',+ {+ 'mutation_queue.clear_cause': 'logout',+ 'mutation_queue.depth': 4,+ 'mutation_queue.operation_names':+ 'SaveEmailDraft,SetEntityProperty,unknown',+ 'mutation_queue.items':+ 'SetEntityProperty:9000,SaveEmailDraft:1000,unknown:unknown,SaveEmailDraft:unknown',+ 'mutation_queue.oldest_item_age_ms': 9_000,+ 'mutation_queue.age_unknown_items': 2,+ }+ );+ });++ it('caps the item list', () => {+ reportMutationQueueCleared(+ Array.from({ length: 30 }, (_, index) => ({+ operationName: 'SaveEmailDraft',+ enqueuedAtMs: 1_000 + index,+ })),+ 'session-expired',+ 10_000+ );+ const [, attributes] = telemetry.warn.mock.calls[0] ?? [];+ expect(+ String(attributes?.['mutation_queue.items']).split(',')+ ).toHaveLength(20);+ expect(attributes?.['mutation_queue.depth']).toBe(30);+ });+});diff --git a/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.tsnew file mode 100644index 0000000000..71e7782d4e--- /dev/null+++ b/apps/web/src/lib/graphql-cache/exchange/mutation-queue-telemetry.ts@@ -0,0 +1,726 @@+/**+ * Telemetry for the durable GraphQL mutation queue.+ *+ * Each claimed attempt is one `graphql.mutation_queue.attempt` span. Its fetch+ * runs inside the span, so `graphql.transport` and the server's request and+ * `graphql.execute` spans join the attempt's trace through `traceparent`.+ *+ * Logs are bounded. An item logs the first occurrence of each outcome and+ * failure class, and the queue logs the first occurrence of each skip reason.+ * Repeats only increment counters, which a summary reports at most once per+ * interval while something other than routine success happened.+ *+ * Attributes are opaque ids, counts, durations, and fixed classes. Variables,+ * responses, and error messages never leave this module.+ *+ * Item ages come from the durable enqueue time, so they survive reloads. When+ * that time is missing or implausible, `mutation_queue.age_known` is false+ * and the age is omitted rather than reported as zero.+ */++import { fireAndForget } from '@core/util/fireAndForget';+import { type Span, Telemetry } from '@macro-inc/observability';+import type { CombinedError } from '@urql/core';+import type { ClaimedMutation } from '../protocol';++export type MutationQueueEnvironment = {+ authenticated: boolean;+ online: boolean;+ websocket: 'connecting' | 'connected' | 'disconnected' | 'none';+};++export type MutationFailureClass =+ | 'network'+ | 'timeout'+ | 'auth'+ | 'server'+ | 'conflict'+ | 'validation'+ | 'graphql'+ | 'no-data'+ | 'local';++export type MutationAttemptOutcome =+ /** The server confirmed the mutation and the cache committed it. */+ | 'committed'+ /** A retryable failure; the head stays queued with a backoff. */+ | 'retry-scheduled'+ /** A permanent failure; the optimistic state was rolled back. */+ | 'rolled-back'+ /** A newer write for the same record replaced this one. */+ | 'superseded'+ /** Too many server failures; the queue gave up and rolled back. */+ | 'retry-exhausted'+ /** Settlement failed after sending; the lease backoff decides later. */+ | 'ack-lost'+ /** A storage reset retired the attempt before it settled. */+ | 'detached'+ /** Admission could not judge the intent yet; it is asked again shortly. */+ | 'deferred'+ /** Admission discarded obsolete local intent without sending it. */+ | 'discarded'+ /** Startup recovery for the head failed; the head was rolled back. */+ | 'recovery-failed';++export type MutationQueueSkipReason =+ | 'host-disabled'+ | 'claim-failed'+ | 'recovery-signed-out'+ | 'recovery-session-inactive'+ | 'recovery-inspection-unavailable'+ | 'recovery-timeout'+ | 'recovery-failed';++export type MutationEnqueueFailure =+ /** The queue may hold the write; the caller cannot safely retry. */+ | 'uncertain'+ | 'storage-changed'+ /** Another context holds the database; the mutation went straight out. */+ | 'owner-lock'+ /** The engine refused the write; the mutation went straight out. */+ | 'rejected';++export type MutationAttemptDetails = {+ failure?: MutationFailureClass;+ /** Sanitized before export; any non-identifier becomes `UNRECOGNIZED`. */+ errorCode?: unknown;+ httpStatus?: number;+ retryDelayMs?: number;+};++export interface MutationAttemptTelemetry {+ /** Runs the network send so its spans join this attempt's trace. */+ run<T>(send: () => T): T;+ sent(mode: 'live' | 'replay'): void;+ /** Ends the attempt; later calls are ignored. */+ finish(+ outcome: MutationAttemptOutcome,+ details?: MutationAttemptDetails+ ): void;+}++export interface MutationEnqueueTelemetry {+ persisted(result: {+ transactionId: string;+ /** Replaced an unsent write for the same record. */+ replaced: boolean;+ /** Claimed as the queue head in the same call. */+ claimed: boolean;+ /** Persisted without its list patches after the engine rejected them. */+ linkPatchesDegraded: boolean;+ }): void;+ failed(reason: MutationEnqueueFailure): void;+}++export type QueuedItemTiming = { enqueuedAtMs?: number };++export interface MutationQueueTelemetry {+ /**+ * Samples the persisted queue once when the runner starts; summaries+ * sample it again through the same callback.+ */+ loaded(+ inspect: () => Promise<ReadonlyArray<QueuedItemTiming> | undefined>+ ): void;+ enqueue(item: {+ uuid?: string;+ operationName?: string;+ }): MutationEnqueueTelemetry;+ attempt(mutation: ClaimedMutation): MutationAttemptTelemetry;+ skipped(reason: MutationQueueSkipReason): void;+ storageReset(): void;+}++export const NOOP_MUTATION_ATTEMPT_TELEMETRY: MutationAttemptTelemetry = {+ run: (send) => send(),+ sent: () => undefined,+ finish: () => undefined,+};++const NOOP_ENQUEUE: MutationEnqueueTelemetry = {+ persisted: () => undefined,+ failed: () => undefined,+};++export const NOOP_MUTATION_QUEUE_TELEMETRY: MutationQueueTelemetry = {+ loaded: () => undefined,+ enqueue: () => NOOP_ENQUEUE,+ attempt: () => NOOP_MUTATION_ATTEMPT_TELEMETRY,+ skipped: () => undefined,+ storageReset: () => undefined,+};++export const MUTATION_QUEUE_SUMMARY_INTERVAL_MS = 5 * 60_000;+/** A tab that only keeps skipping backs its summaries off to this interval. */+const MAX_SUMMARY_INTERVAL_MS = 60 * 60_000;+/** A pending item older than this makes the summary worth reporting. */+const STUCK_ITEM_AGE_MS = 60_000;+/** Bounds per-item log keys for a tab that never settles its items. */+const MAX_LOGGED_KEYS = 1_000;+/** A longer wait means the clock was wrong at enqueue, not a real wait. */+const MAX_PLAUSIBLE_AGE_MS = 365 * 24 * 60 * 60_000;+const MAX_CLEARED_ITEMS = 20;++export type QueueItemAge =+ | { known: true; ageMs: number }+ | { known: false; reason: 'missing' | 'invalid' | 'clock-skew' };++/**+ * Age of a durable enqueue time. Older hosts omit the time and a wall clock+ * can move backwards, so an age is either trustworthy or unknown, never a+ * stand-in zero.+ */+export function queueItemAge(+ enqueuedAtMs: unknown,+ nowMs: number+): QueueItemAge {+ if (enqueuedAtMs === undefined || enqueuedAtMs === null)+ return { known: false, reason: 'missing' };+ if (+ typeof enqueuedAtMs !== 'number' ||+ !Number.isSafeInteger(enqueuedAtMs) ||+ enqueuedAtMs <= 0+ )+ return { known: false, reason: 'invalid' };+ const ageMs = nowMs - enqueuedAtMs;+ if (ageMs < 0 || ageMs > MAX_PLAUSIBLE_AGE_MS)+ return { known: false, reason: 'clock-skew' };+ return { known: true, ageMs };+}++function ageAttributes(+ age: QueueItemAge,+ key: string+): Record<string, string | number | boolean> {+ return age.known+ ? { [key]: age.ageMs, 'mutation_queue.age_known': true }+ : {+ 'mutation_queue.age_known': false,+ 'mutation_queue.age_unknown_reason': age.reason,+ };+}++/** Oldest known age and the count of items whose age is unknown. */+function oldestAge(+ enqueueTimes: Iterable<unknown>,+ nowMs: number+): Record<string, number> {+ let oldest: number | undefined;+ let unknown = 0;+ for (const enqueuedAtMs of enqueueTimes) {+ const age = queueItemAge(enqueuedAtMs, nowMs);+ if (age.known) oldest = Math.max(oldest ?? 0, age.ageMs);+ else unknown += 1;+ }+ const attributes: Record<string, number> = {+ 'mutation_queue.age_unknown_items': unknown,+ };+ if (oldest !== undefined)+ attributes['mutation_queue.oldest_item_age_ms'] = oldest;+ return attributes;+}++const TERMINAL_OUTCOMES: ReadonlySet<MutationAttemptOutcome> = new Set([+ 'committed',+ 'rolled-back',+ 'superseded',+ 'retry-exhausted',+ 'discarded',+ 'recovery-failed',+]);++/** Outcomes that log their first occurrence per item and failure class. */+const LOGGED_OUTCOMES: ReadonlySet<MutationAttemptOutcome> = new Set([+ 'retry-scheduled',+ 'rolled-back',+ 'retry-exhausted',+ 'ack-lost',+ 'recovery-failed',+]);++/** Counters a summary does not report on their own. */+const ROUTINE_COUNTERS: ReadonlySet<string> = new Set([+ 'enqueue.persisted',+ 'attempt.started',+ 'attempt.committed',+ 'attempt.deferred',+ 'attempt.discarded',+ 'attempt.superseded',+]);++const OPERATION_NAME = /^[A-Za-z_][A-Za-z0-9_]{0,99}$/;+const ERROR_CODE = /^[A-Za-z][A-Za-z0-9_]{0,63}$/;++function operationNameAttribute(name: string | undefined): string {+ return name && OPERATION_NAME.test(name) ? name : 'unknown';+}++/** Server codes are identifiers; anything else may carry input text. */+export function errorCodeAttribute(code: unknown): string | undefined {+ if (code === undefined || code === null) return undefined;+ return typeof code === 'string' && ERROR_CODE.test(code)+ ? code+ : 'UNRECOGNIZED';+}++function graphQLErrorCode(error: CombinedError | undefined): unknown {+ return error?.graphQLErrors[0]?.extensions.code;+}++function detailAttributes(+ details: MutationAttemptDetails,+ errorCode: string | undefined+): Record<string, string | number> {+ const attributes: Record<string, string | number> = {};+ if (details.failure) attributes['mutation_queue.failure'] = details.failure;+ if (errorCode) attributes['graphql.error.code'] = errorCode;+ if (details.httpStatus)+ attributes['http.response.status_code'] = details.httpStatus;+ if (details.retryDelayMs !== undefined)+ attributes['mutation_queue.retry_delay_ms'] = details.retryDelayMs;+ return attributes;+}++/** Classifies a failed attempt without retaining its message. */+export function classifyMutationFailure(+ error: CombinedError | undefined,+ httpStatus: number | undefined+): MutationFailureClass {+ if (httpStatus === 401 || httpStatus === 403) return 'auth';+ if (httpStatus === 409) return 'conflict';+ if (httpStatus !== undefined && httpStatus >= 500) return 'server';+ const graphQLError = error?.graphQLErrors[0];+ if (graphQLError) {+ const code = graphQLErrorCode(error);+ const upper = typeof code === 'string' ? code.toUpperCase() : '';+ if (upper.startsWith('LOCAL_')) return 'local';+ if (+ ['UNAUTHENTICATED', 'UNAUTHORIZED', 'FORBIDDEN'].includes(upper) ||+ graphQLError.message === 'authentication required'+ )+ return 'auth';+ if (upper.includes('CONFLICT') || upper.endsWith('ALREADY_SENT'))+ return 'conflict';+ if (upper.startsWith('INTERNAL')) return 'server';+ if (+ upper === 'BAD_USER_INPUT' ||+ upper.startsWith('INVALID') ||+ upper.startsWith('GRAPHQL_') ||+ // Parse and validation errors have neither a code nor a field path.+ (!upper && !graphQLError.path?.length)+ )+ return 'validation';+ return 'graphql';+ }+ if (httpStatus !== undefined && httpStatus >= 400) return 'validation';+ const networkError = error?.networkError;+ if (networkError) {+ return networkError.name === 'TimeoutError' ||+ networkError.name === 'AbortError' ||+ /timed? ?out/i.test(networkError.message)+ ? 'timeout'+ : 'network';+ }+ return 'no-data';+}++/** Failure attributes for an attempt that did not return usable data. */+export function failureDetails(+ error: CombinedError | undefined,+ response: Response | undefined+): MutationAttemptDetails {+ return {+ failure: classifyMutationFailure(error, response?.status),+ errorCode: error?.networkError ? undefined : graphQLErrorCode(error),+ httpStatus: response?.status,+ };+}++/** Maps a recovery failure to a fixed reason; its message is not exported. */+export function recoverySkipReason(error: unknown): MutationQueueSkipReason {+ const message = error instanceof Error ? error.message : String(error);+ if (message.includes('Sign in to save drafts')) return 'recovery-signed-out';+ if (message.includes('no longer active')) return 'recovery-session-inactive';+ if (message.includes('Queue inspection is unavailable'))+ return 'recovery-inspection-unavailable';+ if (message.includes('timed out')) return 'recovery-timeout';+ return 'recovery-failed';+}++export type MutationQueueClearCause = 'logout' | 'session-expired';++/**+ * Reports queued mutations a local sign-out is about to delete, with how long+ * each waited: `mutation_queue.items` lists `operation:age_ms`, oldest first,+ * with `unknown` for an item whose age is unknown.+ */+export function reportMutationQueueCleared(+ queue: ReadonlyArray<{ operationName?: string } & QueuedItemTiming>,+ cause: MutationQueueClearCause,+ nowMs = Date.now()+): void {+ if (queue.length === 0) return;+ const operationNames = [+ ...new Set(queue.map((item) => operationNameAttribute(item.operationName))),+ ].sort();+ const items = queue+ .map((item) => ({+ operationName: operationNameAttribute(item.operationName),+ age: queueItemAge(item.enqueuedAtMs, nowMs),+ }))+ .sort(+ (left, right) =>+ (right.age.known ? right.age.ageMs : -1) -+ (left.age.known ? left.age.ageMs : -1)+ )+ .slice(0, MAX_CLEARED_ITEMS)+ .map(+ ({ operationName, age }) =>+ `${operationName}:${age.known ? age.ageMs : 'unknown'}`+ );+ Telemetry.warn('graphql.mutation_queue.cleared', {+ 'mutation_queue.clear_cause': cause,+ 'mutation_queue.depth': queue.length,+ 'mutation_queue.operation_names': operationNames+ .slice(0, MAX_CLEARED_ITEMS)+ .join(','),+ 'mutation_queue.items': items.join(','),+ ...oldestAge(+ queue.map((item) => item.enqueuedAtMs),+ nowMs+ ),+ });+}++type TrackedItem = {+ firstSeenAt: number;+ /** Claimed from storage written before this runner started. */+ recovered: boolean;+ /** Durable enqueue time from the host, or this tab's enqueue time. */+ enqueuedAtMs?: number;+};++export function createMutationQueueTelemetry({+ environment,+ now = Date.now,+ summaryIntervalMs = MUTATION_QUEUE_SUMMARY_INTERVAL_MS,+}: {+ environment: () => MutationQueueEnvironment;+ now?: () => number;+ summaryIntervalMs?: number;+}): MutationQueueTelemetry {+ const items = new Map<string, TrackedItem>();+ const loggedKeys = new Set<string>();+ const counters = new Map<string, number>();+ let maxAttemptCount = 0;+ let summaryScheduled = false;+ let nextSummaryIntervalMs = summaryIntervalMs;+ let inspectQueue:+ | (() => Promise<ReadonlyArray<QueuedItemTiming> | undefined>)+ | undefined;++ function environmentAttributes(): Record<string, string | boolean> {+ const current = environment();+ return {+ 'auth.state': current.authenticated ? 'signed-in' : 'signed-out',+ 'network.online': current.online,+ 'graphql.websocket.state': current.websocket,+ };+ }++ /**+ * Items this tab enqueued or claimed. `oldest_item_age_ms` comes from+ * durable enqueue times; `oldest_observed_wait_ms` only covers this tab.+ */+ function queueAttributes(): Record<string, number> {+ const at = now();+ let firstSeen: number | undefined;+ for (const item of items.values())+ firstSeen = Math.min(firstSeen ?? item.firstSeenAt, item.firstSeenAt);+ return {+ 'mutation_queue.pending_items': items.size,+ 'mutation_queue.oldest_observed_wait_ms':+ firstSeen === undefined ? 0 : Math.max(0, at - firstSeen),+ ...oldestAge(+ [...items.values()].map((item) => item.enqueuedAtMs),+ at+ ),+ };+ }++ function count(key: string): void {+ counters.set(key, (counters.get(key) ?? 0) + 1);+ if (summaryScheduled) return;+ summaryScheduled = true;+ setTimeout(() => {+ summaryScheduled = false;+ fireAndForget(reportSummary(), 'graphql-cache-mutation-queue');+ }, nextSummaryIntervalMs);+ }++ /** Returns true the first time `key` is seen. */+ function firstOccurrence(key: string): boolean {+ if (loggedKeys.has(key)) return false;+ if (loggedKeys.size >= MAX_LOGGED_KEYS) loggedKeys.clear();+ loggedKeys.add(key);+ return true;+ }++ function forgetItem(uuid: string): void {+ items.delete(uuid);+ for (const key of loggedKeys)+ if (key.startsWith(`${uuid}|`)) loggedKeys.delete(key);+ }++ /** Depth and ages across every persisted item, from storage. */+ async function sampleQueue(): Promise<Record<string, number> | undefined> {+ try {+ const queue = await inspectQueue?.();+ if (!queue) return undefined;+ return {+ 'mutation_queue.depth': queue.length,+ ...oldestAge(+ queue.map((item) => item.enqueuedAtMs),+ now()+ ),+ };+ } catch {+ return undefined;+ }+ }++ async function reportSummary(): Promise<void> {+ const queue = queueAttributes();+ const stuck =+ Math.max(+ queue['mutation_queue.oldest_item_age_ms'] ?? 0,+ queue['mutation_queue.oldest_observed_wait_ms'] ?? 0+ ) >= STUCK_ITEM_AGE_MS;+ const notable = [...counters.keys()].some(+ (key) => !ROUTINE_COUNTERS.has(key)+ );+ const snapshot = new Map(counters);+ counters.clear();+ const intervalMs = nextSummaryIntervalMs;+ // A signed-out or retired tab skips every poll for as long as it stays+ // open; only a change in queue activity resets the interval.+ nextSummaryIntervalMs = [...snapshot.keys()].every((key) =>+ key.startsWith('skip.')+ )+ ? Math.min(intervalMs * 2, MAX_SUMMARY_INTERVAL_MS)+ : summaryIntervalMs;+ if (!notable && !stuck) return;+ const stored = await sampleQueue();+ // Storage covers items this tab never saw, so its ages replace the tab's.+ const {+ 'mutation_queue.oldest_item_age_ms': _trackedOldest,+ 'mutation_queue.age_unknown_items': _trackedUnknown,+ ...observed+ } = queue;+ const attributes: Record<string, string | number | boolean> = {+ ...environmentAttributes(),+ ...(stored ? { ...observed, ...stored } : queue),+ 'mutation_queue.max_attempt_count': maxAttemptCount,+ 'mutation_queue.summary_interval_ms': intervalMs,+ };+ for (const [key, value] of snapshot)+ attributes[`mutation_queue.count.${key}`] = value;+ Telemetry.warn('graphql.mutation_queue.summary', attributes);+ }++ function attempt(mutation: ClaimedMutation): MutationAttemptTelemetry {+ const startedAt = now();+ let item = items.get(mutation.uuid);+ if (!item) {+ item = { firstSeenAt: startedAt, recovered: true };+ items.set(mutation.uuid, item);+ }+ if (mutation.enqueuedAtMs !== undefined)+ item.enqueuedAtMs = mutation.enqueuedAtMs;+ const tracked = item;+ const waited = (at: number) => ({+ 'mutation_queue.observed_wait_ms': at - tracked.firstSeenAt,+ ...ageAttributes(+ queueItemAge(tracked.enqueuedAtMs, at),+ 'mutation_queue.item_age_ms'+ ),+ });+ maxAttemptCount = Math.max(maxAttemptCount, mutation.attemptCount);+ const operationName = operationNameAttribute(mutation.operationName);+ const identity = {+ 'mutation_queue.item_id': mutation.uuid,+ 'mutation_queue.transaction_id': mutation.transactionId,+ 'graphql.operation.name': operationName,+ 'mutation_queue.attempt_count': mutation.attemptCount,+ 'mutation_queue.server_failure_count': mutation.serverFailureCount,+ };+ const claimAttributes = {+ ...identity,+ ...environmentAttributes(),+ ...queueAttributes(),+ 'mutation_queue.superseded': mutation.superseded,+ 'mutation_queue.recovered': tracked.recovered,+ ...waited(startedAt),+ };+ // Opened on first use: a head whose admission keeps deferring is claimed+ // every second and should only count, not emit a span per claim.+ let openedSpan: Span | undefined;+ const span = (): Span => {+ if (openedSpan) return openedSpan;+ openedSpan = Telemetry.span('graphql.mutation_queue.attempt');+ for (const [key, value] of Object.entries(claimAttributes))+ openedSpan.setAttr(key, value);+ return openedSpan;+ };+ count('attempt.started');+ let finished = false;+ return {+ run: (send) => span().run(send),+ sent(mode) {+ span().setAttr('mutation_queue.send_mode', mode);+ span().setAttr('mutation_queue.prepare_ms', now() - startedAt);+ span().event('sent');+ },+ finish(outcome, details = {}) {+ if (finished) return;+ finished = true;+ if (outcome === 'deferred' && !openedSpan) {+ count('attempt.deferred');+ return;+ }+ const finishedAt = now();+ const errorCode = errorCodeAttribute(details.errorCode);+ const age = queueItemAge(tracked.enqueuedAtMs, finishedAt);+ const attributes: Record<string, string | number | boolean> = {+ ...identity,+ 'mutation_queue.outcome': outcome,+ 'mutation_queue.attempt_ms': finishedAt - startedAt,+ ...waited(finishedAt),+ ...detailAttributes(details, errorCode),+ };+ if (TERMINAL_OUTCOMES.has(outcome) && age.known)+ attributes['mutation_queue.end_to_end_ms'] = age.ageMs;+ const attemptSpan = span();+ for (const [key, value] of Object.entries(attributes))+ attemptSpan.setAttr(key, value);+ if (details.failure)+ attemptSpan.error(`mutation ${outcome}: ${details.failure}`);+ count(`attempt.${outcome}`);+ if (details.failure) count(`failure.${details.failure}`);++ const hadLoggedFailure = loggedKeys.has(`${mutation.uuid}|failed`);+ if (+ LOGGED_OUTCOMES.has(outcome) &&+ firstOccurrence(+ `${mutation.uuid}|${outcome}|${details.failure ?? ''}|${errorCode ?? ''}`+ )+ ) {+ loggedKeys.add(`${mutation.uuid}|failed`);+ attemptSpan.run(() =>+ Telemetry.warn(+ `graphql.mutation_queue.${outcome.replaceAll('-', '_')}`,+ {+ ...attributes,+ ...environmentAttributes(),+ 'mutation_queue.recovered': tracked.recovered,+ }+ )+ );+ } else if (outcome === 'committed' && hadLoggedFailure) {+ attemptSpan.run(() =>+ Telemetry.info('graphql.mutation_queue.committed_after_failure', {+ ...attributes,+ ...environmentAttributes(),+ })+ );+ }+ attemptSpan.end();+ if (TERMINAL_OUTCOMES.has(outcome)) forgetItem(mutation.uuid);+ },+ };+ }++ return {+ loaded(inspect) {+ inspectQueue = inspect;+ fireAndForget(+ Telemetry.span('graphql.mutation_queue.loaded', async (span) => {+ const sampled = await sampleQueue();+ span.setAttr('mutation_queue.depth_available', sampled !== undefined);+ if (sampled === undefined) return;+ for (const [key, value] of Object.entries(sampled))+ span.setAttr(key, value);+ if ((sampled['mutation_queue.depth'] ?? 0) > 0)+ Telemetry.info('graphql.mutation_queue.loaded', {+ ...environmentAttributes(),+ ...sampled,+ });+ }),+ 'graphql-cache-mutation-queue'+ );+ },+ enqueue({ uuid, operationName }) {+ const startedAt = now();+ const span = Telemetry.span('graphql.mutation_queue.enqueue');+ span.setAttr(+ 'graphql.operation.name',+ operationNameAttribute(operationName)+ );+ if (uuid) span.setAttr('mutation_queue.item_id', uuid);+ return {+ persisted({ transactionId, replaced, claimed, linkPatchesDegraded }) {+ // A replacement inherits a durable time this tab learns at claim.+ if (uuid && !items.has(uuid))+ items.set(uuid, {+ firstSeenAt: startedAt,+ recovered: false,+ enqueuedAtMs: replaced ? undefined : startedAt,+ });+ span.setAttr('mutation_queue.transaction_id', transactionId);+ span.setAttr('mutation_queue.replaced_pending', replaced);+ span.setAttr('mutation_queue.claimed_on_enqueue', claimed);+ span.setAttr(+ 'mutation_queue.link_patches_degraded',+ linkPatchesDegraded+ );+ if (linkPatchesDegraded) count('enqueue.link-patches-degraded');+ span.setAttr('mutation_queue.enqueue_ms', now() - startedAt);+ for (const [key, value] of Object.entries(queueAttributes()))+ span.setAttr(key, value);+ count('enqueue.persisted');+ span.end();+ },+ failed(reason) {+ span.setAttr('mutation_queue.enqueue_failure', reason);+ span.setAttr('mutation_queue.enqueue_ms', now() - startedAt);+ span.error(`mutation enqueue failed: ${reason}`);+ count(`enqueue.failed.${reason}`);+ if (firstOccurrence(`enqueue|${reason}`))+ span.run(() =>+ Telemetry.warn('graphql.mutation_queue.enqueue_failed', {+ ...environmentAttributes(),+ 'graphql.operation.name': operationNameAttribute(operationName),+ 'mutation_queue.enqueue_failure': reason,+ })+ );+ span.end();+ },+ };+ },+ attempt,+ skipped(reason) {+ count(`skip.${reason}`);+ if (firstOccurrence(`skip|${reason}`))+ Telemetry.warn('graphql.mutation_queue.skipped', {+ ...environmentAttributes(),+ ...queueAttributes(),+ 'mutation_queue.skip_reason': reason,+ });+ },+ storageReset() {+ count('storage-reset');+ items.clear();+ loggedKeys.clear();+ },+ };+}diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.tsindex 5132e0e40f..325f06225f 100644--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts@@ -55,6 +55,7 @@ import {type WriteResult,} from '../protocol';import { entityFromArgument } from './entity-resolvers';+import { NOOP_MUTATION_QUEUE_TELEMETRY } from './mutation-queue-telemetry';import {HYDRATE_ONLY_CONTEXT_KEY,type NormalizedCacheExchangeOptions,@@ -3107,7 +3108,7 @@ describe('normalizedCacheExchange', () => {'replays unrelated mutations when recovery initialization %s',async (failure) => {vi.useFakeTimers();- const warning = vi.spyOn(console, 'warn').mockImplementation(() => {});+ const skipped = vi.fn();try {host.seedQueued({uuid: 'unrelated',@@ -3122,29 +3123,41 @@ describe('normalizedCacheExchange', () => {throw new Error('IndexedDB unavailable');await new Promise<void>(() => {});},+ mutationQueueTelemetry: {+ ...NOOP_MUTATION_QUEUE_TELEMETRY,+ skipped,+ },});await vi.advanceTimersByTimeAsync(4_010);expect(forwarded).toHaveLength(1);expect(host.commits).toMatchObject([{ transactionId: '1' }]);- expect(warning).toHaveBeenCalled();+ expect(skipped).toHaveBeenCalledWith(+ failure === 'rejects' ? 'recovery-failed' : 'recovery-timeout'+ );} finally {- warning.mockRestore();vi.useRealTimers();}});- it('warns once per recovery failure while the queue keeps polling', async () => {+ it('reports each recovery failure while the queue keeps polling', async () => {vi.useFakeTimers();const warning = vi.spyOn(console, 'warn').mockImplementation(() => {});+ const skipped = vi.fn();try {const prepareMutationQueue = vi.fn(async () => {throw new Error('Queue inspection is unavailable');});- harness(host, undefined, { prepareMutationQueue });+ harness(host, undefined, {+ prepareMutationQueue,+ mutationQueueTelemetry: { ...NOOP_MUTATION_QUEUE_TELEMETRY, skipped },+ });await vi.advanceTimersByTimeAsync(90_010);expect(prepareMutationQueue).toHaveBeenCalledTimes(4);- expect(warning).toHaveBeenCalledOnce();+ expect(skipped.mock.calls).toEqual(+ Array(4).fill(['recovery-inspection-unavailable'])+ );+ expect(warning).not.toHaveBeenCalled();} finally {warning.mockRestore();vi.useRealTimers();diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.tsindex 51fc85772c..89b1b07749 100644--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts@@ -96,6 +96,10 @@ import {replayDocument,withQueueRequestTimeout,} from './mutation-queue-runner';+import {+ type MutationQueueTelemetry,+ NOOP_MUTATION_QUEUE_TELEMETRY,+} from './mutation-queue-telemetry';import type { QueryRevalidation } from './optimistic';import {normalizedEntityKey,@@ -419,6 +423,8 @@ export interface NormalizedCacheExchangeOptions {* a later background attempt. Defaults to `false`.*/shouldRetryMutation?: (error: CombinedError) => boolean | Promise<boolean>;+ /** Spans, bounded logs, and counters for the durable mutation queue. */+ mutationQueueTelemetry?: MutationQueueTelemetry;}export function normalizedCacheExchange(@@ -847,6 +853,8 @@ export function normalizedCacheExchange(};const queueOwner = `exchange:${host.clientId}`;+ const queueTelemetry =+ options.mutationQueueTelemetry ?? NOOP_MUTATION_QUEUE_TELEMETRY;const mutationQueue = createMutationQueueRunner({host,client,@@ -1067,18 +1075,26 @@ export function normalizedCacheExchange(nowMs: now,leaseExpiresAtMs: now + QUEUE_LEASE_MS,};+ const enqueueTelemetry = queueTelemetry.enqueue({+ uuid: args.uuid,+ operationName: args.operationName,+ });let enqueue: EnqueueOptimisticMutationResult;+ let linkPatchesDegraded = false;try {enqueue = await host.enqueueOptimisticMutation(args, claim);} catch (error) {- if (!mutationQueue.isCurrent(epoch))+ if (!mutationQueue.isCurrent(epoch)) {+ enqueueTelemetry.failed('storage-changed');return uncertainEnqueueResult(op,new Error('cache storage changed during enqueue', {cause: error,}));+ }if (isAdmittedEnqueueUncertainError(error)) {+ enqueueTelemetry.failed('uncertain');options.onCacheError?.(error, op);// The old-scope queue may already contain the side effect. It is// unsafe to forward or retry without a coordinator fence.@@ -1087,6 +1103,7 @@ export function normalizedCacheExchange(// Another context holds the database, so the enqueue never reached// an engine. Send it as a disabled cache would.if (isOwnerLockUnavailableError(error)) {+ enqueueTelemetry.failed('owner-lock');enqueueForward(op);return undefined;}@@ -1094,10 +1111,12 @@ export function normalizedCacheExchange(// not expose a partial relation move: retain entity optimism and// the post-success revalidation descriptors instead.if (args.linkPatches.length === 0) {+ enqueueTelemetry.failed('rejected');options.onCacheError?.(error, op);enqueueForward(op);return undefined;}+ linkPatchesDegraded = true;// Deliberately loud: this fallback is durable but invisible until// reconnect — the mutation's optimistic list membership silently// becomes eventual. A systematic rejection (schema drift, a patch@@ -1131,27 +1150,38 @@ export function normalizedCacheExchange(claim);} catch (fallbackError) {- if (!mutationQueue.isCurrent(epoch))+ if (!mutationQueue.isCurrent(epoch)) {+ enqueueTelemetry.failed('storage-changed');return uncertainEnqueueResult(op,new Error('cache storage changed during enqueue', {cause: fallbackError,}));+ }options.onCacheError?.(fallbackError, op);if (isAdmittedEnqueueUncertainError(fallbackError)) {+ enqueueTelemetry.failed('uncertain');return uncertainEnqueueResult(op, fallbackError);}+ enqueueTelemetry.failed('rejected');enqueueForward(op);return undefined;}}if (!mutationQueue.isCurrent(epoch)) {+ enqueueTelemetry.failed('storage-changed');return uncertainEnqueueResult(op,new Error('cache storage changed during enqueue'));}+ enqueueTelemetry.persisted({+ transactionId: enqueue.transactionId,+ replaced: enqueue.upsertKind.kind === 'replaced-pending',+ claimed: enqueue.initialClaim.kind === 'claimed',+ linkPatchesDegraded,+ });return await mutationQueue.admit(op, enqueue, epoch);}diff --git a/apps/web/src/lib/graphql-cache/protocol.ts b/apps/web/src/lib/graphql-cache/protocol.tsindex 6a00ae03e0..4686ae96f8 100644--- a/apps/web/src/lib/graphql-cache/protocol.ts+++ b/apps/web/src/lib/graphql-cache/protocol.ts@@ -656,6 +656,7 @@ export type MutationInspection = Pick<| 'operationName'| 'variables'| 'clientMetadata'+ | 'enqueuedAtMs'> & { optimisticData: unknown };/** Claimed strict queue head, ready to be forwarded through urql. */@@ -675,6 +676,12 @@ export type ClaimedMutation = {attemptCount: number;/** Retryable server failures; transport failures do not consume this budget. */serverFailureCount: number;+ /**+ * Durable first-enqueue wall-clock time in epoch ms, kept when a write+ * replaces an unsent one for the same UUID. Native claims and hosts built+ * before this field omit it, so it is unknown rather than zero.+ */+ enqueuedAtMs?: number;};/** Outcome of the strict-head claim attempted immediately after enqueue. */diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.tsindex d5aa111ec9..383b43879e 100644--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts@@ -357,6 +357,8 @@ const mocks = vi.hoisted(() => {platformFetch,toastFailure,telemetryError: vi.fn(),+ telemetryWarn: vi.fn(),+ traceparent: '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01',normalizedCacheExchange: vi.fn((host: unknown, _options?: NormalizedCacheExchangeOptions) => ({kind: 'cache',@@ -440,10 +442,14 @@ vi.mock('@graphql-cache/exchange/normalized-cache-exchange', () => ({vi.mock('@macro-inc/observability', () => ({Telemetry: {error: mocks.telemetryError,+ warn: mocks.telemetryWarn,span: (_name: string, work: (span: unknown) => Promise<unknown>) =>work({event: vi.fn(),setAttr: vi.fn(),+ injectTraceHeaders: (headers: Record<string, string>) => {+ headers.traceparent = mocks.traceparent;+ },}),},}));@@ -808,6 +814,72 @@ describe('GraphQL Soup browser cache session gate', () => {expect(payload.query).toContain('cacheProjection');expect(payload.query).toContain('@include');expect(payload.variables).toEqual({ includeId: true });+ expect(new Headers(transport.headers).get('traceparent')).toBe(+ mocks.traceparent+ );+ });++ it('reports the queued mutations a sign-out deletes', async () => {+ const now = Date.now();+ mocks.host.inspectMutations.mockResolvedValueOnce([+ {+ operationName: 'SaveEmailDraft',+ variables: { body: 'secret' },+ enqueuedAtMs: now - 5_000,+ },+ { operationName: 'SaveEmailDraft', variables: { body: 'secret' } },+ {+ operationName: 'SetEntityProperty',+ variables: {},+ enqueuedAtMs: now - 90_000,+ },+ ] as never);+ const soup = await import('./graphql-soup');+ soup.getGraphqlSoupClient();++ await soup.recordGraphqlMutationQueueCleared('session-expired');++ expect(mocks.telemetryWarn).toHaveBeenCalledWith(+ 'graphql.mutation_queue.cleared',+ {+ 'mutation_queue.clear_cause': 'session-expired',+ 'mutation_queue.depth': 3,+ 'mutation_queue.operation_names': 'SaveEmailDraft,SetEntityProperty',+ 'mutation_queue.items': expect.stringMatching(+ /^SetEntityProperty:9\d{4},SaveEmailDraft:\d+,SaveEmailDraft:unknown$/+ ),+ 'mutation_queue.oldest_item_age_ms': expect.any(Number),+ 'mutation_queue.age_unknown_items': 1,+ }+ );+ const exported = JSON.stringify(mocks.telemetryWarn.mock.calls);+ expect(exported).not.toContain('secret');+ });++ it('never rejects or stalls sign-out when queue inspection fails', async () => {+ const soup = await import('./graphql-soup');+ soup.getGraphqlSoupClient();++ mocks.host.inspectMutations.mockImplementationOnce(() => {+ throw new Error('worker gone');+ });+ await expect(+ soup.recordGraphqlMutationQueueCleared('logout')+ ).resolves.toBeUndefined();++ vi.useFakeTimers();+ try {+ mocks.host.inspectMutations.mockReturnValueOnce(new Promise(() => {}));+ const pending = soup.recordGraphqlMutationQueueCleared('logout');+ await vi.advanceTimersByTimeAsync(1000);+ await expect(pending).resolves.toBeUndefined();+ } finally {+ vi.useRealTimers();+ }+ expect(mocks.telemetryWarn).not.toHaveBeenCalledWith(+ 'graphql.mutation_queue.cleared',+ expect.anything()+ );});it.each([diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.tsindex 2307c3d5a6..50c3a0fdfa 100644--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts@@ -11,6 +11,12 @@ import { fireAndForget } from '@core/util/fireAndForget';import { isTauri } from '@core/util/platform';import { platformFetch } from '@core/util/platformFetch';import { reloadForNewerBuild } from '@core/util/reloadForNewerBuild';+import {+ createMutationQueueTelemetry,+ type MutationQueueClearCause,+ type MutationQueueEnvironment,+ reportMutationQueueCleared,+} from '@graphql-cache/exchange/mutation-queue-telemetry';import { networkRevalidationExchange } from '@graphql-cache/exchange/network-revalidation-exchange';import {HYDRATE_ONLY_CONTEXT_KEY,@@ -48,6 +54,7 @@ import type { GithubPrMentionLocation } from '@service-notification/generated/scimport type { GithubPrReviewState } from '@service-notification/generated/schemas/githubPrReviewState';import type { NotifEvent } from '@service-notification/generated/schemas/notifEvent';import type { NotificationDocumentSubType } from '@service-notification/generated/schemas/notificationDocumentSubType';+import { raceTimeout } from '@solid-primitives/promise';import {type AnyVariables,type Client,@@ -106,6 +113,7 @@ import {} from './graphql-soup-websocket';const dssHost = SERVER_HOSTS['document-storage-service'];+const QUEUE_CLEAR_INSPECTION_TIMEOUT_MS = 1000;function mergeHeaders(...headers: Array<HeadersInit | undefined>): Headers {const result = new Headers();@@ -313,7 +321,15 @@ export async function dssGraphqlFetch(return Telemetry.span('graphql.transport', async (span) => {const started = performance.now();try {- const transportInit = graphqlSoupTransportRequest(init);+ const traceHeaders: Record<string, string> = {};+ span.injectTraceHeaders(traceHeaders);+ const requestInit = graphqlSoupTransportRequest(init);+ // Links this request, and a queued mutation attempt above it, to the+ // server's request and `graphql.execute` spans.+ const transportInit: RequestInit = {+ ...requestInit,+ headers: mergeHeaders(requestInit?.headers, traceHeaders),+ };span.event('request_dispatch');const response = await authorizedDssGraphqlFetch(input, transportInit);span.setAttr('graphql.response_headers_ms', performance.now() - started);@@ -381,7 +397,9 @@ export function subscribeGraphqlSoupReconnected(}function createGraphqlSoupWebSocketClient(- onConnected: () => void+ onConnected: () => void,+ onState: (state: MutationQueueEnvironment['websocket']) => void = () =>+ undefined): GraphqlWsClient {let connections = 0;const resolveWebSocketUrl = createGraphqlSoupWebSocketUrlResolver({@@ -401,7 +419,10 @@ function createGraphqlSoupWebSocketClient(retryAttempts: SOUP_GRAPHQL_WEBSOCKET_RETRY_ATTEMPTS,retryWait: waitToRetryGraphqlSoupWebSocket,on: {+ connecting: () => onState('connecting'),+ closed: () => onState('disconnected'),connected: () => {+ onState('connected');onConnected();connections += 1;if (connections === 1) return;@@ -597,6 +618,26 @@ function fallbackAfterInitializationFailure(error: unknown): void {setCacheAvailability((version) => version + 1);}+/**+ * Reports the queued mutations a local sign-out deletes. Start it before the+ * cache wipe; it never delays or fails the wipe.+ */+export async function recordGraphqlMutationQueueCleared(+ cause: MutationQueueClearCause+): Promise<void> {+ const host = cachedCacheHost;+ if (!host?.inspectMutations || host.disabled) return;+ try {+ const queue = await raceTimeout(+ host.inspectMutations(),+ QUEUE_CLEAR_INSPECTION_TIMEOUT_MS+ );+ if (queue) reportMutationQueueCleared(queue, cause);+ } catch {+ // A rejected entry makes sign-out rotate the cache scope.+ }+}+/** Returns the persistent normalized-cache host after client initialization. */export function getGraphqlCacheHost(): CacheHost | undefined {cacheAvailability();@@ -720,10 +761,21 @@ export function getGraphqlSoupClient(): Client {onSuperseded: () => reloadForNewerBuild(),}));+ let websocketState: MutationQueueEnvironment['websocket'] = 'none';const graphqlWsClient = createGraphqlSoupWebSocketClient(- subscriptionsLifecycle.connected+ subscriptionsLifecycle.connected,+ (state) => {+ websocketState = state;+ });websocketClient = graphqlWsClient;+ const mutationQueueTelemetry = createMutationQueueTelemetry({+ environment: () => ({+ authenticated: untrack(sessionActive),+ online: navigator.onLine,+ websocket: websocketState,+ }),+ });const client = createClient({url: `${dssHost}/items/soup/graphql`,// See graphqlSoupClient: GET serves GraphiQL on this path.@@ -756,6 +808,7 @@ export function getGraphqlSoupClient(): Client {?.id,// Preserve optimistic intent only while transport failures retry.shouldRetryMutation: shouldRetryGraphqlMutation,+ mutationQueueTelemetry,delegateRevalidation: delegateChannelNotificationRefresh,}),graphqlSoupSubscriptionExchange(graphqlWsClient),diff --git a/crates/client/cache-core/AGENTS.md b/crates/client/cache-core/AGENTS.mdindex 94c4729ce8..8f148654c8 100644--- a/crates/client/cache-core/AGENTS.md+++ b/crates/client/cache-core/AGENTS.md@@ -60,6 +60,10 @@ GraphQL response shape for callers. A concrete `Storage` impl (e.g.- Opaque JSON scalars must preserve floating-point values on reparse — `serde_json`is pulled in with the `float_roundtrip` feature; don't drop it.+- `StoredMutation.created_at_ms` is the first-enqueue time hosts report as+ `enqueuedAtMs`: a row replacing an unsent row for the same UUID inherits the+ earlier time. Lease and backoff checks take `now_ms` separately; never read+ `created_at_ms` as the current time.- Patches structural query changes in place instead of resending full results(#7825); a new query shape change needs a patch path, not just a resend.diff --git a/crates/client/cache-core/src/engine.rs b/crates/client/cache-core/src/engine.rsindex d718014456..e3c3e6e737 100644--- a/crates/client/cache-core/src/engine.rs+++ b/crates/client/cache-core/src/engine.rs@@ -1866,12 +1866,19 @@ impl<S: Storage> Engine<S> {queued.mutation.attempt_count > 0,&queued.optimistic.optimistic_data_json,),+ queued.mutation.created_at_ms,)});let expected_kind = match collision {None => MutationUpsertKind::Inserted,- Some((active_id, true)) => MutationUpsertKind::AppendedAfterActive { active_id },- Some((removed_id, false)) => MutationUpsertKind::ReplacedPending { removed_id },+ Some((active_id, true, _)) => MutationUpsertKind::AppendedAfterActive { active_id },+ Some((removed_id, false, _)) => MutationUpsertKind::ReplacedPending { removed_id },+ };+ // The removed row never reached the server, so its wait carries over;+ // an active row stays queued and keeps reporting its own.+ let enqueued_at_ms = match collision {+ Some((_, false, replaced_at_ms)) => replaced_at_ms.min(created_at_ms),+ _ => created_at_ms,};let patches = deduplicate_patches(&self.schema, link_patches)?;@@ -1908,7 +1915,7 @@ impl<S: Storage> Engine<S> {variables_json: canonical_json(&Json::Object(variables.clone())),identity,},- created_at_ms,+ enqueued_at_ms,),optimistic: PersistedOptimisticLayer {optimistic_data_json: encode_optimistic_source(&source),diff --git a/crates/client/cache-core/src/queue.rs b/crates/client/cache-core/src/queue.rsindex ee94008a6d..e4f7b01517 100644--- a/crates/client/cache-core/src/queue.rs+++ b/crates/client/cache-core/src/queue.rs@@ -50,11 +50,23 @@ pub struct StoredMutation {pub lease_expires_at_ms: Option<i64>,/// Last retryable network error, for diagnostics.pub last_error: Option<String>,- /// Wall-clock enqueue time, for diagnostics.+ /// Wall-clock time the write was first enqueued, for diagnostics. A row+ /// replacing an unsent row for the same UUID keeps the earlier time.pub created_at_ms: i64,}+/// Largest integer a JavaScript number represents exactly.+const MAX_SAFE_JS_INTEGER: i64 = (1 << 53) - 1;+impl StoredMutation {+ /// Enqueue time for JavaScript callers, or `None` when a number cannot+ /// carry it exactly.+ pub fn enqueued_at_ms(&self) -> Option<i64> {+ (0..=MAX_SAFE_JS_INTEGER)+ .contains(&self.created_at_ms)+ .then_some(self.created_at_ms)+ }+/// Creates an unclaimed queue row for `request`.pub fn new(request: MutationRequest, created_at_ms: i64) -> Self {Self {@@ -333,6 +345,9 @@ pub struct MutationInspection {pub client_metadata: Option<Json>,/// Original optimistic response for migrating durable local intent.pub optimistic_data: Json,+ /// Wall-clock first-enqueue time in epoch milliseconds.+ #[serde(default, skip_serializing_if = "Option::is_none")]+ pub enqueued_at_ms: Option<i64>,}impl TryFrom<QueuedMutation> for MutationInspection {@@ -340,6 +355,7 @@ impl TryFrom<QueuedMutation> for MutationInspection {fn try_from(queued: QueuedMutation) -> Result<Self, Self::Error> {let source = decode_optimistic_source(&queued.optimistic.optimistic_data_json)?;+ let enqueued_at_ms = queued.mutation.enqueued_at_ms();let request = queued.mutation.request;Ok(Self {transaction_id: queued.id.to_string(),@@ -351,6 +367,7 @@ impl TryFrom<QueuedMutation> for MutationInspection {.map_err(|error| error.to_string())?,client_metadata: source.client_metadata,optimistic_data: source.mutation_data,+ enqueued_at_ms,})}}diff --git a/crates/client/cache-core/tests/optimistic.rs b/crates/client/cache-core/tests/optimistic.rsindex 6ede1f794c..498db3fe97 100644--- a/crates/client/cache-core/tests/optimistic.rs+++ b/crates/client/cache-core/tests/optimistic.rs@@ -1107,6 +1107,74 @@ fn pending_uuid_replacement_moves_to_tail_and_keeps_only_latest_layer() {});}+#[test]+fn pending_uuid_replacement_keeps_the_first_enqueue_time() {+ block_on(async {+ const X: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a01";+ const Y: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a02";+ let mut engine = engine_with_base("Status", "todo").await;+ begin_value(&mut engine, X, "a", 100).await;+ begin_value(&mut engine, X, "b", 250).await;+ // A clock that moved backwards cannot make the replacement look newer.+ begin_value(&mut engine, Y, "c", 500).await;+ begin_value(&mut engine, Y, "d", 400).await;++ let queued = engine.storage().load_mutation_queue().await.unwrap();+ assert_eq!(+ queued+ .iter()+ .map(|row| row.mutation.created_at_ms)+ .collect::<Vec<_>>(),+ [100, 400]+ );+ let inspected = engine.inspect_mutations().await.unwrap();+ assert_eq!(+ inspected+ .iter()+ .map(|item| item.enqueued_at_ms)+ .collect::<Vec<_>>(),+ [Some(100), Some(400)]+ );+ let wire = serde_json::to_value(&inspected[0]).unwrap();+ assert_eq!(wire["enqueuedAtMs"], json!(100));+ });+}++#[test]+fn active_uuid_replacement_keeps_its_own_enqueue_time() {+ block_on(async {+ const X: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a03";+ let mut engine = engine_with_base("Status", "todo").await;+ begin_value(&mut engine, X, "a", 100).await;+ let (_, _) = claim_head(&mut engine, "runner", 110).await;+ begin_value(&mut engine, X, "b", 300).await;++ let queued = engine.storage().load_mutation_queue().await.unwrap();+ assert!(queued[0].superseded);+ assert_eq!(+ queued+ .iter()+ .map(|row| row.mutation.created_at_ms)+ .collect::<Vec<_>>(),+ [100, 300]+ );+ });+}++#[test]+fn enqueue_time_outside_the_javascript_safe_range_is_unknown() {+ block_on(async {+ const X: &str = "0f0b1a43-1d47-4a52-9b8f-3a1e2f6f7a04";+ let mut engine = engine_with_base("Status", "todo").await;+ begin_value(&mut engine, X, "a", 1 << 60).await;++ let inspected = engine.inspect_mutations().await.unwrap();+ assert_eq!(inspected[0].enqueued_at_ms, None);+ let wire = serde_json::to_value(&inspected[0]).unwrap();+ assert!(wire.get("enqueuedAtMs").is_none());+ });+}+#[test]fn pending_replacement_restores_fields_omitted_by_the_new_intent() {block_on(async {diff --git a/crates/client/cache-wasm/src/shell.rs b/crates/client/cache-wasm/src/shell.rsindex a5ad061cee..948f6e8bc5 100644--- a/crates/client/cache-wasm/src/shell.rs+++ b/crates/client/cache-wasm/src/shell.rs@@ -253,6 +253,8 @@ struct JsClaimedMutation {identity: Option<String>,attempt_count: u32,server_failure_count: u32,+ #[serde(skip_serializing_if = "Option::is_none")]+ enqueued_at_ms: Option<i64>,}impl TryFrom<ClaimedMutation> for JsClaimedMutation {@@ -265,6 +267,7 @@ impl TryFrom<ClaimedMutation> for JsClaimedMutation {).map_err(err_js)?.client_metadata;+ let enqueued_at_ms = claimed.queued.mutation.enqueued_at_ms();let request = claimed.queued.mutation.request;Ok(Self {client_metadata,@@ -279,6 +282,7 @@ impl TryFrom<ClaimedMutation> for JsClaimedMutation {identity: request.identity,attempt_count: claimed.queued.mutation.attempt_count,server_failure_count: claimed.queued.mutation.server_failure_count,+ enqueued_at_ms,})}}diff --git a/crates/graphql_entity_mutation/Cargo.toml b/crates/graphql_entity_mutation/Cargo.tomlindex baf0216a40..052b5b32f4 100644--- a/crates/graphql_entity_mutation/Cargo.toml+++ b/crates/graphql_entity_mutation/Cargo.toml@@ -12,9 +12,11 @@ graphql_permission = { path = "../graphql_permission" }graphql_soup = { path = "../graphql_soup" }model-entity = { path = "../model-entity" }models_permissions = { path = "../models_permissions" }+tracing = { workspace = true }workspace-hack = { version = "0.1", path = "../workspace-hack" }[dev-dependencies]macro_user_id = { path = "../macro_user_id" }tokio = { workspace = true }+tracing-subscriber = { workspace = true }uuid = { workspace = true }diff --git a/crates/graphql_entity_mutation/src/mutations.rs b/crates/graphql_entity_mutation/src/mutations.rsindex 8a236d7197..f6e56d8467 100644--- a/crates/graphql_entity_mutation/src/mutations.rs+++ b/crates/graphql_entity_mutation/src/mutations.rs@@ -313,7 +313,7 @@ fn validate_share_policy_inputs(}/// Stable machine-readable mutation error code.-#[derive(Clone, Copy, Enum, Eq, PartialEq)]+#[derive(Clone, Copy, Debug, Enum, Eq, PartialEq)]pub enum GraphqlEntityMutationErrorCode {/// Operation is unavailable for the requested entity kind.UnsupportedOperation,@@ -463,7 +463,21 @@ pub struct EntityMutationPayload<E> {impl<E> EntityMutationPayload<E> {/// Construct a batch payload from ordered domain mutation outcomes.+ ///+ /// Per-item failures are successful GraphQL responses, so the resolver+ /// span records them for the batch.fn new(results: Vec<MutateEntitiesResult>) -> Self {+ let span = tracing::Span::current();+ let mut failures = results.iter().filter_map(|result| result.as_ref().err());+ if let Some(first) = failures.next() {+ span.record("entity_mutation.failed_count", failures.count() + 1);+ span.record(+ "entity_mutation.error_code",+ tracing::field::debug(GraphqlEntityMutationErrorCode::new(*first)),+ );+ } else {+ span.record("entity_mutation.failed_count", 0);+ }EntityMutationPayload {results,edges: PhantomData,@@ -499,6 +513,15 @@ fn mutation_actor(ctx: &Context<'_>) -> async_graphql::Result<EntityMutationActo#[Object]impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {/// Rename heterogeneous entities in one request.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = inputs.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn rename_entities(&self,ctx: &Context<'_>,@@ -522,6 +545,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {}/// Move heterogeneous entities to a project or to the root.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = inputs.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn move_entities(&self,ctx: &Context<'_>,@@ -545,6 +577,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {}/// Update link, explicit team, and channel share policies across supported entity kinds.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = inputs.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn update_entity_share_policies(&self,ctx: &Context<'_>,@@ -569,6 +610,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {}/// Soft-delete entities with a reversible trash lifecycle.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = entities.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn trash_entities(&self,ctx: &Context<'_>,@@ -589,6 +639,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {}/// Restore reversibly deleted entities.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = entities.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn restore_entities(&self,ctx: &Context<'_>,@@ -609,6 +668,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {}/// Irreversibly delete entities.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = entities.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn delete_entities_permanently(&self,ctx: &Context<'_>,@@ -632,6 +700,15 @@ impl<S: EntityMutationService, E: SoupEntityEdges> EntityMutationRoot<S, E> {}/// Duplicate supported entities.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(+ entity_mutation.count = inputs.len(),+ entity_mutation.failed_count = tracing::field::Empty,+ entity_mutation.error_code = tracing::field::Empty,+ )+ )]async fn duplicate_entities(&self,ctx: &Context<'_>,diff --git a/crates/graphql_entity_mutation/src/mutations/test.rs b/crates/graphql_entity_mutation/src/mutations/test.rsindex 9d858483f0..a0d6a7503f 100644--- a/crates/graphql_entity_mutation/src/mutations/test.rs+++ b/crates/graphql_entity_mutation/src/mutations/test.rs@@ -1,4 +1,4 @@-use std::sync::Arc;+use std::sync::{Arc, Mutex};use async_graphql::{Context, EmptySubscription, MaybeUndefined, Object, Request, Schema, SimpleObject, value,@@ -11,6 +11,14 @@ use graphql_soup::SoupEntityEdges;use macro_user_id::user_id::MacroUserIdStr;use model_entity::{Entity, EntityType};use models_permissions::share_permission::{LinkShare, access_level::AccessLevel};+use tracing::{+ field::{Field, Visit},+ span::{Attributes, Id, Record},+};+use tracing_subscriber::{+ Layer, Registry,+ layer::{Context as LayerContext, SubscriberExt},+};/// Minimal composed Soup edge object used by the isolated mutation schema.#[derive(Clone, SimpleObject)]@@ -456,3 +464,78 @@ fn share_policy_validation_requires_access_levels_for_channel_grants() {let error = super::validate_share_policy_inputs(&inputs).unwrap_err();assert!(error.message.contains("accessLevel is required"));}++/// Captures every span field value, including values recorded after creation.+#[derive(Clone, Default)]+struct RecordedSpanFields(Arc<Mutex<Vec<(String, String)>>>);++impl RecordedSpanFields {+ fn get(&self, name: &str) -> Option<String> {+ self.0+ .lock()+ .unwrap()+ .iter()+ .rev()+ .find(|(field, _)| field == name)+ .map(|(_, value)| value.clone())+ }+}++impl Visit for RecordedSpanFields {+ fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {+ self.0+ .lock()+ .unwrap()+ .push((field.name().to_string(), format!("{value:?}")));+ }+}++impl<S: tracing::Subscriber> Layer<S> for RecordedSpanFields {+ fn on_new_span(&self, attrs: &Attributes<'_>, _id: &Id, _ctx: LayerContext<'_, S>) {+ attrs.record(&mut self.clone());+ }++ fn on_record(&self, _id: &Id, values: &Record<'_>, _ctx: LayerContext<'_, S>) {+ values.record(&mut self.clone());+ }+}++#[tokio::test]+async fn batch_span_records_per_item_failures() {+ let fields = RecordedSpanFields::default();+ let _guard = tracing::subscriber::set_default(Registry::default().with(fields.clone()));+ let actor = EntityMutationActor {+ user_id: MacroUserIdStr::parse_from_str("macro|[email protected]").unwrap(),+ organization_id: Some(42),+ };+ let response = Schema::build(+ QueryRoot,+ crate::EntityMutationRoot::<UnavailableEntityMutationService, TestSoupEdges>::new(),+ EmptySubscription,+ )+ .finish()+ .execute(+ Request::new(+ r#"mutation {+ trashEntities(entities: [+ { type: DOCUMENT, id: "document-1" }+ { type: DOCUMENT, id: "document-2" }+ ]) { results { __typename } }+ }"#,+ )+ .data(Arc::new(UnavailableEntityMutationService))+ .data(actor),+ )+ .await;++ assert!(response.errors.is_empty(), "{:?}", response.errors);+ assert_eq!(fields.get("entity_mutation.count").as_deref(), Some("2"));+ assert_eq!(+ fields.get("entity_mutation.failed_count").as_deref(),+ Some("2")+ );+ assert_eq!(+ fields.get("entity_mutation.error_code").as_deref(),+ Some("UnsupportedOperation")+ );+}diff --git a/crates/graphql_notification/src/mutations.rs b/crates/graphql_notification/src/mutations.rsindex 8727daa014..cb5285167c 100644--- a/crates/graphql_notification/src/mutations.rs+++ b/crates/graphql_notification/src/mutations.rs@@ -181,6 +181,11 @@ whereS: NotificationMutationService,{/// Update authenticated user-owned notifications and return authoritative rows.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(notification_count = input.notification_ids.len())+ )]async fn update_notifications(&self,ctx: &Context<'_>,@@ -202,6 +207,11 @@ where}/// Update all notifications associated with the requested entities for the authenticated user.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(entity_count = input.entities.len())+ )]async fn update_notifications_for_entity(&self,ctx: &Context<'_>,diff --git a/crates/graphql_properties/src/mutations.rs b/crates/graphql_properties/src/mutations.rsindex b57ccaf80b..525128d674 100644--- a/crates/graphql_properties/src/mutations.rs+++ b/crates/graphql_properties/src/mutations.rs@@ -434,6 +434,7 @@ whereT: EntityPropertyWriter,{/// Remove a property assignment and identify its normalized cache record.+ #[tracing::instrument(skip_all, err(Debug))]async fn delete_entity_property(&self,ctx: &Context<'_>,@@ -454,6 +455,7 @@ where}/// Set or attach one property on an entity.+ #[tracing::instrument(skip_all, err(Debug))]async fn set_entity_property(&self,ctx: &Context<'_>,@@ -481,6 +483,11 @@ where}/// Add and remove options across one entity's multi-select properties.+ #[tracing::instrument(+ skip_all,+ err(Debug),+ fields(property_count = input.properties.len())+ )]async fn update_entity_property_options(&self,ctx: &Context<'_>,diff --git a/deny.toml b/deny.tomlindex 4c691e2a21..d4e74eb4ea 100644--- a/deny.toml+++ b/deny.toml@@ -229,6 +229,7 @@ deny = ["documents", # Dev logging only"email", # Dev logging only"genai_telemetry", # GenAI span tests+ "graphql_entity_mutation", # Batch failure span field tests only"lambda_runtime_api_client", # External crate allowed to use tracing-subscriber"loom", # External concurrency testing dependency"macro_entrypoint",diff --git a/docs/agent-guide/email.md b/docs/agent-guide/email.mdindex 0ea3b058b6..9b94f80896 100644--- a/docs/agent-guide/email.md+++ b/docs/agent-guide/email.md@@ -547,7 +547,10 @@ server draft exists. If per-mutation recovery preparation fails or times out,the mutation must fail and release the queue too, including legacy drafts thathave not been copied into recovery storage. Existing local working copies remain;an unmigrated legacy edit can be lost if the queue held its only durable copy.-Verify that an unrelated queued action still completes in both cases.+Verify that an unrelated queued action still completes in both cases. With+browser OTel enabled, the failure appears once per draft as+`graphql.mutation_queue.recovery_failed` (or `graphql.mutation_queue.skipped` with a+`mutation_queue.skip_reason`), never as a line per poll.Repeat with two tabs, a save response arriving after a neweredit, an attachment upload completing during another local save, and Discardwhile an attachment snapshot is still being saved. A snapshot based on an older