Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
Seanaye/test/cache startup retry (#8921)
Sean Ayeb94c22648792
diff --git a/apps/web/AGENTS.md b/apps/web/AGENTS.mdindex 0b32b40b49..fdb72f1d5c 100644--- a/apps/web/AGENTS.md+++ b/apps/web/AGENTS.md@@ -194,7 +194,9 @@ ownership. Cancel, logout, and the ten-minute timeout retire pending callbacks.even with retained user data; preserve session state on transient failures.- Treat a repeated worker-ready frame as a coordinator restart: retire the oldengine and re-register in place, preserving durable storage and never replaying- uncertain enqueues. See `docs/graphql-normalized-cache-plan.md`.+ uncertain enqueues. Space failed coordinator registrations with bounded,+ cancelable waits; a missing startup frame is not proof of untouched storage.+ See `docs/graphql-normalized-cache-plan.md`.- Reject already-routed optimistic enqueues with missing replies as uncertain onevery owner departure; never forward them through network fallback, becausetheir durable queue entries may survive the old engine.diff --git a/apps/web/docs/graphql-normalized-cache-plan.md b/apps/web/docs/graphql-normalized-cache-plan.mdindex d71f3ec98e..ba5068f2ea 100644--- a/apps/web/docs/graphql-normalized-cache-plan.md+++ b/apps/web/docs/graphql-normalized-cache-plan.md@@ -303,8 +303,17 @@ Startup phase budgets are relayed to every page, including late joiners: assetloading gets five minutes, taking the lock gets the retry window plus fiveseconds, and database opening gets twenty seconds, plus a five-second pageresponse grace. Ordinary read deadlines remain ten seconds. Coordinator-registration gets sixty seconds per attempt, with three attempts; engine recovery-retains its bounded five retries. Exhausted bootstrap retries preserve the cache+registration gets sixty seconds per attempt, with three attempts separated by+100 ms and 200 ms waits. Safari can abort a shared worker's download when its+creating page leaves, then briefly reuse that failed startup for immediate+retries. These waits let the browser retire it without changing the worker name,+URL, or storage scope. Navigation and disposal cancel the wait; no new adapter+is created by a canceled initialization, even if its timer already fired.+`graphql_cache.registration_retry` reports the wait duration, retry number,+triggering error class, and whether the wait completed or was canceled. This is+separate from a repeated-ready restart and does not make generic transport+failures storage-safe. Engine recovery retains its bounded five retries.+Exhausted bootstrap retries preserve the cachescope only when the coordinator explicitly proves storage was untouched, not froma page's last-seen phase. Missing/broken transports remain conservativelyuncertain. Pages retry proven untouched bootstrap and owner-lock failures on reconnect,diff --git a/apps/web/src/lib/graphql-cache/host/registration-retry.test.ts b/apps/web/src/lib/graphql-cache/host/registration-retry.test.tsnew file mode 100644index 0000000000..e3bce7823f--- /dev/null+++ b/apps/web/src/lib/graphql-cache/host/registration-retry.test.ts@@ -0,0 +1,286 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import {+ type CacheRequest,+ INITIAL_CACHE_REVISION,+ type WorkerMessage,+} from '../protocol';+import type { CacheTelemetryObservation } from '../telemetry';+import type {+ CacheCoordinatorPageAdapterOptions,+ PageAdapterDisposeOptions,+} from '../worker/coordinator-page-adapter';+import { CacheBootstrapExhaustedError } from '../worker/startup';+import { CacheNavigationError } from './navigation-error';++const adapterFactory = vi.hoisted(() => vi.fn());+vi.mock('../worker/coordinator-page-adapter', () => ({+ createCacheCoordinatorPageAdapter: adapterFactory,+}));++import { createWorkerCacheHost } from './worker-host';++class RegistrationAdapter {+ onmessage: ((event: MessageEvent<WorkerMessage>) => void) | null = null;+ readonly requests: CacheRequest[] = [];+ readonly start = vi.fn(async () => {});+ readonly dispose = vi.fn(+ async (_options: PageAdapterDisposeOptions = {}) => {}+ );++ constructor(readonly options: CacheCoordinatorPageAdapterOptions) {}++ postMessage(request: CacheRequest): void {+ this.requests.push(request);+ this.onmessage?.(+ new MessageEvent('message', {+ data: {+ id: request.id,+ ok: true,+ result: request.kind === 'init' ? null : INITIAL_CACHE_REVISION,+ },+ })+ );+ }+}++const scope = 'cache-parking-registration-retry';+const loadFailure = new Error('SharedWorker transport error');+let available: boolean;+let adapters: RegistrationAdapter[];+let observations: CacheTelemetryObservation[];+const onInitializationError = vi.fn<(error: Error) => void>();+let host: ReturnType<typeof createWorkerCacheHost>;++beforeEach(() => {+ vi.useFakeTimers();+ localStorage.clear();+ localStorage.setItem('graphql-cache:scope', scope);+ vi.stubGlobal('SharedWorker', vi.fn());+ vi.stubGlobal('Worker', vi.fn());+ vi.stubGlobal('MessageChannel', vi.fn());+ vi.stubGlobal('navigator', {+ storage: { getDirectory: vi.fn() },+ locks: {+ request: vi.fn(+ async (_name: string, _options: LockOptions, callback: () => unknown) =>+ callback()+ ),+ },+ });+ available = false;+ adapters = [];+ observations = [];+ onInitializationError.mockReset();+ adapterFactory.mockReset();+ adapterFactory.mockImplementation(+ (options: CacheCoordinatorPageAdapterOptions) => {+ const adapter = new RegistrationAdapter(options);+ adapter.start.mockImplementation(async () => {+ if (available) return;+ options.onTerminalError?.(loadFailure);+ throw loadFailure;+ });+ adapters.push(adapter);+ return adapter;+ }+ );+ host = createWorkerCacheHost({+ scope,+ onInitializationError,+ telemetry: {+ record: (observation) => observations.push(observation),+ flush: vi.fn(),+ },+ });+});++afterEach(() => {+ host.dispose();+ vi.useRealTimers();+ vi.unstubAllGlobals();+ vi.restoreAllMocks();+});++describe('coordinator registration retry', () => {+ it('recovers when the failed startup becomes reusable between attempts', async () => {+ const outcome = host.currentRevision().then(+ (revision) => ({ revision }),+ (error: unknown) => ({ error })+ );+ await vi.advanceTimersByTimeAsync(0);+ available = true;+ await vi.advanceTimersByTimeAsync(100);++ await expect(outcome).resolves.toEqual({+ revision: INITIAL_CACHE_REVISION,+ });+ expect(adapters).toHaveLength(2);+ expect(adapters[0].dispose).toHaveBeenCalledExactlyOnceWith({+ graceful: false,+ });+ expect(adapters[0].requests).toEqual([]);+ expect(adapters[1].requests.map(({ kind }) => kind)).toEqual([+ 'init',+ 'current-revision',+ ]);+ expect(adapters.every((adapter) => adapter.options.scope === scope)).toBe(+ true+ );+ expect(localStorage.getItem('graphql-cache:scope')).toBe(scope);+ expect(onInitializationError).not.toHaveBeenCalled();+ expect(observations).toContainEqual(+ expect.objectContaining({+ name: 'graphql_cache.registration_retry',+ outcome: 'success',+ count: 1,+ durationMs: 100,+ })+ );+ });++ it('spaces both retries and exhausts once without admitting or replaying a mutation', async () => {+ const mutation = host.enqueueOptimisticMutation(+ {+ uuid: '00000000-0000-4000-8000-000000000100',+ query: 'mutation Update { update }',+ data: { update: true },+ },+ { owner: 'runner', nowMs: 1, leaseExpiresAtMs: 101 }+ );+ const outcomes = Promise.all([+ host.currentRevision().catch((error: unknown) => error),+ mutation.catch((error: unknown) => error),+ ]);+ await vi.advanceTimersByTimeAsync(0);+ expect(adapters).toHaveLength(1);+ await vi.advanceTimersByTimeAsync(99);+ expect(adapters).toHaveLength(1);+ await vi.advanceTimersByTimeAsync(1);+ expect(adapters).toHaveLength(2);+ await vi.advanceTimersByTimeAsync(199);+ expect(adapters).toHaveLength(2);+ await vi.advanceTimersByTimeAsync(1);+ expect(adapters).toHaveLength(3);+ await expect(outcomes).resolves.toEqual([loadFailure, loadFailure]);+ expect(onInitializationError).toHaveBeenCalledExactlyOnceWith(loadFailure);+ expect(onInitializationError.mock.calls[0][0]).not.toBeInstanceOf(+ CacheBootstrapExhaustedError+ );+ expect(adapters.flatMap((adapter) => adapter.requests)).toEqual([]);+ await expect(host.currentRevision()).rejects.toBe(loadFailure);+ await vi.advanceTimersByTimeAsync(1_000);+ expect(adapters).toHaveLength(3);+ expect(localStorage.getItem('graphql-cache:scope')).toBe(scope);+ expect(vi.getTimerCount()).toBe(0);+ expect(+ observations.filter(+ ({ name }) => name === 'graphql_cache.registration_retry'+ )+ ).toMatchObject([+ { outcome: 'success', count: 1, durationMs: 100 },+ { outcome: 'success', count: 2, durationMs: 200 },+ ]);+ });++ it.each([0, 100])(+ 'cancels disposal during the wait starting at %s ms',+ async (elapsed) => {+ const outcome = host.currentRevision().catch((error: unknown) => error);+ await vi.advanceTimersByTimeAsync(elapsed);+ expect(adapters).toHaveLength(elapsed === 0 ? 1 : 2);+ const attempts = adapters.length;+ host.dispose();+ await expect(outcome).resolves.toEqual(+ expect.objectContaining({+ message: 'cache worker host was disposed during startup',+ })+ );+ await vi.advanceTimersByTimeAsync(1_000);+ expect(adapters).toHaveLength(attempts);+ expect(onInitializationError).not.toHaveBeenCalled();+ expect(vi.getTimerCount()).toBe(0);+ expect(observations).toContainEqual(+ expect.objectContaining({+ name: 'graphql_cache.registration_retry',+ outcome: 'abrupt',+ durationMs: 0,+ })+ );+ }+ );++ it.each([0, 100])(+ 'cancels pagehide during the wait at %s ms and restores without an orphan retry',+ async (elapsed) => {+ const outcome = host.currentRevision().catch((error: unknown) => error);+ await vi.advanceTimersByTimeAsync(elapsed);+ expect(adapters).toHaveLength(elapsed === 0 ? 1 : 2);+ const attempts = adapters.length;+ dispatchEvent(new PageTransitionEvent('pagehide', { persisted: true }));+ available = true;+ // Restore before the old startup's catch/finally has unwound.+ dispatchEvent(new PageTransitionEvent('pageshow', { persisted: true }));+ await expect(outcome).resolves.toBeInstanceOf(CacheNavigationError);+ await expect(host.currentRevision()).resolves.toBe(+ INITIAL_CACHE_REVISION+ );+ await vi.advanceTimersByTimeAsync(1_000);+ expect(adapters).toHaveLength(attempts + 1);+ expect(onInitializationError).not.toHaveBeenCalled();+ expect(localStorage.getItem('graphql-cache:scope')).toBe(scope);+ }+ );++ it('does not construct another adapter if disposed after the delay fires but before its continuation', async () => {+ const outcome = host.currentRevision().catch((error: unknown) => error);+ await vi.advanceTimersByTimeAsync(0);+ expect(adapters).toHaveLength(1);+ vi.advanceTimersByTime(100);+ host.dispose();+ await expect(outcome).resolves.toEqual(+ expect.objectContaining({+ message: 'cache worker host was disposed during startup',+ })+ );+ await vi.advanceTimersByTimeAsync(1_000);+ expect(adapters).toHaveLength(1);+ expect(onInitializationError).not.toHaveBeenCalled();+ expect(vi.getTimerCount()).toBe(0);+ });++ it('gives the next attempt its own timeout and ignores late success from the retired attempt', async () => {+ host.dispose();+ host = createWorkerCacheHost({+ scope,+ initializationTimeoutMs: 10,+ onInitializationError,+ });+ const first = Promise.withResolvers<void>();+ const second = Promise.withResolvers<void>();+ adapterFactory.mockImplementation(+ (options: CacheCoordinatorPageAdapterOptions) => {+ const adapter = new RegistrationAdapter(options);+ adapter.start.mockImplementation(() =>+ adapters.length === 1 ? first.promise : second.promise+ );+ adapters.push(adapter);+ return adapter;+ }+ );+ const result = host.currentRevision().then(+ (revision) => ({ revision }),+ (error: unknown) => ({ error })+ );+ await vi.advanceTimersByTimeAsync(10);+ first.resolve();+ await vi.advanceTimersByTimeAsync(99);+ expect(adapters).toHaveLength(1);+ expect(adapters[0].requests).toEqual([]);+ await vi.advanceTimersByTimeAsync(10);+ expect(adapters).toHaveLength(2);+ second.resolve();+ await expect(result).resolves.toEqual({ revision: INITIAL_CACHE_REVISION });+ expect(adapters[0].requests).toEqual([]);+ expect(onInitializationError).not.toHaveBeenCalled();+ });+});diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.test.ts b/apps/web/src/lib/graphql-cache/host/worker-host.test.tsindex d320b1c71f..6dff0c1e77 100644--- a/apps/web/src/lib/graphql-cache/host/worker-host.test.ts+++ b/apps/web/src/lib/graphql-cache/host/worker-host.test.ts@@ -677,7 +677,9 @@ describe('createWorkerCacheHost', () => {const rejected = expect(host.currentRevision()).rejects.toThrow('cache worker timeout: registration');- await vi.advanceTimersByTimeAsync(COORDINATOR_CONNECT_TIMEOUT_MS * 3 + 1);+ await vi.advanceTimersByTimeAsync(+ COORDINATOR_CONNECT_TIMEOUT_MS * 3 + 100 + 200 + 1+ );await rejected;expect(adapterFactory).toHaveBeenCalledTimes(3);expect(onInitializationError).toHaveBeenCalledOnce();diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.ts b/apps/web/src/lib/graphql-cache/host/worker-host.tsindex f8832f0793..25aa5ff26a 100644--- a/apps/web/src/lib/graphql-cache/host/worker-host.ts+++ b/apps/web/src/lib/graphql-cache/host/worker-host.ts@@ -67,6 +67,7 @@ import { isUnambiguousCacheScope } from '../worker/stale-databases';import {CacheBootstrapExhaustedError,COORDINATOR_CONNECT_ATTEMPTS,+ COORDINATOR_CONNECT_RETRY_DELAYS_MS,COORDINATOR_CONNECT_TIMEOUT_MS,STARTUP_RESPONSE_GRACE_MS,} from '../worker/startup';@@ -594,6 +595,11 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {attempt < COORDINATOR_CONNECT_ATTEMPTS;attempt += 1) {+ // Cancellation may arrive after a retry timer fires, before this continuation.+ if (initializationId !== initializationAttempt)+ throw new CacheNavigationError();+ if (state !== 'initializing')+ throw new Error('cache worker host was disposed during startup');const current = getAdapter();let timer: ReturnType<typeof setTimeout> | undefined;try {@@ -624,6 +630,28 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {attempt + 1 === COORDINATOR_CONNECT_ATTEMPTS)throw error;+ if (timer !== undefined) clearTimeout(timer);+ const retryStartedAt = now();+ let completed = false;+ try {+ await new Promise<void>((resolve, reject) => {+ cancelRegistration = reject;+ timer = setTimeout(+ resolve,+ COORDINATOR_CONNECT_RETRY_DELAYS_MS[attempt]+ );+ });+ completed = true;+ } finally {+ telemetry.record({+ name: 'graphql_cache.registration_retry',+ operationCategory: 'initialization',+ outcome: completed ? 'success' : 'abrupt',+ errorCode: classifyCacheError(error),+ count: attempt + 1,+ durationMs: now() - retryStartedAt,+ });+ }} finally {if (timer !== undefined) clearTimeout(timer);cancelRegistration = undefined;diff --git a/apps/web/src/lib/graphql-cache/telemetry.ts b/apps/web/src/lib/graphql-cache/telemetry.tsindex fd5132bc58..7d17b5c6ca 100644--- a/apps/web/src/lib/graphql-cache/telemetry.ts+++ b/apps/web/src/lib/graphql-cache/telemetry.ts@@ -9,6 +9,7 @@ export const CACHE_TELEMETRY_EVENT_NAMES = ['graphql_cache.schema_init','graphql_cache.db_ready','graphql_cache.host_ready',+ 'graphql_cache.registration_retry','graphql_cache.host_request','graphql_cache.coordinator_request','graphql_cache.engine_request',diff --git a/apps/web/src/lib/graphql-cache/worker/startup.ts b/apps/web/src/lib/graphql-cache/worker/startup.tsindex bba181a6af..fde293086e 100644--- a/apps/web/src/lib/graphql-cache/worker/startup.ts+++ b/apps/web/src/lib/graphql-cache/worker/startup.ts@@ -21,7 +21,13 @@ export const OWNER_LOCK_WAIT_TIMEOUT_MS =OWNER_LOCK_RETRY_DELAYS_MS.reduce((total, delay) => total + delay, 0) + 5_000;/** Lets the coordinator's phase timeout/retry reach a page before its guard fires. */export const STARTUP_RESPONSE_GRACE_MS = 5_000;-export const COORDINATOR_CONNECT_ATTEMPTS = 3;+/** Safari can briefly reuse a failed SharedWorker startup after its creator+ * leaves during download. Let that entry retire before consuming another try. */+export const COORDINATOR_CONNECT_RETRY_DELAYS_MS: readonly number[] = [+ 100, 200,+];+export const COORDINATOR_CONNECT_ATTEMPTS =+ COORDINATOR_CONNECT_RETRY_DELAYS_MS.length + 1;/** Only the live coordinator can prove that every failed attempt stayed before* the open grant. A missing progress message is not proof of untouched storage.diff --git a/docs/agent-guide/navigation.md b/docs/agent-guide/navigation.mdindex 0ff0c72c0b..ef5b52d47e 100644--- a/docs/agent-guide/navigation.md+++ b/docs/agent-guide/navigation.md@@ -76,6 +76,18 @@ optimistic enqueues must not be sent twice. A `graphql_cache.owner` event with`ownerEvent: coordinator-restarted` distinguishes this from engine replacementor a BFCache restore.+Separately, leaving the creating page while its SharedWorker is still downloading+can fail the survivor before its first ready frame. In an isolated Safari fixture,+hold that download, connect a second page, and navigate the creator through a+normal link. Release the asset and verify the survivor initializes on its bounded+registration retries without a reload. Retries wait 100 ms and 200 ms; navigating+or disposing the survivor during either wait must stop further constructors.+Verify BFCache restoration starts fresh without resurrecting the canceled retry.+Use `graphql_cache.registration_retry` to inspect wait timing and cancellation;+preserve the scope, existing queue, and conservative storage-failure handling.+Do not inject this failure into a live account or mistake it for a repeated-ready+loop.+The same write-safety rule applies when an owner navigates away without restartingthe coordinator: a surviving tab's enqueue may be committed even if its reply waslost. It must report an uncertain outcome, not retry or fall through to a second