Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(cache-core): prevent worker disposal on upgrade (#7140)
teo767a999a5f09
diff --git a/Cargo.lock b/Cargo.lockindex 1e8d4e2e75..888ac94906 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -3304,7 +3304,7 @@ dependencies = [[[package]]name = "cache-wasm"-version = "0.6.30"+version = "0.6.31"dependencies = ["async-lock","cache-core",diff --git a/apps/web/docs/graphql-normalized-cache-plan.md b/apps/web/docs/graphql-normalized-cache-plan.mdindex f598dda6d2..9251cf12eb 100644--- a/apps/web/docs/graphql-normalized-cache-plan.md+++ b/apps/web/docs/graphql-normalized-cache-plan.md@@ -198,18 +198,90 @@ heartbeat without changing epochs or touching storage. A released lock confirmssilent worker termination and triggers the usual fenced recovery. Read deadlinesstill bound callers waiting on an unresponsive engine.-Protocol v3 separates startup into asset loading and database opening. The engine-loads/compiles WASM without touching OPFS, reports `engine-assets-ready`, and waits-for the coordinator's `open-engine` grant. The coordinator records that storage may-be touched before sending this grant. Pre-grant failures preserve the current-open/reset requirement; an earlier required wipe is never forgotten. Startup phase-budgets are relayed to every page, including late joiners: asset loading gets five-minutes and database opening gets twenty seconds, plus a five-second page response-grace. Ordinary read deadlines remain ten seconds. Coordinator registration gets-sixty seconds per attempt, with three attempts; engine recovery retains its bounded-five retries. Exhausted bootstrap retries preserve the cache scope only when the-coordinator explicitly proves storage was untouched, not from a page's last-seen-phase. Missing/broken transports remain conservatively uncertain.+Protocol v6 separates startup into asset loading, taking the database owner+lock, and database opening. The engine loads/compiles WASM without touching OPFS+and reports `engine-assets-ready`. It then takes the owner lock only if it is+free (`ifAvailable`), never queueing behind another holder: a busy lock is retried+for about ten seconds with `owner-lock-busy` reports. Once it holds the lock it+reports `owner-lock-acquired` and waits for the coordinator's `open-engine` grant;+the coordinator records that storage may be touched before sending this grant.+Pre-grant failures, including every failure while taking the lock, preserve the+current open/reset requirement; an earlier required wipe is never forgotten.+Startup phase budgets are relayed to every page, including late joiners: asset+loading gets five minutes, taking the lock gets the retry window plus five+seconds, and database opening gets twenty seconds, plus a five-second page+response grace. Ordinary read deadlines remain ten seconds. Coordinator+registration gets sixty seconds per attempt, with three attempts; engine recovery+retains its bounded five retries. Exhausted bootstrap retries preserve the cache+scope only when the coordinator explicitly proves storage was untouched, not from+a page's last-seen phase. Missing/broken transports remain conservatively+uncertain.++A build's coordinator is usually a separate SharedWorker (its script URL is+content-hashed), so after a deploy two builds can target one database. The newer+build takes it over. Pages report their build time (`__APP_BUILD_TIME__`,+stamped by Vite) when they register. When the lock stays busy while tabs of+another build are alive (liveness locks of tabs this coordinator never+registered; a departed tab's lock can outlast it), the coordinator posts a+takeover request on the cross-build channel+`graphql-cache-takeover:{database name}` (`coordinator-takeover.ts`, a contract+every later build must keep). Only builds that open the same database join it,+and only the coordinator whose engine holds the database answers:++- **Yield**, if the requester is strictly newer. Every tab registered there gets+ `cache-superseded`. Each page retires its host to the network. The owner's+ page leaves the way a navigating page does: it terminates its engine, which+ releases the owner lock without counting as a lost owner. An enqueue that was+ already sent is reported as uncertain rather than sent again, because it may+ be in the durable queue that the newer build replays. Nothing is elected+ again. The requester's engine, still retrying the lock, opens the same+ database with its data and queued mutations.+- **Keep**, if the requester is not newer. The requester fails closed as below.++A superseded page offers a persistent reload prompt. It reloads by itself only+while hidden, online, not holding work (a call or an upload, via+`holdAutomaticReload`), and not focused on a text field that holds text.+Otherwise it retries the next time it is hidden or comes back online. A page+that registers with a superseded coordinator later, for example after a+rollback or from stale HTML, only runs uncached, because a reload could land on+the same build again.++A coordinator that holds nothing but gets a request from a newer build sends+its tabs there too, without answering. When no yield arrives within a second+(builds from before handover never answer), when the holder keeps the+database, or when the engine exhausts its retries, the coordinator fails closed:+it refuses queued requests with `owner-lock-unavailable` and tells every+registered page `cache-unavailable`. Those pages retire their host for the rest+of the page session, without quarantine, toast, or error report; only a tab+that registers later tries again. Nothing waits in line for the lock, and a+build only ever yields to a newer one. A rollback therefore leaves the rolled-back+build uncached until the newer tabs close, instead of reloading tabs in a loop.++The browser can also hand the owner lock to the next engine before a departing+page's terminated worker has let go of the database files. Production logs show+that gap outlasting two seconds after a reload. Every holder of these files+first holds the owner lock, so a busy file seen while holding it is a predecessor+still exiting, not damage. The OPFS adapter waits up to ten seconds for busy+files within one open or wipe. If they stay busy, the engine reports+`storage-busy`, and the coordinator fails closed as above. It keeps that+attempt's open or wipe requirement instead of escalating a failed open to a+wipe, which could not remove open files anyway.++The physical database name embeds the storage versions,+`graphql-cache:{scope}:s{epoch}.v{format}.t{storage}`, so builds with different+compatibility epochs, record formats, or storage schemas use separate files and+owner locks instead of resetting each other's data. Builds at the versions in+use when names began to embed them (epoch 3, format 3, storage 11) keep the+unversioned `graphql-cache:{scope}`, so that change moved no database and the+next build still replays its queued mutations. Once an owner is active and no+other build has live tabs, the coordinator lets that page delete stale+databases of its scope: names for versions no newer than its own that it does+not open, which includes the unversioned name once the versions move past it.+A newer version is kept, because its build may come back after a rollback. The page lists OPFS and, only if a stale name exists,+starts a disposable worker (a file Turso cannot read may poison its OPFS+registry) that deletes each stale database whose owner lock is free and whose+`mutation_queue` is empty. Databases still queueing mutations, or whose queue+cannot be counted, are kept and reported.Browsers missing the required worker, lock, or OPFScapabilities use a storage-free no-op cache host. Tauri@@ -322,8 +394,9 @@ apps/web/src/lib/graphql-cache/ # JS gluecompatibility epoch, record format, and storage schema) before creating thefixture database and again before fault injection. This matched-build checkis intentionally stricter than normal cache compatibility.-- Deferred: stale-namespace DB cleanup (browser), `scan_prefix`/- `approx_size` for GC (hardening phase).+- Stale-version DB cleanup for the current scope (browser) is done; see §4.2.+ Deferred: cleanup of other scopes' databases (quarantined or rotated+ scopes), `scan_prefix`/`approx_size` for GC (hardening phase).**Phase 3 — hosts + JS glue** *(done)*- ~~`cache-wasm`~~: wasm-bindgen shell (async-mutex engine, string op-iddiff --git a/apps/web/src/features/channel/Call/CallContext.tsx b/apps/web/src/features/channel/Call/CallContext.tsxindex e22a04bc1e..7b553d414d 100644--- a/apps/web/src/features/channel/Call/CallContext.tsx+++ b/apps/web/src/features/channel/Call/CallContext.tsx@@ -5,6 +5,7 @@ import {type BackgroundEffect,backgroundProcessorOptions,} from '@core/media/background-effect';+import { holdAutomaticReload } from '@core/util/reloadForNewerBuild';import type { KrispNoiseFilter } from '@livekit/krisp-noise-filter';import type { BackgroundProcessorWrapper } from '@livekit/track-processors';import {@@ -37,7 +38,7 @@ import {createCallSessionController,stopPrejoinTracks,} from './CallSessionController';-import { createCallLifecycle } from './call-lifecycle';+import { type CallLifecycleState, createCallLifecycle } from './call-lifecycle';import { publishCallResolution } from './call-resolution';import { createLatestAsyncRequestQueue } from './latest-async-request-queue';import {@@ -1630,9 +1631,22 @@ function createCallState() {const [lifecycleSnapshot, setLifecycleSnapshot] = createSignal(lifecycle.getState());- const unsubscribeLifecycle = lifecycle.subscribe((state) =>- setLifecycleSnapshot(state)- );+ // Reloading into a newer app build would cut the call off.+ let releaseReloadHold: (() => void) | undefined;+ const holdReloadDuringCall = (state: CallLifecycleState) => {+ const inCall = state.t !== 'idle' && state.t !== 'failed';+ if (inCall && !releaseReloadHold) {+ releaseReloadHold = holdAutomaticReload();+ } else if (!inCall && releaseReloadHold) {+ releaseReloadHold();+ releaseReloadHold = undefined;+ }+ };+ holdReloadDuringCall(lifecycle.getState());+ const unsubscribeLifecycle = lifecycle.subscribe((state) => {+ setLifecycleSnapshot(state);+ holdReloadDuringCall(state);+ });const unsubscribeNative =isNativeIosCallKitEnabled() && nativeCall@@ -1649,6 +1663,7 @@ function createCallState() {onCleanup(() => {unsubscribeNative?.();unsubscribeLifecycle();+ releaseReloadHold?.();lifecycle.dispose();disposed = true;browserConnectGeneration += 1;diff --git a/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx b/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsxindex 8684176de6..caa8f6dea7 100644--- a/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx+++ b/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx@@ -360,18 +360,25 @@ export function createQuickAccessValue(): QuickAccessContextValue {equals: equalActivityMaps,});const isConnectedSecondaryInbox = useIsConnectedSecondaryInbox();- const graphqlCacheHost = getGraphqlSoupCacheHost();- const cacheHost = graphqlCacheHost?.disabled ? undefined : graphqlCacheHost;+ // Read reactively: if the session abandons its cache mid-session, Quick+ // Access falls back to the REST channel list instead of the dead host. The+ // memo keeps the subscription below until the host itself changes.+ const cacheHost = createMemo(() => {+ const graphqlCacheHost = getGraphqlSoupCacheHost();+ return graphqlCacheHost?.disabled ? undefined : graphqlCacheHost;+ });const [cacheRevision, setCacheRevision] = createSignal(0);const cachedChannelsQuery = useCachedGraphqlChannelsQuery(cacheHost);- if (cacheHost) {+ createEffect(() => {+ const host = cacheHost();+ if (!host) return;onCleanup(- subscribeToVisibleCacheChanges(cacheHost, () => {+ subscribeToVisibleCacheChanges(host, () => {setCacheRevision((revision) => revision + 1);return cachedChannelsQuery.refetch({ cancelRefetch: false });}));- }+ });const instructionsIdQuery = useInstructionsMdIdQuery();const { query: crmCompaniesQuery, companies: crmCompaniesAccessor } =useQuickAccessCrmCompaniesQuery();@@ -490,7 +497,8 @@ export function createQuickAccessValue(): QuickAccessContextValue {// The GraphQL cache is authoritative while enabled. Otherwise preserve the// existing channel-list source unchanged.- const channelData = cacheHost+ const usesCache = cacheHost() !== undefined;+ const channelData = usesCache? queryReadyGate(cachedChannelsQuery)? cachedChannelsQuery.data: []@@ -946,10 +954,12 @@ export function createQuickAccessValue(): QuickAccessContextValue {? searchQuickAccessItems(baseList(), options.searchTerm?.() ?? ''): baseList());+ // A retired host degrades this list to local items; later lists skip it.+ const projectionHost = options ? cacheHost() : undefined;const projected =- options && cacheHost+ options && projectionHost? createProjectedList<QuickAccessItem>({- host: cacheHost,+ host: projectionHost,get buckets() {return projectedBuckets();},@@ -968,9 +978,9 @@ export function createQuickAccessValue(): QuickAccessContextValue {);const [historyItems, cachedChannelItems, cachedCompanies] =await Promise.all([- materializeCachedGraphqlHistoryItems(cacheHost, missing),- materializeCachedGraphqlChannels(cacheHost, missing),- materializeCachedGraphqlCrmCompanies(cacheHost, missing),+ materializeCachedGraphqlHistoryItems(projectionHost, missing),+ materializeCachedGraphqlChannels(projectionHost, missing),+ materializeCachedGraphqlCrmCompanies(projectionHost, missing),]);const historyById = new Map(historyItems.map((item) => [item.id, item])@@ -1066,7 +1076,7 @@ export function createQuickAccessValue(): QuickAccessContextValue {Boolean(projected?.isLoading() ||historyQuery.isLoading ||- (cacheHost ? cachedChannelsQuery.isLoading : channelsLoading())+ (cacheHost() ? cachedChannelsQuery.isLoading : channelsLoading())),isLoadingMore: () => projected?.isLoadingMore() ?? false,loadMore: async () => {@@ -1079,10 +1089,10 @@ export function createQuickAccessValue(): QuickAccessContextValue {// resolves rather than gating quick access on a slower/failing CRM fetch.const isLoading = () =>historyQuery.isLoading ||- (cacheHost ? cachedChannelsQuery.isLoading : channelsLoading());+ (cacheHost() ? cachedChannelsQuery.isLoading : channelsLoading());const refresh = () => {- if (cacheHost) {+ if (cacheHost()) {setCacheRevision((revision) => revision + 1);void cachedChannelsQuery.refetch();}@@ -1096,7 +1106,7 @@ export function createQuickAccessValue(): QuickAccessContextValue {return {useList,usesRecordSelection: () => false,- usesSearchProjection: () => cacheHost !== undefined,+ usesSearchProjection: () => cacheHost() !== undefined,isLoading,refresh,getById,diff --git a/apps/web/src/lib/core/util/reloadForNewerBuild.test.ts b/apps/web/src/lib/core/util/reloadForNewerBuild.test.tsnew file mode 100644index 0000000000..9e8d8b787d--- /dev/null+++ b/apps/web/src/lib/core/util/reloadForNewerBuild.test.ts@@ -0,0 +1,138 @@+import { beforeEach, describe, expect, it, vi } from 'vitest';++vi.mock('@core/component/Toast/Toast', () => ({+ toast: { custom: vi.fn() },+}));++import {+ holdAutomaticReload,+ type ReloadPage,+ reloadForNewerBuild,+ resetReloadForNewerBuildForTests,+} from './reloadForNewerBuild';++function fakePage(state: {+ hidden: boolean;+ offline: boolean;+ editingText?: boolean;+}) {+ const listeners: Array<() => void> = [];+ const prompts: Array<() => void> = [];+ const page: ReloadPage & { reload: ReturnType<typeof vi.fn> } = {+ isHidden: () => state.hidden,+ isOffline: () => state.offline,+ isEditingText: () => state.editingText ?? false,+ reload: vi.fn<() => void>(),+ onHiddenOrOnline: (callback) => listeners.push(callback),+ promptReload: (reload) => prompts.push(reload),+ };+ const notify = () => {+ for (const listener of listeners) listener();+ };+ return {+ page,+ prompts,+ state,+ hide() {+ state.hidden = true;+ notify();+ },+ show() {+ state.hidden = false;+ },+ goOnline() {+ state.offline = false;+ notify();+ },+ };+}++describe('reloadForNewerBuild', () => {+ beforeEach(() => {+ resetReloadForNewerBuildForTests();+ });++ it('reloads a hidden tab right away and still offers the reload', () => {+ const { page, prompts } = fakePage({ hidden: true, offline: false });++ reloadForNewerBuild(page);++ expect(page.reload).toHaveBeenCalledOnce();+ expect(prompts).toHaveLength(1);+ });++ it('prompts on the visible tab and reloads when the user switches away', () => {+ const { page, prompts, hide } = fakePage({ hidden: false, offline: false });++ reloadForNewerBuild(page);++ expect(page.reload).not.toHaveBeenCalled();+ expect(prompts).toHaveLength(1);+ hide();+ expect(page.reload).toHaveBeenCalledOnce();+ });++ it('reloads from the prompt without waiting to be hidden', () => {+ const { page, prompts } = fakePage({ hidden: false, offline: false });++ reloadForNewerBuild(page);+ prompts[0]?.();++ expect(page.reload).toHaveBeenCalledOnce();+ });++ it('waits until the device is online again, and only while hidden', () => {+ const hidden = fakePage({ hidden: true, offline: true });+ reloadForNewerBuild(hidden.page);+ expect(hidden.page.reload).not.toHaveBeenCalled();+ hidden.goOnline();+ expect(hidden.page.reload).toHaveBeenCalledOnce();++ resetReloadForNewerBuildForTests();+ const woken = fakePage({ hidden: true, offline: true });+ reloadForNewerBuild(woken.page);+ // The device wakes up with the tab in front: the user may be typing.+ woken.show();+ woken.goOnline();+ expect(woken.page.reload).not.toHaveBeenCalled();+ woken.hide();+ expect(woken.page.reload).toHaveBeenCalledOnce();+ });++ it('never reloads by itself during held work or while text is written', () => {+ const release = holdAutomaticReload();+ const { page, prompts, state, hide, show } = fakePage({+ hidden: true,+ offline: false,+ editingText: true,+ });++ reloadForNewerBuild(page);+ expect(page.reload).not.toHaveBeenCalled();++ release();+ show();+ hide();+ expect(page.reload).not.toHaveBeenCalled();++ state.editingText = false;+ show();+ hide();+ expect(page.reload).toHaveBeenCalledOnce();+ // The prompt still reloads whenever the user asks.+ prompts[0]?.();+ expect(page.reload).toHaveBeenCalledTimes(2);+ });++ it('schedules one reload however many times the cache asks', () => {+ const first = fakePage({ hidden: false, offline: false });+ const second = fakePage({ hidden: true, offline: false });++ reloadForNewerBuild(first.page);+ reloadForNewerBuild(second.page);++ expect(first.prompts).toHaveLength(1);+ expect(second.prompts).toHaveLength(0);+ expect(second.page.reload).not.toHaveBeenCalled();+ });+});diff --git a/apps/web/src/lib/core/util/reloadForNewerBuild.ts b/apps/web/src/lib/core/util/reloadForNewerBuild.tsnew file mode 100644index 0000000000..685c2b4374--- /dev/null+++ b/apps/web/src/lib/core/util/reloadForNewerBuild.ts@@ -0,0 +1,98 @@+import { toast } from '@core/component/Toast/Toast';+import { deviceLooksOffline } from './connectivity';+import { isEditableInput } from './isEditableInput';++/** What the reload decision needs from the page; replaced in tests. */+export interface ReloadPage {+ isHidden(): boolean;+ isOffline(): boolean;+ /** Whether focus is in a text field that holds text a reload would lose. */+ isEditingText(): boolean;+ reload(): void;+ /** Runs `callback` each time the page is hidden or comes back online. */+ onHiddenOrOnline(callback: () => void): void;+ /** Offers the user a reload now. */+ promptReload(reload: () => void): void;+}++const browserPage: ReloadPage = {+ isHidden: () => document.visibilityState === 'hidden',+ isOffline: deviceLooksOffline,+ isEditingText() {+ const focused = document.activeElement;+ if (!isEditableInput(focused)) return false;+ const text =+ focused instanceof HTMLInputElement ||+ focused instanceof HTMLTextAreaElement+ ? focused.value+ : focused?.textContent;+ return Boolean(text?.trim());+ },+ reload: () => window.location.reload(),+ onHiddenOrOnline(callback) {+ // Run a task later: a page being unloaded also turns hidden, and its+ // timers never run, so leaving the app never becomes a reload.+ const later = () => setTimeout(callback, 0);+ document.addEventListener('visibilitychange', () => {+ if (document.visibilityState === 'hidden') later();+ });+ window.addEventListener('online', later);+ },+ promptReload(reload) {+ toast.custom(+ {+ title: 'A new version of Macro is ready',+ actions: [{ label: 'Reload', onClick: reload }],+ },+ { persistent: true }+ );+ },+};++const holds = new Set<symbol>();+let scheduled = false;++/**+ * Keeps this tab from reloading by itself into a newer build while work a+ * reload would cut off is running, such as a call or an upload. Returns the+ * release. The user can still reload from the prompt.+ */+export function holdAutomaticReload(): () => void {+ const hold = Symbol('automatic reload hold');+ holds.add(hold);+ return () => {+ holds.delete(hold);+ };+}++/**+ * A newer build of the app took over this tab's local cache, so the tab keeps+ * working without it and should move to that build. It offers a reload, and+ * reloads by itself only while it is in the background and the reload loses+ * nothing: the device is online, nothing holds the tab, and no text is being+ * written. Otherwise it tries again the next time it is hidden or comes back+ * online.+ */+export function reloadForNewerBuild(page: ReloadPage = browserPage): void {+ if (scheduled) return;+ scheduled = true;+ const reloadIfIdle = () => {+ if (+ page.isHidden() &&+ !page.isOffline() &&+ holds.size === 0 &&+ !page.isEditingText()+ ) {+ page.reload();+ }+ };+ page.onHiddenOrOnline(reloadIfIdle);+ page.promptReload(() => page.reload());+ reloadIfIdle();+}++/** Test seam: forget the scheduled reload and every hold. */+export function resetReloadForNewerBuildForTests(): void {+ scheduled = false;+ holds.clear();+}diff --git a/apps/web/src/lib/core/util/upload.ts b/apps/web/src/lib/core/util/upload.tsindex e45688bfd3..e9a99b7461 100644--- a/apps/web/src/lib/core/util/upload.ts+++ b/apps/web/src/lib/core/util/upload.ts@@ -21,6 +21,7 @@ import {blockAcceptsFileExtension,} from '@core/constant/allBlocks';import { heicConversionService } from '@core/heic/service';+import { holdAutomaticReload } from '@core/util/reloadForNewerBuild';import {createStaticUploadFile,createUploadFile,@@ -411,6 +412,8 @@ export async function uploadFile(dssOptions: DssUploadFileOptions = {}): Promise<MaybeUploadResult> {const uploadSource = createUploadFile(file);+ // Reloading into a newer app build would cut the upload off.+ const releaseReloadHold = holdAutomaticReload();try {validateFileSize(uploadSource);@@ -448,6 +451,8 @@ export async function uploadFile(: new UploadError(uploadSource, 'dss', new Error(String(error))),name,};+ } finally {+ releaseReloadHold();}}diff --git a/apps/web/src/lib/graphql-cache/app-build.ts b/apps/web/src/lib/graphql-cache/app-build.tsnew file mode 100644index 0000000000..395531cbfc--- /dev/null+++ b/apps/web/src/lib/graphql-cache/app-build.ts@@ -0,0 +1,6 @@+/**+ * When this app build was made, in ms since the epoch, or 0 where the bundler+ * did not stamp it (tests and harnesses). A newer build takes the local cache+ * database over from tabs of an older one.+ */+export const APP_BUILD_TIME: number = import.meta.env.__APP_BUILD_TIME__ ?? 0;diff --git a/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts b/apps/web/src/lib/graphql-cache/exchange/inspection.test.tsindex 4b9c3cf5f7..a239ce1daa 100644--- a/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts+++ b/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts@@ -129,7 +129,7 @@ describe('typed generated query inspection', () => {it('returns no selections through the no-op host', async () => {const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});try {- const host = createNoopCacheHost('test');+ const host = createNoopCacheHost();const selection = selectAll(GroupSoupMembershipDocument).field('user').field('groupSoup');diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.tsindex 17c65a4955..b90a22cff0 100644--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts@@ -33,6 +33,7 @@ import {type EnqueueOptimisticMutationResult,INITIAL_CACHE_REVISION,type MutationClaim,+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE,type ReadResult,type WriteResult,} from '../protocol';@@ -3067,6 +3068,63 @@ describe('normalizedCacheExchange', () => {expect(host.commits).toHaveLength(1);});+ it('sends a patched mutation directly while another context owns the database', async () => {+ const base = makeMutationOp(1, optimistic);+ const op = makeOperation(base.kind, base, {+ ...base.context,+ normalizedCacheOptimistic: {+ uuid: crypto.randomUUID(),+ optimisticResponse: optimistic,+ linkPatches: [+ {+ query:+ 'query Group { user { groupSoup { bins { items { id } } } } }',+ operationName: 'Group',+ variablesJson: '{}',+ path: [+ { field: 'user' },+ { field: 'groupSoup' },+ { field: 'bins' },+ ],+ operation: {+ kind: 'remove',+ entityKey: 'GraphqlSoupItem:task-1',+ },+ },+ ],+ revalidations: [],+ },+ });+ host.enqueueOptimisticMutation = vi.fn().mockRejectedValue(+ Object.assign(new Error('owner lock is held by another build'), {+ errorCode: OWNER_LOCK_UNAVAILABLE_ERROR_CODE,+ })+ );+ const warn = vi+ .spyOn(console, 'warn')+ .mockImplementation(() => undefined);+ const onCacheError = vi.fn();+ const { ops, results, forwarded } = harness(host, undefined, {+ onCacheError,+ });+ try {+ ops.next(op);+ await tick();++ // No link-patch degradation retry: the refusal says nothing about+ // the patches, and nothing was admitted to a durable queue.+ expect(host.enqueueOptimisticMutation).toHaveBeenCalledOnce();+ expect(forwarded.map((forwardedOp) => forwardedOp.kind)).toEqual([+ 'mutation',+ ]);+ expect(results[0]?.data).toEqual({ from: 'network' });+ expect(onCacheError).not.toHaveBeenCalled();+ expect(warn).not.toHaveBeenCalled();+ } finally {+ warn.mockRestore();+ }+ });+it('uses only explicit recovery queries when a record-rooted parent is missing', async () => {const base = makeMutationOp(1, optimistic);const recovery = {diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.tsindex 242f06b850..ed291dca1a 100644--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts@@ -72,6 +72,7 @@ import {isAdmittedEnqueueUncertainError,isCacheRevision,isOwnerEpochLostError,+ isOwnerLockUnavailableError,type QueryRevalidationWire,} from '../protocol';import { createDeferredQueryRereads } from './deferred-query-rereads';@@ -1101,6 +1102,12 @@ export function normalizedCacheExchange(// unsafe to forward or retry without a coordinator fence.return uncertainEnqueueResult(op, error);}+ // Another context holds the database, so the enqueue never reached+ // an engine. Send it as a disabled cache would.+ if (isOwnerLockUnavailableError(error)) {+ enqueueForward(op);+ return undefined;+ }// A cached bin/page may disappear between inspect and enqueue. Do// not expose a partial relation move: retain entity optimism and// the post-success revalidation descriptors instead.diff --git a/apps/web/src/lib/graphql-cache/host/noop-host.ts b/apps/web/src/lib/graphql-cache/host/noop-host.tsindex ff0a047e50..9f217f71dd 100644--- a/apps/web/src/lib/graphql-cache/host/noop-host.ts+++ b/apps/web/src/lib/graphql-cache/host/noop-host.ts@@ -16,13 +16,12 @@ const emptyWriteResult = (): WriteResult => ({});/**- * CacheHost used when the platform cannot run the shared cache engine.- * It never stores data. Its disabled marker makes the exchange bypass- * optimistic persistence and forward mutations directly.+ * CacheHost used when the platform cannot run the shared cache engine, or+ * after a page stops using it. It never stores data. Its disabled marker+ * makes the exchange bypass optimistic persistence and forward mutations+ * directly.*/-export function createNoopCacheHost(reason: string): CacheHost {- console.warn(`[graphql-cache] disabled: ${reason}`);-+export function createNoopCacheHost(): CacheHost {return {clientId: 'noop',disabled: true,diff --git a/apps/web/src/lib/graphql-cache/host/retirable-host.test.ts b/apps/web/src/lib/graphql-cache/host/retirable-host.test.tsnew file mode 100644index 0000000000..fc0c01d13b--- /dev/null+++ b/apps/web/src/lib/graphql-cache/host/retirable-host.test.ts@@ -0,0 +1,87 @@+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';+import { INITIAL_CACHE_REVISION } from '../protocol';+import { createNoopCacheHost } from './noop-host';+import { createRetirableCacheHost } from './retirable-host';+import type { CacheHost } from './types';++const QUERY = { query: 'query Q { user { id } }', operationName: 'Q' };++/** A host whose every call fails like a disposed worker host. */+function disposedLikeHost(): CacheHost {+ const rejecting = createNoopCacheHost();+ const disposed = () =>+ Promise.reject(new Error('cache worker host was disposed'));+ return {+ ...rejecting,+ clientId: 'inner-client',+ disabled: false,+ currentStorageGeneration: vi.fn(async () => 'generation-1'),+ readQuery: vi.fn(disposed),+ writeQuery: vi.fn(disposed),+ search: vi.fn(disposed),+ teardown: vi.fn(disposed),+ dispose: vi.fn(),+ };+}++describe('createRetirableCacheHost', () => {+ beforeEach(() => {+ vi.spyOn(console, 'warn');+ });++ afterEach(() => {+ vi.restoreAllMocks();+ });++ it('delegates to the wrapped host until it is retired', async () => {+ const inner = disposedLikeHost();+ const host = createRetirableCacheHost(inner);++ expect(host.clientId).toBe('inner-client');+ expect(host.disabled).toBe(false);+ await expect(host.currentStorageGeneration()).resolves.toBe('generation-1');+ await expect(host.readQuery(QUERY)).rejects.toThrow(+ 'cache worker host was disposed'+ );+ expect(inner.readQuery).toHaveBeenCalledOnce();+ });++ it('turns every captured reference into a disabled cache on dispose', async () => {+ const inner = disposedLikeHost();+ const host = createRetirableCacheHost(inner);++ host.dispose();++ expect(inner.dispose).toHaveBeenCalledOnce();+ // The owner already logged why it retired the cache.+ expect(console.warn).not.toHaveBeenCalled();+ expect(host.disabled).toBe(true);+ expect(host.clientId).toBe('inner-client');+ // Durable checkpoints must not trust a generation from a retired cache.+ await expect(host.currentStorageGeneration()).rejects.toThrow(+ 'normalized GraphQL cache is unavailable'+ );+ await expect(host.readQuery(QUERY)).resolves.toEqual({ kind: 'miss' });+ await expect(+ host.writeQuery({ ...QUERY, data: { user: { id: 'u' } } })+ ).resolves.toMatchObject({ revision: INITIAL_CACHE_REVISION });+ await expect(+ host.search({ profile: 'quick-access-v1', query: '', limit: 10 })+ ).resolves.toEqual({ documents: [], nextCursor: null });+ await expect(host.teardown(1)).resolves.toBeUndefined();+ expect(inner.readQuery).not.toHaveBeenCalled();+ expect(inner.writeQuery).not.toHaveBeenCalled();+ expect(inner.search).not.toHaveBeenCalled();+ expect(inner.teardown).not.toHaveBeenCalled();+ });++ it('retires once', () => {+ const inner = disposedLikeHost();+ const host = createRetirableCacheHost(inner);++ host.dispose();+ host.dispose();++ expect(inner.dispose).toHaveBeenCalledOnce();+ });+});diff --git a/apps/web/src/lib/graphql-cache/host/retirable-host.ts b/apps/web/src/lib/graphql-cache/host/retirable-host.tsnew file mode 100644index 0000000000..f0c1946248--- /dev/null+++ b/apps/web/src/lib/graphql-cache/host/retirable-host.ts@@ -0,0 +1,58 @@+import { createNoopCacheHost } from './noop-host';+import type { CacheHost } from './types';++/**+ * Wraps a host that its owner may abandon mid-session. `dispose()` releases+ * the wrapped host and routes every later call to the storage-free no-op+ * host. References captured before retirement (the replaced urql client's+ * exchange, query closures) then see a disabled cache: reads miss to the+ * network, writes are dropped, and mutations bypass the durable queue,+ * instead of every call rejecting with "cache worker host was disposed".+ */+export function createRetirableCacheHost(inner: CacheHost): CacheHost {+ let current = inner;+ return {+ clientId: inner.clientId,+ get disabled() {+ return current.disabled;+ },+ currentRevision: () => current.currentRevision(),+ currentStorageGeneration: () => current.currentStorageGeneration(),+ readQuery: (args) => current.readQuery(args),+ readRecordsByKeys: (args) => current.readRecordsByKeys(args),+ search: (args) => current.search(args),+ entityFilter: (args) => current.entityFilter(args),+ writeQuery: (args) => current.writeQuery(args),+ hydrateQuery: (args) => current.hydrateQuery(args),+ enqueueOptimisticMutation: (args, claim) =>+ current.enqueueOptimisticMutation(args, claim),+ inspectQueryVariants: (args) => current.inspectQueryVariants(args),+ inspectQuery: (args) => current.inspectQuery(args),+ claimNextMutation: (owner, nowMs, leaseExpiresAtMs) =>+ current.claimNextMutation(owner, nowMs, leaseExpiresAtMs),+ deferOptimisticWrite: (transactionId, claim, nextAttemptAtMs, error) =>+ current.deferOptimisticWrite(+ transactionId,+ claim,+ nextAttemptAtMs,+ error+ ),+ commitOptimisticWrite: (transactionId, claim, args) =>+ current.commitOptimisticWrite(transactionId, claim, args),+ rollbackOptimisticWrite: (transactionId, claim, error) =>+ current.rollbackOptimisticWrite(transactionId, claim, error),+ invalidate: (keys) => current.invalidate(keys),+ deleteRecords: (keys) => current.deleteRecords(keys),+ teardown: (opKey) => current.teardown(opKey),+ clear: () => current.clear(),+ onOpsAffected: (cb) => current.onOpsAffected(cb),+ onCacheChanged: (cb, options) => current.onCacheChanged(cb, options),+ onCacheGenerationChanged: (cb) => current.onCacheGenerationChanged(cb),+ onMutationSettled: (cb) => current.onMutationSettled(cb),+ dispose() {+ if (current !== inner) return;+ current = createNoopCacheHost();+ inner.dispose();+ },+ };+}diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts b/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.tsindex efab6bed68..95f1477d19 100644--- a/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts+++ b/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts@@ -132,7 +132,8 @@ class IntegrationCore {private readonly logs: RuntimeLog[],private readonly activationGates: Map<number, Promise<void>>,private readonly hostInitGates: Map<number, Promise<void>>,- private readonly enqueueGates: Map<string, Promise<void>>+ private readonly enqueueGates: Map<string, Promise<void>>,+ private readonly onOwnerLockAcquired?: () => Promise<void>) {}addPort(): void {}@@ -148,6 +149,8 @@ class IntegrationCore {query: 'query' in request ? request.query : undefined,});if (request.kind === 'init') {+ // Like WASM: hold the owner lock and get the storage grant, then open.+ if (request.id === 0) await this.onOwnerLockAcquired?.();const gate =request.id === 0? this.activationGates.get(this.epoch)@@ -208,13 +211,14 @@ class IntegrationDedicatedWorker {) {installCacheEngineWorker({scope: this.scope,- createCore: () =>+ createCore: (options) =>new IntegrationCore(epoch,logs,activationGates,hostInitGates,- enqueueGates+ enqueueGates,+ options.onOwnerLockAcquired),ownerLockIsHeld: async () => true,});diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.test.ts b/apps/web/src/lib/graphql-cache/host/worker-host.test.tsindex 629bddefcb..13e24df94f 100644--- a/apps/web/src/lib/graphql-cache/host/worker-host.test.ts+++ b/apps/web/src/lib/graphql-cache/host/worker-host.test.ts@@ -138,6 +138,14 @@ class FakePageAdapter {this.options.onTerminalError?.(error);}+ unavailable(reason: string): void {+ this.options.onCacheUnavailable?.(reason);+ }++ superseded(reason: string): void {+ this.options.onCacheSuperseded?.(reason);+ }+private emit(message: WorkerMessage): void {this.onmessage?.({ data: message } as MessageEvent<WorkerMessage>);}@@ -225,6 +233,18 @@ describe('createWorkerCacheHost', () => {).resolves.toEqual(EMPTY_WRITE);});+ it("refuses a scope whose database would read as another scope's WAL file", () => {+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});++ const host = createWorkerCacheHost({ scope: 'scope-1-wal' });++ expect(warn).toHaveBeenCalledWith(+ '[graphql-cache] disabled: cache scope must not end with -wal'+ );+ expect(adapterFactory).not.toHaveBeenCalled();+ expect(host.disabled).toBe(true);+ });+it('routes hydration through the payload-projecting RPC', async () => {const host = createWorkerCacheHost({ scope: 'scope-1' });@@ -1287,6 +1307,139 @@ describe('createWorkerCacheHost', () => {);});+ it('stops using the cache without quarantine when another context holds the database', async () => {+ configureAdapter = (fake) => {+ fake.ignoredKinds.add('init');+ };+ localStorage.setItem('graphql-cache:scope', 'scope-1');+ const onInitializationError = vi.fn();+ const host = createWorkerCacheHost({+ scope: 'scope-1',+ onInitializationError,+ });+ const read = host.readQuery({ query: 'query Q { q }' });+ const unavailable = expect(read).rejects.toMatchObject({+ errorCode: 'owner-lock-unavailable',+ });+ await vi.waitFor(() =>+ expect(requireAdapter().requests.map(({ kind }) => kind)).toContain(+ 'init'+ )+ );+ const adapter = requireAdapter();++ adapter.unavailable('cache database owner lock is held by another context');++ await unavailable;+ expect(onInitializationError).toHaveBeenCalledExactlyOnceWith(+ expect.objectContaining({ errorCode: 'owner-lock-unavailable' })+ );+ expect(adapter.dispose).toHaveBeenCalledWith({ graceful: false });+ // Storage was never touched, so the scope stays reusable next load.+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');+ await expect(+ host.readQuery({ query: 'query Q { q }' })+ ).rejects.toMatchObject({ errorCode: 'owner-lock-unavailable' });+ adapter.unavailable('duplicate notice');+ expect(onInitializationError).toHaveBeenCalledOnce();+ });++ it('retires a ready host once a request is refused because the database is unavailable', async () => {+ localStorage.setItem('graphql-cache:scope', 'scope-1');+ const onInitializationError = vi.fn();+ const host = createWorkerCacheHost({+ scope: 'scope-1',+ onInitializationError,+ });+ await host.clear();+ const adapter = requireAdapter();+ adapter.ignoredKinds.add('read');+ const read = host.readQuery({ query: 'query Q { q }' });+ await vi.waitFor(() => expect(adapter.requests.at(-1)?.kind).toBe('read'));++ adapter.reject(+ adapter.requests.at(-1)?.id ?? -1,+ 'cache database owner lock is held by another context',+ 'owner-lock-unavailable'+ );++ await expect(read).rejects.toMatchObject({+ errorCode: 'owner-lock-unavailable',+ });+ expect(onInitializationError).toHaveBeenCalledOnce();+ expect(adapter.dispose).toHaveBeenCalledWith({ graceful: false });+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');+ });++ it('retires quietly and asks the app to reload when a newer build takes over', async () => {+ localStorage.setItem('graphql-cache:scope', 'scope-1');+ const onInitializationError = vi.fn();+ const onSuperseded = vi.fn();+ const host = createWorkerCacheHost({+ scope: 'scope-1',+ onInitializationError,+ onSuperseded,+ });+ await host.clear();+ const adapter = requireAdapter();++ adapter.superseded('a newer version of the app took over the local cache');+ adapter.superseded('duplicate notice');++ expect(onSuperseded).toHaveBeenCalledOnce();+ expect(onInitializationError).toHaveBeenCalledExactlyOnceWith(+ expect.objectContaining({ errorCode: 'owner-lock-unavailable' })+ );+ // Leaving like a navigating page stops the engine without counting as a+ // lost owner, so the newer build opens the same database.+ expect(adapter.dispose).toHaveBeenCalledWith({+ graceful: false,+ preserveDatabase: true,+ });+ // The newer build opens the same database, so the scope is kept.+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');+ await expect(+ host.readQuery({ query: 'query Q { q }' })+ ).rejects.toMatchObject({ errorCode: 'owner-lock-unavailable' });+ });++ it('reports an enqueue in flight at the handover as uncertain, never unsent', async () => {+ localStorage.setItem('graphql-cache:scope', 'scope-1');+ const host = createWorkerCacheHost({ scope: 'scope-1' });+ await host.clear();+ const adapter = requireAdapter();+ adapter.ignoredKinds.add('enqueue-optimistic-mutation');+ adapter.ignoredKinds.add('read');+ const mutation = host.enqueueOptimisticMutation(+ {+ uuid: '00000000-0000-4000-8000-000000000101',+ query: 'mutation Update { update }',+ data: { update: true },+ },+ { owner: 'runner', nowMs: 1, leaseExpiresAtMs: 101 }+ );+ const read = host.readQuery({ query: 'query Q { q }' });+ // The newer build replays the queued row, so the page must not send the+ // mutation itself; the read never touched durable state.+ const mutationRejected = expect(mutation).rejects.toMatchObject({+ errorCode: 'admitted-enqueue-uncertain',+ });+ const readRejected = expect(read).rejects.toMatchObject({+ errorCode: 'owner-lock-unavailable',+ });+ await vi.waitFor(() =>+ expect(adapter.requests.map(({ kind }) => kind)).toEqual(+ expect.arrayContaining(['enqueue-optimistic-mutation', 'read'])+ )+ );++ adapter.superseded('a newer version of the app took over the local cache');++ await mutationRejected;+ await readRejected;+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');+ });+it('quarantines transport scope before invoking the product failure callback', async () => {let runLock!: () => Promise<void>;vi.stubGlobal('navigator', {diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.ts b/apps/web/src/lib/graphql-cache/host/worker-host.tsindex 9b3f535908..56672fb7fd 100644--- a/apps/web/src/lib/graphql-cache/host/worker-host.ts+++ b/apps/web/src/lib/graphql-cache/host/worker-host.ts@@ -25,6 +25,7 @@ import {type MutationClaim,type MutationSettlement,OWNER_EPOCH_LOST_ERROR_CODE,+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE,parseStorageGeneration,type ReadRecordsByKeysArgs,type ReadRecordsByKeysResult,@@ -52,6 +53,7 @@ import {createCacheCoordinatorPageAdapter,} from '../worker/coordinator-page-adapter';import type { EngineOpenOutcome } from '../worker/coordinator-protocol';+import { isUnambiguousCacheScope } from '../worker/stale-databases';import {CacheBootstrapExhaustedError,COORDINATOR_CONNECT_ATTEMPTS,@@ -114,6 +116,9 @@ export interface WorkerHostOptions {initializationTimeoutMs?: number;/** Reports terminal initialization or coordinator-transport failure. */onInitializationError?: (error: Error) => void;+ /** A newer app build took the local cache over. The host has already+ * retired to the network; the app should reload the page into that build. */+ onSuperseded?: () => void;/** Allowlisted rollout cohort attached to browser-cache telemetry. */rolloutCohort?: CacheRolloutCohort;/** Injectable recorder for deterministic tests or alternate exporters. */@@ -177,6 +182,10 @@ const isOwnerEpochLoss = (error: unknown): error is CacheResponseError =>error instanceof CacheResponseError &&error.errorCode === OWNER_EPOCH_LOST_ERROR_CODE;+const isOwnerLockUnavailable = (error: unknown): error is CacheResponseError =>+ error instanceof CacheResponseError &&+ error.errorCode === OWNER_LOCK_UNAVAILABLE_ERROR_CODE;+export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {let pageTelemetry = options.telemetry? undefined@@ -188,7 +197,11 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {options.telemetry ?? pageTelemetry?.recorder);const now = (): number => globalThis.performance?.now() ?? Date.now();- const unsupportedReason = unsupportedBrowserReason();+ const unsupportedReason =+ unsupportedBrowserReason() ??+ (isUnambiguousCacheScope(options.scope)+ ? undefined+ : 'cache scope must not end with -wal');if (unsupportedReason) {telemetry?.record({name: 'graphql_cache.host_ready',@@ -197,7 +210,8 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {errorCode: 'unsupported',});telemetry?.flush();- return createNoopCacheHost(unsupportedReason);+ console.warn(`[graphql-cache] disabled: ${unsupportedReason}`);+ return createNoopCacheHost();}const clientId = crypto.randomUUID();@@ -233,6 +247,7 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {let latestReplacementEpoch = 0;let failureReported = false;let terminalFailureHandled = false;+ let superseded = false;let adapter: CacheCoordinatorPageAdapter | undefined;let registeredAdapter: CacheCoordinatorPageAdapter | undefined;let adapterDisposePromise: Promise<void> | undefined;@@ -318,6 +333,7 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {if (entry.timer !== undefined) clearTimeout(entry.timer);recordRequestOutcome(entry, 'error', error);entry.reject(error);+ if (isOwnerLockUnavailable(error)) retireUnavailable(msg.error);finishGracefulDisposeIfDrained();return;}@@ -450,6 +466,15 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {if (adapter === created && !registrationInProgress && !suspension)failTransport(error);},+ onCacheUnavailable: (reason) => {+ if (adapter === created) retireUnavailable(reason);+ },+ onCacheSuperseded: (reason) => {+ if (adapter !== created || superseded) return;+ superseded = true;+ retireUnavailable(reason, true);+ options.onSuperseded?.();+ },telemetry,});created.onmessage = (event) => {@@ -605,13 +630,66 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {durationMs:initializationStartedAt > 0 ? now() - initializationStartedAt : 0,});+ stopForSession(error);+ reportFailure(error);+ }++ /**+ * Fails this host for the rest of the page session: pending requests reject+ * (an admitted enqueue as uncertain when `uncertain`), subscribers are+ * dropped, and the coordinator connection closes.+ */+ function stopForSession(+ error: Error,+ { uncertain = false, preserveDatabase = false } = {}+ ): void {state = 'failed';initialization = undefined;initializationError = error;- rejectPending(error);+ rejectPending(error, uncertain);clearSubscribers();unregisterPagehide();- void disposeAdapter(false).then(finishTelemetry);+ void disposeAdapter(false, preserveDatabase).then(finishTelemetry);+ }++ /**+ * Another context, usually a tab on another deployed build, holds the+ * database, and the coordinator touched no storage. Nothing is quarantined:+ * this page stops using the cache until it reloads, and a later page load+ * tries again. A page handing the database to a newer build leaves like a+ * navigating page: its engine stops without counting as a lost owner, and+ * an enqueue already sent may have reached the durable queue, so it is+ * reported uncertain rather than sent again.+ */+ function retireUnavailable(reason: string, leaveForNewerBuild = false): void {+ if (+ terminalFailureHandled ||+ state === 'failed' ||+ state === 'suspended' ||+ state === 'disposing' ||+ state === 'disposed'+ )+ return;+ terminalFailureHandled = true;+ const error = new CacheResponseError(+ reason,+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE+ );+ if (state === 'initializing') {+ telemetry?.record({+ name: 'graphql_cache.host_ready',+ operationCategory: 'initialization',+ outcome: 'error',+ errorCode: 'lock',+ durationMs:+ initializationStartedAt > 0 ? now() - initializationStartedAt : 0,+ });+ }+ stopStorageHealthSampling();+ stopForSession(error, {+ uncertain: leaveForNewerBuild,+ preserveDatabase: leaveForNewerBuild,+ });reportFailure(error);}@@ -630,13 +708,7 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {);if (state === 'disposed' && !admittedWorkIsUncertain) return;terminalFailureHandled = true;- state = 'failed';- initialization = undefined;- initializationError = error;- rejectPending(error, true);- clearSubscribers();- unregisterPagehide();- void disposeAdapter(false).then(finishTelemetry);+ stopForSession(error, { uncertain: true });// Product failure handling may immediately construct another host. Make// the matching old scope unreachable before invoking that callback.const quarantine = storageWasUntoucheddiff --git a/apps/web/src/lib/graphql-cache/protocol.ts b/apps/web/src/lib/graphql-cache/protocol.tsindex 3aeea6820a..af6d6026e9 100644--- a/apps/web/src/lib/graphql-cache/protocol.ts+++ b/apps/web/src/lib/graphql-cache/protocol.ts@@ -585,12 +585,21 @@ export type CacheRequest = { id: number } & (/** Stable machine-readable cache RPC rejection codes. */export type CacheResponseErrorCode =| 'owner-epoch-lost'+ | 'owner-lock-unavailable'| 'admitted-enqueue-uncertain';/** Old-owner work was rejected after fenced engine loss and was not replayed. */export const OWNER_EPOCH_LOST_ERROR_CODE: CacheResponseErrorCode ='owner-epoch-lost';+/**+ * Another context holds the database, so this page's cache stays off until+ * reload: typically another deployed build keeps its owner lock, or a closing+ * tab's worker still has its files open. The request never reached an engine.+ */+export const OWNER_LOCK_UNAVAILABLE_ERROR_CODE: CacheResponseErrorCode =+ 'owner-lock-unavailable';+/** An enqueue send was admitted before its unfenced transport became uncertain. */export const ADMITTED_ENQUEUE_UNCERTAIN_ERROR_CODE: CacheResponseErrorCode ='admitted-enqueue-uncertain';@@ -640,6 +649,7 @@ export const isCacheResponseErrorCode = (value: unknown): value is CacheResponseErrorCode =>value === OWNER_EPOCH_LOST_ERROR_CODE ||+ value === OWNER_LOCK_UNAVAILABLE_ERROR_CODE ||value === ADMITTED_ENQUEUE_UNCERTAIN_ERROR_CODE;/** Identifies a coordinator-fenced rejection from a lost owner epoch. */@@ -650,6 +660,14 @@ export const isOwnerEpochLostError = ('errorCode' in value &&value.errorCode === OWNER_EPOCH_LOST_ERROR_CODE;+/** Identifies a request refused because another context holds the database. */+export const isOwnerLockUnavailableError = (+ value: unknown+): value is Error & { errorCode: 'owner-lock-unavailable' } =>+ value instanceof Error &&+ 'errorCode' in value &&+ value.errorCode === OWNER_LOCK_UNAVAILABLE_ERROR_CODE;+/** Identifies the host-only uncertainty result for an admitted enqueue send. */export const isAdmittedEnqueueUncertainError = (value: unknowndiff --git a/apps/web/src/lib/graphql-cache/telemetry.ts b/apps/web/src/lib/graphql-cache/telemetry.tsindex 1edd8436bd..36c26e6693 100644--- a/apps/web/src/lib/graphql-cache/telemetry.ts+++ b/apps/web/src/lib/graphql-cache/telemetry.ts@@ -133,6 +133,12 @@ export const CACHE_OWNER_EVENTS = ['abrupt-loss','replacement','multiple-owner-detected',+ 'owner-lock-unavailable',+ 'storage-busy',+ 'takeover-granted',+ 'superseded',+ 'stale-databases-removed',+ 'stale-database-kept',] as const;export type CacheOwnerEvent = (typeof CACHE_OWNER_EVENTS)[number];diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.tsindex 4285e09996..138577934e 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts@@ -299,12 +299,14 @@ test('production cache-wasm Turso engine preserves graceful data and atomicallyrealTursoDataPreservedGracefully: boolean;gracefulCloseReleasedOwnerLock: boolean;gracefulReplacementWaitedForPhysicalLock: boolean;- gracefulPendingOwnerLockRequests: number;+ gracefulQueuedOwnerLockRequests: number;+ gracefulRetriedHeldOwnerLock: boolean;abruptInflightRejected: boolean;abruptRequestReplayCount: number;abruptOwnerPageStayedAlive: boolean;recoveryReplacementWaitedForPhysicalLock: boolean;- recoveryPendingOwnerLockRequests: number;+ recoveryQueuedOwnerLockRequests: number;+ recoveryRetriedHeldOwnerLock: boolean;atomicRecoveryOpenWipedToMiss: boolean;recoveryDatabaseAction: string;ownerEpochs: number[];@@ -316,12 +318,14 @@ test('production cache-wasm Turso engine preserves graceful data and atomicallyrealTursoDataPreservedGracefully: true,gracefulCloseReleasedOwnerLock: true,gracefulReplacementWaitedForPhysicalLock: true,- gracefulPendingOwnerLockRequests: 1,+ gracefulQueuedOwnerLockRequests: 0,+ gracefulRetriedHeldOwnerLock: true,abruptInflightRejected: true,abruptRequestReplayCount: 1,abruptOwnerPageStayedAlive: true,recoveryReplacementWaitedForPhysicalLock: true,- recoveryPendingOwnerLockRequests: 1,+ recoveryQueuedOwnerLockRequests: 0,+ recoveryRetriedHeldOwnerLock: true,atomicRecoveryOpenWipedToMiss: true,recoveryDatabaseAction: 'wipe-before-open',ownerEpochs: [1, 2, 3],@@ -329,3 +333,88 @@ test('production cache-wasm Turso engine preserves graceful data and atomically});expect(browserErrors).toEqual([]);});++test('production engine waits out a departing owner that still holds the files, and never wipes them', async ({+ context,+ page,+}, testInfo) => {+ const browserErrors: string[] = [];+ const watch = (candidate: typeof page): void => {+ candidate.on('console', (message) => {+ if (message.type() === 'error') browserErrors.push(message.text());+ });+ candidate.on('pageerror', (error) => browserErrors.push(error.message));+ };+ watch(page);+ context.on('page', watch);++ await page.goto(harnessPath(testInfo.project.name, 'production-busy.html'));+ const result = page.locator('#result');+ await expect(result).toHaveAttribute('data-status', 'passed', {+ timeout: 80_000,+ });+ const report = JSON.parse((await result.textContent()) ?? '') as Record<+ string,+ unknown+ >;++ expect(report).toEqual({+ passed: true,+ firstReplacementDatabaseAction: 'open-existing',+ firstReplacementWaitedForFiles: true,+ firstReplacementKeptData: true,+ firstRecoveryAttempts: 0,+ busyReplacementDatabaseAction: 'open-existing',+ busyReplacementOpened: false,+ busyReplacementTerminated: true,+ survivorToldFilesAreOpen: true,+ survivorReadFellBack: true,+ retriedWhileBusy: false,+ laterTabDatabaseAction: 'open-existing',+ laterTabKeptData: true,+ protocolErrors: [],+ });+ expect(browserErrors).toEqual([]);+});++test('a newer build takes the production database over and older builds are turned away', async ({+ context,+ page,+}, testInfo) => {+ const browserErrors: string[] = [];+ const watch = (candidate: typeof page): void => {+ candidate.on('console', (message) => {+ if (message.type() === 'error') browserErrors.push(message.text());+ });+ candidate.on('pageerror', (error) => browserErrors.push(error.message));+ };+ watch(page);+ context.on('page', watch);++ await page.goto(+ harnessPath(testInfo.project.name, 'production-takeover.html')+ );+ const result = page.locator('#result');+ await expect(result).toHaveAttribute('data-status', 'passed', {+ timeout: 80_000,+ });+ const report = JSON.parse((await result.textContent()) ?? '') as Record<+ string,+ unknown+ >;++ expect(report).toEqual({+ passed: true,+ newBuildDatabaseAction: 'open-existing',+ newBuildKeptData: true,+ oldEngineStoppedLikeNavigation: true,+ oldTabsSentOn: true,+ olderBuildTurnedAway: true,+ olderBuildSentOn: false,+ lateOldTabRunsUncached: true,+ lateOldTabSentOn: false,+ newBuildSentOn: false,+ protocolErrors: [],+ });+ expect(browserErrors).toEqual([]);+});diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.tsindex 0afe763de6..202beb638d 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts@@ -13,10 +13,11 @@ import {validateCoordinatorToEngineEnvelope,validatePageToEngineEnvelope,} from '../coordinator-protocol';+import { OWNER_LOCK_RETRY_DELAYS_MS } from '../startup';declare const self: DedicatedWorkerGlobalScope;-const withVersion = <T extends { coordinatorVersion: 5 }>(+const withVersion = <T extends { coordinatorVersion: 6 }>(value: T extends unknown ? Omit<T, 'coordinatorVersion'> : never): T =>({@@ -157,7 +158,9 @@ async function activate(activationValue = activation;controlPort = port;telemetry = new BroadcastChannel(`graphql-cache-wp08:${activation.scope}`);- await new Promise<void>((resolve) => {+ // Like the production engine: announce assets, take the owner lock, then+ // touch storage only after the coordinator's `open-engine` grant.+ const storageGrant = new Promise<void>((resolve) => {port.onmessage = ({ data }) => {const parsed = validateCoordinatorToEngineEnvelope(Array.isArray(data) ? data[1] : data@@ -169,180 +172,218 @@ async function activate()resolve();};- port.start();- port.postMessage([0]);- sendEngine(- withVersion<EngineToCoordinatorEnvelope>({- kind: 'engine-assets-ready',- tabId: activation.tabId,- ownerEpoch: activation.ownerEpoch,- })- );});- await navigator.locks.request(- activation.ownerLockName,- { mode: 'exclusive' },- async (lock) => {- if (!lock) throw new Error('owner lock was not acquired');- telemetry?.postMessage({- kind: 'lock-acquired',- tabId: activation.tabId,- ownerEpoch: activation.ownerEpoch,- });- let database: IDBDatabase | undefined;- const engineState: FakeEngineState = { revision: 0n };- let queue = Promise.resolve();- let draining = false;- let requestShutdown: (() => void) | undefined;- const shutdown = new Promise<void>((resolve) => {- requestShutdown = resolve;- });- try {- if (activation.databaseAction === 'wipe-before-open') {- telemetry?.postMessage({- kind: 'wipe-started',- ownerEpoch: activation.ownerEpoch,- });- await deleteDatabase(activation.scope);- telemetry?.postMessage({- kind: 'wipe-completed',- ownerEpoch: activation.ownerEpoch,- });+ port.start();+ port.postMessage([0]);+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'engine-assets-ready',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ );+ // Never queue for the owner lock: retry it briefly, then give up without+ // touching storage, exactly like the production engine.+ for (let attempt = 1; ; attempt += 1) {+ let lockWasBusy = false;+ await navigator.locks.request(+ activation.ownerLockName,+ { mode: 'exclusive', ifAvailable: true },+ async (lock) => {+ if (!lock) {+ lockWasBusy = true;+ return;}- database = await openDatabase(activation.scope);- const activeDatabase = database;- port.onmessage = (event: MessageEvent<unknown>) => {- if (Array.isArray(event.data) && event.data[0] === 1) {- requestShutdown?.();- return;+ telemetry?.postMessage({+ kind: 'lock-acquired',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ });+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'owner-lock-acquired',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ );+ await storageGrant;+ let database: IDBDatabase | undefined;+ const engineState: FakeEngineState = { revision: 0n };+ let queue = Promise.resolve();+ let draining = false;+ let requestShutdown: (() => void) | undefined;+ const shutdown = new Promise<void>((resolve) => {+ requestShutdown = resolve;+ });+ try {+ if (activation.databaseAction === 'wipe-before-open') {+ telemetry?.postMessage({+ kind: 'wipe-started',+ ownerEpoch: activation.ownerEpoch,+ });+ await deleteDatabase(activation.scope);+ telemetry?.postMessage({+ kind: 'wipe-completed',+ ownerEpoch: activation.ownerEpoch,+ });}- const payload =- Array.isArray(event.data) && event.data[0] === 0- ? event.data[1]- : event.data;- const parsed = validateCoordinatorToEngineEnvelope(payload);- if (!parsed.ok) return;- const message: CoordinatorToEngineEnvelope = parsed.value;- if (message.ownerEpoch !== activation.ownerEpoch) return;- switch (message.kind) {- case 'heartbeat':- if (!ignoreHeartbeats) {- sendEngine(- withVersion<EngineToCoordinatorEnvelope>({- kind: 'heartbeat-ack',- ownerEpoch: activation.ownerEpoch,- heartbeatId: message.heartbeatId,- })- );- }- break;- case 'drain-engine':- if (draining) return;- draining = true;- void queue.finally(() => requestShutdown?.());- break;- case 'engine-request':- if (draining) return;- telemetry?.postMessage({- kind: 'request-started',- tabId: activation.tabId,- ownerEpoch: activation.ownerEpoch,- routeId: message.routeId,- requestKind: message.request.kind,- requestId: message.request.id,- slow:- message.request.kind === 'read' &&- message.request.query.includes('Slow'),- });- queue = queue.then(async () => {- let response: CacheResponse;- try {- response = await execute(- activeDatabase,- message.request,- engineState+ database = await openDatabase(activation.scope);+ const activeDatabase = database;+ port.onmessage = (event: MessageEvent<unknown>) => {+ if (Array.isArray(event.data) && event.data[0] === 1) {+ requestShutdown?.();+ return;+ }+ const payload =+ Array.isArray(event.data) && event.data[0] === 0+ ? event.data[1]+ : event.data;+ const parsed = validateCoordinatorToEngineEnvelope(payload);+ if (!parsed.ok) return;+ const message: CoordinatorToEngineEnvelope = parsed.value;+ if (message.ownerEpoch !== activation.ownerEpoch) return;+ switch (message.kind) {+ case 'heartbeat':+ if (!ignoreHeartbeats) {+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'heartbeat-ack',+ ownerEpoch: activation.ownerEpoch,+ heartbeatId: message.heartbeatId,+ }));- } catch (error) {- response = {- id: message.request.id,- ok: false,- error:- error instanceof Error ? error.message : String(error),- };}- sendEngine(- withVersion<EngineToCoordinatorEnvelope>({- kind: 'engine-response',- ownerEpoch: activation.ownerEpoch,- routeId: message.routeId,- response,- })- );- if (- response.ok &&- (message.request.kind === 'write' ||- message.request.kind === 'clear')- ) {+ break;+ case 'drain-engine':+ if (draining) return;+ draining = true;+ void queue.finally(() => requestShutdown?.());+ break;+ case 'engine-request':+ if (draining) return;+ telemetry?.postMessage({+ kind: 'request-started',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ routeId: message.routeId,+ requestKind: message.request.kind,+ requestId: message.request.id,+ slow:+ message.request.kind === 'read' &&+ message.request.query.includes('Slow'),+ });+ queue = queue.then(async () => {+ let response: CacheResponse;+ try {+ response = await execute(+ activeDatabase,+ message.request,+ engineState+ );+ } catch (error) {+ response = {+ id: message.request.id,+ ok: false,+ error:+ error instanceof Error ? error.message : String(error),+ };+ }sendEngine(withVersion<EngineToCoordinatorEnvelope>({- kind: 'engine-push',+ kind: 'engine-response',ownerEpoch: activation.ownerEpoch,- push: {- kind: 'cache-changed',- revision:- engineState.revision.toString() as typeof INITIAL_CACHE_REVISION,- },+ routeId: message.routeId,+ response,}));- }- });- break;- }- };- sendEngine(- withVersion<EngineToCoordinatorEnvelope>({- kind: 'engine-ready',+ if (+ response.ok &&+ (message.request.kind === 'write' ||+ message.request.kind === 'clear')+ ) {+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'engine-push',+ ownerEpoch: activation.ownerEpoch,+ push: {+ kind: 'cache-changed',+ revision:+ engineState.revision.toString() as typeof INITIAL_CACHE_REVISION,+ },+ })+ );+ }+ });+ break;+ }+ };+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'engine-ready',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ ownerLockName: activation.ownerLockName,+ ownerLockHeld: true,+ databaseActionProof:+ activation.databaseAction === 'wipe-before-open'+ ? 'wiped-before-open'+ : 'opened-existing',+ openOutcome:+ activation.databaseAction === 'wipe-before-open'+ ? 'reset-storage-uncertain'+ : 'opened-existing',+ })+ );+ telemetry?.postMessage({+ kind: 'ready',tabId: activation.tabId,ownerEpoch: activation.ownerEpoch,- ownerLockName: activation.ownerLockName,- ownerLockHeld: true,- databaseActionProof:- activation.databaseAction === 'wipe-before-open'- ? 'wiped-before-open'- : 'opened-existing',- openOutcome:- activation.databaseAction === 'wipe-before-open'- ? 'reset-storage-uncertain'- : 'opened-existing',- })- );- telemetry?.postMessage({- kind: 'ready',- tabId: activation.tabId,- ownerEpoch: activation.ownerEpoch,- });+ });- await shutdown;- await queue;- database.close();- database = undefined;- sendEngine(- withVersion<EngineToCoordinatorEnvelope>({- kind: 'engine-drained',+ await shutdown;+ await queue;+ database.close();+ database = undefined;+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'engine-drained',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ );+ } finally {+ database?.close();+ telemetry?.postMessage({+ kind: 'lock-releasing',tabId: activation.tabId,ownerEpoch: activation.ownerEpoch,- })- );- } finally {- database?.close();- telemetry?.postMessage({- kind: 'lock-releasing',+ });+ }+ }+ );+ if (!lockWasBusy) break;+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'owner-lock-busy',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ );+ const delayMs = OWNER_LOCK_RETRY_DELAYS_MS[attempt - 1];+ if (delayMs === undefined) {+ sendEngine(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'owner-lock-unavailable',tabId: activation.tabId,ownerEpoch: activation.ownerEpoch,- });- }+ })+ );+ return;}- );+ await new Promise<void>((resolve) => {+ setTimeout(resolve, delayMs);+ });+ }port.close();telemetry?.close();self.close();diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.tsindex bce3d68b6a..3a2e7ef1f9 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts@@ -22,7 +22,9 @@ export type ProductionHarnessCommand =commandId: string;request: ProductionCacheRequestWithoutId;}- | { kind: 'terminate-worker'; commandId: string };+ | { kind: 'terminate-worker'; commandId: string }+ /** Leaves the page the way `pagehide` does on navigation or reload. */+ | { kind: 'navigate-away'; commandId: string };export type ProductionHarnessEnvelope =| {@@ -49,5 +51,7 @@ export type ProductionHarnessEnvelope =ok: false;error: string;}- | { kind: 'protocol-error'; error: string };+ | { kind: 'protocol-error'; error: string }+ | { kind: 'cache-unavailable'; reason: string }+ | { kind: 'cache-superseded'; reason: string };};diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy-harness.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy-harness.tsnew file mode 100644index 0000000000..d1b743c4d9--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy-harness.ts@@ -0,0 +1,344 @@+import {+ cacheDatabaseIdentity,+ databaseOwnerLockName,+} from '../coordinator-protocol';+import type {+ ProductionHarnessCommand,+ ProductionHarnessEnvelope,+} from './production-browser-wire';+import type {+ FileHolderRequest,+ FileHolderResponse,+} from './production-cache.file-holder-worker';++// A page that leaves while it owns the cache terminates its engine worker. The+// browser can hand the owner Web Lock to the next engine before that worker+// lets go of the database files. This harness recreates that gap with a worker+// that keeps the files open without the lock.++const resultElement = document.querySelector<HTMLElement>('#result');+if (!resultElement) throw new Error('missing result element');++type CommandWithoutId = ProductionHarnessCommand extends infer Command+ ? Command extends ProductionHarnessCommand+ ? Omit<Command, 'commandId'>+ : never+ : never;++type RuntimeTelemetry = {+ kind: string;+ tabId: string;+ ownerEpoch: number;+ databaseAction: 'open-existing' | 'wipe-before-open';+ receivedAt: number;+};++/** Longer than both the retry this change replaced and the gap seen in+ * production, and well inside the engine's busy-file wait. */+const HOLD_AFTER_OWNER_LOCK_MS = 3_000;++const assert: (condition: unknown, message: string) => asserts condition = (+ condition,+ message+) => {+ if (!condition) throw new Error(message);+};++const sleep = (ms: number) =>+ new Promise<void>((resolve) => setTimeout(resolve, ms));++const waitUntil = async (+ description: string,+ predicate: () => boolean | Promise<boolean>,+ timeoutMs = 30_000+): Promise<void> => {+ const started = performance.now();+ while (!(await predicate())) {+ if (performance.now() - started > timeoutMs) {+ throw new Error(`timed out waiting for ${description}`);+ }+ await sleep(20);+ }+};++const run = async (): Promise<Record<string, unknown>> => {+ const scope = `busy-files-${crypto.randomUUID()}`;+ const identity = cacheDatabaseIdentity(scope);+ const ownerLockName = databaseOwnerLockName(scope);+ const tabChannel = new BroadcastChannel(+ `graphql-cache-wp08-production-tabs:${scope}`+ );+ const telemetryChannel = new BroadcastChannel(+ `graphql-cache-wp08-production:${scope}`+ );+ const popups = new Map<string, Window>();+ const registered = new Set<string>();+ const workers: Array<{ tabId: string; ownerEpoch: number }> = [];+ const terminated: Array<{ ownerEpoch: number; reason: string }> = [];+ const unavailable = new Map<string, string>();+ const telemetry: RuntimeTelemetry[] = [];+ const protocolErrors: string[] = [];+ const pendingCommands = new Map<+ string,+ {+ resolve: (value: unknown) => void;+ reject: (error: Error) => void;+ timer: ReturnType<typeof setTimeout>;+ }+ >();++ tabChannel.onmessage = (event: MessageEvent<ProductionHarnessEnvelope>) => {+ const message = event.data;+ if (message.source !== 'tab') return;+ switch (message.event.kind) {+ case 'registered':+ registered.add(message.tabId);+ break;+ case 'worker-created':+ workers.push({+ tabId: message.tabId,+ ownerEpoch: message.event.ownerEpoch,+ });+ break;+ case 'worker-terminated':+ terminated.push({+ ownerEpoch: message.event.ownerEpoch,+ reason: message.event.reason,+ });+ break;+ case 'cache-unavailable':+ unavailable.set(message.tabId, message.event.reason);+ break;+ case 'protocol-error':+ protocolErrors.push(message.event.error);+ break;+ case 'command-result': {+ const pending = pendingCommands.get(message.event.commandId);+ if (!pending) return;+ pendingCommands.delete(message.event.commandId);+ clearTimeout(pending.timer);+ if (message.event.ok) pending.resolve(message.event.result);+ else pending.reject(new Error(message.event.error));+ break;+ }+ }+ };+ telemetryChannel.onmessage = (+ event: MessageEvent<Omit<RuntimeTelemetry, 'receivedAt'>>+ ) => {+ telemetry.push({ ...event.data, receivedAt: performance.now() });+ };++ const command = async (+ targetTabId: string,+ value: CommandWithoutId+ ): Promise<unknown> => {+ const commandId = crypto.randomUUID();+ const result = new Promise<unknown>((resolve, reject) => {+ const timer = setTimeout(() => {+ pendingCommands.delete(commandId);+ reject(new Error(`busy-file command timed out: ${value.kind}`));+ }, 30_000);+ pendingCommands.set(commandId, { resolve, reject, timer });+ });+ tabChannel.postMessage({+ source: 'harness',+ targetTabId,+ command: { ...value, commandId } as ProductionHarnessCommand,+ } satisfies ProductionHarnessEnvelope);+ return await result;+ };++ const openTab = (tabId: string): void => {+ const url = new URL('./production-tab.html', location.href);+ url.searchParams.set('scope', scope);+ url.searchParams.set('tabId', tabId);+ const popup = window.open(url, `${scope}:${tabId}`);+ if (!popup) throw new Error(`popup blocked for ${tabId}`);+ popups.set(tabId, popup);+ };++ const ownerOf = (ownerEpoch: number) => {+ const owner = workers.find((worker) => worker.ownerEpoch === ownerEpoch);+ assert(owner, `missing owner for epoch ${ownerEpoch}`);+ return owner;+ };++ const event = (kind: string, ownerEpoch: number) =>+ telemetry.find(+ (candidate) =>+ candidate.kind === kind && candidate.ownerEpoch === ownerEpoch+ );++ const heldOwnerLocks = async (): Promise<number> =>+ (await navigator.locks.query()).held?.filter(+ (lock) => lock.name === ownerLockName+ ).length ?? 0;++ const holder = new Worker(+ new URL('./production-cache.file-holder-worker.ts', import.meta.url),+ { type: 'module', name: `busy-file-holder:${scope}` }+ );+ const askHolder = (request: FileHolderRequest): Promise<void> =>+ new Promise((resolve, reject) => {+ holder.onmessage = (message: MessageEvent<FileHolderResponse>) => {+ if (message.data.kind === 'error')+ reject(new Error(message.data.error));+ else resolve();+ };+ holder.postMessage(request);+ });++ /**+ * The owner leaves like a navigating page. While the harness holds the+ * owner lock the holder opens both files, then the next engine may take the+ * lock and finds the files busy. Resolves when that engine holds the lock.+ */+ const leaveWhileFilesStayOpen = async (+ ownerEpoch: number+ ): Promise<number> => {+ let releaseLock!: () => void;+ let markAcquired!: () => void;+ const acquired = new Promise<void>((resolve) => {+ markAcquired = resolve;+ });+ const lockReleased = navigator.locks.request(+ ownerLockName,+ { mode: 'exclusive' },+ async () => {+ markAcquired();+ await new Promise<void>((resolve) => {+ releaseLock = resolve;+ });+ }+ );+ await command(ownerOf(ownerEpoch).tabId, { kind: 'navigate-away' });+ await acquired;+ await askHolder({ kind: 'hold', paths: [identity, `${identity}-wal`] });+ releaseLock();+ await lockReleased;+ await waitUntil(+ `epoch ${ownerEpoch + 1} to take the owner lock`,+ async () => (await heldOwnerLocks()) === 1+ );+ return performance.now();+ };++ for (const tabId of ['busy-a', 'busy-b', 'busy-c']) openTab(tabId);+ await waitUntil('three registrations', () => registered.size === 3);+ await waitUntil('epoch 1 ready', () => event('ready', 1) !== undefined);+ const standby = [...registered].find((tabId) => tabId !== ownerOf(1).tabId);+ assert(standby, 'missing standby');+ await command(standby, { kind: 'write', value: 'busy-preserved' });++ // Phase 1: the files stay open for 3 seconds after the next engine takes+ // the owner lock. It must wait and reopen the same database.+ const firstLockedAt = await leaveWhileFilesStayOpen(1);+ assert(+ event('activation-started', 2)?.databaseAction === 'open-existing',+ 'navigation replacement did not start with open-existing'+ );+ await sleep(HOLD_AFTER_OWNER_LOCK_MS);+ assert(!event('ready', 2), 'epoch 2 opened while its files were busy');+ await askHolder({ kind: 'release' });+ const firstReleasedAt = performance.now();+ await waitUntil('epoch 2 ready', () => event('ready', 2) !== undefined);+ const secondOwner = ownerOf(2).tabId;+ expectHit(await command(secondOwner, { kind: 'read' }), 'busy-preserved');+ const firstRecoveryAttempts = telemetry.filter(+ (candidate) =>+ candidate.kind === 'activation-started' && candidate.ownerEpoch > 2+ ).length;++ // Phase 2: the files stay open past the engine's whole wait. The cache+ // must turn off quietly without wiping, and a later tab reopens the data.+ const survivor = [...registered].find(+ (tabId) => tabId !== ownerOf(1).tabId && tabId !== secondOwner+ );+ assert(survivor, 'missing surviving tab');+ await leaveWhileFilesStayOpen(2);+ await waitUntil(+ 'the surviving tab to be told the cache is unavailable',+ () => unavailable.has(survivor),+ 30_000+ );+ let survivorRead = 'hit';+ try {+ await command(survivor, { kind: 'read' });+ } catch (error) {+ survivorRead = error instanceof Error ? error.message : String(error);+ }+ await sleep(1_000);+ const retriedWhileBusy = telemetry.some(+ (candidate) =>+ candidate.kind === 'activation-started' && candidate.ownerEpoch > 3+ );+ await askHolder({ kind: 'release' });+ openTab('busy-d');+ await waitUntil('epoch 4 ready', () => event('ready', 4) !== undefined);+ expectHit(+ await command(ownerOf(4).tabId, { kind: 'read' }),+ 'busy-preserved'+ );++ const result = {+ passed: true,+ firstReplacementDatabaseAction: event('activation-started', 2)+ ?.databaseAction,+ firstReplacementWaitedForFiles:+ (event('ready', 2)?.receivedAt ?? 0) >= firstReleasedAt &&+ firstReleasedAt - firstLockedAt >= HOLD_AFTER_OWNER_LOCK_MS,+ firstReplacementKeptData: true,+ firstRecoveryAttempts,+ busyReplacementDatabaseAction: event('activation-started', 3)+ ?.databaseAction,+ busyReplacementOpened: event('ready', 3) !== undefined,+ busyReplacementTerminated: terminated.some(+ (candidate) =>+ candidate.ownerEpoch === 3 &&+ candidate.reason.includes('files stayed busy')+ ),+ survivorToldFilesAreOpen:+ unavailable.get(survivor)?.includes('files are still open') ?? false,+ survivorReadFellBack: survivorRead.includes('files are still open'),+ retriedWhileBusy,+ laterTabDatabaseAction: event('activation-started', 4)?.databaseAction,+ laterTabKeptData: true,+ protocolErrors,+ };++ holder.terminate();+ for (const popup of popups.values()) popup.close();+ tabChannel.close();+ telemetryChannel.close();+ return result;+};++function expectHit(value: unknown, expected: string): void {+ assert(+ typeof value === 'object' &&+ value !== null &&+ (value as { kind?: unknown }).kind === 'hit' &&+ (+ value as {+ data?: { user?: { soup?: { items?: Array<{ id?: unknown }> } } };+ }+ ).data?.user?.soup?.items?.[0]?.id === expected,+ `expected cache hit ${expected}`+ );+}++void (async () => {+ try {+ const result = await run();+ resultElement.dataset.status = 'passed';+ resultElement.textContent = JSON.stringify(result, null, 2);+ } catch (error) {+ resultElement.dataset.status = 'failed';+ // Firefox stacks leave out the message.+ resultElement.textContent =+ error instanceof Error+ ? `${error.message}\n${error.stack ?? ''}`+ : String(error);+ }+})();diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy.html b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy.htmlnew file mode 100644index 0000000000..e82d0f2e1b--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy.html@@ -0,0 +1,12 @@+<!doctype html>+<html lang="en">+ <head>+ <meta charset="UTF-8" />+ <link rel="icon" href="data:," />+ <title>Production Turso busy-file harness</title>+ </head>+ <body>+ <pre id="result" data-status="running">running</pre>+ <script type="module" src="./production-busy-harness.ts"></script>+ </body>+</html>diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.tsindex 65298ffbf3..0432637d84 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts@@ -50,6 +50,7 @@ const report = (event: CacheEngineRuntimeEvent): void => {event.request.kind === 'read' &&event.request.query.includes('CacheAdmissionBarrier'),reason: event.kind === 'fatal' ? event.reason : undefined,+ attempt: event.kind === 'owner-lock-busy' ? event.attempt : undefined,fatalCode: event.kind === 'fatal' ? event.fatalCode : undefined,performanceMemoryAvailable:event.kind === 'ready' && workerPerformance.memory !== undefined,diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.file-holder-worker.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.file-holder-worker.tsnew file mode 100644index 0000000000..2d7d666a2d--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.file-holder-worker.ts@@ -0,0 +1,64 @@+/// <reference lib="webworker" />++/**+ * Keeps database files open without their owner Web Lock, standing in for a+ * page's cache worker that is still being torn down after the page left.+ */+export type FileHolderRequest =+ | { kind: 'hold'; paths: string[] }+ | { kind: 'release' };++export type FileHolderResponse =+ | { kind: 'holding' }+ | { kind: 'released' }+ | { kind: 'error'; error: string };++const worker = self as unknown as DedicatedWorkerGlobalScope;+let handles: FileSystemSyncAccessHandle[] = [];++const isBusy = (error: unknown): boolean =>+ error instanceof DOMException && error.name === 'NoModificationAllowedError';++/** The terminated owner may still be letting go of the file. */+const openWhenFree = async (+ root: FileSystemDirectoryHandle,+ path: string+): Promise<FileSystemSyncAccessHandle> => {+ const file = await root.getFileHandle(path);+ for (let attempt = 1; ; attempt += 1) {+ try {+ return await file.createSyncAccessHandle();+ } catch (error) {+ if (!isBusy(error) || attempt >= 500) throw error;+ await new Promise((resolve) => setTimeout(resolve, 10));+ }+ }+};++const handle = async (+ request: FileHolderRequest+): Promise<FileHolderResponse> => {+ if (request.kind === 'hold') {+ const root = await navigator.storage.getDirectory();+ for (const path of request.paths) {+ handles.push(await openWhenFree(root, path));+ }+ return { kind: 'holding' };+ }+ for (const held of handles) held.close();+ handles = [];+ return { kind: 'released' };+};++worker.onmessage = async (event: MessageEvent<FileHolderRequest>) => {+ let response: FileHolderResponse;+ try {+ response = await handle(event.data);+ } catch (error) {+ response = {+ kind: 'error',+ error: error instanceof Error ? error.message : String(error),+ };+ }+ worker.postMessage(response);+};diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.tsindex bd9af4ece7..6b2bfd993b 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts@@ -155,10 +155,12 @@ const run = async (): Promise<Record<string, unknown>> => {);return { acquired, release, completed };};- const waitForPendingReplacementLock = async (+ // The replacement retries the contender's lock without queueing behind it.+ const waitForRetriedReplacementLock = async (+ ownerEpoch: number,description: string- ): Promise<{ held: number; pending: number }> => {- let evidence = { held: 0, pending: 0 };+ ): Promise<{ held: number; pending: number; busyAttempts: number }> => {+ let evidence = { held: 0, pending: 0, busyAttempts: 0 };await waitUntil(description, async () => {const snapshot = await navigator.locks.query();evidence = {@@ -168,8 +170,16 @@ const run = async (): Promise<Record<string, unknown>> => {pending:snapshot.pending?.filter((lock) => lock.name === ownerLockName).length ?? 0,+ busyAttempts: telemetry.filter(+ (event) =>+ event.kind === 'owner-lock-busy' && event.ownerEpoch === ownerEpoch+ ).length,};- return evidence.held === 1 && evidence.pending === 1;+ return (+ evidence.held === 1 &&+ evidence.pending === 0 &&+ evidence.busyAttempts > 0+ );});return evidence;};@@ -215,8 +225,9 @@ const run = async (): Promise<Record<string, unknown>> => {(event) => event.kind === 'activation-started' && event.ownerEpoch === 2));- const gracefulLockEvidence = await waitForPendingReplacementLock(- 'epoch 2 exact owner lock request to become pending'+ const gracefulLockEvidence = await waitForRetriedReplacementLock(+ 2,+ 'epoch 2 to retry the held owner lock');const gracefulWaitedForPhysicalLock = !telemetry.some((event) => event.kind === 'ready' && event.ownerEpoch === 2@@ -277,8 +288,9 @@ const run = async (): Promise<Record<string, unknown>> => {(event) => event.kind === 'activation-started' && event.ownerEpoch === 3));- const recoveryLockEvidence = await waitForPendingReplacementLock(- 'epoch 3 exact owner lock request to become pending'+ const recoveryLockEvidence = await waitForRetriedReplacementLock(+ 3,+ 'epoch 3 to retry the held owner lock');const recoveryWaitedForPhysicalLock = !telemetry.some((event) => event.kind === 'ready' && event.ownerEpoch === 3@@ -318,12 +330,14 @@ const run = async (): Promise<Record<string, unknown>> => {(event) => event.kind === 'drained' && event.ownerEpoch === 1),gracefulReplacementWaitedForPhysicalLock: gracefulWaitedForPhysicalLock,- gracefulPendingOwnerLockRequests: gracefulLockEvidence.pending,+ gracefulQueuedOwnerLockRequests: gracefulLockEvidence.pending,+ gracefulRetriedHeldOwnerLock: gracefulLockEvidence.busyAttempts > 0,abruptInflightRejected: abruptError.includes('owner epoch 2 was lost'),abruptRequestReplayCount: slowAdmissions.length,abruptOwnerPageStayedAlive: !secondPage.closed,recoveryReplacementWaitedForPhysicalLock: recoveryWaitedForPhysicalLock,- recoveryPendingOwnerLockRequests: recoveryLockEvidence.pending,+ recoveryQueuedOwnerLockRequests: recoveryLockEvidence.pending,+ recoveryRetriedHeldOwnerLock: recoveryLockEvidence.busyAttempts > 0,atomicRecoveryOpenWipedToMiss: true,recoveryDatabaseAction: telemetry.find((event) => event.kind === 'ready' && event.ownerEpoch === 3diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.tsindex 1b90e367d5..5e81d4520d 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts@@ -13,6 +13,10 @@ const parameters = new URLSearchParams(location.search);const tabId = parameters.get('tabId') ?? '';const scope = parameters.get('scope') ?? '';if (!tabId || !scope) throw new Error('missing production harness parameters');+// A named build gets its own coordinator, as each deployed build does, and+// reports its build time so the newer build can take the database over.+const build = parameters.get('build');+const buildTime = Number(parameters.get('buildTime') ?? 0);const QUERY = `query Soup($input: SoupInput!) {user {@@ -49,6 +53,19 @@ const pending = new Map<number, (response: CacheResponse) => void>();const adapter = createCacheCoordinatorPageAdapter({scope,tabId,+ buildTime,+ ...(build+ ? {+ createSharedWorker: (workerScope: string) =>+ new SharedWorker(+ new URL('../cache.coordinator.shared-worker.ts', import.meta.url),+ {+ type: 'module',+ name: `graphql-cache-coordinator:${workerScope}:build-${build}`,+ }+ ),+ }+ : {}),createDedicatedWorker: (_workerScope, ownerEpoch) =>new Worker(new URL('./production-cache.engine-worker.ts', import.meta.url),@@ -68,6 +85,15 @@ const adapter = createCacheCoordinatorPageAdapter({onProtocolError: (error) => {report({ kind: 'protocol-error', error: error.message });},+ onCacheUnavailable: (reason) => {+ report({ kind: 'cache-unavailable', reason });+ },+ onCacheSuperseded: (reason) => {+ report({ kind: 'cache-superseded', reason });+ // What the app's host does: leave like a navigating page, which stops+ // this tab's engine so the newer build can open the database.+ void adapter.dispose({ graceful: false, preserveDatabase: true });+ },});adapter.onmessage = (event) => {@@ -203,6 +229,17 @@ const handleCommand = (command: ProductionHarnessCommand): void => {currentWorker.terminate();report({ kind: 'command-result', commandId: value.commandId, ok: true });})+ .with({ kind: 'navigate-away' }, (value) => {+ void (async () => {+ await adapter.dispose({ graceful: false, preserveDatabase: true });+ report({+ kind: 'command-result',+ commandId: value.commandId,+ ok: true,+ });+ setTimeout(() => window.close());+ })();+ }).exhaustive();};diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover-harness.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover-harness.tsnew file mode 100644index 0000000000..ec173c4ac9--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover-harness.ts@@ -0,0 +1,239 @@+import type {+ ProductionHarnessCommand,+ ProductionHarnessEnvelope,+} from './production-browser-wire';++// Tabs of three simulated builds share one scope. Each build runs its own+// coordinator over the one database, as deployed builds do. The newer build+// must take the database over without wiping it, and send the older build's+// tabs on; an even older build must be turned away.++const resultElement = document.querySelector<HTMLElement>('#result');+if (!resultElement) throw new Error('missing result element');++type CommandWithoutId = ProductionHarnessCommand extends infer Command+ ? Command extends ProductionHarnessCommand+ ? Omit<Command, 'commandId'>+ : never+ : never;++type RuntimeTelemetry = {+ kind: string;+ tabId: string;+ ownerEpoch: number;+ databaseAction: 'open-existing' | 'wipe-before-open';+};++const BUILDS = { older: 500, old: 1_000, new: 2_000 } as const;++const assert: (condition: unknown, message: string) => asserts condition = (+ condition,+ message+) => {+ if (!condition) throw new Error(message);+};++const waitUntil = async (+ description: string,+ predicate: () => boolean,+ timeoutMs = 30_000+): Promise<void> => {+ const started = performance.now();+ while (!predicate()) {+ if (performance.now() - started > timeoutMs) {+ throw new Error(`timed out waiting for ${description}`);+ }+ await new Promise<void>((resolve) => setTimeout(resolve, 20));+ }+};++const run = async (): Promise<Record<string, unknown>> => {+ const scope = `takeover-${crypto.randomUUID()}`;+ const tabChannel = new BroadcastChannel(+ `graphql-cache-wp08-production-tabs:${scope}`+ );+ const telemetryChannel = new BroadcastChannel(+ `graphql-cache-wp08-production:${scope}`+ );+ const popups = new Map<string, Window>();+ const registered = new Set<string>();+ const superseded = new Map<string, string>();+ const unavailable = new Map<string, string>();+ const terminated: Array<{ tabId: string; reason: string }> = [];+ const telemetry: RuntimeTelemetry[] = [];+ const protocolErrors: string[] = [];+ const pendingCommands = new Map<+ string,+ {+ resolve: (value: unknown) => void;+ reject: (error: Error) => void;+ timer: ReturnType<typeof setTimeout>;+ }+ >();++ tabChannel.onmessage = (event: MessageEvent<ProductionHarnessEnvelope>) => {+ const message = event.data;+ if (message.source !== 'tab') return;+ switch (message.event.kind) {+ case 'registered':+ registered.add(message.tabId);+ break;+ case 'cache-superseded':+ superseded.set(message.tabId, message.event.reason);+ break;+ case 'cache-unavailable':+ unavailable.set(message.tabId, message.event.reason);+ break;+ case 'worker-terminated':+ terminated.push({+ tabId: message.tabId,+ reason: message.event.reason,+ });+ break;+ case 'protocol-error':+ protocolErrors.push(message.event.error);+ break;+ case 'command-result': {+ const pending = pendingCommands.get(message.event.commandId);+ if (!pending) return;+ pendingCommands.delete(message.event.commandId);+ clearTimeout(pending.timer);+ if (message.event.ok) pending.resolve(message.event.result);+ else pending.reject(new Error(message.event.error));+ break;+ }+ }+ };+ telemetryChannel.onmessage = (event: MessageEvent<RuntimeTelemetry>) => {+ telemetry.push(event.data);+ };++ const command = async (+ targetTabId: string,+ value: CommandWithoutId+ ): Promise<unknown> => {+ const commandId = crypto.randomUUID();+ const result = new Promise<unknown>((resolve, reject) => {+ const timer = setTimeout(() => {+ pendingCommands.delete(commandId);+ reject(new Error(`takeover command timed out: ${value.kind}`));+ }, 30_000);+ pendingCommands.set(commandId, { resolve, reject, timer });+ });+ tabChannel.postMessage({+ source: 'harness',+ targetTabId,+ command: { ...value, commandId } as ProductionHarnessCommand,+ } satisfies ProductionHarnessEnvelope);+ return await result;+ };++ const openTab = (tabId: string, build: keyof typeof BUILDS): void => {+ const url = new URL('./production-tab.html', location.href);+ url.searchParams.set('scope', scope);+ url.searchParams.set('tabId', tabId);+ url.searchParams.set('build', build);+ url.searchParams.set('buildTime', String(BUILDS[build]));+ const popup = window.open(url, `${scope}:${tabId}`);+ if (!popup) throw new Error(`popup blocked for ${tabId}`);+ popups.set(tabId, popup);+ };++ const readyFor = (tabId: string) =>+ telemetry.find((event) => event.kind === 'ready' && event.tabId === tabId);++ // The old build owns the database and holds data.+ openTab('old-a', 'old');+ openTab('old-b', 'old');+ await waitUntil('both old-build tabs to register', () =>+ ['old-a', 'old-b'].every((tabId) => registered.has(tabId))+ );+ await waitUntil('the old build to open the database', () =>+ ['old-a', 'old-b'].some((tabId) => readyFor(tabId))+ );+ const oldOwner = ['old-a', 'old-b'].find((tabId) => readyFor(tabId));+ assert(oldOwner, 'missing old-build owner');+ await command('old-b', { kind: 'write', value: 'takeover-preserved' });++ // A tab of the newer build opens and takes the database over.+ openTab('new-a', 'new');+ await waitUntil('the new build to open the database', () =>+ Boolean(readyFor('new-a'))+ );+ expectHit(await command('new-a', { kind: 'read' }), 'takeover-preserved');+ await waitUntil('both old-build tabs to be sent on', () =>+ ['old-a', 'old-b'].every((tabId) => superseded.has(tabId))+ );++ // A tab of an even older build is turned away; the new build keeps it.+ openTab('older-a', 'older');+ await waitUntil('the older-build tab to be told to run uncached', () =>+ unavailable.has('older-a')+ );+ const olderBuildSentOn = superseded.has('older-a');+ // A page of the old build that loads after the handover, as after a+ // rollback, runs uncached rather than reloading, which could land on the+ // old build again. Its coordinator either was superseded or, once the old+ // tabs let it go, starts fresh and is turned away by the newer build.+ openTab('old-c', 'old');+ await waitUntil('the late old-build tab to be told to run uncached', () =>+ unavailable.has('old-c')+ );+ expectHit(await command('new-a', { kind: 'read' }), 'takeover-preserved');++ const result = {+ passed: true,+ newBuildDatabaseAction: telemetry.find(+ (event) => event.kind === 'activation-started' && event.tabId === 'new-a'+ )?.databaseAction,+ newBuildKeptData: true,+ // The old owner stopped its engine the way a navigating page does.+ oldEngineStoppedLikeNavigation:+ terminated.find((event) => event.tabId === oldOwner)?.reason ===+ 'page navigation',+ oldTabsSentOn: ['old-a', 'old-b'].every((tabId) =>+ superseded.get(tabId)?.includes('newer version of the app')+ ),+ olderBuildTurnedAway:+ unavailable.get('older-a')?.includes('owner lock is held') ?? false,+ olderBuildSentOn,+ lateOldTabRunsUncached: unavailable.has('old-c'),+ lateOldTabSentOn: superseded.has('old-c'),+ newBuildSentOn: superseded.has('new-a'),+ protocolErrors,+ };++ for (const popup of popups.values()) popup.close();+ tabChannel.close();+ telemetryChannel.close();+ return result;+};++function expectHit(value: unknown, expected: string): void {+ assert(+ typeof value === 'object' &&+ value !== null &&+ (value as { kind?: unknown }).kind === 'hit' &&+ (+ value as {+ data?: { user?: { soup?: { items?: Array<{ id?: unknown }> } } };+ }+ ).data?.user?.soup?.items?.[0]?.id === expected,+ `expected cache hit ${expected}`+ );+}++void (async () => {+ try {+ const result = await run();+ resultElement.dataset.status = 'passed';+ resultElement.textContent = JSON.stringify(result, null, 2);+ } catch (error) {+ resultElement.dataset.status = 'failed';+ // Firefox stacks leave out the message.+ resultElement.textContent =+ error instanceof Error+ ? `${error.message}\n${error.stack ?? ''}`+ : String(error);+ }+})();diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover.html b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover.htmlnew file mode 100644index 0000000000..61a563b1c9--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover.html@@ -0,0 +1,12 @@+<!doctype html>+<html lang="en">+ <head>+ <meta charset="UTF-8" />+ <link rel="icon" href="data:," />+ <title>Production Turso takeover harness</title>+ </head>+ <body>+ <pre id="result" data-status="running">running</pre>+ <script type="module" src="./production-takeover-harness.ts"></script>+ </body>+</html>diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.tsindex 1a616ded5d..e4df0f10fd 100644--- a/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts@@ -42,6 +42,8 @@ export default defineConfig(({ command }) => ({'host.html','cutover.html','production.html',+ 'production-busy.html',+ 'production-takeover.html','production-tab.html','tab.html','performance.html',diff --git a/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts b/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.tsindex bd6ff68338..f362664c75 100644--- a/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts@@ -44,7 +44,7 @@ class FakeEndpoint extends EventTarget {}const message = {- coordinatorVersion: 5,+ coordinatorVersion: 6,kind: 'disconnect-tab',tabId: 'tab-a',reason: 'test',@@ -90,7 +90,7 @@ describe('cache coordinator runtime', () => {expect(() =>ports[0]?.postMessage({- coordinatorVersion: 5,+ coordinatorVersion: 6,kind: 'protocol-error',error: 'test',} satisfies CoordinatorToTabEnvelope)diff --git a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.tsindex 119f8169d6..e84f3f9046 100644--- a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts@@ -3,10 +3,11 @@ import type { CacheRequest } from '../protocol';import { INITIAL_CACHE_REVISION } from '../protocol';import {type CacheEngineRuntimeOptions,- installCacheEngineWorker,+ installCacheEngineWorker as installRuntime,} from './cache-engine-runtime';import {CACHE_COORDINATOR_PROTOCOL_VERSION,+ cacheDatabaseIdentity,databaseOwnerLockName,type PageToEngineEnvelope,} from './coordinator-protocol';@@ -18,6 +19,8 @@ import {EFFECT_WORKER_REQUEST_TAG,EFFECT_WORKER_RESPONSE_TAG,} from './effect-worker-transport';+import { OWNER_LOCK_RETRY_DELAYS_MS } from './startup';+import type { CacheWasmModule } from './wasm-module';import type { CacheWorkerCoreOptions } from './worker-core';class FakePort extends EventTarget {@@ -33,7 +36,7 @@ class FakePort extends EventTarget {const payload = effectPayload(message) as| { kind?: string; ownerEpoch?: number }| undefined;- if (this.autoOpen && payload?.kind === 'engine-assets-ready') {+ if (this.autoOpen && payload?.kind === 'owner-lock-acquired') {queueMicrotask(() =>this.receive({...version,@@ -83,6 +86,33 @@ const version = {coordinatorVersion: CACHE_COORDINATOR_PROTOCOL_VERSION,} as const;+type RuntimeCore = ReturnType<+ NonNullable<CacheEngineRuntimeOptions['createCore']>+>;++/** Like CacheWorkerCore over WASM: initialization holds the owner lock and+ * asks for the storage grant before its first storage access. */+const installCacheEngineWorker = (options: CacheEngineRuntimeOptions): void => {+ const createCore = options.createCore;+ installRuntime({+ ...options,+ createCore:+ createCore &&+ ((coreOptions: CacheWorkerCoreOptions): RuntimeCore => {+ const core = createCore(coreOptions);+ return {+ ...core,+ handleRequest: async (port, request) => {+ if (request.kind === 'init') {+ await coreOptions.onOwnerLockAcquired?.();+ }+ return await core.handleRequest(port, request);+ },+ };+ }),+ });+};+const activation = (databaseAction: 'open-existing' | 'wipe-before-open' = 'open-existing'): PageToEngineEnvelope => ({@@ -122,6 +152,7 @@ const registerTab = async (scope: 'scope',tabId,livenessLockName: `graphql-cache-tab:scope:${tabId}`,+ buildTime: 0,});};@@ -184,7 +215,7 @@ const cacheResponses = (port: FakePort) =>);describe('cache engine worker runtime', () => {- it('does not touch storage until assets load and the coordinator grants opening', async () => {+ it('does not touch storage until it holds the owner lock and the coordinator grants opening', async () => {const scope = new FakeWorkerScope();const direct = new FakePort();direct.autoOpen = false;@@ -214,8 +245,9 @@ describe('cache engine worker runtime', () => {expect(messagesOfKind(direct, 'engine-assets-ready')).toHaveLength(0);finishLoading();await vi.waitFor(() =>- expect(messagesOfKind(direct, 'engine-assets-ready')).toHaveLength(1)+ expect(messagesOfKind(direct, 'owner-lock-acquired')).toHaveLength(1));+ expect(messagesOfKind(direct, 'engine-assets-ready')).toHaveLength(1);expect(handleRequest).not.toHaveBeenCalled();direct.receive({ ...version, kind: 'open-engine', ownerEpoch: 7 });await vi.waitFor(() =>@@ -224,6 +256,31 @@ describe('cache engine worker runtime', () => {expect(handleRequest).toHaveBeenCalledOnce();});+ it('fatals on a storage grant before it holds the owner lock', async () => {+ const scope = new FakeWorkerScope();+ const direct = new FakePort();+ direct.autoOpen = false;+ const handleRequest = vi.fn();+ installCacheEngineWorker({+ scope,+ createCore: () => ({+ prepare: () => new Promise<void>(() => undefined),+ handleRequest,+ addPort: vi.fn(),+ drain: vi.fn(),+ }),+ });+ scope.activate(activation(), direct);+ direct.receive({ ...version, kind: 'open-engine', ownerEpoch: 7 });++ await vi.waitFor(() =>+ expect(messagesOfKind(direct, 'engine-fatal')).toEqual([+ expect.objectContaining({ reason: 'unexpected database-open grant' }),+ ])+ );+ expect(handleRequest).not.toHaveBeenCalled();+ });+it('reports failed asset loading without initializing or opening storage', async () => {const scope = new FakeWorkerScope();const direct = new FakePort();@@ -248,6 +305,171 @@ describe('cache engine worker runtime', () => {expect(messagesOfKind(direct, 'engine-ready')).toHaveLength(0);});+ it('retries a busy owner lock, reporting each attempt, and opens once it frees', async () => {+ const scope = new FakeWorkerScope();+ const direct = new FakePort();+ const sleep = vi.fn(async (_delayMs: number) => undefined);+ installRuntime({+ scope,+ sleep,+ ownerLockIsHeld: async () => true,+ createCore: (coreOptions) => ({+ handleRequest: async (port, request) => {+ if (request.kind === 'init') {+ for (const attempt of [1, 2]) {+ await coreOptions.onOwnerLockBusy?.(attempt);+ }+ await coreOptions.onOwnerLockAcquired?.();+ }+ port.postMessage({ id: request.id, ok: true, result: null });+ },+ addPort: vi.fn(),+ drain: vi.fn(),+ }),+ });+ scope.activate(activation(), direct);++ await vi.waitFor(() =>+ expect(messagesOfKind(direct, 'engine-ready')).toHaveLength(1)+ );+ expect(messagesOfKind(direct, 'owner-lock-busy')).toEqual([+ expect.objectContaining({ tabId: 'tab-a', ownerEpoch: 7 }),+ expect.objectContaining({ tabId: 'tab-a', ownerEpoch: 7 }),+ ]);+ expect(sleep.mock.calls.map(([delayMs]) => delayMs)).toEqual(+ OWNER_LOCK_RETRY_DELAYS_MS.slice(0, 2)+ );+ });++ it('gives up on an owner lock that stays busy without touching storage', async () => {+ const scope = new FakeWorkerScope();+ const direct = new FakePort();+ installRuntime({+ scope,+ sleep: async () => undefined,+ createCore: (coreOptions) => ({+ handleRequest: async (port, request) => {+ try {+ for (let attempt = 1; ; attempt += 1) {+ await coreOptions.onOwnerLockBusy?.(attempt);+ }+ } catch (error) {+ port.postMessage({+ id: request.id,+ ok: false,+ error: (error as Error).message,+ });+ }+ },+ addPort: vi.fn(),+ drain: vi.fn(),+ }),+ });+ scope.activate(activation(), direct);++ await vi.waitFor(() =>+ expect(messagesOfKind(direct, 'owner-lock-unavailable')).toEqual([+ {+ ...version,+ kind: 'owner-lock-unavailable',+ tabId: 'tab-a',+ ownerEpoch: 7,+ },+ ])+ );+ expect(messagesOfKind(direct, 'owner-lock-busy')).toHaveLength(+ OWNER_LOCK_RETRY_DELAYS_MS.length + 1+ );+ expect(messagesOfKind(direct, 'activation-failed')).toHaveLength(0);+ expect(messagesOfKind(direct, 'owner-lock-acquired')).toHaveLength(0);+ });++ it('reports database files another context kept open as storage-busy', async () => {+ for (const databaseAction of [+ 'open-existing',+ 'wipe-before-open',+ ] as const) {+ const scope = new FakeWorkerScope();+ const direct = new FakePort();+ installRuntime({+ scope,+ ownerLockIsHeld: async () => true,+ createCore: (coreOptions) => ({+ handleRequest: async (port, request) => {+ await coreOptions.onOwnerLockAcquired?.();+ coreOptions.onStorageBusy?.();+ port.postMessage({+ id: request.id,+ ok: false,+ error:+ 'OPFS sync handle open failed (NoModificationAllowedError)',+ });+ },+ addPort: vi.fn(),+ drain: vi.fn(),+ }),+ });+ scope.activate(activation(databaseAction), direct);++ await vi.waitFor(() =>+ expect(messagesOfKind(direct, 'activation-failed')).toEqual([+ expect.objectContaining({+ tabId: 'tab-a',+ ownerEpoch: 7,+ reason: 'OPFS sync handle open failed (NoModificationAllowedError)',+ failureCode: 'storage-busy',+ }),+ ])+ );+ expect(messagesOfKind(direct, 'engine-ready')).toHaveLength(0);+ }+ });++ it('deletes stale databases when started as a cleanup worker', async () => {+ const scope = new FakeWorkerScope();+ const reply = new FakePort();+ const removeStaleCacheDatabase = vi.fn(+ async (_scope: string, identity: string) =>+ identity === 'graphql-cache:scope:s0.v0.t1'+ ? { outcome: 'removed' as const }+ : { outcome: 'queued-mutations' as const, queuedMutations: 2 }+ );+ installRuntime({+ scope,+ loadWasm: async () =>+ ({ removeStaleCacheDatabase }) as unknown as CacheWasmModule,+ listOpfsRootNames: async () => [+ 'graphql-cache:scope:s0.v0.t1',+ 'graphql-cache:scope:s0.v0.t1-wal',+ 'graphql-cache:scope:s0.v0.t0',+ cacheDatabaseIdentity('scope'),+ `${cacheDatabaseIdentity('scope')}-wal`,+ ],+ });++ scope.onmessage?.({+ data: { ...version, kind: 'remove-stale-databases', scope: 'scope' },+ ports: [reply],+ } as unknown as MessageEvent);++ await vi.waitFor(() => expect(scope.closed).toBe(true));+ expect(removeStaleCacheDatabase.mock.calls).toEqual([+ ['scope', 'graphql-cache:scope:s0.v0.t1'],+ ['scope', 'graphql-cache:scope:s0.v0.t0'],+ ]);+ expect(reply.messages).toEqual([+ {+ ...version,+ kind: 'stale-databases-removed',+ removed: 1,+ inUse: 0,+ keptWithQueuedMutations: 1,+ failed: false,+ },+ ]);+ expect(reply.closed).toBe(true);+ });+it('selects atomic recovery-open and only proves wipe after initialization', async () => {const scope = new FakeWorkerScope();const direct = new FakePort();diff --git a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.tsindex 7b0f8e2dae..c5c98238d3 100644--- a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts+++ b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts@@ -15,21 +15,35 @@ import {import { workerCacheTelemetry } from '../telemetry-relay';import {CACHE_COORDINATOR_PROTOCOL_VERSION,+ type CleanupToPageEnvelope,type CoordinatorToEngineEnvelope,type EngineFatalCode,type EngineOpenOutcome,type EngineToCoordinatorEnvelope,isCachePush,isCacheResponse,+ type PageToCleanupEnvelope,type PageToEngineEnvelope,validateCoordinatorToEngineEnvelope,+ validatePageToCleanupEnvelope,validatePageToEngineEnvelope,} from './coordinator-protocol';import {createEffectWorkerRunnerTransport,type EffectWorkerRunnerTransport,} from './effect-worker-transport';-import { cacheWasmLinearMemoryBytes } from './wasm-module';+import {+ listOpfsRootNames,+ removeStaleCacheDatabases,+ type StaleDatabaseCleanupTally,+ staleCacheDatabaseIdentities,+} from './stale-databases';+import { OWNER_LOCK_RETRY_DELAYS_MS } from './startup';+import {+ type CacheWasmModule,+ cacheWasmLinearMemoryBytes,+ loadCacheWasm,+} from './wasm-module';import { CacheWorkerCore, type CacheWorkerCoreOptions } from './worker-core';export type CacheEngineRuntimeEvent =@@ -39,6 +53,11 @@ export type CacheEngineRuntimeEvent =activation: PageToEngineEnvelope;request: CacheRequest;}+ | {+ kind: 'owner-lock-busy';+ activation: PageToEngineEnvelope;+ attempt: number;+ }| { kind: 'ready'; activation: PageToEngineEnvelope }| { kind: 'drained'; activation: PageToEngineEnvelope }| {@@ -80,11 +99,14 @@ export interface CacheEngineRuntimeOptions {telemetry?: CacheTelemetryRecorderLike;/** Injectable clock/memory source for throttling tests. */now?: () => number;+ sleep?: (delayMs: number) => Promise<void>;+ loadWasm?: () => Promise<CacheWasmModule>;+ listOpfsRootNames?: () => Promise<string[]>;readLinearMemoryBytes?: () => number;memoryTelemetryIntervalMs?: number;}-const withVersion = <T extends { coordinatorVersion: 5 }>(+const withVersion = <T extends { coordinatorVersion: 6 }>(value: T extends unknown ? Omit<T, 'coordinatorVersion'> : never): T =>({@@ -95,6 +117,11 @@ const withVersion = <T extends { coordinatorVersion: 5 }>(const errorMessage = (error: unknown): string =>error instanceof Error ? error.message : String(error);+const sleep = (delayMs: number): Promise<void> =>+ new Promise((resolve) => {+ setTimeout(resolve, delayMs);+ });+async function initializeCore(core: CacheWorkerCoreLike,activation: PageToEngineEnvelope@@ -181,7 +208,11 @@ async function activate(};emitEvent({ kind: 'activation-started', activation });let failed = false;- let assetsReady = false;+ let ownerLockAcquired = false;+ // Set when retries for a busy owner lock ran out; storage stayed untouched.+ let ownerLockGaveUp = false;+ // Set when opening outwaited database files another context kept open.+ let storageBusy = false;let openGranted = false;let resolveOpen!: () => void;const openPermission = new Promise<void>((resolve) => {@@ -246,9 +277,50 @@ async function activate(((coreOptions: CacheWorkerCoreOptions) => new CacheWorkerCore(coreOptions));const core = createCore({recoveryOpen: activation.databaseAction === 'wipe-before-open',+ // WASM holds the owner lock but has not touched OPFS yet. The coordinator+ // records that this epoch may change storage before granting the open.+ onOwnerLockAcquired: async () => {+ if (failed || ownerLockAcquired) {+ throw new Error('cache engine was not waiting for a storage grant');+ }+ ownerLockAcquired = true;+ post(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'owner-lock-acquired',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ );+ await openPermission;+ if (failed)+ throw new Error('cache engine failed before its storage grant');+ },+ // Never queue for the lock. The coordinator decides from these reports+ // whether another build holds it and may end this activation early.+ onOwnerLockBusy: async (attempt) => {+ if (failed) throw new Error('cache engine failed while waiting to open');+ emitEvent({ kind: 'owner-lock-busy', activation, attempt });+ post(+ withVersion<EngineToCoordinatorEnvelope>({+ kind: 'owner-lock-busy',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ );+ const delayMs = OWNER_LOCK_RETRY_DELAYS_MS[attempt - 1];+ if (delayMs === undefined) {+ ownerLockGaveUp = true;+ throw new Error('cache database owner lock stayed unavailable');+ }+ await (options.sleep ?? sleep)(delayMs);+ if (failed) throw new Error('cache engine failed while waiting to open');+ },onStorageResetRequired: () => {fatal('cache storage requested physical reset', 'storage-reset-required');},+ onStorageBusy: () => {+ storageBusy = true;+ },onInitializationOutcome: (outcome) => {initializationOpenOutcome = outcome;},@@ -322,7 +394,7 @@ async function activate(}match(message).with({ kind: 'open-engine' }, () => {- if (!assetsReady || openGranted) {+ if (!ownerLockAcquired || openGranted) {fatal('unexpected database-open grant');return;}@@ -397,7 +469,6 @@ async function activate(try {await core.prepare?.();if (failed) return;- assetsReady = true;post(withVersion<EngineToCoordinatorEnvelope>({kind: 'engine-assets-ready',@@ -405,10 +476,8 @@ async function activate(ownerEpoch: activation.ownerEpoch,}));- // Sending readiness is not permission: the coordinator must first record- // that this epoch may touch storage before the first OPFS operation.- await openPermission;- if (failed) return;+ // Opening first takes the database owner lock, retrying briefly while+ // another context holds it. Storage is touched only after `open-engine`.await initializeCore(core, activation);if (failed) return;const ownerLockIsHeld = options.ownerLockIsHeld ?? defaultOwnerLockIsHeld;@@ -447,19 +516,72 @@ async function activate(durationMs: now() - activationStartedAt,});telemetry.flush();- post(- withVersion<EngineToCoordinatorEnvelope>({- kind: 'activation-failed',- tabId: activation.tabId,- ownerEpoch: activation.ownerEpoch,- reason: errorMessage(error),- failureCode:- activation.databaseAction === 'wipe-before-open'- ? 'recovery-open-failed'- : 'initialization-failed',+ // A fatal report already ended this activation; the open only unwound.+ if (!failed) {+ post(+ ownerLockGaveUp+ ? withVersion<EngineToCoordinatorEnvelope>({+ kind: 'owner-lock-unavailable',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ })+ : withVersion<EngineToCoordinatorEnvelope>({+ kind: 'activation-failed',+ tabId: activation.tabId,+ ownerEpoch: activation.ownerEpoch,+ reason: errorMessage(error),+ failureCode: storageBusy+ ? 'storage-busy'+ : activation.databaseAction === 'wipe-before-open'+ ? 'recovery-open-failed'+ : 'initialization-failed',+ })+ );+ }+ await closeLifecycle();+ }+}++/** Deletes this scope's stale databases in a worker used for nothing else,+ * because a file Turso cannot read can poison the worker's OPFS registry. */+async function removeStaleDatabases(+ request: PageToCleanupEnvelope,+ port: MessagePort,+ workerScope: DedicatedWorkerScopeLike,+ options: CacheEngineRuntimeOptions+): Promise<void> {+ const failed: StaleDatabaseCleanupTally = {+ removed: 0,+ inUse: 0,+ keptWithQueuedMutations: 0,+ failed: true,+ };+ let tally = failed;+ try {+ const wasm = await (options.loadWasm ?? loadCacheWasm)();+ const removeOne = wasm.removeStaleCacheDatabase;+ if (removeOne) {+ const identities = staleCacheDatabaseIdentities(+ request.scope,+ await (options.listOpfsRootNames ?? listOpfsRootNames)()+ );+ tally = await removeStaleCacheDatabases(identities, (identity) =>+ removeOne(request.scope, identity)+ );+ }+ } catch {+ tally = failed;+ }+ try {+ port.postMessage(+ withVersion<CleanupToPageEnvelope>({+ kind: 'stale-databases-removed',+ ...tally,}));- await closeLifecycle();+ } finally {+ port.close();+ workerScope.close();}}@@ -471,12 +593,27 @@ export function installCacheEngineWorker(options.scope ?? (self as unknown as DedicatedWorkerScopeLike);let activated = false;workerScope.onmessage = (event: MessageEvent<unknown>) => {- const parsed = validatePageToEngineEnvelope(event.data);const directPort = event.ports[0];- if (!parsed.ok || event.ports.length !== 1 || !directPort || activated) {+ if (event.ports.length !== 1 || !directPort || activated) {for (const port of event.ports) port.close();return;}+ const cleanup = validatePageToCleanupEnvelope(event.data);+ if (cleanup.ok) {+ activated = true;+ void removeStaleDatabases(+ cleanup.value,+ directPort,+ workerScope,+ options+ );+ return;+ }+ const parsed = validatePageToEngineEnvelope(event.data);+ if (!parsed.ok) {+ directPort.close();+ return;+ }activated = true;void activate(parsed.value, directPort, workerScope, options);};diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.tsindex 4546351c80..df121af9a6 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts@@ -14,13 +14,23 @@ const action = <K extends CoordinatorAction['kind']>(return found as Extract<CoordinatorAction, { kind: K }>;};+/** Assets loaded, owner lock acquired, storage granted. */+const grantStorage = (+ core: CoordinatorCore,+ tabId: string,+ ownerEpoch: number+): void => {+ core.beginOwnerLockWait(tabId, ownerEpoch);+ core.beginEngineOpen(tabId, ownerEpoch);+};+const ready = (core: CoordinatorCore,tabId: string,ownerEpoch: number,databaseActionProof: DatabaseActionProof): CoordinatorAction[] => {- core.beginEngineOpen(tabId, ownerEpoch);+ grantStorage(core, tabId, ownerEpoch);return core.engineReady({tabId,ownerEpoch,@@ -40,6 +50,226 @@ const init = (id: number) => ({ id, kind: 'init', scope: 'scope' }) as const;const clear = (id: number) => ({ id, kind: 'clear' }) as const;describe('CoordinatorCore', () => {+ it('preserves storage when the engine is lost while trying the owner lock', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ expect(core.beginOwnerLockWait('tab-a', 1)).toBe(true);++ core.ownerLost('tab-a', 1, 'owner tab closed while the lock was busy');++ // Trying to take the lock cannot have changed OPFS.+ expect(action(core.resumeAfterLoss(), 'elect-owner').databaseAction).toBe(+ 'open-existing'+ );+ });++ it('fails closed when another context keeps the owner lock', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ core.registerTab('tab-b');+ expect(core.request('tab-a', init(1))).toEqual([]);+ expect(core.request('tab-b', clear(2))).toEqual([]);+ core.beginOwnerLockWait('tab-a', 1);++ const actions = core.ownerLockUnavailable('tab-a', 1);++ const unavailable = {+ kind: 'reject-request',+ error: expect.stringContaining('owner lock is held'),+ errorCode: 'owner-lock-unavailable',+ };+ expect(actions).toEqual([+ { ...unavailable, tabId: 'tab-a', requestId: 1 },+ { ...unavailable, tabId: 'tab-b', requestId: 2 },+ { kind: 'close-engine-route', tabId: 'tab-a', ownerEpoch: 1 },+ {+ kind: 'broadcast-cache-unavailable',+ reason: expect.stringContaining('another app version'),+ },+ ]);+ expect(core.state).toEqual({+ kind: 'waiting-for-tab',+ nextDatabaseAction: 'open-existing',+ });+ // Tabs told to stay uncached are refused and never elected again.+ expect(core.request('tab-b', init(3))).toEqual([+ { ...unavailable, tabId: 'tab-b', requestId: 3 },+ ]);+ expect(core.ownerLockUnavailable('tab-a', 1)).toEqual([]);+ expect(action(core.registerTab('tab-c'), 'elect-owner')).toEqual({+ kind: 'elect-owner',+ tabId: 'tab-c',+ ownerEpoch: 2,+ databaseAction: 'open-existing',+ });+ expect(core.request('tab-c', init(4))).toEqual([]);+ });++ it('never fails closed once storage may have been touched', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ grantStorage(core, 'tab-a', 1);++ expect(core.ownerLockUnavailable('tab-a', 1)).toEqual([]);+ expect(core.state).toMatchObject({+ kind: 'activating',+ phase: 'opening-database',+ });+ });++ it('keeps a pending recovery wipe for the attempt after failing closed', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ core.registerTab('tab-b');+ ready(core, 'tab-a', 1, 'opened-existing');+ core.tabLost('tab-a');+ expect(action(core.resumeAfterLoss(), 'elect-owner')).toMatchObject({+ tabId: 'tab-b',+ databaseAction: 'wipe-before-open',+ });+ core.beginOwnerLockWait('tab-b', 2);++ core.ownerLockUnavailable('tab-b', 2);++ expect(action(core.registerTab('tab-c'), 'elect-owner')).toMatchObject({+ tabId: 'tab-c',+ databaseAction: 'wipe-before-open',+ });+ });++ it('fails closed without a wipe when the database files stay busy', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ core.registerTab('tab-b');+ expect(core.request('tab-b', init(1))).toEqual([]);+ // Only an engine granted storage can find the files busy.+ expect(core.storageBusy('tab-a', 1)).toEqual([]);+ grantStorage(core, 'tab-a', 1);++ const actions = core.storageBusy('tab-a', 1);++ const unavailable = {+ kind: 'reject-request',+ error: expect.stringContaining('files are still open'),+ errorCode: 'owner-lock-unavailable',+ };+ expect(actions).toEqual([+ { ...unavailable, tabId: 'tab-b', requestId: 1 },+ { kind: 'close-engine-route', tabId: 'tab-a', ownerEpoch: 1 },+ {+ kind: 'broadcast-cache-unavailable',+ reason: expect.stringContaining('files are still open'),+ },+ ]);+ // The open changed nothing, so no attempt is scheduled to repair it.+ expect(core.state).toEqual({+ kind: 'waiting-for-tab',+ nextDatabaseAction: 'open-existing',+ });+ expect(core.request('tab-a', init(2))).toEqual([+ { ...unavailable, tabId: 'tab-a', requestId: 2 },+ ]);+ expect(core.storageBusy('tab-a', 1)).toEqual([]);+ expect(action(core.registerTab('tab-c'), 'elect-owner')).toEqual({+ kind: 'elect-owner',+ tabId: 'tab-c',+ ownerEpoch: 2,+ databaseAction: 'open-existing',+ });+ });++ it('hands the database to a newer build and never elects again', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ core.registerTab('tab-b');+ ready(core, 'tab-a', 1, 'opened-existing');+ expect(core.request('tab-b', init(1))).toMatchObject([+ { kind: 'route-request', ownerEpoch: 1 },+ ]);++ expect(core.supersede()).toEqual([+ {+ kind: 'broadcast-cache-superseded',+ reason: expect.stringContaining('newer version of the app'),+ },+ ]);+ expect(core.superseded).toBe(true);+ expect(core.supersede()).toEqual([]);+ // Routed work still gets its answer; new work is refused quietly.+ expect(core.state).toMatchObject({ kind: 'active', tabId: 'tab-a' });+ expect(core.request('tab-b', clear(2))).toEqual([+ expect.objectContaining({+ kind: 'reject-request',+ requestId: 2,+ errorCode: 'owner-lock-unavailable',+ }),+ ]);++ // The owner's page then leaves like a navigation, freeing the database.+ const departure = core.departForNavigation(+ 'tab-a',+ 1,+ 'a newer build took over'+ );+ expect(departure).toContainEqual({+ kind: 'close-engine-route',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ expect(departure.some((action) => action.kind === 'elect-owner')).toBe(+ false+ );+ expect(core.state.kind).toBe('waiting-for-tab');+ // A page of this build that loads afterwards could reload into this+ // build again, so it runs uncached instead of reloading.+ expect(core.registerTab('tab-c')).toEqual([+ {+ kind: 'notify-cache-unavailable',+ tabId: 'tab-c',+ reason: expect.stringContaining('newer version of the app'),+ },+ ]);+ });++ it('hands over only while no engine is changing the files', () => {+ const activating = new CoordinatorCore('scope');+ activating.registerTab('tab-a');+ grantStorage(activating, 'tab-a', 1);+ expect(activating.supersede()).toEqual([]);++ // An idle coordinator's tabs, already uncached, move on too.+ const idle = new CoordinatorCore('scope');+ idle.registerTab('tab-a');+ idle.beginOwnerLockWait('tab-a', 1);+ idle.ownerLockUnavailable('tab-a', 1);+ expect(idle.supersede()).toEqual([+ {+ kind: 'broadcast-cache-superseded',+ reason: expect.stringContaining('newer version of the app'),+ },+ ]);+ });++ it('keeps a pending recovery wipe when its files stay busy', () => {+ const core = new CoordinatorCore('scope');+ core.registerTab('tab-a');+ core.registerTab('tab-b');+ ready(core, 'tab-a', 1, 'opened-existing');+ core.tabLost('tab-a');+ expect(action(core.resumeAfterLoss(), 'elect-owner')).toMatchObject({+ tabId: 'tab-b',+ databaseAction: 'wipe-before-open',+ });+ grantStorage(core, 'tab-b', 2);++ core.storageBusy('tab-b', 2);++ expect(action(core.registerTab('tab-c'), 'elect-owner')).toMatchObject({+ tabId: 'tab-c',+ databaseAction: 'wipe-before-open',+ });+ });+it('preserves storage after bootstrap loss but requires recovery once opening was granted', () => {const core = new CoordinatorCore('scope');core.registerTab('tab-a');@@ -47,7 +277,10 @@ describe('CoordinatorCore', () => {expect(action(core.resumeAfterLoss(), 'elect-owner').databaseAction).toBe('open-existing');- expect(core.beginEngineOpen('tab-a', 1)).toBe(false);+ expect(core.beginOwnerLockWait('tab-a', 1)).toBe(false);+ // Storage is granted only once the engine holds the owner lock.+ expect(core.beginEngineOpen('tab-a', 2)).toBe(false);+ expect(core.beginOwnerLockWait('tab-a', 2)).toBe(true);expect(core.beginEngineOpen('tab-a', 2)).toBe(true);expect(core.beginEngineOpen('tab-a', 2)).toBe(false);core.ownerLost('tab-a', 2, 'database open timeout');@@ -150,7 +383,7 @@ describe('CoordinatorCore', () => {expect(action(handoff, 'elect-owner').databaseAction).toBe('open-existing');- core.beginEngineOpen('tab-b', 2);+ grantStorage(core, 'tab-b', 2);const actions = core.engineReady({tabId: 'tab-b',ownerEpoch: 2,@@ -455,7 +688,7 @@ describe('CoordinatorCore', () => {core.registerTab('tab-a');core.registerTab('tab-b');- core.beginEngineOpen('tab-a', 1);+ grantStorage(core, 'tab-a', 1);const wrongLock = core.engineReady({tabId: 'tab-a',ownerEpoch: 1,@@ -469,7 +702,7 @@ describe('CoordinatorCore', () => {'wrong physical owner lock');core.resumeAfterLoss();- core.beginEngineOpen('tab-b', 2);+ grantStorage(core, 'tab-b', 2);const wrongProof = core.engineReady({tabId: 'tab-b',ownerEpoch: 2,diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-core.tsindex bb11c8ec2e..f7f507b555 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts@@ -4,6 +4,7 @@ import {type CacheResponse,type CacheResponseErrorCode,OWNER_EPOCH_LOST_ERROR_CODE,+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE,} from '../protocol';import type {DatabaseAction,@@ -86,6 +87,9 @@ export type CoordinatorAction =| { kind: 'drop-tab'; tabId: string }| { kind: 'retire-tab'; tabId: string; ownerEpoch: OwnerEpoch }| { kind: 'schedule-reset-activation' }+ | { kind: 'broadcast-cache-unavailable'; reason: string }+ | { kind: 'broadcast-cache-superseded'; reason: string }+ | { kind: 'notify-cache-unavailable'; tabId: string; reason: string }| {kind: 'broadcast-engine-replaced';ownerEpoch: OwnerEpoch;@@ -113,6 +117,19 @@ export type EngineReady = {const proofFor = (outcome: EngineOpenOutcome): DatabaseActionProof =>outcome.startsWith('reset-') ? 'wiped-before-open' : 'opened-existing';+/** Only `opening-database` follows the storage grant; earlier phases, including+ * waiting for the owner lock, cannot have changed OPFS. */+export const startupPhaseMayTouchStorage = (+ phase: EngineStartupPhase+): boolean => phase === 'opening-database';++const OWNER_LOCK_UNAVAILABLE_REASON =+ 'cache database owner lock is held by another context, such as another app version';+const STORAGE_BUSY_REASON =+ 'cache database files are still open in another context, such as a closing tab';+const SUPERSEDED_REASON =+ 'a newer version of the app took over the local cache';+/** Pure election, epoch, routing, drain, and abrupt-loss state machine. */export class CoordinatorCore {private stateValue: CoordinatorState = {@@ -121,6 +138,11 @@ export class CoordinatorCore {};private readonly tabs: string[] = [];private readonly retiringTabs = new Set<string>();+ /** Tabs told the database is unavailable, with the reason they were given;+ * their pages stay uncached. */+ private readonly unavailableTabs = new Map<string, string>();+ /** Set once a newer app build took the database over; never cleared. */+ private supersededReason: string | undefined;private readonly queuedRequests: QueuedRequest[] = [];private readonly inFlight = new Map<RouteId, InFlightRequest>();private currentEpoch = 0;@@ -150,9 +172,27 @@ export class CoordinatorCore {};}+ /** Whether a newer app build took the database over. */+ get superseded(): boolean {+ return this.supersededReason !== undefined;+ }+registerTab(tabId: string): CoordinatorAction[] {if (this.tabs.includes(tabId)) return [];this.tabs.push(tabId);+ if (this.supersededReason !== undefined) {+ // A page of this build loaded after the handover, as after a rollback+ // or from stale HTML. Reloading could land on this build again, so it+ // only runs uncached.+ this.unavailableTabs.set(tabId, this.supersededReason);+ return [+ {+ kind: 'notify-cache-unavailable',+ tabId,+ reason: this.supersededReason,+ },+ ];+ }if (this.stateValue.kind === 'failed') {return [{@@ -180,6 +220,10 @@ export class CoordinatorCore {if (this.stateValue.kind === 'failed') {return [this.reject(tabId, request.id, this.stateValue.reason)];}+ const unavailable = this.unavailableTabs.get(tabId);+ if (unavailable !== undefined) {+ return [this.rejectUnavailable(tabId, request.id, unavailable)];+ }const queued = { tabId, request };if (this.stateValue.kind !== 'active') {@@ -189,8 +233,75 @@ export class CoordinatorCore {return [this.route(queued, this.stateValue)];}- /** Grant storage access only after assets have loaded. Before this grant an- * abandoned worker cannot have changed OPFS, so its successor preserves it.+ /** The engine loaded its assets and tries to take the database owner lock.+ * Trying grants no storage access. */+ beginOwnerLockWait(tabId: string, ownerEpoch: OwnerEpoch): boolean {+ const state = this.stateValue;+ if (+ state.kind !== 'activating' ||+ state.tabId !== tabId ||+ state.ownerEpoch !== ownerEpoch ||+ state.phase !== 'loading-assets'+ )+ return false;+ this.stateValue = { ...state, phase: 'awaiting-owner-lock' };+ return true;+ }++ /**+ * Another context, usually another deployed build, keeps the database owner+ * lock. Stop without touching storage: refuse every queued request and tell+ * every tab registered now to stay uncached for the rest of its page+ * session. Only a tab that registers later starts a new attempt.+ */+ ownerLockUnavailable(+ tabId: string,+ ownerEpoch: OwnerEpoch+ ): CoordinatorAction[] {+ const state = this.stateValue;+ if (+ state.kind !== 'activating' ||+ state.tabId !== tabId ||+ state.ownerEpoch !== ownerEpoch ||+ startupPhaseMayTouchStorage(state.phase)+ )+ return [];+ return this.failClosed(+ tabId,+ ownerEpoch,+ state.databaseAction,+ OWNER_LOCK_UNAVAILABLE_REASON+ );+ }++ /**+ * The engine held the owner lock, but another context kept the database+ * files open through the engine's bounded wait, usually a predecessor worker+ * that is still being torn down. Wiping cannot remove files that are open,+ * and they need no repair, so fail closed like an unavailable owner lock.+ * The attempt's database action carries over: an open changed nothing, and+ * a wipe that removed one file must still finish.+ */+ storageBusy(tabId: string, ownerEpoch: OwnerEpoch): CoordinatorAction[] {+ const state = this.stateValue;+ if (+ state.kind !== 'activating' ||+ state.tabId !== tabId ||+ state.ownerEpoch !== ownerEpoch ||+ state.phase !== 'opening-database'+ )+ return [];+ return this.failClosed(+ tabId,+ ownerEpoch,+ state.databaseAction,+ STORAGE_BUSY_REASON+ );+ }++ /** Grant storage access only once the engine holds the owner lock. Before+ * this grant an abandoned worker cannot have changed OPFS, so its successor+ * preserves it.*/beginEngineOpen(tabId: string, ownerEpoch: OwnerEpoch): boolean {const state = this.stateValue;@@ -198,7 +309,7 @@ export class CoordinatorCore {state.kind !== 'activating' ||state.tabId !== tabId ||state.ownerEpoch !== ownerEpoch ||- state.phase !== 'loading-assets'+ state.phase !== 'awaiting-owner-lock')return false;this.stateValue = { ...state, phase: 'opening-database' };@@ -344,6 +455,35 @@ export class CoordinatorCore {return this.transitionToAbruptLoss(tabId, ownerEpoch, reason);}+ /**+ * A newer app build takes the database over. Every tab registered now stops+ * using the cache and moves to that build; a tab that registers later only+ * runs uncached. The owner's page leaves the way a navigating page does,+ * which stops its engine and frees the database, and nothing is elected+ * afterwards. Refused while an engine is starting, recovering, or draining,+ * because its files may be changing.+ */+ supersede(): CoordinatorAction[] {+ const { kind } = this.stateValue;+ if (+ this.supersededReason !== undefined ||+ (kind !== 'active' && kind !== 'waiting-for-tab' && kind !== 'failed')+ ) {+ return [];+ }+ const reason = SUPERSEDED_REASON;+ this.supersededReason = reason;+ const actions: CoordinatorAction[] = this.queuedRequests+ .splice(0)+ .map(({ tabId, request }) =>+ this.rejectUnavailable(tabId, request.id, reason)+ );+ for (const tabId of this.tabs) this.unavailableTabs.set(tabId, reason);+ actions.push({ kind: 'broadcast-cache-superseded', reason });+ this.assertInvariants();+ return actions;+ }+/** Stops owner recovery and fails every current or future page connection. */terminalFailure(reason: string): CoordinatorAction[] {if (this.stateValue.kind === 'failed') return [];@@ -499,7 +639,7 @@ export class CoordinatorCore {previousEpoch: ownerEpoch,nextEpoch: this.currentEpoch + 1,databaseAction:- state.kind === 'activating' && state.phase === 'loading-assets'+ state.kind === 'activating' && !startupPhaseMayTouchStorage(state.phase)? state.databaseAction: 'wipe-before-open',reason,@@ -552,7 +692,10 @@ export class CoordinatorCore {}private chooseCandidate(avoid?: string): string | undefined {- const eligible = this.tabs.filter((tabId) => !this.retiringTabs.has(tabId));+ const eligible = this.tabs.filter(+ (tabId) =>+ !this.retiringTabs.has(tabId) && !this.unavailableTabs.has(tabId)+ );return eligible.find((tabId) => tabId !== avoid) ?? eligible[0];}@@ -582,6 +725,7 @@ export class CoordinatorCore {private removeTabRecord(tabId: string): void {const index = this.tabs.indexOf(tabId);if (index >= 0) this.tabs.splice(index, 1);+ this.unavailableTabs.delete(tabId);for (let index = this.queuedRequests.length - 1; index >= 0; index -= 1) {if (this.queuedRequests[index]?.tabId === tabId) {this.queuedRequests.splice(index, 1);@@ -615,6 +759,44 @@ export class CoordinatorCore {};}+ /** Refuses queued work, keeps every registered tab uncached, and waits for+ * a tab that registers later to try again. */+ private failClosed(+ tabId: string,+ ownerEpoch: OwnerEpoch,+ nextDatabaseAction: DatabaseAction,+ reason: string+ ): CoordinatorAction[] {+ const actions: CoordinatorAction[] = this.queuedRequests+ .splice(0)+ .map(({ tabId: requester, request }) =>+ this.rejectUnavailable(requester, request.id, reason)+ );+ for (const registered of this.tabs) {+ this.unavailableTabs.set(registered, reason);+ }+ actions.push(+ { kind: 'close-engine-route', tabId, ownerEpoch },+ { kind: 'broadcast-cache-unavailable', reason }+ );+ this.stateValue = { kind: 'waiting-for-tab', nextDatabaseAction };+ this.assertInvariants();+ return actions;+ }++ private rejectUnavailable(+ tabId: string,+ requestId: number,+ reason: string+ ): Extract<CoordinatorAction, { kind: 'reject-request' }> {+ return this.reject(+ tabId,+ requestId,+ reason,+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE+ );+ }+private dropStale(ownerEpoch: OwnerEpoch,routeId: RouteId | undefined,diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.tsindex 82614459b2..fc96cbc709 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts@@ -4,7 +4,10 @@ import {type DedicatedWorkerLike,type SharedWorkerLike,} from './coordinator-page-adapter';-import { CACHE_COORDINATOR_PROTOCOL_VERSION } from './coordinator-protocol';+import {+ CACHE_COORDINATOR_PROTOCOL_VERSION,+ cacheDatabaseIdentity,+} from './coordinator-protocol';class FakeCoordinatorPort extends EventTarget {readonly messages: Array<{ message: unknown; transfer: Transferable[] }> = [];@@ -208,6 +211,205 @@ describe('CacheCoordinatorPageAdapter', () => {expect(port.closed).toBe(true);});+ it('forwards the owner-lock phases in order and rejects stepping back', async () => {+ const port = new FakeCoordinatorPort();+ const progress = vi.fn();+ const error = vi.fn();+ const adapter = createCacheCoordinatorPageAdapter({+ scope: 'scope',+ tabId: 'tab-a',+ lockManager: heldLockManager(),+ createSharedWorker: () => ({ port: port as unknown as MessagePort }),+ createDedicatedWorker: () => new FakeWorker(),+ onStartupProgress: progress,+ onTerminalError: error,+ });+ const started = adapter.start();+ await vi.waitFor(() => expect(port.messages).toHaveLength(1));+ port.receive({ ...version, kind: 'registered', tabId: 'tab-a' });+ await started;+ const phase = (name: string, timeoutMs: number) => ({+ ...version,+ kind: 'engine-startup',+ ownerEpoch: 1,+ phase: name,+ databaseAction: 'open-existing',+ timeoutMs,+ });+ port.receive(phase('loading-assets', 300_000));+ port.receive(phase('awaiting-owner-lock', 15_000));+ port.receive(phase('awaiting-owner-lock', 15_000));+ port.receive(phase('opening-database', 20_000));+ expect(progress.mock.calls.map(([message]) => message.phase)).toEqual([+ 'loading-assets',+ 'awaiting-owner-lock',+ 'opening-database',+ ]);+ expect(error).not.toHaveBeenCalled();++ port.receive(phase('awaiting-owner-lock', 15_000));++ expect(error).toHaveBeenCalledOnce();+ expect(port.closed).toBe(true);+ });++ const registeredAdapter = async (+ options: Partial<Parameters<typeof createCacheCoordinatorPageAdapter>[0]>+ ) => {+ const port = new FakeCoordinatorPort();+ const adapter = createCacheCoordinatorPageAdapter({+ scope: 'scope',+ tabId: 'tab-a',+ lockManager: heldLockManager(),+ createSharedWorker: () => ({ port: port as unknown as MessagePort }),+ createDedicatedWorker: () => new FakeWorker(),+ ...options,+ });+ const started = adapter.start();+ await vi.waitFor(() => expect(port.messages).toHaveLength(1));+ port.receive({ ...version, kind: 'registered', tabId: 'tab-a' });+ await started;+ return port;+ };++ const removeStale = (ownerEpoch: number) => ({+ ...version,+ kind: 'remove-stale-databases',+ tabId: 'tab-a',+ ownerEpoch,+ });++ it('relays that another context holds the database', async () => {+ const onCacheUnavailable = vi.fn();+ const port = await registeredAdapter({ onCacheUnavailable });++ port.receive({+ ...version,+ kind: 'cache-unavailable',+ reason: 'another app version holds the database',+ });++ expect(onCacheUnavailable).toHaveBeenCalledExactlyOnceWith(+ 'another app version holds the database'+ );+ });++ it('registers with its build time and relays a newer build taking over', async () => {+ const onCacheSuperseded = vi.fn();+ const port = await registeredAdapter({+ buildTime: 1_790_000_000_000,+ onCacheSuperseded,+ });+ expect(port.messages[0]?.message).toMatchObject({+ kind: 'register-tab',+ buildTime: 1_790_000_000_000,+ });++ port.receive({+ ...version,+ kind: 'cache-superseded',+ reason: 'a newer version of the app took over the local cache',+ });++ expect(onCacheSuperseded).toHaveBeenCalledExactlyOnceWith(+ 'a newer version of the app took over the local cache'+ );+ });++ it('registers as an unstamped build outside app bundles', async () => {+ const port = await registeredAdapter({});++ expect(port.messages[0]?.message).toMatchObject({+ kind: 'register-tab',+ buildTime: 0,+ });+ });++ it('deletes stale databases in a disposable worker, once, for its own engine', async () => {+ const cleanupWorker = new FakeWorker();+ const createCleanupWorker = vi.fn(() => cleanupWorker);+ const listOpfsRootNames = vi.fn(async () => [+ 'graphql-cache:scope:s1.v1.t1',+ 'graphql-cache:scope:s1.v1.t1-wal',+ cacheDatabaseIdentity('scope'),+ 'unrelated.db',+ ]);+ const observations: Array<Record<string, unknown>> = [];+ const port = await registeredAdapter({+ createCleanupWorker,+ listOpfsRootNames,+ telemetry: {+ record: (observation) => observations.push(observation),+ flush: vi.fn(),+ },+ });++ // Only the engine's current owner cleans up.+ port.receive(removeStale(1));+ await Promise.resolve();+ expect(listOpfsRootNames).not.toHaveBeenCalled();+ port.receive(election(1));+ port.receive(removeStale(1));++ await vi.waitFor(() => expect(cleanupWorker.messages).toHaveLength(1));+ expect(cleanupWorker.messages[0]?.message).toEqual({+ ...version,+ kind: 'remove-stale-databases',+ scope: 'scope',+ });+ const reply = FakeMessageChannel.instances.at(-1)?.port1 as unknown as {+ onmessage: (event: MessageEvent<unknown>) => void;+ };+ reply.onmessage(+ new MessageEvent('message', {+ data: {+ ...version,+ kind: 'stale-databases-removed',+ removed: 1,+ inUse: 0,+ keptWithQueuedMutations: 1,+ failed: false,+ },+ })+ );+ await vi.waitFor(() => expect(cleanupWorker.terminated).toBe(true));+ expect(+ observations.filter((observation) =>+ String(observation.ownerEvent).startsWith('stale-')+ )+ ).toEqual([+ expect.objectContaining({+ ownerEvent: 'stale-databases-removed',+ outcome: 'success',+ count: 1,+ }),+ expect.objectContaining({ ownerEvent: 'stale-database-kept', count: 1 }),+ ]);++ port.receive(removeStale(1));+ await Promise.resolve();+ expect(createCleanupWorker).toHaveBeenCalledOnce();+ });++ it('starts no cleanup worker when only its own database exists', async () => {+ const createCleanupWorker = vi.fn(() => new FakeWorker());+ const listOpfsRootNames = vi.fn(async () => [+ cacheDatabaseIdentity('scope'),+ `${cacheDatabaseIdentity('scope')}-wal`,+ ]);+ const port = await registeredAdapter({+ createCleanupWorker,+ listOpfsRootNames,+ });+ port.receive(election(1));++ port.receive(removeStale(1));++ await vi.waitFor(() => expect(listOpfsRootNames).toHaveBeenCalledOnce());+ await Promise.resolve();+ expect(createCleanupWorker).not.toHaveBeenCalled();+ });+it('terminates and clears a failed worker before reporting owner loss', async () => {const order: string[] = [];const coordinatorPort = new FakeCoordinatorPort();diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.tsindex dc9aa58280..e37235d745 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts@@ -1,4 +1,5 @@import * as Effect from 'effect/Effect';+import { APP_BUILD_TIME } from '../app-build';import type { CacheRequest, WorkerMessage } from '../protocol';import {type CacheTelemetryRecorderLike,@@ -7,18 +8,26 @@ import {} from '../telemetry';import {CACHE_COORDINATOR_PROTOCOL_VERSION,+ type CleanupToPageEnvelope,type CoordinatorToTabEnvelope,+ ENGINE_STARTUP_PHASES,type EngineOpenOutcome,isCacheRequest,+ type PageToCleanupEnvelope,type PageToEngineEnvelope,type TabToCoordinatorEnvelope,tabLivenessLockName,+ validateCleanupToPageEnvelope,validateCoordinatorToTabEnvelope,} from './coordinator-protocol';import {createEffectWorkerTransport,type EffectWorkerTransport,} from './effect-worker-transport';+import {+ listOpfsRootNames,+ staleCacheDatabaseIdentities,+} from './stale-databases';import { CacheBootstrapExhaustedError } from './startup';export interface SharedWorkerLike {@@ -51,6 +60,17 @@ export interface CacheCoordinatorPageAdapterOptions {progress: Extract<CoordinatorToTabEnvelope, { kind: 'engine-startup' }>) => void;onOwnerChanged?: (ownerEpoch: number | undefined) => void;+ /** Another context holds the database, so the cache stays off until reload. */+ onCacheUnavailable?: (reason: string) => void;+ /** A newer app build took the database over; this page should reload. */+ onCacheSuperseded?: (reason: string) => void;+ /** When this page's app build was made, in ms since the epoch. Defaults to+ * the bundler's stamp; a newer build takes the database over. */+ buildTime?: number;+ /** Disposable worker, running the engine script, that deletes stale databases. */+ createCleanupWorker?: (scope: string) => DedicatedWorkerLike;+ listOpfsRootNames?: () => Promise<string[]>;+ cleanupTimeoutMs?: number;onWorkerCreated?: (worker: DedicatedWorkerLike, ownerEpoch: number) => void;onWorkerTerminated?: (ownerEpoch: number, reason: string) => void;/** Advisory coordinator diagnostic after successful registration. */@@ -74,8 +94,9 @@ interface CoordinatorConnection {}const DEFAULT_GRACEFUL_TIMEOUT_MS = 10_000;+const DEFAULT_CLEANUP_TIMEOUT_MS = 30_000;-const withVersion = <T extends { coordinatorVersion: 5 }>(+const withVersion = <T extends { coordinatorVersion: 6 }>(value: T extends unknown ? Omit<T, 'coordinatorVersion'> : never): T =>({@@ -98,6 +119,12 @@ const defaultDedicatedWorkerFactory = (name: `graphql-cache-engine:${scope}:${ownerEpoch}`,});+const defaultCleanupWorkerFactory = (scope: string): DedicatedWorkerLike =>+ new Worker(new URL('./cache.engine-worker.ts', import.meta.url), {+ type: 'module',+ name: `graphql-cache-cleanup:${scope}`,+ });+/*** Import-safe page endpoint for the browser coordinator topology.* SharedWorker creation starts on first use; DedicatedWorker creation starts@@ -136,6 +163,8 @@ export class CacheCoordinatorPageAdapter {| Extract<CoordinatorToTabEnvelope, { kind: 'engine-startup' }>| undefined;private closed = false;+ private staleCleanupStarted = false;+ private cleanupWorker: DedicatedWorkerLike | undefined;private readonly telemetry: CacheTelemetryRecorderLike;private readonly now = (): number =>globalThis.performance?.now() ?? Date.now();@@ -461,6 +490,7 @@ export class CacheCoordinatorPageAdapter {tabId: this.tabId,livenessLockName,hotCapacity: this.options.hotCapacity,+ buildTime: this.options.buildTime ?? APP_BUILD_TIME,}));}@@ -561,7 +591,8 @@ export class CacheCoordinatorPageAdapter {return;if (previous?.ownerEpoch === message.ownerEpoch &&- previous.phase === 'opening-database'+ ENGINE_STARTUP_PHASES.indexOf(message.phase) <+ ENGINE_STARTUP_PHASES.indexOf(previous.phase)) {this.failTerminal(new Error('engine startup phase moved backwards'));return;@@ -595,6 +626,125 @@ export class CacheCoordinatorPageAdapter {: new Error(message.error));break;+ case 'cache-unavailable':+ this.options.onCacheUnavailable?.(message.reason);+ break;+ case 'cache-superseded':+ this.options.onCacheSuperseded?.(message.reason);+ break;+ case 'remove-stale-databases':+ // Only the engine's current owner cleans up, once per page.+ if (+ message.tabId === this.tabId &&+ message.ownerEpoch === this.ownerEpoch+ ) {+ void this.removeStaleDatabases();+ }+ break;+ }+ }++ /** Lists OPFS first so the second WASM instance starts only when needed. */+ private async removeStaleDatabases(): Promise<void> {+ if (this.staleCleanupStarted || this.closed) return;+ this.staleCleanupStarted = true;+ const scope = this.options.scope;+ let stale: string[];+ try {+ stale = staleCacheDatabaseIdentities(+ scope,+ await (this.options.listOpfsRootNames ?? listOpfsRootNames)()+ );+ } catch {+ return;+ }+ if (stale.length === 0 || this.closed) return;+ let worker: DedicatedWorkerLike;+ try {+ worker = (+ this.options.createCleanupWorker ?? defaultCleanupWorkerFactory+ )(scope);+ } catch {+ this.recordStaleCleanup(undefined);+ return;+ }+ this.cleanupWorker = worker;+ const channel = new MessageChannel();+ const result = await new Promise<CleanupToPageEnvelope | undefined>(+ (resolve) => {+ const timer = setTimeout(+ () => resolve(undefined),+ this.options.cleanupTimeoutMs ?? DEFAULT_CLEANUP_TIMEOUT_MS+ );+ const settle = (value: CleanupToPageEnvelope | undefined) => {+ clearTimeout(timer);+ resolve(value);+ };+ channel.port1.onmessage = (event) => {+ const parsed = validateCleanupToPageEnvelope(event.data);+ settle(parsed.ok ? parsed.value : undefined);+ };+ worker.onerror = (event) => {+ event.preventDefault();+ settle(undefined);+ };+ try {+ worker.postMessage(+ withVersion<PageToCleanupEnvelope>({+ kind: 'remove-stale-databases',+ scope,+ }),+ [channel.port2]+ );+ } catch {+ settle(undefined);+ }+ }+ );+ this.closePort(channel.port1);+ this.terminateCleanupWorker();+ this.recordStaleCleanup(result);+ }++ private recordStaleCleanup(result: CleanupToPageEnvelope | undefined): void {+ if (!result || result.failed) {+ this.telemetry.record({+ name: 'graphql_cache.owner',+ operationCategory: 'storage',+ outcome: 'error',+ ownerEvent: 'stale-databases-removed',+ errorCode: result ? 'opfs-io' : 'timeout',+ count: result?.removed ?? 0,+ });+ } else if (result.removed > 0) {+ this.telemetry.record({+ name: 'graphql_cache.owner',+ operationCategory: 'storage',+ outcome: 'success',+ ownerEvent: 'stale-databases-removed',+ count: result.removed,+ });+ }+ if (result && result.keptWithQueuedMutations > 0) {+ this.telemetry.record({+ name: 'graphql_cache.owner',+ operationCategory: 'storage',+ outcome: 'success',+ ownerEvent: 'stale-database-kept',+ count: result.keptWithQueuedMutations,+ });+ }+ }++ private terminateCleanupWorker(): void {+ const worker = this.cleanupWorker;+ this.cleanupWorker = undefined;+ if (!worker) return;+ try {+ worker.onerror = null;+ worker.terminate();+ } catch {+ // A failed cleanup worker only delays deletion to a later owner.}}@@ -773,6 +923,7 @@ export class CacheCoordinatorPageAdapter {this.terminalErrorReported = true;this.closed = true;this.registered = false;+ this.terminateCleanupWorker();this.clearGracefulTimeout();if (this.ownerEpoch !== undefined) {this.terminateEngine(this.ownerEpoch, error.message, false);@@ -791,6 +942,7 @@ export class CacheCoordinatorPageAdapter {private finishDispose(): void {this.closed = true;this.registered = false;+ this.terminateCleanupWorker();this.clearGracefulTimeout();this.closeCoordinatorPort();this.releaseLiveness();diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.tsindex bac80c4977..439282674a 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts@@ -1,14 +1,25 @@+import { readFileSync } from 'node:fs';+import { dirname, resolve } from 'node:path';+import { fileURLToPath } from 'node:url';import { describe, expect, it } from 'vitest';import { INITIAL_CACHE_REVISION } from '../protocol';import {CACHE_COORDINATOR_PROTOCOL_VERSION,+ CACHE_STORAGE_VERSION,+ cacheDatabaseIdentity,databaseOwnerLockName,isCachePush,isCacheRequest,isCacheResponse,+ isStaleCacheDatabaseIdentity,+ tabIdFromLivenessLockName,+ tabLivenessLockName,+ UNVERSIONED_STORAGE_VERSION,+ validateCleanupToPageEnvelope,validateCoordinatorToEngineEnvelope,validateCoordinatorToTabEnvelope,validateEngineToCoordinatorEnvelope,+ validatePageToCleanupEnvelope,validatePageToEngineEnvelope,validateTabToCoordinatorEnvelope,} from './coordinator-protocol';@@ -75,6 +86,108 @@ describe('coordinator runtime protocol', () => {validateCoordinatorToEngineEnvelope({ ...grant, coordinatorVersion: 2 }).ok).toBe(false);+ // Protocol 5 and earlier granted storage right after assets loaded,+ // before the lock.+ expect(+ validateCoordinatorToEngineEnvelope({ ...grant, coordinatorVersion: 5 })+ .ok+ ).toBe(false);+ });++ it('validates owner-lock retries, unavailability, and stale cleanup envelopes', () => {+ const waiting = {+ ...version,+ kind: 'engine-startup',+ ownerEpoch: 1,+ phase: 'awaiting-owner-lock',+ databaseAction: 'open-existing',+ timeoutMs: 15_000,+ };+ expect(validateCoordinatorToTabEnvelope(waiting).ok).toBe(true);+ for (const invalid of [+ { ...waiting, timeoutMs: undefined },+ { ...waiting, phase: 'owner-lock-blocked' },+ ]) {+ expect(validateCoordinatorToTabEnvelope(invalid).ok).toBe(false);+ }+ const acquired = {+ ...version,+ kind: 'owner-lock-acquired',+ tabId: 'tab',+ ownerEpoch: 1,+ };+ const busy = { ...acquired, kind: 'owner-lock-busy' };+ const unavailable = { ...acquired, kind: 'owner-lock-unavailable' };+ for (const valid of [acquired, busy, unavailable]) {+ expect(validateEngineToCoordinatorEnvelope(valid).ok).toBe(true);+ }+ for (const invalid of [+ { ...acquired, tabId: '' },+ { ...acquired, ownerEpoch: 0 },+ { ...acquired, extra: true },+ { ...busy, attempt: 2 },+ { ...unavailable, reason: 'extra' },+ ]) {+ expect(validateEngineToCoordinatorEnvelope(invalid).ok).toBe(false);+ }++ const cacheUnavailable = {+ ...version,+ kind: 'cache-unavailable',+ reason: 'another app version holds the database',+ };+ const superseded = {+ ...version,+ kind: 'cache-superseded',+ reason: 'a newer version of the app took over the local cache',+ };+ const removeStale = {+ ...version,+ kind: 'remove-stale-databases',+ tabId: 'tab',+ ownerEpoch: 2,+ };+ for (const valid of [cacheUnavailable, superseded, removeStale]) {+ expect(validateCoordinatorToTabEnvelope(valid).ok).toBe(true);+ }+ for (const invalid of [+ { ...cacheUnavailable, reason: '' },+ { ...superseded, reason: '' },+ { ...superseded, buildTime: 1 },+ { ...removeStale, ownerEpoch: 0 },+ ]) {+ expect(validateCoordinatorToTabEnvelope(invalid).ok).toBe(false);+ }++ const cleanup = { ...version, kind: 'remove-stale-databases', scope: 's' };+ expect(validatePageToCleanupEnvelope(cleanup).ok).toBe(true);+ expect(validatePageToCleanupEnvelope({ ...cleanup, scope: '' }).ok).toBe(+ false+ );+ const tally = {+ ...version,+ kind: 'stale-databases-removed',+ removed: 1,+ inUse: 0,+ keptWithQueuedMutations: 2,+ failed: false,+ };+ expect(validateCleanupToPageEnvelope(tally).ok).toBe(true);+ for (const invalid of [+ { ...tally, removed: -1 },+ { ...tally, failed: 'no' },+ { ...tally, extra: true },+ ]) {+ expect(validateCleanupToPageEnvelope(invalid).ok).toBe(false);+ }+ expect(+ isCacheResponse({+ id: 3,+ ok: false,+ error: 'owner lock is held elsewhere',+ errorCode: 'owner-lock-unavailable',+ })+ ).toBe(true);});it('validates bounded reconciliation evidence without changing exact requests', () => {@@ -265,6 +378,7 @@ describe('coordinator runtime protocol', () => {scope: 'scope',tabId: 'tab',livenessLockName: 'graphql-cache-tab:scope:tab',+ buildTime: 1_790_000_000_000,},{...version,@@ -473,6 +587,26 @@ describe('coordinator runtime protocol', () => {failureCode: 'recovery-open-failed',}).ok).toBe(true);+ expect(+ validateEngineToCoordinatorEnvelope({+ ...version,+ kind: 'activation-failed',+ tabId: 'tab',+ ownerEpoch: 1,+ reason: 'OPFS sync handle open failed (NoModificationAllowedError)',+ failureCode: 'storage-busy',+ }).ok+ ).toBe(true);+ expect(+ validateEngineToCoordinatorEnvelope({+ ...version,+ kind: 'activation-failed',+ tabId: 'tab',+ ownerEpoch: 1,+ reason: 'open failed',+ failureCode: 'busy',+ }).ok+ ).toBe(false);expect(validateEngineToCoordinatorEnvelope({...version,@@ -496,6 +630,17 @@ describe('coordinator runtime protocol', () => {scope: 'scope',tabId: 'tab',livenessLockName: 'lock',+ buildTime: 0,+ }).ok+ ).toBe(false);+ // Every page reports its build, which decides database takeovers.+ expect(+ validateTabToCoordinatorEnvelope({+ ...version,+ kind: 'register-tab',+ scope: 'scope',+ tabId: 'tab',+ livenessLockName: 'lock',}).ok).toBe(false);expect(@@ -533,6 +678,8 @@ describe('coordinator runtime protocol', () => {});it('derives the exact UTF-8 canonical turso-opfs lock name', () => {+ // The current versions keep the unversioned name, and so the lock that+ // builds before versioned names hold.expect(databaseOwnerLockName('scope')).toBe('macro:turso-opfs:v1:19:graphql-cache:scope');@@ -540,4 +687,87 @@ describe('coordinator runtime protocol', () => {'macro:turso-opfs:v1:16:graphql-cache:é');});++ it('mirrors the storage versions the Rust crates embed in database names', () => {+ const repositoryRoot = resolve(+ dirname(fileURLToPath(import.meta.url)),+ '../../../../../..'+ );+ const rustSource = (path: string): string =>+ readFileSync(resolve(repositoryRoot, path), 'utf8');+ const rustConstant = (path: string, name: string): number => {+ const found = new RegExp(`pub const ${name}: u32 = (\\d+);`).exec(+ rustSource(path)+ );+ if (!found?.[1]) throw new Error(`${name} not found in ${path}`);+ return Number(found[1]);+ };+ const unversioned =+ /pub const UNVERSIONED_STORAGE_VERSIONS: \(u32, u32, u32\) = \((\d+), (\d+), (\d+)\);/.exec(+ rustSource('crates/client/cache-core/src/codec.rs')+ );+ if (!unversioned) throw new Error('UNVERSIONED_STORAGE_VERSIONS not found');+ expect(UNVERSIONED_STORAGE_VERSION).toEqual({+ schemaCompatibilityEpoch: Number(unversioned[1]),+ formatVersion: Number(unversioned[2]),+ storageSchemaVersion: Number(unversioned[3]),+ });+ expect(CACHE_STORAGE_VERSION).toEqual({+ schemaCompatibilityEpoch: rustConstant(+ 'crates/client/cache-core/src/codec.rs',+ 'CACHE_SCHEMA_COMPATIBILITY_EPOCH'+ ),+ formatVersion: rustConstant(+ 'crates/client/cache-core/src/codec.rs',+ 'CACHE_FORMAT_VERSION'+ ),+ storageSchemaVersion: rustConstant(+ 'crates/client/cache-turso/src/storage.rs',+ 'STORAGE_SCHEMA_VERSION'+ ),+ });+ });++ it("recognizes only this scope's older databases as stale", () => {+ const { schemaCompatibilityEpoch, formatVersion, storageSchemaVersion } =+ CACHE_STORAGE_VERSION;+ const own = cacheDatabaseIdentity('a');+ const versioned = (storage: number): string =>+ `graphql-cache:a:s${schemaCompatibilityEpoch}.v${formatVersion}.t${storage}`;+ // The current versions still open the unversioned name.+ expect(own).toBe('graphql-cache:a');+ expect(isStaleCacheDatabaseIdentity('a', own)).toBe(false);+ // Another spelling of this build's own versions is never opened again.+ expect(+ isStaleCacheDatabaseIdentity('a', versioned(storageSchemaVersion))+ ).toBe(true);+ expect(isStaleCacheDatabaseIdentity('a', 'graphql-cache:a:s1.v2.t3')).toBe(+ true+ );+ expect(+ isStaleCacheDatabaseIdentity('a', versioned(storageSchemaVersion - 1))+ ).toBe(true);+ for (const other of [+ // A newer build may come back after a rollback, so it keeps its file.+ versioned(storageSchemaVersion + 1),+ 'graphql-cache:a:s99.v0.t0',+ 'graphql-cache:b:s1.v2.t3',+ 'graphql-cache:ab',+ 'graphql-cache:a-wal',+ 'graphql-cache:a:s1.v2',+ 'graphql-cache:a:s1.v2.t3.x',+ 'graphql-cache:quarantine:a',+ ]) {+ expect(isStaleCacheDatabaseIdentity('a', other)).toBe(false);+ }+ expect(+ tabIdFromLivenessLockName('a', tabLivenessLockName('a', 'tab-1'))+ ).toBe('tab-1');+ expect(+ tabIdFromLivenessLockName('a', tabLivenessLockName('ab', 'tab-1'))+ ).toBeUndefined();+ expect(+ tabIdFromLivenessLockName('a', 'graphql-cache-tab:a:')+ ).toBeUndefined();+ });});diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.tsindex 08a108bf00..084873809e 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts@@ -20,8 +20,23 @@ import {export { isCachePush, isCacheResponse, isWorkerMessage };/** Version of the topology envelope and routed cache RPC surface. */-export const CACHE_COORDINATOR_PROTOCOL_VERSION = 5 as const;-export type EngineStartupPhase = 'loading-assets' | 'opening-database';+export const CACHE_COORDINATOR_PROTOCOL_VERSION = 6 as const;+/**+ * Startup phases in order. The engine may touch storage only in+ * `opening-database`, which begins once it holds the database owner lock. It+ * never queues for that lock: while another context holds it, the engine+ * retries briefly and then gives up without touching storage.+ */+export type EngineStartupPhase =+ | 'loading-assets'+ | 'awaiting-owner-lock'+ | 'opening-database';++export const ENGINE_STARTUP_PHASES: readonly EngineStartupPhase[] = [+ 'loading-assets',+ 'awaiting-owner-lock',+ 'opening-database',+];export type OwnerEpoch = number;export type RouteId = number;@@ -34,54 +49,61 @@ export type EngineOpenOutcome =| 'reset-corrupt'| 'reset-storage-uncertain';export type EngineFatalCode = 'storage-reset-required' | 'runtime-failure';+/** `storage-busy`: another context kept the database files open through the+ * engine's bounded wait. An open changed nothing; a wipe may have removed one+ * file, so the coordinator keeps the attempt's database action. */export type ActivationFailureCode =| 'initialization-failed'- | 'recovery-open-failed';+ | 'recovery-open-failed'+ | 'storage-busy';export type TabToCoordinatorEnvelope =| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'register-tab';scope: string;tabId: string;livenessLockName: string;hotCapacity?: number;+ /** When the page's app build was made, in ms since the epoch; 0 if+ * unstamped. A newer build takes the database over from an older one. */+ buildTime: number;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'cache-request';tabId: string;request: CacheRequest;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'attach-engine-port';tabId: string;ownerEpoch: OwnerEpoch;enginePort: MessagePort;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'graceful-departure';tabId: string;ownerEpoch: OwnerEpoch;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'navigation-departure';tabId: string;ownerEpoch: OwnerEpoch;reason: string;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-lost';tabId: string;ownerEpoch: OwnerEpoch;reason: string;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'disconnect-tab';tabId: string;reason: string;@@ -89,12 +111,12 @@ export type TabToCoordinatorEnvelope =export type CoordinatorToTabEnvelope =| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'registered';tabId: string;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'become-owner';scope: string;tabId: string;@@ -104,25 +126,25 @@ export type CoordinatorToTabEnvelope =hotCapacity?: number;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'cache-message';message: WorkerMessage;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'terminate-engine';tabId: string;ownerEpoch: OwnerEpoch;reason: string;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'retire-complete';tabId: string;ownerEpoch: OwnerEpoch;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-startup';ownerEpoch: OwnerEpoch;phase: EngineStartupPhase;@@ -130,26 +152,46 @@ export type CoordinatorToTabEnvelope =timeoutMs: number;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-replaced';ownerEpoch: OwnerEpoch;/** Whether this engine reopened durable data or created/reset it. */openOutcome: EngineOpenOutcome;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'protocol-error';error: string;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'terminal-error';error: string;storageUntouched?: true;+ }+ | {+ /** Another context holds the database; stop using the cache until reload. */+ coordinatorVersion: 6;+ kind: 'cache-unavailable';+ reason: string;+ }+ | {+ /** A newer app build took the database over. The page stops using the+ * cache and reloads into that build. */+ coordinatorVersion: 6;+ kind: 'cache-superseded';+ reason: string;+ }+ | {+ /** No other build has live tabs, so the owner may delete stale databases. */+ coordinatorVersion: 6;+ kind: 'remove-stale-databases';+ tabId: string;+ ownerEpoch: OwnerEpoch;};export type PageToEngineEnvelope = {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'activate-engine';scope: string;tabId: string;@@ -161,24 +203,25 @@ export type PageToEngineEnvelope = {export type CoordinatorToEngineEnvelope =| {- coordinatorVersion: 5;+ /** Storage grant, sent only after `owner-lock-acquired`. */+ coordinatorVersion: 6;kind: 'open-engine';ownerEpoch: OwnerEpoch;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-request';ownerEpoch: OwnerEpoch;routeId: RouteId;request: CacheRequest;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'drain-engine';ownerEpoch: OwnerEpoch;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'heartbeat';ownerEpoch: OwnerEpoch;heartbeatId: number;@@ -186,13 +229,34 @@ export type CoordinatorToEngineEnvelope =export type EngineToCoordinatorEnvelope =| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-assets-ready';tabId: string;ownerEpoch: OwnerEpoch;}| {- coordinatorVersion: 5;+ /** Holds the owner lock, has not touched storage, awaits `open-engine`. */+ coordinatorVersion: 6;+ kind: 'owner-lock-acquired';+ tabId: string;+ ownerEpoch: OwnerEpoch;+ }+ | {+ /** The owner lock was held elsewhere on this attempt; retrying. */+ coordinatorVersion: 6;+ kind: 'owner-lock-busy';+ tabId: string;+ ownerEpoch: OwnerEpoch;+ }+ | {+ /** Gave up on the owner lock without touching storage. */+ coordinatorVersion: 6;+ kind: 'owner-lock-unavailable';+ tabId: string;+ ownerEpoch: OwnerEpoch;+ }+ | {+ coordinatorVersion: 6;kind: 'engine-ready';tabId: string;ownerEpoch: OwnerEpoch;@@ -202,26 +266,26 @@ export type EngineToCoordinatorEnvelope =openOutcome: EngineOpenOutcome;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-response';ownerEpoch: OwnerEpoch;routeId: RouteId;response: CacheResponse;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-push';ownerEpoch: OwnerEpoch;push: CachePush;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-drained';tabId: string;ownerEpoch: OwnerEpoch;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'engine-fatal';tabId: string;ownerEpoch: OwnerEpoch;@@ -229,7 +293,7 @@ export type EngineToCoordinatorEnvelope =fatalCode: EngineFatalCode;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'activation-failed';tabId: string;ownerEpoch: OwnerEpoch;@@ -237,12 +301,30 @@ export type EngineToCoordinatorEnvelope =failureCode: ActivationFailureCode;}| {- coordinatorVersion: 5;+ coordinatorVersion: 6;kind: 'heartbeat-ack';ownerEpoch: OwnerEpoch;heartbeatId: number;};+/** Starts a disposable worker that deletes this scope's stale databases. */+export type PageToCleanupEnvelope = {+ coordinatorVersion: 6;+ kind: 'remove-stale-databases';+ scope: string;+};++/** Tally from a stale-database cleanup worker. */+export type CleanupToPageEnvelope = {+ coordinatorVersion: 6;+ kind: 'stale-databases-removed';+ removed: number;+ inUse: number;+ keptWithQueuedMutations: number;+ /** Cleanup stopped at an error; remaining databases wait for a later run. */+ failed: boolean;+};+export type EnvelopeValidation<T> =| { ok: true; value: T }| { ok: false; error: string };@@ -296,6 +378,9 @@ const hasVersion = (record: UnknownRecord): boolean =>const isDatabaseAction = (value: unknown): value is DatabaseAction =>value === 'open-existing' || value === 'wipe-before-open';+const isEngineStartupPhase = (value: unknown): value is EngineStartupPhase =>+ ENGINE_STARTUP_PHASES.includes(value as EngineStartupPhase);+const isDatabaseActionProof = (value: unknown): value is DatabaseActionProof =>value === 'opened-existing' || value === 'wiped-before-open';@@ -315,7 +400,9 @@ const isEngineFatalCode = (value: unknown): value is EngineFatalCode =>const isActivationFailureCode = (value: unknown): value is ActivationFailureCode =>- value === 'initialization-failed' || value === 'recovery-open-failed';+ value === 'initialization-failed' ||+ value === 'recovery-open-failed' ||+ value === 'storage-busy';const isOptionalRecord = (value: unknown@@ -675,10 +762,12 @@ export function validateTabToCoordinatorEnvelope('tabId','livenessLockName','hotCapacity',+ 'buildTime',]) &&isNonEmptyString(value.scope) &&isNonEmptyString(value.livenessLockName) &&- isOptionalPositiveInteger(value.hotCapacity)+ isOptionalPositiveInteger(value.hotCapacity) &&+ isSafeNonNegativeInteger(value.buildTime)) {return pass(value as TabToCoordinatorEnvelope);}@@ -853,8 +942,7 @@ export function validateCoordinatorToTabEnvelope('timeoutMs',]) &&isPositiveInteger(value.ownerEpoch) &&- (value.phase === 'loading-assets' ||- value.phase === 'opening-database') &&+ isEngineStartupPhase(value.phase) &&isDatabaseAction(value.databaseAction) &&isPositiveInteger(value.timeoutMs)) {@@ -897,10 +985,75 @@ export function validateCoordinatorToTabEnvelope(return pass(value as CoordinatorToTabEnvelope);}break;+ case 'cache-unavailable':+ case 'cache-superseded':+ if (+ hasOnlyKeys(value, ['coordinatorVersion', 'kind', 'reason']) &&+ isNonEmptyString(value.reason)+ ) {+ return pass(value as CoordinatorToTabEnvelope);+ }+ break;+ case 'remove-stale-databases':+ if (+ hasOnlyKeys(value, [+ 'coordinatorVersion',+ 'kind',+ 'tabId',+ 'ownerEpoch',+ ]) &&+ isNonEmptyString(value.tabId) &&+ isPositiveInteger(value.ownerEpoch)+ ) {+ return pass(value as CoordinatorToTabEnvelope);+ }+ break;}return fail(`invalid ${value.kind} coordinator envelope`);}+/** Validates the page's message starting a stale-database cleanup worker. */+export function validatePageToCleanupEnvelope(+ value: unknown+): EnvelopeValidation<PageToCleanupEnvelope> {+ if (+ isRecord(value) &&+ hasVersion(value) &&+ value.kind === 'remove-stale-databases' &&+ hasOnlyKeys(value, ['coordinatorVersion', 'kind', 'scope']) &&+ isNonEmptyString(value.scope)+ ) {+ return pass(value as PageToCleanupEnvelope);+ }+ return fail('invalid remove-stale-databases envelope');+}++/** Validates a cleanup worker's tally. */+export function validateCleanupToPageEnvelope(+ value: unknown+): EnvelopeValidation<CleanupToPageEnvelope> {+ if (+ isRecord(value) &&+ hasVersion(value) &&+ value.kind === 'stale-databases-removed' &&+ hasOnlyKeys(value, [+ 'coordinatorVersion',+ 'kind',+ 'removed',+ 'inUse',+ 'keptWithQueuedMutations',+ 'failed',+ ]) &&+ isSafeNonNegativeInteger(value.removed) &&+ isSafeNonNegativeInteger(value.inUse) &&+ isSafeNonNegativeInteger(value.keptWithQueuedMutations) &&+ typeof value.failed === 'boolean'+ ) {+ return pass(value as CleanupToPageEnvelope);+ }+ return fail('invalid stale-databases-removed envelope');+}+/** Validates the page's one-time DedicatedWorker activation message. */export function validatePageToEngineEnvelope(value: unknown@@ -1046,6 +1199,9 @@ export function validateEngineToCoordinatorEnvelope(}break;case 'engine-assets-ready':+ case 'owner-lock-acquired':+ case 'owner-lock-busy':+ case 'owner-lock-unavailable':case 'engine-drained':if (hasOnlyKeys(value, [@@ -1114,12 +1270,123 @@ export function validateEngineToCoordinatorEnvelope(return fail(`invalid ${value.kind} coordinator envelope`);}+const TAB_LIVENESS_LOCK_PREFIX = 'graphql-cache-tab:';+export const tabLivenessLockName = (scope: string, tabId: string): string =>- `graphql-cache-tab:${scope}:${tabId}`;+ `${TAB_LIVENESS_LOCK_PREFIX}${scope}:${tabId}`;++/** The tab id in a liveness lock name for `scope`, if it is one. */+export function tabIdFromLivenessLockName(+ scope: string,+ lockName: string+): string | undefined {+ const prefix = `${TAB_LIVENESS_LOCK_PREFIX}${scope}:`;+ if (!lockName.startsWith(prefix)) return;+ const tabId = lockName.slice(prefix.length);+ return tabId.length > 0 ? tabId : undefined;+}++/**+ * Storage versions embedded in the physical database name. Mirrors+ * `CACHE_SCHEMA_COMPATIBILITY_EPOCH`, `CACHE_FORMAT_VERSION` and+ * `STORAGE_SCHEMA_VERSION` in the Rust crates: a parity test compares them,+ * and the engine checks its WASM before touching storage.+ */+export const CACHE_STORAGE_VERSION = {+ schemaCompatibilityEpoch: 3,+ formatVersion: 3,+ storageSchemaVersion: 11,+} as const;++/**+ * Storage versions in use when database names began to embed versions; a+ * build at exactly these keeps the unversioned name. Mirrors+ * `UNVERSIONED_STORAGE_VERSIONS` in cache-core.+ */+export const UNVERSIONED_STORAGE_VERSION = {+ schemaCompatibilityEpoch: 3,+ formatVersion: 3,+ storageSchemaVersion: 11,+} as const;++const DATABASE_PREFIX = 'graphql-cache:';++type StorageVersionTuple = readonly [number, number, number];++const versionTuple = (+ version: typeof CACHE_STORAGE_VERSION | typeof UNVERSIONED_STORAGE_VERSION+): StorageVersionTuple => [+ version.schemaCompatibilityEpoch,+ version.formatVersion,+ version.storageSchemaVersion,+];++/** Compared as a tuple, like the WASM: the first version that differs decides. */+const compareVersions = (+ left: StorageVersionTuple,+ right: StorageVersionTuple+): number => {+ const index = left.findIndex((version, at) => version !== right[at]);+ return index < 0 ? 0 : (left[index] ?? 0) - (right[index] ?? 0);+};++/** Mirrors `cache_database_name`: the physical database this build opens. */+export function cacheDatabaseIdentity(scope: string): string {+ const unversioned = `${DATABASE_PREFIX}${scope}`;+ const own = versionTuple(CACHE_STORAGE_VERSION);+ if (compareVersions(own, versionTuple(UNVERSIONED_STORAGE_VERSION)) === 0) {+ return unversioned;+ }+ return `${unversioned}:s${own[0]}.v${own[1]}.t${own[2]}`;+}++/**+ * Whether `identity` is one of this scope's databases that this build does+ * not open and that no newer build could want: the unversioned name once this+ * build's versions have moved past it, or another name for versions no newer+ * than this build's. A newer version belongs to a build that may come back,+ * for example after a rollback. Mirrors the WASM's own check.+ */+export function isStaleCacheDatabaseIdentity(+ scope: string,+ identity: string+): boolean {+ const unversioned = `${DATABASE_PREFIX}${scope}`;+ const version =+ identity === unversioned+ ? versionTuple(UNVERSIONED_STORAGE_VERSION)+ : storageVersion(unversioned, identity);+ return (+ identity !== cacheDatabaseIdentity(scope) &&+ version !== undefined &&+ compareVersions(version, versionTuple(CACHE_STORAGE_VERSION)) <= 0+ );+}++/** The `[epoch, format, storage]` versions embedded in a database name. */+function storageVersion(+ unversioned: string,+ identity: string+): StorageVersionTuple | undefined {+ if (!identity.startsWith(unversioned)) return;+ const match = /^:s(\d+)\.v(\d+)\.t(\d+)$/.exec(+ identity.slice(unversioned.length)+ );+ if (!match) return;+ const [epoch, format, storage] = match.slice(1).map(Number);+ if (+ !Number.isSafeInteger(epoch) ||+ !Number.isSafeInteger(format) ||+ !Number.isSafeInteger(storage)+ ) {+ return;+ }+ return [epoch ?? 0, format ?? 0, storage ?? 0];+}/** Mirrors turso-opfs's canonical lock derivation without exposing a new lock. */export function databaseOwnerLockName(scope: string): string {- const databaseIdentity = `graphql-cache:${scope}`;+ const databaseIdentity = cacheDatabaseIdentity(scope);const byteLength = new TextEncoder().encode(databaseIdentity).byteLength;return `macro:turso-opfs:v1:${byteLength}:${databaseIdentity}`;}diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.tsindex d9839d0a4b..71cc76246c 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts@@ -8,9 +8,11 @@ import {type CoordinatorMessagePort,CoordinatorRouter,} from './coordinator-router';+import type { CacheTakeoverMessage } from './coordinator-takeover';class FakePort extends EventTarget {readonly messages: unknown[] = [];+ readonly received: unknown[] = [];readonly events: string[] = [];closed = false;started = false;@@ -38,6 +40,7 @@ class FakePort extends EventTarget {}receive(message: unknown): void {+ this.received.push(message);if (this.effectProtocol) {this.dispatchEvent(new MessageEvent('message', { data: [1, message] }));return;@@ -60,7 +63,8 @@ const version = {const register = async (router: CoordinatorRouter,port: FakePort,- tabId: string+ tabId: string,+ buildTime = 0): Promise<void> => {await router.handleTabMessage(port as CoordinatorMessagePort, {...version,@@ -68,6 +72,7 @@ const register = async (scope: 'scope',tabId,livenessLockName: `graphql-cache-tab:scope:${tabId}`,+ buildTime,});};@@ -94,7 +99,11 @@ const ready = (ownerEpoch: number,proof: 'opened-existing' | 'wiped-before-open'): void => {- if (messagesOfKind(enginePort, 'open-engine').length === 0) {+ const sent = (kind: string) =>+ enginePort.received.some(+ (message) => (message as { kind?: unknown }).kind === kind+ );+ if (!sent('engine-assets-ready')) {enginePort.receive({...version,kind: 'engine-assets-ready',@@ -102,6 +111,14 @@ const ready = (ownerEpoch,});}+ if (!sent('owner-lock-acquired')) {+ enginePort.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId,+ ownerEpoch,+ });+ }enginePort.receive({...version,kind: 'engine-ready',@@ -234,6 +251,12 @@ describe('CoordinatorRouter', () => {tabId: 'tab-a',ownerEpoch: 1,});+ initialEngine.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });initialEngine.receive({...version,kind: 'activation-failed',@@ -265,6 +288,12 @@ describe('CoordinatorRouter', () => {tabId,ownerEpoch: state.ownerEpoch,});+ engine.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId,+ ownerEpoch: state.ownerEpoch,+ });engine.receive({...version,kind: 'activation-failed',@@ -345,6 +374,7 @@ describe('CoordinatorRouter', () => {scope: 'scope',tabId: 'stale-tab',livenessLockName: 'graphql-cache-tab:scope:stale-tab',+ buildTime: 0,});stalePort.onmessageerror?.();@@ -691,10 +721,712 @@ describe('CoordinatorRouter', () => {);});+ const heldLocks = (names: () => string[]) => async (): Promise<string[]> =>+ names();++ const livenessLocks = (...tabIds: string[]) =>+ tabIds.map((tabId) => `graphql-cache-tab:scope:${tabId}`);++ const busy = (engine: FakePort): void => {+ engine.receive({+ ...version,+ kind: 'owner-lock-busy',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ };++ const fakeTakeoverChannel = () => {+ const posted: CacheTakeoverMessage[] = [];+ let deliver: ((message: CacheTakeoverMessage) => void) | undefined;+ return {+ posted,+ open: (+ _scope: string,+ onMessage: (message: CacheTakeoverMessage) => void+ ) => {+ deliver = onMessage;+ return {+ post: (message: CacheTakeoverMessage) => posted.push(message),+ close: () => {},+ };+ },+ deliver: (message: CacheTakeoverMessage) => deliver?.(message),+ };+ };++ /** Two tabs of this build queue work while another build's tab is alive+ * and the engine finds the owner lock busy. */+ const askForAnotherBuildsDatabase = async () => {+ const observations: Array<{ name: string; ownerEvent?: string }> = [];+ const channel = fakeTakeoverChannel();+ const router = new CoordinatorRouter({+ takeoverReplyTimeoutMs: 100,+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ queryHeldLockNames: heldLocks(() =>+ livenessLocks('tab-a', 'tab-b', 'other-build-tab')+ ),+ openTakeoverChannel: channel.open,+ telemetry: {+ record: (observation) => observations.push(observation),+ flush: vi.fn(),+ },+ });+ const tabA = new FakePort();+ const tabB = new FakePort();+ await register(router, tabA, 'tab-a', 2_000);+ await register(router, tabB, 'tab-b', 1_900);+ for (const [tab, tabId, request] of [+ [tabA, 'tab-a', { id: 1, kind: 'init', scope: 'scope' }],+ [tabB, 'tab-b', { id: 2, kind: 'clear' }],+ ] as const) {+ await router.handleTabMessage(tab as CoordinatorMessagePort, {+ ...version,+ kind: 'cache-request',+ tabId,+ request,+ });+ }+ const engine = new FakePort();+ await attach(router, tabA, 'tab-a', 1, engine);+ engine.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ busy(engine);+ await vi.advanceTimersByTimeAsync(0);+ return { router, channel, observations, tabA, tabB, engine };+ };++ const expectFailedClosed = (+ tabA: FakePort,+ tabB: FakePort,+ reason: string+ ): void => {+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([+ expect.objectContaining({+ ownerEpoch: 1,+ reason: expect.stringContaining(reason),+ }),+ ]);+ for (const [tab, requestId] of [+ [tabA, 1],+ [tabB, 2],+ ] as const) {+ expect(messagesOfKind(tab, 'cache-message')).toEqual([+ expect.objectContaining({+ message: expect.objectContaining({+ id: requestId,+ ok: false,+ errorCode: 'owner-lock-unavailable',+ }),+ }),+ ]);+ expect(messagesOfKind(tab, 'cache-unavailable')).toHaveLength(1);+ }+ };++ it('asks another build for its database and fails closed if nobody answers', async () => {+ vi.useFakeTimers();+ const { router, channel, observations, tabA, tabB, engine } =+ await askForAnotherBuildsDatabase();++ // The request carries the newest build among this coordinator's tabs.+ expect(channel.posted).toEqual([+ {+ takeover: 1,+ kind: 'request',+ scope: 'scope',+ requestId: expect.stringMatching(/^1:/),+ buildTime: 2_000,+ },+ ]);+ busy(engine);+ await vi.advanceTimersByTimeAsync(99);+ expect(channel.posted).toHaveLength(1);+ expect(messagesOfKind(tabB, 'cache-unavailable')).toHaveLength(0);++ // Builds from before handover never answer.+ await vi.advanceTimersByTimeAsync(1);+ expectFailedClosed(tabA, tabB, 'another build keeps the database');+ expect(+ observations.filter(+ (observation) => observation.ownerEvent === 'owner-lock-unavailable'+ )+ ).toHaveLength(1);+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');++ // Nothing waits in line for the lock; a new tab starts a fresh attempt.+ await vi.advanceTimersByTimeAsync(10 * 60_000);+ const tabC = new FakePort();+ await register(router, tabC, 'tab-c', 2_000);+ expect(messagesOfKind(tabC, 'become-owner')).toEqual([+ expect.objectContaining({+ ownerEpoch: 2,+ databaseAction: 'open-existing',+ }),+ ]);+ });++ it('keeps waiting when the unknown lock was a tab of this build still registering', async () => {+ vi.useFakeTimers();+ const channel = fakeTakeoverChannel();+ const router = new CoordinatorRouter({+ takeoverReplyTimeoutMs: 100,+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ // tab-late holds its liveness lock before it sends register-tab.+ queryHeldLockNames: heldLocks(() => livenessLocks('tab-a', 'tab-late')),+ openTakeoverChannel: channel.open,+ });+ const tab = new FakePort();+ await register(router, tab, 'tab-a', 2_000);+ const engine = new FakePort();+ await attach(router, tab, 'tab-a', 1, engine);+ engine.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ busy(engine);+ await vi.advanceTimersByTimeAsync(0);+ expect(channel.posted).toHaveLength(1);++ await register(router, new FakePort(), 'tab-late', 2_000);+ await vi.advanceTimersByTimeAsync(100);++ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([]);+ expect(router.snapshot()?.state).toMatchObject({+ kind: 'activating',+ phase: 'awaiting-owner-lock',+ });+ // Later busy reports find only this build's tabs and ask nobody.+ busy(engine);+ await vi.advanceTimersByTimeAsync(5_000);+ expect(channel.posted).toHaveLength(1);+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([]);+ });++ it('fails closed at once when the holding build keeps its database', async () => {+ vi.useFakeTimers();+ const { channel, tabA, tabB } = await askForAnotherBuildsDatabase();+ const [request] = channel.posted;+ if (request?.kind !== 'request') throw new Error('missing request');++ channel.deliver({+ takeover: 1,+ kind: 'reply',+ scope: 'scope',+ requestId: request.requestId,+ decision: 'keep',+ });++ expectFailedClosed(tabA, tabB, 'another build keeps the database');+ });++ it('keeps retrying the lock while the holding build hands it over', async () => {+ vi.useFakeTimers();+ const { router, channel, observations, tabA, tabB, engine } =+ await askForAnotherBuildsDatabase();+ const [request] = channel.posted;+ if (request?.kind !== 'request') throw new Error('missing request');++ // A reply to some other request changes nothing.+ channel.deliver({+ takeover: 1,+ kind: 'reply',+ scope: 'scope',+ requestId: 'another-request',+ decision: 'keep',+ });+ channel.deliver({+ takeover: 1,+ kind: 'reply',+ scope: 'scope',+ requestId: request.requestId,+ decision: 'yield',+ });+ for (let attempt = 2; attempt <= 4; attempt += 1) busy(engine);+ await vi.advanceTimersByTimeAsync(5_000);+ expect(messagesOfKind(tabB, 'cache-unavailable')).toHaveLength(0);+ expect(channel.posted).toHaveLength(1);+ expect(+ observations.filter(+ (observation) => observation.ownerEvent === 'takeover-granted'+ )+ ).toHaveLength(1);++ ready(engine, 'tab-a', 1, 'opened-existing');+ expect(router.snapshot()?.state).toMatchObject({+ kind: 'active',+ tabId: 'tab-a',+ });+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([]);+ });++ it('asks for nothing once the engine took the lock during the lock query', async () => {+ vi.useFakeTimers();+ const channel = fakeTakeoverChannel();+ let answerQuery: ((names: string[]) => void) | undefined;+ const router = new CoordinatorRouter({+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ queryHeldLockNames: () =>+ new Promise<string[]>((resolve) => {+ answerQuery = resolve;+ }),+ openTakeoverChannel: channel.open,+ });+ const tab = new FakePort();+ await register(router, tab, 'tab-a', 2_000);+ const engine = new FakePort();+ await attach(router, tab, 'tab-a', 1, engine);+ engine.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });++ busy(engine);+ engine.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ answerQuery?.(livenessLocks('tab-a', 'other-build-tab'));+ await vi.advanceTimersByTimeAsync(5_000);++ expect(channel.posted).toEqual([]);+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([]);+ expect(router.snapshot()?.state).toMatchObject({+ kind: 'activating',+ phase: 'opening-database',+ });+ });++ it("never takes a departed owner's lingering liveness lock for another build", async () => {+ vi.useFakeTimers();+ const channel = fakeTakeoverChannel();+ const router = new CoordinatorRouter({+ takeoverReplyTimeoutMs: 100,+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ // The departed page's lock is still held while its document unloads.+ queryHeldLockNames: heldLocks(() => livenessLocks('tab-a', 'tab-b')),+ openTakeoverChannel: channel.open,+ });+ const tabA = new FakePort();+ const tabB = new FakePort();+ await register(router, tabA, 'tab-a', 1_000);+ await register(router, tabB, 'tab-b', 1_000);+ const first = new FakePort();+ await attach(router, tabA, 'tab-a', 1, first);+ ready(first, 'tab-a', 1, 'opened-existing');+ await router.handleTabMessage(tabA as CoordinatorMessagePort, {+ ...version,+ kind: 'navigation-departure',+ tabId: 'tab-a',+ ownerEpoch: 1,+ reason: 'page navigation',+ });+ const next = new FakePort();+ await attach(router, tabB, 'tab-b', 2, next);+ next.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-b',+ ownerEpoch: 2,+ });++ // The departing engine still holds the lock for a moment.+ next.receive({+ ...version,+ kind: 'owner-lock-busy',+ tabId: 'tab-b',+ ownerEpoch: 2,+ });+ await vi.advanceTimersByTimeAsync(5_000);++ expect(channel.posted).toEqual([]);+ expect(messagesOfKind(tabB, 'cache-unavailable')).toEqual([]);+ expect(router.snapshot()?.state).toMatchObject({+ kind: 'activating',+ tabId: 'tab-b',+ phase: 'awaiting-owner-lock',+ });+ });++ /** An active engine of an older build, plus a second tab of that build. */+ const holdTheDatabase = async (buildTime: number) => {+ vi.useFakeTimers();+ const observations: Array<{ name: string; ownerEvent?: string }> = [];+ const channel = fakeTakeoverChannel();+ const router = new CoordinatorRouter({+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ openTakeoverChannel: channel.open,+ telemetry: {+ record: (observation) => observations.push(observation),+ flush: vi.fn(),+ },+ });+ const tabA = new FakePort();+ const tabB = new FakePort();+ await register(router, tabA, 'tab-a', buildTime);+ await register(router, tabB, 'tab-b', buildTime);+ const engine = new FakePort();+ await attach(router, tabA, 'tab-a', 1, engine);+ ready(engine, 'tab-a', 1, 'opened-existing');+ const ask = (requestBuildTime: number) =>+ channel.deliver({+ takeover: 1,+ kind: 'request',+ scope: 'scope',+ requestId: 'request-1',+ buildTime: requestBuildTime,+ });+ return { router, channel, observations, tabA, tabB, engine, ask };+ };++ it('keeps its database for a build that is not newer', async () => {+ const { router, channel, tabA, tabB, engine, ask } =+ await holdTheDatabase(2_000);++ ask(2_000);+ ask(1_000);++ expect(channel.posted).toEqual([+ expect.objectContaining({ kind: 'reply', decision: 'keep' }),+ expect.objectContaining({ kind: 'reply', decision: 'keep' }),+ ]);+ expect(router.snapshot()?.state.kind).toBe('active');+ expect(messagesOfKind(engine, 'drain-engine')).toEqual([]);+ for (const tab of [tabA, tabB]) {+ expect(messagesOfKind(tab, 'cache-superseded')).toEqual([]);+ }+ });++ it('hands its database to a newer build and sends every tab there', async () => {+ const { router, channel, observations, tabA, tabB, engine, ask } =+ await holdTheDatabase(1_000);++ ask(2_000);++ expect(channel.posted).toEqual([+ {+ takeover: 1,+ kind: 'reply',+ scope: 'scope',+ requestId: 'request-1',+ decision: 'yield',+ },+ ]);+ for (const tab of [tabA, tabB]) {+ expect(messagesOfKind(tab, 'cache-superseded')).toHaveLength(1);+ }+ // The owner's page stops its engine itself when it leaves.+ expect(messagesOfKind(engine, 'drain-engine')).toEqual([]);+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([]);+ expect(+ observations.filter(+ (observation) => observation.ownerEvent === 'superseded'+ )+ ).toHaveLength(1);+ // Superseded tabs are refused quietly while they wait to reload.+ await router.handleTabMessage(tabB as CoordinatorMessagePort, {+ ...version,+ kind: 'cache-request',+ tabId: 'tab-b',+ request: { id: 5, kind: 'clear' },+ });+ expect(messagesOfKind(tabB, 'cache-message')).toEqual([+ expect.objectContaining({+ message: expect.objectContaining({+ id: 5,+ ok: false,+ errorCode: 'owner-lock-unavailable',+ }),+ }),+ ]);++ await router.handleTabMessage(tabA as CoordinatorMessagePort, {+ ...version,+ kind: 'navigation-departure',+ tabId: 'tab-a',+ ownerEpoch: 1,+ reason: 'page navigation',+ });+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');+ expect(+ observations.filter(+ (observation) => observation.ownerEvent === 'abrupt-loss'+ )+ ).toEqual([]);++ // Nothing is elected again. A late tab of this build runs uncached rather+ // than reloading, which could land on this build again, and a second+ // request finds nothing left to hand over.+ await vi.advanceTimersByTimeAsync(10 * 60_000);+ const tabC = new FakePort();+ await register(router, tabC, 'tab-c', 1_000);+ for (const tab of [tabA, tabB, tabC]) {+ expect(messagesOfKind(tab, 'become-owner')).toHaveLength(+ tab === tabA ? 1 : 0+ );+ }+ expect(messagesOfKind(tabC, 'cache-superseded')).toEqual([]);+ expect(messagesOfKind(tabC, 'cache-unavailable')).toHaveLength(1);+ ask(3_000);+ expect(channel.posted).toHaveLength(1);+ });++ it('sends the tabs of an idle older build to the newer one without answering', async () => {+ vi.useFakeTimers();+ const channel = fakeTakeoverChannel();+ const router = new CoordinatorRouter({+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ openTakeoverChannel: channel.open,+ });+ const tab = new FakePort();+ await register(router, tab, 'tab-a', 1_000);+ const engine = new FakePort();+ await attach(router, tab, 'tab-a', 1, engine);+ engine.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ engine.receive({+ ...version,+ kind: 'owner-lock-unavailable',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');++ channel.deliver({+ takeover: 1,+ kind: 'request',+ scope: 'scope',+ requestId: 'request-1',+ buildTime: 2_000,+ });++ expect(channel.posted).toEqual([]);+ expect(messagesOfKind(tab, 'cache-superseded')).toHaveLength(1);+ });++ it('retries a busy lock within one build until the engine gives up', async () => {+ vi.useFakeTimers();+ const router = new CoordinatorRouter({+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ queryHeldLockNames: heldLocks(() => livenessLocks('tab-a')),+ });+ const tab = new FakePort();+ await register(router, tab, 'tab-a');+ const engine = new FakePort();+ await attach(router, tab, 'tab-a', 1, engine);+ engine.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });++ busy(engine);+ await vi.advanceTimersByTimeAsync(0);+ expect(router.snapshot()?.state).toMatchObject({+ kind: 'activating',+ phase: 'awaiting-owner-lock',+ });++ engine.receive({+ ...version,+ kind: 'owner-lock-unavailable',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });+ expect(messagesOfKind(tab, 'cache-unavailable')).toHaveLength(1);+ expect(messagesOfKind(tab, 'terminate-engine')).toEqual([+ expect.objectContaining({+ reason: expect.stringContaining('the engine stopped retrying'),+ }),+ ]);+ });++ it('fails closed without a wipe when an open finds the files still busy', async () => {+ vi.useFakeTimers();+ const observations: Array<{ name: string; ownerEvent?: string }> = [];+ const router = new CoordinatorRouter({+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ telemetry: {+ record: (observation) => observations.push(observation),+ flush: vi.fn(),+ },+ });+ const tabA = new FakePort();+ const tabB = new FakePort();+ await register(router, tabA, 'tab-a');+ await register(router, tabB, 'tab-b');+ await router.handleTabMessage(tabB as CoordinatorMessagePort, {+ ...version,+ kind: 'cache-request',+ tabId: 'tab-b',+ request: { id: 1, kind: 'clear' },+ });+ const engine = new FakePort();+ await attach(router, tabA, 'tab-a', 1, engine);+ for (const kind of [+ 'engine-assets-ready',+ 'owner-lock-acquired',+ ] as const) {+ engine.receive({ ...version, kind, tabId: 'tab-a', ownerEpoch: 1 });+ }++ // A predecessor, such as a reloading tab's worker, kept the files open.+ engine.receive({+ ...version,+ kind: 'activation-failed',+ tabId: 'tab-a',+ ownerEpoch: 1,+ reason: 'OPFS sync handle open failed (NoModificationAllowedError)',+ failureCode: 'storage-busy',+ });++ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([+ expect.objectContaining({+ ownerEpoch: 1,+ reason: expect.stringContaining('files stayed busy'),+ }),+ ]);+ expect(messagesOfKind(tabB, 'cache-message')).toEqual([+ expect.objectContaining({+ message: expect.objectContaining({+ id: 1,+ ok: false,+ errorCode: 'owner-lock-unavailable',+ }),+ }),+ ]);+ for (const tab of [tabA, tabB]) {+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([+ expect.objectContaining({+ reason: expect.stringContaining('files are still open'),+ }),+ ]);+ expect(messagesOfKind(tab, 'terminal-error')).toEqual([]);+ }+ expect(+ observations.filter(+ (observation) => observation.ownerEvent === 'storage-busy'+ )+ ).toHaveLength(1);+ expect(+ observations.filter(+ (observation) =>+ observation.name === 'graphql_cache.reset_wipe' ||+ observation.name === 'graphql_cache.storage_reset_required'+ )+ ).toEqual([]);++ // Nothing retries in the background. A tab that registers later opens+ // the existing database instead of wiping it.+ await vi.advanceTimersByTimeAsync(10 * 60_000);+ expect(messagesOfKind(tabB, 'become-owner')).toEqual([]);+ const tabC = new FakePort();+ await register(router, tabC, 'tab-c');+ expect(messagesOfKind(tabC, 'become-owner')).toEqual([+ expect.objectContaining({+ ownerEpoch: 2,+ databaseAction: 'open-existing',+ }),+ ]);+ });++ it('lets the owner delete stale databases only when no other build has tabs', async () => {+ vi.useFakeTimers();+ let locks = livenessLocks('tab-a', 'old-build-tab');+ const setup = async () => {+ const router = new CoordinatorRouter({+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ queryHeldLockNames: heldLocks(() => locks),+ });+ const tab = new FakePort();+ await register(router, tab, 'tab-a');+ const engine = new FakePort();+ await attach(router, tab, 'tab-a', 1, engine);+ ready(engine, 'tab-a', 1, 'opened-existing');+ await vi.advanceTimersByTimeAsync(0);+ return tab;+ };++ expect(+ messagesOfKind(await setup(), 'remove-stale-databases')+ ).toHaveLength(0);++ locks = livenessLocks('tab-a');+ expect(messagesOfKind(await setup(), 'remove-stale-databases')).toEqual([+ {+ ...version,+ kind: 'remove-stale-databases',+ tabId: 'tab-a',+ ownerEpoch: 1,+ },+ ]);+ });++ it('replaces an engine that stops reporting while it tries the owner lock', async () => {+ vi.useFakeTimers();+ const router = new CoordinatorRouter({+ ownerLockWaitTimeoutMs: 30,+ verifyTabLockHeld: async () => true,+ watchTabLock: () => () => {},+ });+ const tabA = new FakePort();+ const tabB = new FakePort();+ await register(router, tabA, 'tab-a');+ await register(router, tabB, 'tab-b');+ const engine = new FakePort();+ await attach(router, tabA, 'tab-a', 1, engine);+ engine.receive({+ ...version,+ kind: 'engine-assets-ready',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });++ await vi.advanceTimersByTimeAsync(30);++ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([+ expect.objectContaining({+ reason: 'engine owner lock watchdog timed out',+ }),+ ]);+ // It never had the storage grant, so its replacement keeps the database.+ expect(messagesOfKind(tabB, 'become-owner')).toEqual([+ expect.objectContaining({+ ownerEpoch: 2,+ databaseAction: 'open-existing',+ }),+ ]);+ });+it('allows slow asset loading and reports phase budgets to owners and late joiners', async () => {vi.useFakeTimers();const router = new CoordinatorRouter({assetLoadTimeoutMs: 100,+ ownerLockWaitTimeoutMs: 30,activationTimeoutMs: 10,verifyTabLockHeld: async () => true,watchTabLock: () => () => {},@@ -721,6 +1453,18 @@ describe('CoordinatorRouter', () => {tabId: 'tab-a',ownerEpoch: 1,});+ // Loading assets grants no storage access; the lock wait comes first.+ expect(messagesOfKind(engine, 'open-engine')).toHaveLength(0);+ expect(messagesOfKind(tabB, 'engine-startup').at(-1)).toMatchObject({+ phase: 'awaiting-owner-lock',+ timeoutMs: 30,+ });+ engine.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });expect(messagesOfKind(engine, 'open-engine')).toHaveLength(1);expect(messagesOfKind(tabB, 'engine-startup').at(-1)).toMatchObject({phase: 'opening-database',@@ -968,6 +1712,12 @@ describe('CoordinatorRouter', () => {tabId: 'tab-a',ownerEpoch: 1,});+ openingEngine.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });await vi.advanceTimersByTimeAsync(11);expect(messagesOfKind(tabA, 'terminate-engine')).toContainEqual(expect.objectContaining({@@ -1156,6 +1906,7 @@ describe('CoordinatorRouter', () => {scope: 'scope',tabId: 'tab-a',livenessLockName: 'graphql-cache-tab:scope:tab-a',+ buildTime: 0,});await vi.waitFor(() =>expect(messagesOfKind(tabA, 'registered')).toHaveLength(1)@@ -1335,6 +2086,12 @@ describe('CoordinatorRouter', () => {tabId: 'tab-a',ownerEpoch: 1,});+ engine.receive({+ ...version,+ kind: 'owner-lock-acquired',+ tabId: 'tab-a',+ ownerEpoch: 1,+ });engine.receive({ kind: 'engine-ready', ownerEpoch: 1 });await Promise.resolve();diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-router.tsindex 3032126f72..decae5f253 100644--- a/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts@@ -16,6 +16,7 @@ import {type CoordinatorAction,CoordinatorCore,type CoordinatorSnapshot,+ startupPhaseMayTouchStorage,} from './coordinator-core';import {type ActivationFailureCode,@@ -27,10 +28,18 @@ import {type EngineOpenOutcome,type EngineToCoordinatorEnvelope,type TabToCoordinatorEnvelope,+ tabIdFromLivenessLockName,tabLivenessLockName,validateEngineToCoordinatorEnvelope,validateTabToCoordinatorEnvelope,} from './coordinator-protocol';+import {+ CACHE_TAKEOVER_VERSION,+ type CacheTakeoverChannel,+ type CacheTakeoverMessage,+ type CacheTakeoverRequest,+ openCacheTakeoverChannel,+} from './coordinator-takeover';import {createEffectWorkerTransport,type EffectWorkerTransport,@@ -38,6 +47,8 @@ import {import {ENGINE_ASSET_LOAD_TIMEOUT_MS,ENGINE_DATABASE_OPEN_TIMEOUT_MS,+ OWNER_LOCK_WAIT_TIMEOUT_MS,+ TAKEOVER_REPLY_TIMEOUT_MS,} from './startup';export interface CoordinatorMessagePort {@@ -52,11 +63,20 @@ export type CancelLivenessWatch = () => void;export interface CoordinatorRouterOptions {assetLoadTimeoutMs?: number;+ ownerLockWaitTimeoutMs?: number;+ /** How long a takeover request waits for the holding build to agree. */+ takeoverReplyTimeoutMs?: number;+ openTakeoverChannel?: (+ scope: string,+ onMessage: (message: CacheTakeoverMessage) => void+ ) => CacheTakeoverChannel | undefined;activationTimeoutMs?: number;heartbeatIntervalMs?: number;heartbeatTimeoutMs?: number;verifyTabLockHeld?: (lockName: string) => Promise<boolean>;verifyOwnerLockHeld?: (lockName: string) => Promise<boolean>;+ /** Names of every Web Lock currently held in this storage partition. */+ queryHeldLockNames?: () => Promise<string[]>;watchTabLock?: (lockName: string,onReleased: () => void@@ -72,7 +92,16 @@ type TabConnection = {cancelLivenessWatch: CancelLivenessWatch;};-type PendingRegistration = { cancelled: boolean };+type PendingRegistration = { cancelled: boolean; tabId: string };++/** This coordinator's request for another build's database, per activation. */+type PendingTakeover = {+ tabId: string;+ ownerEpoch: number;+ requestId: string;+ granted: boolean;+ timer: ReturnType<typeof setTimeout> | undefined;+};type EngineRoute = {tabId: string;@@ -82,6 +111,8 @@ type EngineRoute = {const DEFAULT_HEARTBEAT_INTERVAL_MS = 2_000;const DEFAULT_HEARTBEAT_TIMEOUT_MS = 5_000;+/** How many departed tab ids a long-lived coordinator remembers. */+const DEPARTED_TAB_MEMORY = 256;const RECOVERY_RETRY_LIMIT = 5;const RECOVERY_RETRY_BASE_DELAY_MS = 100;@@ -122,7 +153,7 @@ type WithoutVersion<T> = T extends unknown? Omit<T, 'coordinatorVersion'>: never;-const envelope = <T extends { coordinatorVersion: 5 }>(+const envelope = <T extends { coordinatorVersion: 6 }>(value: WithoutVersion<T>): T =>({@@ -141,6 +172,13 @@ export async function verifyExclusiveLockHeld();}+export async function queryHeldLockNames(): Promise<string[]> {+ const snapshot = await navigator.locks.query();+ return (snapshot.held ?? []).flatMap((lock) =>+ lock.name === undefined ? [] : [lock.name]+ );+}+/** Waits for the page's liveness lock to be released or abandoned. */export function watchTabLivenessLock(lockName: string,@@ -171,6 +209,9 @@ export class CoordinatorRouter {private coreValue: CoordinatorCore | undefined;private hotCapacity: number | undefined;private readonly tabs = new Map<string, TabConnection>();+ /** Recently departed tabs, whose liveness locks can outlast their+ * departure but never belong to another build. */+ private readonly departedTabIds = new Set<string>();private readonly portTabs = new Map<CoordinatorMessagePort, string>();private readonly pendingRegistrations = new Map<CoordinatorMessagePort,@@ -186,11 +227,21 @@ export class CoordinatorRouter {| undefined;private readonly assetLoadTimeoutMs: number;+ private readonly ownerLockWaitTimeoutMs: number;+ private readonly takeoverReplyTimeoutMs: number;+ private readonly openTakeoverChannel: NonNullable<+ CoordinatorRouterOptions['openTakeoverChannel']+ >;+ private takeoverChannel: CacheTakeoverChannel | undefined;+ private takeover: PendingTakeover | undefined;+ /** The newest app build among this coordinator's registered pages. */+ private buildTime = 0;private readonly activationTimeoutMs: number;private readonly heartbeatIntervalMs: number;private readonly heartbeatTimeoutMs: number;private readonly verifyTabLockHeld: (lockName: string) => Promise<boolean>;private readonly verifyOwnerLockHeld: (lockName: string) => Promise<boolean>;+ private readonly queryHeldLockNames: () => Promise<string[]>;private readonly watchTabLock: (lockName: string,onReleased: () => void@@ -222,6 +273,12 @@ export class CoordinatorRouter {constructor(options: CoordinatorRouterOptions = {}) {this.assetLoadTimeoutMs =options.assetLoadTimeoutMs ?? ENGINE_ASSET_LOAD_TIMEOUT_MS;+ this.ownerLockWaitTimeoutMs =+ options.ownerLockWaitTimeoutMs ?? OWNER_LOCK_WAIT_TIMEOUT_MS;+ this.takeoverReplyTimeoutMs =+ options.takeoverReplyTimeoutMs ?? TAKEOVER_REPLY_TIMEOUT_MS;+ this.openTakeoverChannel =+ options.openTakeoverChannel ?? openCacheTakeoverChannel;this.activationTimeoutMs =options.activationTimeoutMs ?? ENGINE_DATABASE_OPEN_TIMEOUT_MS;this.heartbeatIntervalMs =@@ -232,6 +289,7 @@ export class CoordinatorRouter {options.verifyTabLockHeld ?? verifyExclusiveLockHeld;this.verifyOwnerLockHeld =options.verifyOwnerLockHeld ?? verifyExclusiveLockHeld;+ this.queryHeldLockNames = options.queryHeldLockNames ?? queryHeldLockNames;this.watchTabLock = options.watchTabLock ?? watchTabLivenessLock;this.setTimeoutFn =options.setTimeout ?? globalThis.setTimeout.bind(globalThis);@@ -374,7 +432,10 @@ export class CoordinatorRouter {return;}- const registration: PendingRegistration = { cancelled: false };+ const registration: PendingRegistration = {+ cancelled: false,+ tabId: message.tabId,+ };this.pendingRegistrations.set(port, registration);let lockHeld = false;try {@@ -426,7 +487,12 @@ export class CoordinatorRouter {if (!this.coreValue) {this.coreValue = new CoordinatorCore(message.scope);this.hotCapacity = message.hotCapacity;+ this.takeoverChannel = this.openTakeoverChannel(+ message.scope,+ (takeover) => this.handleTakeoverMessage(takeover)+ );}+ this.buildTime = Math.max(this.buildTime, message.buildTime);let connection: TabConnection;const cancelLivenessWatch = this.watchTabLock(message.livenessLockName,@@ -544,7 +610,9 @@ export class CoordinatorRouter {switch (message.kind) {case 'engine-assets-ready': {- if (!core.beginEngineOpen(message.tabId, message.ownerEpoch)) {+ // The engine now tries to take the owner lock without touching+ // storage, and gives up by itself if another context keeps it.+ if (!core.beginOwnerLockWait(message.tabId, message.ownerEpoch)) {this.failOwner(message.tabId,message.ownerEpoch,@@ -553,6 +621,19 @@ export class CoordinatorRouter {break;}this.armStartupWatchdog(message.tabId, message.ownerEpoch);+ break;+ }+ case 'owner-lock-acquired': {+ if (!core.beginEngineOpen(message.tabId, message.ownerEpoch)) {+ this.failOwner(+ message.tabId,+ message.ownerEpoch,+ 'unexpected owner lock acquisition'+ );+ break;+ }+ this.clearTakeover();+ this.armStartupWatchdog(message.tabId, message.ownerEpoch);this.sendToEngine(route,envelope<CoordinatorToEngineEnvelope>({@@ -562,6 +643,16 @@ export class CoordinatorRouter {);break;}+ case 'owner-lock-busy':+ void this.checkBusyOwnerLock(route);+ break;+ case 'owner-lock-unavailable':+ this.closeUnavailable(+ route.tabId,+ route.ownerEpoch,+ 'the engine stopped retrying'+ );+ break;case 'engine-ready': {const actions = core.engineReady({...message,@@ -626,6 +717,7 @@ export class CoordinatorRouter {this.clearActivationTimer();this.resetRecoveryRetries();this.scheduleHeartbeat(message.ownerEpoch);+ void this.allowStaleDatabaseRemoval(route);}break;}@@ -684,6 +776,15 @@ export class CoordinatorRouter {);break;case 'activation-failed':+ if (message.failureCode === 'storage-busy') {+ this.closeUnavailable(+ route.tabId,+ route.ownerEpoch,+ message.reason,+ 'storage-busy'+ );+ break;+ }this.failOwner(message.tabId,message.ownerEpoch,@@ -844,6 +945,31 @@ export class CoordinatorRouter {case 'schedule-reset-activation':this.scheduleResetActivation();break;+ case 'broadcast-cache-unavailable':+ this.broadcast(+ envelope<CoordinatorToTabEnvelope>({+ kind: 'cache-unavailable',+ reason: action.reason,+ })+ );+ break;+ case 'broadcast-cache-superseded':+ this.broadcast(+ envelope<CoordinatorToTabEnvelope>({+ kind: 'cache-superseded',+ reason: action.reason,+ })+ );+ break;+ case 'notify-cache-unavailable':+ this.postToTab(+ action.tabId,+ envelope<CoordinatorToTabEnvelope>({+ kind: 'cache-unavailable',+ reason: action.reason,+ })+ );+ break;case 'broadcast-engine-replaced':this.telemetry.record({name: 'graphql_cache.owner',@@ -912,10 +1038,11 @@ export class CoordinatorRouter {ownerEpoch: state.ownerEpoch,phase: state.phase,databaseAction: state.databaseAction,- timeoutMs:- state.phase === 'loading-assets'- ? this.assetLoadTimeoutMs- : this.activationTimeoutMs,+ timeoutMs: match(state.phase)+ .with('loading-assets', () => this.assetLoadTimeoutMs)+ .with('awaiting-owner-lock', () => this.ownerLockWaitTimeoutMs)+ .with('opening-database', () => this.activationTimeoutMs)+ .exhaustive(),});}@@ -924,15 +1051,138 @@ export class CoordinatorRouter {const progress = this.startupProgress();if (!progress) return;this.broadcast(progress);+ const { phase, timeoutMs } = progress;this.activationTimer = this.setTimeoutFn(() => {this.failOwner(tabId,ownerEpoch,- progress.phase === 'loading-assets'- ? 'engine asset loading watchdog timed out'- : 'engine database opening watchdog timed out'+ match(phase)+ .with(+ 'loading-assets',+ () => 'engine asset loading watchdog timed out'+ )+ .with(+ 'awaiting-owner-lock',+ () => 'engine owner lock watchdog timed out'+ )+ .with(+ 'opening-database',+ () => 'engine database opening watchdog timed out'+ )+ .exhaustive());- }, progress.timeoutMs);+ }, timeoutMs);+ }++ /** Tab ids holding this scope's liveness locks without registering here,+ * so they belong to another coordinator: another deployed build. */+ private async otherBuildTabIds(): Promise<string[] | undefined> {+ const core = this.coreValue;+ if (!core) return;+ let names: string[];+ try {+ names = await this.queryHeldLockNames();+ } catch {+ return;+ }+ const known = new Set([...this.tabs.keys(), ...this.departedTabIds]);+ for (const registration of this.pendingRegistrations.values()) {+ known.add(registration.tabId);+ }+ return names.flatMap((name) => {+ const tabId = tabIdFromLivenessLockName(core.scope, name);+ return tabId === undefined || known.has(tabId) ? [] : [tabId];+ });+ }++ /** This build's previous owner frees the lock within milliseconds. While+ * another build has live tabs, the lock is likely theirs, so ask for it. */+ private async checkBusyOwnerLock(route: EngineRoute): Promise<void> {+ if (this.takeover?.ownerEpoch === route.ownerEpoch) return;+ const others = await this.otherBuildTabIds();+ const state = this.coreValue?.state;+ if (+ !others?.length ||+ this.engineRoute !== route ||+ state?.kind !== 'activating' ||+ state.ownerEpoch !== route.ownerEpoch ||+ state.phase !== 'awaiting-owner-lock'+ ) {+ return;+ }+ this.requestTakeover(route);+ }++ /** Fails closed without touching storage again: the owner lock stayed with+ * another context, or it kept the database files open (`storage-busy`). */+ private closeUnavailable(+ tabId: string,+ ownerEpoch: number,+ detail: string,+ cause: 'owner-lock' | 'storage-busy' = 'owner-lock'+ ): void {+ const core = this.coreValue;+ if (!core) return;+ const state = core.state;+ const actions =+ cause === 'storage-busy'+ ? core.storageBusy(tabId, ownerEpoch)+ : core.ownerLockUnavailable(tabId, ownerEpoch);+ if (actions.length === 0) return;+ this.activationStarted.delete(ownerEpoch);+ // Storage was not changed, or a busy wipe removed only part of it, so a+ // pending recovery wipe carries over to the next attempt exactly as after+ // a navigation departure.+ const interruptedResetReason =+ state.kind === 'activating' && state.databaseAction === 'wipe-before-open'+ ? this.pendingRecoveryResetEpochs.get(ownerEpoch)+ : undefined;+ if (interruptedResetReason !== undefined) {+ this.pendingRecoveryResetEpochs.delete(ownerEpoch);+ this.nextRecoveryResetReason = interruptedResetReason;+ }+ this.telemetry.record({+ name: 'graphql_cache.owner',+ operationCategory: 'lifecycle',+ outcome: 'error',+ ownerEvent:+ cause === 'storage-busy' ? 'storage-busy' : 'owner-lock-unavailable',+ errorCode: cause === 'storage-busy' ? 'opfs-io' : 'lock',+ });+ this.postTerminateEngine(+ tabId,+ ownerEpoch,+ cause === 'storage-busy'+ ? `cache database files stayed busy: ${detail}`+ : `cache database owner lock is unavailable: ${detail}`+ );+ this.clearEngineWatchdogs();+ this.resetRecoveryRetries();+ this.applyActions(actions);+ }++ /** Stale databases are deleted only once no other build could still use+ * them; the cleanup itself also skips any database whose lock is held. */+ private async allowStaleDatabaseRemoval(route: EngineRoute): Promise<void> {+ const others = await this.otherBuildTabIds();+ const state = this.coreValue?.state;+ if (+ others === undefined ||+ others.length > 0 ||+ this.engineRoute !== route ||+ state?.kind !== 'active' ||+ state.ownerEpoch !== route.ownerEpoch+ ) {+ return;+ }+ this.postToTab(+ route.tabId,+ envelope<CoordinatorToTabEnvelope>({+ kind: 'remove-stale-databases',+ tabId: route.tabId,+ ownerEpoch: route.ownerEpoch,+ })+ );}private scheduleResetActivation(): void {@@ -1091,7 +1341,7 @@ export class CoordinatorRouter {this.activationStarted.delete(state.ownerEpoch);if (state.kind !== 'activating' ||- state.phase !== 'loading-assets' ||+ startupPhaseMayTouchStorage(state.phase) ||state.databaseAction === 'wipe-before-open') {const pendingReason = this.recordPendingResetFailure(@@ -1258,6 +1508,12 @@ export class CoordinatorRouter {const connection = this.tabs.get(tabId);if (!connection) return;this.tabs.delete(tabId);+ // Tab ids are fresh per page load, so the oldest can be forgotten.+ this.departedTabIds.add(tabId);+ if (this.departedTabIds.size > DEPARTED_TAB_MEMORY) {+ const [oldest] = this.departedTabIds;+ if (oldest !== undefined) this.departedTabIds.delete(oldest);+ }this.portTabs.delete(connection.port);connection.cancelLivenessWatch();connection.port.close();@@ -1390,5 +1646,137 @@ export class CoordinatorRouter {private clearEngineWatchdogs(): void {this.clearActivationTimer();this.clearHeartbeatTimers();+ this.clearTakeover();+ }++ /**+ * Asks the build that holds the database to hand it over. The engine keeps+ * retrying the lock meanwhile. Unless the holder agrees promptly, or the+ * unknown tabs turn out to be this build's, this activation fails closed.+ */+ private requestTakeover(route: EngineRoute): void {+ const core = this.coreValue;+ if (!core || this.takeover?.ownerEpoch === route.ownerEpoch) return;+ const channel = this.takeoverChannel;+ if (!channel) {+ this.closeUnavailable(+ route.tabId,+ route.ownerEpoch,+ 'another build has live tabs'+ );+ return;+ }+ const requestId = `${route.ownerEpoch}:${crypto.randomUUID()}`;+ const takeover: PendingTakeover = {+ tabId: route.tabId,+ ownerEpoch: route.ownerEpoch,+ requestId,+ granted: false,+ timer: undefined,+ };+ takeover.timer = this.setTimeoutFn(() => {+ takeover.timer = undefined;+ void this.expireTakeover(takeover);+ }, this.takeoverReplyTimeoutMs);+ this.takeover = takeover;+ channel.post({+ takeover: CACHE_TAKEOVER_VERSION,+ kind: 'request',+ scope: core.scope,+ requestId,+ buildTime: this.buildTime,+ });+ }++ /**+ * Nobody answered. The unknown liveness lock that prompted the request may+ * belong to a tab of this build that had not registered yet, so look again:+ * once every live tab is known, keep waiting for the engine's own retries.+ */+ private async expireTakeover(takeover: PendingTakeover): Promise<void> {+ if (this.takeover !== takeover || takeover.granted) return;+ const others = await this.otherBuildTabIds();+ if (this.takeover !== takeover || takeover.granted) return;+ if (others?.length === 0) {+ this.clearTakeover();+ return;+ }+ this.closeUnavailable(+ takeover.tabId,+ takeover.ownerEpoch,+ 'another build keeps the database'+ );+ }++ private handleTakeoverMessage(message: CacheTakeoverMessage): void {+ if (message.kind === 'request') {+ this.answerTakeover(message);+ return;+ }+ const takeover = this.takeover;+ if (+ !takeover ||+ takeover.requestId !== message.requestId ||+ takeover.granted+ ) {+ return;+ }+ if (takeover.timer !== undefined) this.clearTimeoutFn(takeover.timer);+ takeover.timer = undefined;+ if (message.decision === 'yield') {+ takeover.granted = true;+ this.telemetry.record({+ name: 'graphql_cache.owner',+ operationCategory: 'lifecycle',+ outcome: 'success',+ ownerEvent: 'takeover-granted',+ });+ return;+ }+ this.closeUnavailable(+ takeover.tabId,+ takeover.ownerEpoch,+ 'another build keeps the database'+ );+ }++ /**+ * Only the coordinator whose engine holds the database answers, and it+ * yields only to a strictly newer build, so builds never trade the database+ * back and forth and a reload never lands on an older build. An idle+ * coordinator's tabs already run on the network; they move on silently.+ */+ private answerTakeover(request: CacheTakeoverRequest): void {+ const core = this.coreValue;+ if (!core || core.superseded) return;+ const newer = request.buildTime > this.buildTime;+ const holdsDatabase = core.state.kind === 'active';+ if (holdsDatabase) {+ this.takeoverChannel?.post({+ takeover: CACHE_TAKEOVER_VERSION,+ kind: 'reply',+ scope: core.scope,+ requestId: request.requestId,+ decision: newer ? 'yield' : 'keep',+ });+ }+ if (!newer) return;+ const actions = core.supersede();+ if (actions.length === 0) return;+ if (holdsDatabase) {+ this.telemetry.record({+ name: 'graphql_cache.owner',+ operationCategory: 'lifecycle',+ outcome: 'graceful',+ ownerEvent: 'superseded',+ });+ }+ this.applyActions(actions);+ }++ private clearTakeover(): void {+ const takeover = this.takeover;+ this.takeover = undefined;+ if (takeover?.timer !== undefined) this.clearTimeoutFn(takeover.timer);}}diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.test.tsnew file mode 100644index 0000000000..305600b3a1--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.test.ts@@ -0,0 +1,61 @@+import { describe, expect, it } from 'vitest';+import { cacheDatabaseIdentity } from './coordinator-protocol';+import {+ cacheTakeoverChannelName,+ parseCacheTakeoverMessage,+} from './coordinator-takeover';++describe('cache takeover messages', () => {+ const request = {+ takeover: 1,+ kind: 'request',+ scope: 'scope',+ requestId: 'r1',+ buildTime: 1_790_000_000_000,+ };+ const reply = {+ takeover: 1,+ kind: 'reply',+ scope: 'scope',+ requestId: 'r1',+ decision: 'yield',+ };++ it('names one channel per database, shared only by builds that open it', () => {+ expect(cacheTakeoverChannelName('scope')).toBe(+ `graphql-cache-takeover:${cacheDatabaseIdentity('scope')}`+ );+ expect(cacheTakeoverChannelName('scope')).toBe(+ 'graphql-cache-takeover:graphql-cache:scope'+ );+ });++ it('reads requests and replies from any build, ignoring added fields', () => {+ expect(parseCacheTakeoverMessage(request)).toEqual(request);+ expect(parseCacheTakeoverMessage(reply)).toEqual(reply);+ expect(+ parseCacheTakeoverMessage({ ...request, addedByALaterBuild: true })+ ).toEqual(request);+ expect(parseCacheTakeoverMessage({ ...reply, decision: 'keep' })).toEqual({+ ...reply,+ decision: 'keep',+ });+ });++ it('ignores anything it cannot act on', () => {+ for (const invalid of [+ undefined,+ 'request',+ [request],+ { ...request, takeover: 2 },+ { ...request, scope: '' },+ { ...request, requestId: '' },+ { ...request, buildTime: -1 },+ { ...request, buildTime: 1.5 },+ { ...request, kind: 'announce' },+ { ...reply, decision: 'maybe' },+ ]) {+ expect(parseCacheTakeoverMessage(invalid)).toBeUndefined();+ }+ });+});diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.tsnew file mode 100644index 0000000000..3c4cc7eccd--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.ts@@ -0,0 +1,110 @@+import { cacheDatabaseIdentity } from './coordinator-protocol';++/**+ * Handover of the cache database between app builds.+ *+ * A build usually runs its own coordinator, because the coordinator's script+ * URL is content-hashed, yet builds with the same storage version open one+ * database. When a newer build finds that database held by an older one, it+ * asks on this channel, which only builds opening the same database join.+ * The holder either yields, closing the database and telling its tabs to+ * reload into the newer build, or keeps it.+ *+ * Builds answer each other across releases, so the channel name and these+ * message shapes must never change. Extend them only with fields that older+ * builds may ignore, or add a new `takeover` version beside this one.+ */+export const CACHE_TAKEOVER_VERSION = 1;++export const cacheTakeoverChannelName = (scope: string): string =>+ `graphql-cache-takeover:${cacheDatabaseIdentity(scope)}`;++export type CacheTakeoverRequest = {+ takeover: typeof CACHE_TAKEOVER_VERSION;+ kind: 'request';+ scope: string;+ requestId: string;+ /** The requesting build's time; only a strictly newer build is yielded to. */+ buildTime: number;+};++export type CacheTakeoverReply = {+ takeover: typeof CACHE_TAKEOVER_VERSION;+ kind: 'reply';+ scope: string;+ requestId: string;+ /** Sent only by the build whose engine holds the database. */+ decision: 'yield' | 'keep';+};++export type CacheTakeoverMessage = CacheTakeoverRequest | CacheTakeoverReply;++const isRecord = (value: unknown): value is Record<string, unknown> =>+ typeof value === 'object' && value !== null && !Array.isArray(value);++const isNonEmptyString = (value: unknown): value is string =>+ typeof value === 'string' && value.length > 0;++/** Reads a message from any build. Unknown fields are ignored, not rejected,+ * so later builds can add them. */+export function parseCacheTakeoverMessage(+ value: unknown+): CacheTakeoverMessage | undefined {+ if (+ !isRecord(value) ||+ value.takeover !== CACHE_TAKEOVER_VERSION ||+ !isNonEmptyString(value.scope) ||+ !isNonEmptyString(value.requestId)+ ) {+ return;+ }+ const { scope, requestId } = value;+ if (+ value.kind === 'request' &&+ Number.isSafeInteger(value.buildTime) &&+ (value.buildTime as number) >= 0+ ) {+ return {+ takeover: CACHE_TAKEOVER_VERSION,+ kind: 'request',+ scope,+ requestId,+ buildTime: value.buildTime as number,+ };+ }+ if (+ value.kind === 'reply' &&+ (value.decision === 'yield' || value.decision === 'keep')+ ) {+ return {+ takeover: CACHE_TAKEOVER_VERSION,+ kind: 'reply',+ scope,+ requestId,+ decision: value.decision,+ };+ }+}++export interface CacheTakeoverChannel {+ post(message: CacheTakeoverMessage): void;+ close(): void;+}++/** Opens the scope's takeover channel, or returns undefined where the+ * browser has no BroadcastChannel; builds then never hand over. */+export function openCacheTakeoverChannel(+ scope: string,+ onMessage: (message: CacheTakeoverMessage) => void+): CacheTakeoverChannel | undefined {+ if (typeof BroadcastChannel !== 'function') return;+ const channel = new BroadcastChannel(cacheTakeoverChannelName(scope));+ channel.onmessage = (event: MessageEvent<unknown>) => {+ const message = parseCacheTakeoverMessage(event.data);+ if (message?.scope === scope) onMessage(message);+ };+ return {+ post: (message) => channel.postMessage(message),+ close: () => channel.close(),+ };+}diff --git a/apps/web/src/lib/graphql-cache/worker/stale-databases.test.ts b/apps/web/src/lib/graphql-cache/worker/stale-databases.test.tsnew file mode 100644index 0000000000..c24a9ca328--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/stale-databases.test.ts@@ -0,0 +1,54 @@+import { describe, expect, it, vi } from 'vitest';+import { cacheDatabaseIdentity } from './coordinator-protocol';+import {+ isUnambiguousCacheScope,+ removeStaleCacheDatabases,+ staleCacheDatabaseIdentities,+} from './stale-databases';++describe('stale cache databases', () => {+ it('lists each stale database of the scope once, whichever file remains', () => {+ const own = cacheDatabaseIdentity('scope');+ expect(+ staleCacheDatabaseIdentities('scope', [+ own,+ `${own}-wal`,+ 'graphql-cache:scope:s1.v1.t0',+ 'graphql-cache:scope:s1.v1.t0-wal',+ 'graphql-cache:scope:s1.v1.t1-wal',+ 'graphql-cache:other-scope',+ 'unrelated.db',+ ])+ ).toEqual(['graphql-cache:scope:s1.v1.t0', 'graphql-cache:scope:s1.v1.t1']);+ });++ it('refuses scopes whose database would read as another WAL file', () => {+ expect(+ isUnambiguousCacheScope('4f7c9d1e-0b2a-4c8e-9f31-7a6d5e4c3b2a')+ ).toBe(true);+ expect(isUnambiguousCacheScope('quarantine:scope')).toBe(true);+ expect(isUnambiguousCacheScope('scope-wal')).toBe(false);+ });++ it('tallies each outcome and stops at the first failure', async () => {+ const removeOne = vi+ .fn()+ .mockResolvedValueOnce({ outcome: 'removed' })+ .mockResolvedValueOnce({ outcome: 'in-use' })+ .mockResolvedValueOnce({+ outcome: 'queued-mutations',+ queuedMutations: 3,+ })+ .mockRejectedValueOnce(new Error('unreadable database'));++ expect(+ await removeStaleCacheDatabases(['a', 'b', 'c', 'd', 'e'], removeOne)+ ).toEqual({+ removed: 1,+ inUse: 1,+ keptWithQueuedMutations: 1,+ failed: true,+ });+ expect(removeOne).toHaveBeenCalledTimes(4);+ });+});diff --git a/apps/web/src/lib/graphql-cache/worker/stale-databases.ts b/apps/web/src/lib/graphql-cache/worker/stale-databases.tsnew file mode 100644index 0000000000..30576c71ba--- /dev/null+++ b/apps/web/src/lib/graphql-cache/worker/stale-databases.ts@@ -0,0 +1,76 @@+import { isStaleCacheDatabaseIdentity } from './coordinator-protocol';++const WAL_SUFFIX = '-wal';++/**+ * A database's WAL file is its name plus `-wal`, so the database of a scope+ * ending in `-wal` would read as another scope's WAL file and could be+ * removed with it. App scopes are UUIDs; a host refuses any other such scope.+ */+export const isUnambiguousCacheScope = (scope: string): boolean =>+ !scope.endsWith(WAL_SUFFIX);++/** This scope's databases this build does not open, from OPFS entry names. */+export function staleCacheDatabaseIdentities(+ scope: string,+ entryNames: Iterable<string>+): string[] {+ const identities = new Set<string>();+ for (const name of entryNames) {+ const identity = name.endsWith(WAL_SUFFIX)+ ? name.slice(0, -WAL_SUFFIX.length)+ : name;+ if (isStaleCacheDatabaseIdentity(scope, identity)) identities.add(identity);+ }+ return [...identities];+}++/** Lists the origin-private root without opening any file. */+export async function listOpfsRootNames(): Promise<string[]> {+ const root = await navigator.storage.getDirectory();+ const names: string[] = [];+ for await (const name of root.keys()) names.push(name);+ return names;+}++export type StaleDatabaseRemovalOutcome = {+ outcome: 'removed' | 'in-use' | 'queued-mutations';+ queuedMutations?: number;+};++export type StaleDatabaseCleanupTally = {+ removed: number;+ inUse: number;+ keptWithQueuedMutations: number;+ failed: boolean;+};++/**+ * Deletes each stale database whose owner lock is free and whose mutation+ * queue is empty. Stops at the first failure: in the disposable cleanup worker+ * it can leave the worker-local OPFS registry poisoned.+ */+export async function removeStaleCacheDatabases(+ identities: readonly string[],+ removeOne: (identity: string) => Promise<StaleDatabaseRemovalOutcome>+): Promise<StaleDatabaseCleanupTally> {+ const tally: StaleDatabaseCleanupTally = {+ removed: 0,+ inUse: 0,+ keptWithQueuedMutations: 0,+ failed: false,+ };+ for (const identity of identities) {+ let result: StaleDatabaseRemovalOutcome;+ try {+ result = await removeOne(identity);+ } catch {+ tally.failed = true;+ break;+ }+ if (result.outcome === 'removed') tally.removed += 1;+ else if (result.outcome === 'in-use') tally.inUse += 1;+ else tally.keptWithQueuedMutations += 1;+ }+ return tally;+}diff --git a/apps/web/src/lib/graphql-cache/worker/startup.ts b/apps/web/src/lib/graphql-cache/worker/startup.tsindex 59f842a172..bba181a6af 100644--- a/apps/web/src/lib/graphql-cache/worker/startup.ts+++ b/apps/web/src/lib/graphql-cache/worker/startup.ts@@ -4,6 +4,21 @@export const COORDINATOR_CONNECT_TIMEOUT_MS = 60_000;export const ENGINE_ASSET_LOAD_TIMEOUT_MS = 5 * 60_000;export const ENGINE_DATABASE_OPEN_TIMEOUT_MS = 20_000;+/** Delays between attempts to take a busy database owner lock. The engine+ * never queues for it: a handoff between owners of one build frees it within+ * milliseconds, and after these ~10s the engine gives up without touching+ * storage, so pages use the network until they reload.+ */+export const OWNER_LOCK_RETRY_DELAYS_MS: readonly number[] = [+ 25, 50, 100, 200, 400, 800, 1_600, 3_200, 3_200,+];+/** When another build has live tabs, a busy lock is almost certainly theirs.+ * The coordinator asks that build to hand the database over and gives up+ * unless the holder agrees this soon; builds from before handover never do. */+export const TAKEOVER_REPLY_TIMEOUT_MS = 1_000;+/** Coordinator backstop for an engine that stops reporting while it waits. */+export const OWNER_LOCK_WAIT_TIMEOUT_MS =+ OWNER_LOCK_RETRY_DELAYS_MS.reduce((total, delay) => total + delay, 0) + 5_000;/** Lets the coordinator's phase timeout/retry reach a page before its guard fires. */export const STARTUP_RESPONSE_GRACE_MS = 5_000;export const COORDINATOR_CONNECT_ATTEMPTS = 3;diff --git a/apps/web/src/lib/graphql-cache/worker/wasm-module.ts b/apps/web/src/lib/graphql-cache/worker/wasm-module.tsindex f4c86c70eb..1e39dc0c77 100644--- a/apps/web/src/lib/graphql-cache/worker/wasm-module.ts+++ b/apps/web/src/lib/graphql-cache/worker/wasm-module.ts@@ -31,6 +31,7 @@ import type {WriteResult,} from '../protocol';import { workerCacheTelemetry } from '../telemetry-relay';+import type { StaleDatabaseRemovalOutcome } from './stale-databases';/** Stable, payload-free marker latched on reset-required WASM errors. */export interface CacheStorageResetRequiredError extends Error {@@ -176,22 +177,45 @@ export interface CacheEngine {export interface CacheWasmModule {default: (input?: { module_or_path?: unknown }) => Promise<unknown>;openCache(scope: string, hotCapacity?: number): Promise<CacheEngine>;- /** Additive open API with a coarse, payload-free recovery outcome. */+ /**+ * Additive open API with a coarse, payload-free recovery outcome. When+ * given, `onOwnerLockAcquired` runs after the owner lock is held and before+ * any OPFS access; opening continues once its promise resolves. With+ * `ifAvailable`, a held owner lock rejects at once, touching nothing, with an+ * error whose `cacheOwnerLockUnavailable` property is `true`. Files another+ * context keeps open through a bounded wait reject with an error whose+ * `cacheStorageBusy` property is `true`; a plain open has then changed+ * neither file.+ */openCacheWithOutcome?(scope: string,- hotCapacity?: number+ hotCapacity?: number,+ onOwnerLockAcquired?: () => Promise<void>,+ ifAvailable?: boolean): Promise<CacheOpenResult>;/** Atomically wipes before Turso open while retaining one OPFS owner lock. */openCacheForRecovery(scope: string,hotCapacity?: number): Promise<CacheEngine>;- /** Additive recovery-open API with its coarse wipe outcome. */+ /** Additive recovery-open API with its coarse wipe outcome; see above for+ * `onOwnerLockAcquired` and `ifAvailable`, which precede the recovery wipe. */openCacheForRecoveryWithOutcome?(scope: string,- hotCapacity?: number+ hotCapacity?: number,+ onOwnerLockAcquired?: () => Promise<void>,+ ifAvailable?: boolean): Promise<CacheOpenResult>;destroyCache(scope: string): Promise<void>;+ /** Physical database this build opens for `scope`; absent before databases+ * were named by storage version. */+ cacheDatabaseIdentity?(scope: string): string;+ /** Deletes one stale database of `scope` if unused and without queued+ * mutations; see `staleCacheDatabaseIdentities`. */+ removeStaleCacheDatabase?(+ scope: string,+ identity: string+ ): Promise<StaleDatabaseRemovalOutcome>;/** Optional while older cached WASM artifacts are still in circulation. */setSlowQueryCallback?(callback: (diff --git a/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts b/apps/web/src/lib/graphql-cache/worker/worker-core.test.tsindex 1d46812905..456b1dfb59 100644--- a/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts+++ b/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts@@ -5,6 +5,7 @@ const loadCacheWasmMock = vi.hoisted(() => vi.fn());vi.mock('./wasm-module', () => ({ loadCacheWasm: loadCacheWasmMock }));+import { cacheDatabaseIdentity } from './coordinator-protocol';import { CacheWorkerCore } from './worker-core';describe('CacheWorkerCore', () => {@@ -1132,6 +1133,176 @@ describe('CacheWorkerCore', () => {expect(order).toEqual(['read:start', 'read:done', 'response', 'close']);});+ it.each([+ [false, 'openCacheWithOutcome'],+ [true, 'openCacheForRecoveryWithOutcome'],+ ] as const)(+ 'asks for the storage grant from inside the WASM open (recovery: %s)',+ async (recoveryOpen, openName) => {+ const order: string[] = [];+ const open = vi.fn(+ async (+ _scope: string,+ _hotCapacity: number | undefined,+ grant?: () => Promise<void>+ ) => {+ order.push('owner-lock');+ await grant?.();+ order.push('storage');+ return { engine: {}, outcome: 'opened-existing' };+ }+ );+ loadCacheWasmMock.mockResolvedValue({ [openName]: open });+ const messages: unknown[] = [];+ const core = new CacheWorkerCore({+ recoveryOpen,+ onOwnerLockAcquired: async () => {+ order.push('grant');+ },+ });++ await core.handleRequest(+ { postMessage: (message: unknown) => messages.push(message) },+ { id: 1, kind: 'init', scope: 'scope-1' }+ );++ expect(order).toEqual(['owner-lock', 'grant', 'storage']);+ expect(messages).toEqual([{ id: 1, ok: true, result: null }]);+ }+ );++ it('refuses a WASM open that never asked for the storage grant', async () => {+ loadCacheWasmMock.mockResolvedValue({+ openCacheWithOutcome: vi.fn(async () => ({+ engine: {},+ outcome: 'opened-existing',+ })),+ });+ const messages: unknown[] = [];+ const core = new CacheWorkerCore({+ onOwnerLockAcquired: async () => undefined,+ });++ await core.handleRequest(+ { postMessage: (message: unknown) => messages.push(message) },+ { id: 1, kind: 'init', scope: 'scope-1' }+ );++ expect(messages).toEqual([+ {+ id: 1,+ ok: false,+ error: 'cache WASM does not support the owner-lock grant',+ },+ ]);+ });++ const busyOwnerLock = () =>+ Object.assign(new Error('owner lock is held by another context'), {+ cacheOwnerLockUnavailable: true,+ });++ const initMessages = async (core: CacheWorkerCore): Promise<unknown[]> => {+ const messages: unknown[] = [];+ await core.handleRequest(+ { postMessage: (message: unknown) => messages.push(message) },+ { id: 1, kind: 'init', scope: 'scope-1' }+ );+ return messages;+ };++ it.each([+ [false, 'openCacheWithOutcome'],+ [true, 'openCacheForRecoveryWithOutcome'],+ ] as const)(+ 'opens only a free owner lock, asking before each retry (recovery: %s)',+ async (recoveryOpen, openName) => {+ const open = vi+ .fn()+ .mockRejectedValueOnce(busyOwnerLock())+ .mockRejectedValueOnce(busyOwnerLock())+ .mockImplementation(+ async (+ _scope: string,+ _hotCapacity: number | undefined,+ grant?: () => Promise<void>+ ) => {+ await grant?.();+ return { engine: {}, outcome: 'opened-existing' };+ }+ );+ loadCacheWasmMock.mockResolvedValue({+ [openName]: open,+ cacheDatabaseIdentity,+ });+ const onOwnerLockBusy = vi.fn(async (_attempt: number) => undefined);+ const core = new CacheWorkerCore({+ recoveryOpen,+ onOwnerLockAcquired: async () => undefined,+ onOwnerLockBusy,+ });++ expect(await initMessages(core)).toEqual([+ { id: 1, ok: true, result: null },+ ]);+ expect(onOwnerLockBusy.mock.calls).toEqual([[1], [2]]);+ // Every attempt declines to queue behind another holder.+ expect(open.mock.calls.map((call) => call[3])).toEqual([+ true,+ true,+ true,+ ]);+ }+ );++ it('gives up on a busy owner lock when told to stop retrying', async () => {+ const open = vi.fn().mockRejectedValue(busyOwnerLock());+ loadCacheWasmMock.mockResolvedValue({+ openCacheWithOutcome: open,+ cacheDatabaseIdentity,+ });+ const core = new CacheWorkerCore({+ onOwnerLockAcquired: async () => undefined,+ onOwnerLockBusy: async () => {+ throw new Error('owner lock stayed unavailable');+ },+ });++ expect(await initMessages(core)).toEqual([+ { id: 1, ok: false, error: 'owner lock stayed unavailable' },+ ]);+ expect(open).toHaveBeenCalledOnce();+ });++ it.each([+ [+ 'predates storage versions',+ {},+ 'cache WASM predates storage-versioned databases; rebuild it',+ ],+ [+ 'names another version',+ { cacheDatabaseIdentity: () => 'graphql-cache:scope-1:s0.v0.t0' },+ 'cache WASM storage version does not match this build',+ ],+ ])(+ 'refuses a WASM build that %s before opening anything',+ async (_label, identity, error) => {+ const open = vi.fn();+ loadCacheWasmMock.mockResolvedValue({+ openCacheWithOutcome: open,+ ...identity,+ });+ const core = new CacheWorkerCore({+ onOwnerLockAcquired: async () => undefined,+ onOwnerLockBusy: async () => undefined,+ });++ expect(await initMessages(core)).toEqual([{ id: 1, ok: false, error }]);+ expect(open).not.toHaveBeenCalled();+ }+ );+it('uses atomic recovery-open instead of opening before a reset', async () => {const openCache = vi.fn();const openCacheForRecovery = vi.fn().mockResolvedValue({});@@ -1700,4 +1871,33 @@ describe('CacheWorkerCore', () => {{ id: 3, ok: false, error: 'cache storage reset required' },]);});++ it('reports an open that gave up on busy database files', async () => {+ const busyError = Object.assign(+ new Error('OPFS sync handle open failed (NoModificationAllowedError)'),+ { cacheStorageBusy: true as const }+ );+ loadCacheWasmMock.mockResolvedValue({+ openCache: vi.fn().mockRejectedValue(busyError),+ });+ const onStorageBusy = vi.fn();+ const onStorageResetRequired = vi.fn();+ const messages: unknown[] = [];+ const core = new CacheWorkerCore({ onStorageBusy, onStorageResetRequired });++ await core.handleRequest(+ { postMessage: (message: unknown) => messages.push(message) },+ { id: 1, kind: 'init', scope: 'scope-1' }+ );++ expect(onStorageBusy).toHaveBeenCalledOnce();+ expect(onStorageResetRequired).not.toHaveBeenCalled();+ expect(messages).toEqual([+ {+ id: 1,+ ok: false,+ error: 'OPFS sync handle open failed (NoModificationAllowedError)',+ },+ ]);+ });});diff --git a/apps/web/src/lib/graphql-cache/worker/worker-core.ts b/apps/web/src/lib/graphql-cache/worker/worker-core.tsindex cb6b66ef25..3b8c643b7b 100644--- a/apps/web/src/lib/graphql-cache/worker/worker-core.ts+++ b/apps/web/src/lib/graphql-cache/worker/worker-core.ts@@ -26,12 +26,27 @@ import {isStorageTransactionRequest,operationCategoryForRequest,} from '../telemetry';+import { cacheDatabaseIdentity } from './coordinator-protocol';import {type CacheEngine,type CacheOpenOutcome,+ type CacheOpenResult,loadCacheWasm,} from './wasm-module';+const isOwnerLockUnavailable = (error: unknown): boolean =>+ typeof error === 'object' &&+ error !== null &&+ 'cacheOwnerLockUnavailable' in error &&+ error.cacheOwnerLockUnavailable === true;++/** WASM gave up on database files another context kept open. */+const isStorageBusy = (error: unknown): boolean =>+ typeof error === 'object' &&+ error !== null &&+ 'cacheStorageBusy' in error &&+ error.cacheStorageBusy === true;+type PortLike = {postMessage(msg: unknown): void;};@@ -53,8 +68,22 @@ export interface CacheWorkerCoreOptions {recoveryOpen?: boolean;/** Called once when WASM latches a reset-required storage failure. */onStorageResetRequired?: (error: Error) => void;+ /** Called when opening gave up on database files another context kept+ * open. The failed request's response carries only the message. */+ onStorageBusy?: () => void;/** Reports the bounded open outcome to the coordinator transport. */onInitializationOutcome?: (outcome: CacheOpenOutcome) => void;+ /**+ * Runs once WASM holds the database owner lock and before it touches OPFS;+ * opening proceeds when the returned promise resolves.+ */+ onOwnerLockAcquired?: () => Promise<void>;+ /**+ * With `onOwnerLockAcquired`, opening never queues for the owner lock. This+ * runs with the 1-based attempt that found it held elsewhere: resolve to+ * try again, or reject to give up without touching storage.+ */+ onOwnerLockBusy?: (attempt: number) => Promise<void>;telemetry?: CacheTelemetryRecorderLike;/** Injectable clocks and cadence for payload-free diagnostics tests. */monotonicNow?: () => number;@@ -288,6 +317,7 @@ export class CacheWorkerCore {});}this.reportResetRequired(error);+ if (isStorageBusy(error)) this.options.onStorageBusy?.();respond({id: request.id,ok: false,@@ -858,14 +888,50 @@ export class CacheWorkerCore {this.hotCapacity = hotCapacity;this.initPromise = (async () => {const wasm = await loadCacheWasm();- const schemaStartedAt = this.now();+ let schemaStartedAt = this.now();+ const onOwnerLockAcquired = this.options.onOwnerLockAcquired;+ const onOwnerLockBusy = this.options.onOwnerLockBusy;+ let storageGrantRequested = false;+ const awaitStorageGrant = onOwnerLockAcquired+ ? async () => {+ storageGrantRequested = true;+ // Time the open itself, not the wait for another owner.+ schemaStartedAt = this.now();+ await onOwnerLockAcquired();+ }+ : undefined;try {let openOutcome: CacheOpenOutcome;- if (this.options.recoveryOpen) {+ if (awaitStorageGrant && onOwnerLockBusy) {+ const open = this.options.recoveryOpen+ ? wasm.openCacheForRecoveryWithOutcome+ : wasm.openCacheWithOutcome;+ // Checked before any open: an older artifact would queue for the+ // lock and touch storage without the coordinator's grant.+ if (!open || !wasm.cacheDatabaseIdentity) {+ throw new Error(+ 'cache WASM predates storage-versioned databases; rebuild it'+ );+ }+ if (+ wasm.cacheDatabaseIdentity(scope) !== cacheDatabaseIdentity(scope)+ ) {+ throw new Error(+ 'cache WASM storage version does not match this build'+ );+ }+ const opened = await this.openWhenOwnerLockIsFree(+ () => open(scope, hotCapacity, awaitStorageGrant, true),+ onOwnerLockBusy+ );+ this.engine = opened.engine;+ openOutcome = opened.outcome;+ } else if (this.options.recoveryOpen) {if (wasm.openCacheForRecoveryWithOutcome) {const opened = await wasm.openCacheForRecoveryWithOutcome(scope,- hotCapacity+ hotCapacity,+ awaitStorageGrant);this.engine = opened.engine;openOutcome = opened.outcome;@@ -874,13 +940,21 @@ export class CacheWorkerCore {openOutcome = 'reset-storage-uncertain';}} else if (wasm.openCacheWithOutcome) {- const opened = await wasm.openCacheWithOutcome(scope, hotCapacity);+ const opened = await wasm.openCacheWithOutcome(+ scope,+ hotCapacity,+ awaitStorageGrant+ );this.engine = opened.engine;openOutcome = opened.outcome;} else {this.engine = await wasm.openCache(scope, hotCapacity);openOutcome = 'opened-existing';}+ if (awaitStorageGrant && !storageGrantRequested) {+ // A stale artifact opened storage without asking the coordinator.+ throw new Error('cache WASM does not support the owner-lock grant');+ }this.options.onInitializationOutcome?.(openOutcome);this.telemetry.record({name: 'graphql_cache.schema_init',@@ -905,6 +979,20 @@ export class CacheWorkerCore {await this.initPromise;}+ private async openWhenOwnerLockIsFree(+ open: () => Promise<CacheOpenResult>,+ onOwnerLockBusy: (attempt: number) => Promise<void>+ ): Promise<CacheOpenResult> {+ for (let attempt = 1; ; attempt += 1) {+ try {+ return await open();+ } catch (error) {+ if (!isOwnerLockUnavailable(error)) throw error;+ await onOwnerLockBusy(attempt);+ }+ }+ }+private reportResetRequired(error: unknown): void {if (this.resetRequiredReported ||diff --git a/apps/web/src/lib/queries/channel/channels.ts b/apps/web/src/lib/queries/channel/channels.tsindex 69b1bebcaf..d2b8903936 100644--- a/apps/web/src/lib/queries/channel/channels.ts+++ b/apps/web/src/lib/queries/channel/channels.ts@@ -14,6 +14,7 @@ import type { CreateChannelRequest } from '@service-storage/generated/schemas/crimport type { CreateChannelResponse } from '@service-storage/generated/schemas/createChannelResponse';import type { PatchChannelRequest } from '@service-storage/generated/schemas/patchChannelRequest';import { useMutation, useQuery } from '@tanstack/solid-query';+import type { Accessor } from 'solid-js';import { invalidateChannelParticipants } from './channel-participants';import {type CachedGraphqlChannel,@@ -52,9 +53,12 @@ export function useListChannelsQuery() {}/** Reads the recent Quick Access channel list from the normalized GraphQL cache. */-export function useCachedGraphqlChannelsQuery(cacheHost?: CacheHost) {+export function useCachedGraphqlChannelsQuery(+ cacheHost: Accessor<CacheHost | undefined>+) {return useQuery(() => {- const enabledCacheHost = cacheHost?.disabled ? undefined : cacheHost;+ const host = cacheHost();+ const enabledCacheHost = host?.disabled ? undefined : host;return {queryKey: channelKeys.quickAccessGraphql(enabledCacheHost?.clientId ?? 'disabled'diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.tsindex bfeb217e59..363b7b265c 100644--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts@@ -1,4 +1,5 @@import type { NormalizedCacheExchangeOptions } from '@graphql-cache/exchange/normalized-cache-exchange';+import type { CacheHost } from '@graphql-cache/host/types';import type { BrowserTursoCacheRolloutDecision } from '@graphql-cache/rollout-policy';import type { Operation } from '@urql/core';import { parse } from 'graphql';@@ -117,6 +118,7 @@ const mocks = vi.hoisted(() => {});let queuedMutationCount = 0;let initializationErrorHandler: ((error: Error) => void) | undefined;+ let supersededHandler: (() => void) | undefined;const cleanupOrder: string[] = [];const host = {disabled: false,@@ -174,6 +176,7 @@ const mocks = vi.hoisted(() => {queuedMutationCount = 0;cleanupOrder.length = 0;initializationErrorHandler = undefined;+ supersededHandler = undefined;},queueDepth: () => queuedMutationCount,recordSubscriptionDisposal: () => cleanupOrder.push('subscriptions'),@@ -181,6 +184,8 @@ const mocks = vi.hoisted(() => {failInitialization: (error = new Error('injected initialization failure')) => initializationErrorHandler?.(error),+ supersede: () => supersededHandler?.(),+ reloadForNewerBuild: vi.fn(),plainClient,realtimeClient,replaceSubscriptions,@@ -194,8 +199,12 @@ const mocks = vi.hoisted(() => {})),createWorkerCacheHost: vi.fn(- (options: { onInitializationError?: (error: Error) => void }) => {+ (options: {+ onInitializationError?: (error: Error) => void;+ onSuperseded?: () => void;+ }) => {initializationErrorHandler = options.onInitializationError;+ supersededHandler = options.onSuperseded;return host;}),@@ -220,6 +229,9 @@ vi.mock('@core/constant/servers', () => ({SERVER_HOSTS: { 'document-storage-service': 'http://dss.test' },}));vi.mock('@core/util/fetchWithToken', () => ({ fetchToken: vi.fn() }));+vi.mock('@core/util/reloadForNewerBuild', () => ({+ reloadForNewerBuild: mocks.reloadForNewerBuild,+}));vi.mock('@core/util/platform', () => ({ isTauri: () => mocks.tauri }));vi.mock('@core/util/platformFetch', () => ({platformFetch: mocks.platformFetch,@@ -707,6 +719,12 @@ describe('GraphQL Soup browser cache session gate', () => {'@graphql-cache/host/navigation-error');report?.(new CacheNavigationError(), operation);+ report?.(+ Object.assign(new Error('owner lock is held by another context'), {+ errorCode: 'owner-lock-unavailable',+ }),+ operation+ );expect(mocks.telemetryError).not.toHaveBeenCalled();report?.(error, operation);@@ -818,6 +836,84 @@ describe('GraphQL Soup browser cache session gate', () => {);});+ it('retires a failed host so the replaced client degrades instead of rejecting', async () => {+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ const soup = await import('./graphql-soup');+ soup.getGraphqlSoupClient();+ const exchangeHost = mocks.normalizedCacheExchange.mock.calls[0]?.[0] as+ | CacheHost+ | undefined;+ expect(exchangeHost?.disabled).toBe(false);++ mocks.failInitialization();++ // Operations still running on the old client, and callers that captured+ // the host, now miss to the network rather than "host was disposed".+ expect(mocks.host.dispose).toHaveBeenCalledOnce();+ expect(exchangeHost?.disabled).toBe(true);+ await expect(+ exchangeHost?.readQuery({ query: 'query Q { user { id } }' })+ ).resolves.toEqual({ kind: 'miss' });+ await expect(+ exchangeHost?.search({ profile: 'quick-access-v1', limit: 10 })+ ).resolves.toEqual({ documents: [], nextCursor: null });+ });++ it('moves reactive cache readers off a host abandoned mid-session', async () => {+ vi.spyOn(console, 'warn').mockImplementation(() => {});+ const soup = await import('./graphql-soup');+ soup.getGraphqlSoupClient();+ const { createMemo, createRoot } = await import('solid-js');+ const readers = createRoot((dispose) => ({+ dispose,+ enabled: createMemo(() => soup.graphqlCacheEnabled()),+ host: createMemo(() => soup.getGraphqlSoupCacheHost()),+ }));+ expect(readers.enabled()).toBe(true);+ expect(readers.host()).toBeDefined();++ mocks.failInitialization();++ expect(readers.enabled()).toBe(false);+ expect(readers.host()).toBeUndefined();+ readers.dispose();+ });++ it('falls back quietly while another context holds the database', async () => {+ const info = vi.spyOn(console, 'info').mockImplementation(() => {});+ const soup = await import('./graphql-soup');+ const cachedClient = soup.getGraphqlSoupClient();++ mocks.failInitialization(+ Object.assign(new Error('owner lock is held by another context'), {+ errorCode: 'owner-lock-unavailable',+ })+ );++ // Expected after a deploy or while a closing tab lets go: no toast and no+ // error report, but the cache is off for this page session and its host+ // is retired.+ expect(mocks.toastFailure).not.toHaveBeenCalled();+ expect(mocks.telemetryError).not.toHaveBeenCalled();+ expect(info).toHaveBeenCalledExactlyOnceWith(+ '[graphql-cache] owner lock is held by another context; using the network until reload'+ );+ expect(mocks.host.dispose).toHaveBeenCalledOnce();+ expect(soup.graphqlCacheEnabled()).toBe(false);+ expect(soup.getGraphqlSoupClient()).not.toBe(cachedClient);+ });++ it('moves the page to a newer build that took the database over', async () => {+ const soup = await import('./graphql-soup');+ soup.getGraphqlSoupClient();+ expect(mocks.reloadForNewerBuild).not.toHaveBeenCalled();++ mocks.supersede();++ expect(mocks.reloadForNewerBuild).toHaveBeenCalledOnce();+ expect(mocks.toastFailure).not.toHaveBeenCalled();+ });+it('imports and uses the native path without constructing browser workers', async () => {const WorkerConstructor = vi.fn(() => {throw new Error('browser worker must not be constructed on Tauri');diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.tsindex 72c4eec347..aa83b7cf9c 100644--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts@@ -8,11 +8,13 @@ import { SERVER_HOSTS } from '@core/constant/servers';import { fetchToken } from '@core/util/fetchWithToken';import { isTauri } from '@core/util/platform';import { platformFetch } from '@core/util/platformFetch';+import { reloadForNewerBuild } from '@core/util/reloadForNewerBuild';import {HYDRATE_ONLY_CONTEXT_KEY,normalizedCacheExchange,} from '@graphql-cache/exchange/normalized-cache-exchange';import { CacheNavigationError } from '@graphql-cache/host/navigation-error';+import { createRetirableCacheHost } from '@graphql-cache/host/retirable-host';import type { CacheHost } from '@graphql-cache/host/types';import {createTauriCacheHost,@@ -20,6 +22,7 @@ import {entityFromArgument,} from '@graphql-cache/index';import { registerCacheHost } from '@graphql-cache/lifecycle';+import { isOwnerLockUnavailableError } from '@graphql-cache/protocol';import { getBrowserTursoCacheRolloutDecision } from '@graphql-cache/rollout';import { getOrCreateCacheScope } from '@graphql-cache/scope';import { Telemetry } from '@macro-inc/observability';@@ -50,6 +53,7 @@ import {createClient as createGraphqlWsClient,type Client as GraphqlWsClient,} from 'graphql-ws';+import { createSignal } from 'solid-js';import { match } from 'ts-pattern';import { delegateChannelNotificationRefresh } from '../../queries/channel/notification-refresh';import type { SoupApiItem } from './generated/schemas/soupApiItem';@@ -348,6 +352,9 @@ function getUncachedRealtimeClient(): Client {}let cacheInitializationFailed = false;+// Changes when the session abandons its cache so reactive readers of the+// availability getters below (query options, effects) move to network paths.+const [cacheAvailability, setCacheAvailability] = createSignal(0);/*** Whether the normalized cache is active for soup GraphQL queries.@@ -355,6 +362,7 @@ let cacheInitializationFailed = false;* process (graphql_cache_plugin).*/export function graphqlCacheEnabled(): boolean {+ cacheAvailability();if (cacheInitializationFailed) return false;if (!isTauri() && browserCacheClientActivated) return true;return getBrowserTursoCacheRolloutDecision().enabled;@@ -379,10 +387,12 @@ function fallbackAfterInitializationFailure(): void {} catch {// Initialization-failure cleanup cannot alter GraphQL transport fallback.}+ setCacheAvailability((version) => version + 1);}/** Returns the persistent normalized-cache host after client initialization. */export function getGraphqlCacheHost(): CacheHost | undefined {+ cacheAvailability();return cachedCacheHost?.disabled ? undefined : cachedCacheHost;}@@ -416,9 +426,14 @@ export function getGraphqlSoupClient(): Client {operationKind?: Operation['kind']) => {try {- // Navigation deliberately rejects outstanding reads. Do not turn that- // expected shutdown into an error; unexpected disposal still reports.- if (error instanceof CacheNavigationError) return;+ // Navigation deliberately rejects outstanding reads, and a tab on+ // another deployed build can own the database. Neither is an error;+ // unexpected disposal still reports.+ if (+ error instanceof CacheNavigationError ||+ isOwnerLockUnavailableError(error)+ )+ return;// Caught cache failures never reach window.unhandledrejection. Report// them through the Datadog-bound exporter without query/variable data.Telemetry.error(error, {@@ -442,11 +457,23 @@ export function getGraphqlSoupClient(): Client {// Unsubscribing emits urql teardown operations; keep the cache host// available until those best-effort registration removals are issued.subscriptionsLifecycle.dispose();+ // Retires the host: operations still running on this client, and any+ // caller that captured the host, fall back to the network from here.host?.dispose();if (websocketClient) void websocketClient.dispose();};const onInitializationError = (error: Error) => {if (!host || cachedCacheHost !== host) return;+ // Another context holds the database: a tab on another deployed build,+ // or a closing tab whose worker still has the files open. This page+ // simply runs uncached until it reloads.+ if (isOwnerLockUnavailableError(error)) {+ fallbackAfterInitializationFailure();+ console.info(+ `[graphql-cache] ${error.message}; using the network until reload`+ );+ return;+ }reportCacheError(error, 'initialization');fallbackAfterInitializationFailure();toast.failure('Local cache unavailable', {@@ -459,13 +486,17 @@ export function getGraphqlSoupClient(): Client {};try {const scope = getOrCreateCacheScope();- host = native- ? createTauriCacheHost({ scope, onInitializationError })- : createWorkerCacheHost({- scope,- onInitializationError,- rolloutCohort: rollout.cohort,- });+ host = createRetirableCacheHost(+ native+ ? createTauriCacheHost({ scope, onInitializationError })+ : createWorkerCacheHost({+ scope,+ onInitializationError,+ rolloutCohort: rollout.cohort,+ // A newer deploy took the local cache over: move to it.+ onSuperseded: () => reloadForNewerBuild(),+ })+ );const graphqlWsClient = createGraphqlSoupWebSocketClient(subscriptionsLifecycle.connected);@@ -519,6 +550,7 @@ export function getGraphqlSoupClient(): Client {cachedCacheHost = undefined;cachedCacheCleanup = undefined;cacheInitializationFailed = true;+ setCacheAvailability((version) => version + 1);console.warn('graphql cache init failed; using uncached client', error);return isFeatureEnabled(enableGraphqlSoup)? getUncachedRealtimeClient()diff --git a/apps/web/src/vite-env.d.ts b/apps/web/src/vite-env.d.tsindex 9a4afc616c..7d089cb8da 100644--- a/apps/web/src/vite-env.d.ts+++ b/apps/web/src/vite-env.d.ts@@ -1,5 +1,7 @@interface ImportMetaEnv {readonly __APP_VERSION__: string;+ /** When this build was made, in ms since the epoch. Unset outside app builds. */+ readonly __APP_BUILD_TIME__?: number;readonly __LOCAL_JWT__: string;readonly __GIT_BRANCH__: string;diff --git a/apps/web/vite.base.ts b/apps/web/vite.base.tsindex d732e6ddb1..a4e550f571 100644--- a/apps/web/vite.base.ts+++ b/apps/web/vite.base.ts@@ -25,6 +25,8 @@ function readShortSha(): string {const shortSha = readShortSha();const appVersion = `${version}+${shortSha}`;+/** Orders builds: a newer build takes the local cache over from older tabs. */+const appBuildTime = Date.now();function readGitBranch(): string {try {@@ -252,6 +254,7 @@ function defineEnv(mode: string, command: string) {});return {'import.meta.env.__APP_VERSION__': JSON.stringify(appVersion),+ 'import.meta.env.__APP_BUILD_TIME__': JSON.stringify(appBuildTime),'import.meta.env.ASSETS_PATH': JSON.stringify(getAssetsPath(mode, command)),'import.meta.env.__LOCAL_DOCKER__': process.env.LOCAL_DOCKER === 'true','import.meta.env.__LOCAL_JWT__': JSON.stringify(process.env.LOCAL_JWT),diff --git a/crates/client/cache-core/src/codec.rs b/crates/client/cache-core/src/codec.rsindex b1eeb58079..e1c3db1609 100644--- a/crates/client/cache-core/src/codec.rs+++ b/crates/client/cache-core/src/codec.rs@@ -1,10 +1,11 @@//! Binary record codec shared by all persistent Turso backends (browser and//! native hosts). Records are stored as postcard bytes; the logical namespace-//! embeds [`CACHE_FORMAT_VERSION`] and [`CACHE_SCHEMA_COMPATIBILITY_EPOCH`].-//! A healthy, graceful browser reopen with matching versions preserves records,-//! queued mutations, and optimistic layers. A compatibility/format mismatch or-//! abrupt/uncertain browser owner loss physically resets the database and-//! discards all three.+//! embeds [`CACHE_FORMAT_VERSION`] and [`CACHE_SCHEMA_COMPATIBILITY_EPOCH`],+//! and so does the browser's physical database name after+//! [`UNVERSIONED_STORAGE_VERSIONS`]. A healthy, graceful+//! browser reopen with matching versions preserves records, queued mutations,+//! and optimistic layers. A compatibility/format mismatch or abrupt/uncertain+//! browser owner loss physically resets the database and discards all three.use crate::normalize::RecordUpdates;use crate::queue::{PersistedOptimisticLayer, StoredMutation};@@ -86,15 +87,41 @@ pub fn cache_namespace(scope: &str) -> String {format!("graphql-cache:{scope}:s{CACHE_SCHEMA_COMPATIBILITY_EPOCH}:v{CACHE_FORMAT_VERSION}")}-/// Stable physical database name for a cache scope.+/// Storage versions `(schema compatibility epoch, cache format, storage+/// schema)` in use when database names began to embed versions. A build at+/// exactly these versions keeps the unversioned name, so that change moved no+/// database: the next build opens the same file, queued mutations included.+pub const UNVERSIONED_STORAGE_VERSIONS: (u32, u32, u32) = (3, 3, 11);++/// Physical database name for a cache scope at one storage version.///-/// Unlike [`cache_namespace`], this deliberately excludes the schema-/// compatibility epoch and cache format version so the browser can acquire the-/// same main/WAL paths and validate their metadata. Only a healthy, graceful-/// close and compatible reopen preserves those files. A compatibility/format-/// mismatch or abrupt/uncertain owner loss physically resets both files,-/// discarding every row, including queued mutations and optimistic layers.-pub fn cache_database_name(scope: &str) -> String {+/// After [`UNVERSIONED_STORAGE_VERSIONS`], it embeds the schema compatibility+/// epoch, the cache format version, and the backend's+/// `storage_schema_version`, so builds that cannot share records use separate+/// files and owner locks instead of resetting each other's database while tabs+/// of both stay open after a deploy. Builds with matching versions share one+/// database: only a healthy, graceful close and reopen preserves its files.+/// Metadata is still validated on open, and a mismatch or an abrupt/uncertain+/// owner loss physically resets both files, discarding every row, including+/// queued mutations and optimistic layers.+pub fn cache_database_name(scope: &str, storage_schema_version: u32) -> String {+ let unversioned = legacy_cache_database_name(scope);+ if (+ CACHE_SCHEMA_COMPATIBILITY_EPOCH,+ CACHE_FORMAT_VERSION,+ storage_schema_version,+ ) == UNVERSIONED_STORAGE_VERSIONS+ {+ return unversioned;+ }+ format!(+ "{unversioned}:s{CACHE_SCHEMA_COMPATIBILITY_EPOCH}.v{CACHE_FORMAT_VERSION}.t{storage_schema_version}"+ )+}++/// Physical name every storage version shared before names embedded versions,+/// and still the name at [`UNVERSIONED_STORAGE_VERSIONS`].+pub fn legacy_cache_database_name(scope: &str) -> String {format!("graphql-cache:{scope}")}@@ -145,4 +172,21 @@ mod tests {"graphql-cache:client-token-1:s3:v3");}++ #[test]+ fn database_name_separates_storage_versions() {+ // The versions of the last unversioned builds keep their file.+ assert_eq!(+ cache_database_name("client-token-1", 11),+ "graphql-cache:client-token-1"+ );+ assert_eq!(+ cache_database_name("client-token-1", 12),+ "graphql-cache:client-token-1:s3.v3.t12"+ );+ assert_eq!(+ legacy_cache_database_name("client-token-1"),+ "graphql-cache:client-token-1"+ );+ }}diff --git a/crates/client/cache-turso/src/lib.rs b/crates/client/cache-turso/src/lib.rsindex 4bb4b3f4cf..9786d83561 100644--- a/crates/client/cache-turso/src/lib.rs+++ b/crates/client/cache-turso/src/lib.rs@@ -17,13 +17,14 @@ mod storage;pub use error::{PhysicalResetReason, TursoStorageError};pub use storage::{- STORAGE_SCHEMA_VERSION, TursoStorage, TursoStorageCloseOutcome, TursoStorageOpenOutcome,+ QueuedMutationCount, STORAGE_SCHEMA_VERSION, TursoStorage, TursoStorageCloseOutcome,+ TursoStorageOpenOutcome,};#[cfg(target_arch = "wasm32")]pub use storage::{HealthyTursoStorageClosed, ResetRequiredTursoStorageClosed, TursoStorageCloseFailure,- TursoStorageOpenFailure, TursoStorageResetFailure,+ TursoStorageOpenFailure, TursoStorageResetFailure, count_queued_mutations,};#[cfg(not(target_arch = "wasm32"))]pub use storage::{TursoFileDatabase, TursoMemoryDatabase};diff --git a/crates/client/cache-turso/src/storage.rs b/crates/client/cache-turso/src/storage.rsindex c7052841a7..d8bbb2c1c5 100644--- a/crates/client/cache-turso/src/storage.rs+++ b/crates/client/cache-turso/src/storage.rs@@ -331,6 +331,42 @@ impl TursoStorage {}}+/// Mutations still queued in a database of any storage schema version.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum QueuedMutationCount {+ /// The file has a mutation queue holding this many rows.+ Queued(u64),+ /// The file has no mutation queue, so it never held queued work.+ NoQueue,+}++/// Counts queued mutations without validating or changing the metadata.+///+/// Every storage schema version keeps its queue in `mutation_queue` and+/// deletes a row once the mutation settles, so this is safe to call on a+/// database written by an older build before deciding whether to delete it.+#[cfg(target_arch = "wasm32")]+pub fn count_queued_mutations(+ session: &ConnectedOpfsSession,+) -> Result<QueuedMutationCount, TursoStorageError> {+ let connection = session.connection();+ let count = |sql: &str| -> Result<u64, TursoStorageError> {+ let rows = driver::query(&connection, sql, Vec::new())?;+ let [row] = rows.as_slice() else {+ return Err(invariant());+ };+ u64::try_from(required_i64(row, 0)?).map_err(|_| invariant())+ };+ if count("SELECT COUNT(*) FROM sqlite_schema WHERE type = 'table' AND name = 'mutation_queue'")?+ == 0+ {+ return Ok(QueuedMutationCount::NoQueue);+ }+ Ok(QueuedMutationCount::Queued(count(+ "SELECT COUNT(*) FROM mutation_queue",+ )?))+}+/// Result of consuming and gracefully closing a browser Turso storage.#[cfg(target_arch = "wasm32")]#[derive(Debug)]diff --git a/crates/client/cache-wasm/Cargo.toml b/crates/client/cache-wasm/Cargo.tomlindex d3b9f736cd..bb9290d17e 100644--- a/crates/client/cache-wasm/Cargo.toml+++ b/crates/client/cache-wasm/Cargo.toml@@ -2,8 +2,10 @@edition = "2024"name = "cache-wasm"publish = false-# Rebuild for entity-only Soup hydration and bounded page retention.-version = "0.6.30"+# Rebuild for entity-only Soup hydration, bounded page retention,+# storage-version database names, stale-database cleanup, and waiting out+# files a departing worker holds.+version = "0.6.31"[features]default = []diff --git a/crates/client/cache-wasm/src/shell.rs b/crates/client/cache-wasm/src/shell.rsindex ed6abc3bf7..dc026e5d59 100644--- a/crates/client/cache-wasm/src/shell.rs+++ b/crates/client/cache-wasm/src/shell.rs@@ -1,5 +1,8 @@use async_lock::Mutex;-use cache_core::codec::cache_database_name;+use cache_core::codec::{+ CACHE_FORMAT_VERSION, CACHE_SCHEMA_COMPATIBILITY_EPOCH, UNVERSIONED_STORAGE_VERSIONS,+ cache_database_name, legacy_cache_database_name,+};use cache_core::deps::OpId;use cache_core::engine::{BeginOptimisticWrite, CommitOptimisticWriteResult, DeferOptimisticWriteResult, Engine,@@ -18,8 +21,8 @@ use cache_core::search::SearchRequest;use cache_core::store::QueueDiagnosticsAvailability;use cache_core::value::EntityKey;use cache_turso::{- PhysicalResetReason, TursoStorage, TursoStorageCloseOutcome, TursoStorageError,- TursoStorageOpenOutcome,+ PhysicalResetReason, QueuedMutationCount, TursoStorage, TursoStorageCloseOutcome,+ TursoStorageError, TursoStorageOpenOutcome, count_queued_mutations,};use predicate_index::RecordKey;use serde::{Deserialize, Serialize};@@ -33,9 +36,9 @@ use soup_filter_cache_adapter::{use std::cell::RefCell;use std::collections::HashMap;use std::rc::Rc;-use turso_opfs::{OpenResult, OpfsOwner};+use turso_opfs::{OpenResult, OpfsError, OpfsErrorKind, OpfsOwner};use wasm_bindgen::prelude::*;-use wasm_bindgen_futures::future_to_promise;+use wasm_bindgen_futures::{JsFuture, future_to_promise};#[cfg(test)]mod test;@@ -492,7 +495,45 @@ pub struct CacheEngine {}fn database_identity(scope: &str) -> String {- cache_database_name(scope)+ cache_database_name(scope, cache_turso::STORAGE_SCHEMA_VERSION)+}++/// Whether `identity` names one of this scope's databases that this build does+/// not open and that no newer build could want: the unversioned name once this+/// build's versions have moved past it, or another name for versions no newer+/// than this build's. A newer version belongs to a build that may come back,+/// for example after a rollback, so it is never stale.+fn is_stale_identity(scope: &str, identity: &str) -> bool {+ let unversioned = legacy_cache_database_name(scope);+ let version = if identity == unversioned {+ Some(UNVERSIONED_STORAGE_VERSIONS)+ } else {+ storage_version(&unversioned, identity)+ };+ let own = (+ CACHE_SCHEMA_COMPATIBILITY_EPOCH,+ CACHE_FORMAT_VERSION,+ cache_turso::STORAGE_SCHEMA_VERSION,+ );+ identity != database_identity(scope) && version.is_some_and(|version| version <= own)+}++/// The `(epoch, format, storage)` versions embedded in a scope's database name.+fn storage_version(unversioned: &str, identity: &str) -> Option<(u32, u32, u32)> {+ let mut parts = identity+ .strip_prefix(unversioned)?+ .strip_prefix(":s")?+ .split('.');+ let number = |part: &str| -> Option<u32> {+ if part.is_empty() || !part.bytes().all(|byte| byte.is_ascii_digit()) {+ return None;+ }+ part.parse().ok()+ };+ let epoch = number(parts.next()?)?;+ let format = number(parts.next()?.strip_prefix('v')?)?;+ let storage = number(parts.next()?.strip_prefix('t')?)?;+ parts.next().is_none().then_some((epoch, format, storage))}fn build_engine(storage: TursoStorage, hot_capacity: Option<u32>) -> BrowserEngine {@@ -515,7 +556,7 @@ async fn open_owner(owner: OpfsOwner) -> Result<OpenResult, JsValue> {match owner.open().await {Ok(opened) => Ok(opened),Err(failure) => {- let error = err_js(&failure);+ let error = storage_err_js(failure.error());if let Some(owner) = failure.into_owner() {owner.release().await.map_err(err_js)?;}@@ -552,12 +593,21 @@ async fn open_storage(scope: &str, owner: OpfsOwner) -> Result<OpenedStorage, Js.physical_reset_reason().expect("reset-required error has a reset reason"),);- (failure.reset().await.map_err(err_js)?, outcome)+ (+ failure+ .reset()+ .await+ .map_err(|failure| storage_err_js(failure.error()))?,+ outcome,+ )}}}OpenResult::ResetRequired(session) => (- session.reset().await.map_err(err_js)?,+ session+ .reset()+ .await+ .map_err(|failure| storage_err_js(failure.error()))?,CacheOpenOutcome::ResetStorageUncertain,),};@@ -572,7 +622,10 @@ async fn open_storage(scope: &str, owner: OpfsOwner) -> Result<OpenedStorage, Js}),Err(failure) => {let reset_required = failure.error().requires_physical_reset();- let owner = failure.reset().await.map_err(err_js)?;+ let owner = failure+ .reset()+ .await+ .map_err(|failure| storage_err_js(failure.error()))?;owner.release().await.map_err(err_js)?;Err(if reset_required {reset_required_js_error()@@ -583,19 +636,60 @@ async fn open_storage(scope: &str, owner: OpfsOwner) -> Result<OpenedStorage, Js}}OpenResult::ResetRequired(session) => {- let owner = session.reset().await.map_err(err_js)?;+ let owner = session+ .reset()+ .await+ .map_err(|failure| storage_err_js(failure.error()))?;owner.release().await.map_err(err_js)?;Err(reset_required_js_error())}}}-async fn acquire_storage(scope: &str, recovery_wipe: bool) -> Result<OpenedStorage, JsValue> {- let owner = OpfsOwner::acquire(&database_identity(scope))- .await- .map_err(err_js)?;+/// Waits until the caller may touch storage while this owner holds the lock.+/// Holding the lock only proves no other engine uses the files; the browser+/// coordinator must also record that this engine's epoch may change them.+async fn await_storage_grant(+ owner: OpfsOwner,+ grant: &js_sys::Function,+) -> Result<OpfsOwner, JsValue> {+ let granted = match grant.call0(&JsValue::UNDEFINED) {+ Ok(value) => JsFuture::from(js_sys::Promise::resolve(&value)).await,+ Err(error) => Err(error),+ };+ match granted {+ Ok(_) => Ok(owner),+ Err(error) => {+ owner.release().await.map_err(err_js)?;+ Err(error)+ }+ }+}++async fn acquire_storage(+ scope: &str,+ recovery_wipe: bool,+ storage_grant: Option<&js_sys::Function>,+ if_available: bool,+) -> Result<OpenedStorage, JsValue> {+ let identity = database_identity(scope);+ let owner = if if_available {+ OpfsOwner::try_acquire(&identity)+ .await+ .map_err(err_js)?+ .ok_or_else(owner_lock_unavailable_js_error)?+ } else {+ OpfsOwner::acquire(&identity).await.map_err(err_js)?+ };+ let owner = match storage_grant {+ Some(grant) => await_storage_grant(owner, grant).await?,+ None => owner,+ };let owner = if recovery_wipe {- owner.recovery_wipe().await.map_err(err_js)?+ owner+ .recovery_wipe()+ .await+ .map_err(|failure| storage_err_js(failure.error()))?} else {owner};@@ -610,9 +704,12 @@ async fn open_cache_inner(scope: String,hot_capacity: Option<u32>,recovery_wipe: bool,+ storage_grant: Option<js_sys::Function>,+ if_available: bool,) -> Result<(CacheEngine, CacheOpenOutcome), JsValue> {validate_hot_capacity(hot_capacity)?;- let OpenedStorage { storage, outcome } = acquire_storage(&scope, recovery_wipe).await?;+ let OpenedStorage { storage, outcome } =+ acquire_storage(&scope, recovery_wipe, storage_grant.as_ref(), if_available).await?;Ok((CacheEngine {state: Rc::new(Mutex::new(CacheState {@@ -648,18 +745,40 @@ fn cache_open_result(engine: CacheEngine, outcome: CacheOpenOutcome) -> Result<J/// incomplete or incompatible files are reset and reopened before returning.#[wasm_bindgen(js_name = openCache)]pub async fn open_cache(scope: String, hot_capacity: Option<u32>) -> Result<CacheEngine, JsValue> {- open_cache_inner(scope, hot_capacity, false)+ open_cache_inner(scope, hot_capacity, false, None, false).await.map(|(engine, _)| engine)}/// Additive open API returning the engine and payload-free recovery outcome.+///+/// When given, `on_owner_lock_acquired` is called once the owner lock is held+/// and before any OPFS access; opening continues after its promise resolves.+/// A rejection releases the lock without touching storage.+///+/// With `if_available`, opening never waits behind another holder of the+/// owner lock: it rejects at once, touching nothing, with an error whose+/// `cacheOwnerLockUnavailable` property is `true`.+///+/// When another context keeps the database files open through the adapter's+/// bounded wait, opening rejects with an error whose `cacheStorageBusy`+/// property is `true`. A plain open has then changed neither file; a recovery+/// wipe may have removed one of them.#[wasm_bindgen(js_name = openCacheWithOutcome)]pub async fn open_cache_with_outcome(scope: String,hot_capacity: Option<u32>,+ on_owner_lock_acquired: Option<js_sys::Function>,+ if_available: Option<bool>,) -> Result<JsValue, JsValue> {- let (engine, outcome) = open_cache_inner(scope, hot_capacity, false).await?;+ let (engine, outcome) = open_cache_inner(+ scope,+ hot_capacity,+ false,+ on_owner_lock_acquired,+ if_available.unwrap_or(false),+ )+ .await?;cache_open_result(engine, outcome)}@@ -670,21 +789,105 @@ pub async fn open_cache_for_recovery(scope: String,hot_capacity: Option<u32>,) -> Result<CacheEngine, JsValue> {- open_cache_inner(scope, hot_capacity, true)+ open_cache_inner(scope, hot_capacity, true, None, false).await.map(|(engine, _)| engine)}/// Additive recovery-open API returning the engine and coarse wipe outcome.+/// `on_owner_lock_acquired` and `if_available` behave as in+/// [`openCacheWithOutcome`](open_cache_with_outcome); both precede the wipe.#[wasm_bindgen(js_name = openCacheForRecoveryWithOutcome)]pub async fn open_cache_for_recovery_with_outcome(scope: String,hot_capacity: Option<u32>,+ on_owner_lock_acquired: Option<js_sys::Function>,+ if_available: Option<bool>,) -> Result<JsValue, JsValue> {- let (engine, outcome) = open_cache_inner(scope, hot_capacity, true).await?;+ let (engine, outcome) = open_cache_inner(+ scope,+ hot_capacity,+ true,+ on_owner_lock_acquired,+ if_available.unwrap_or(false),+ )+ .await?;cache_open_result(engine, outcome)}+/// Physical database identity this build opens for `scope`; it also names the+/// OPFS files and derives the owner Web Lock name.+#[wasm_bindgen(js_name = cacheDatabaseIdentity)]+pub fn cache_database_identity(scope: String) -> String {+ database_identity(&scope)+}++#[derive(Serialize)]+#[serde(rename_all = "camelCase")]+struct StaleDatabaseCleanup {+ outcome: &'static str,+ #[serde(skip_serializing_if = "Option::is_none")]+ queued_mutations: Option<u64>,+}++/// Deletes one of this scope's databases that an older build left behind (see+/// [`cacheDatabaseIdentity`](cache_database_identity)), unless another context+/// holds its owner lock (`in-use`) or it still queues mutations+/// (`queued-mutations`). Never waits for the lock. A file Turso cannot read,+/// or whose queue cannot be counted, is kept and rejects; the caller should+/// use a disposable worker, because such a failure can poison the+/// worker-local OPFS registry.+#[wasm_bindgen(js_name = removeStaleCacheDatabase)]+pub async fn remove_stale_cache_database(+ scope: String,+ identity: String,+) -> Result<JsValue, JsValue> {+ if !is_stale_identity(&scope, &identity) {+ return Err(err_js("not a stale cache database for this scope"));+ }+ let Some(owner) = OpfsOwner::try_acquire(&identity).await.map_err(err_js)? else {+ return to_js(&StaleDatabaseCleanup {+ outcome: "in-use",+ queued_mutations: None,+ });+ };+ let owner = match open_owner(owner).await? {+ OpenResult::Ready(session) => {+ let connected = session.connect().map_err(err_js)?;+ let counted = count_queued_mutations(&connected);+ let owner = connected+ .try_close()+ .map_err(err_js)?+ .preserve()+ .map_err(err_js)?;+ match counted {+ Ok(QueuedMutationCount::Queued(queued)) if queued > 0 => {+ owner.release().await.map_err(err_js)?;+ return to_js(&StaleDatabaseCleanup {+ outcome: "queued-mutations",+ queued_mutations: Some(queued),+ });+ }+ // A queue that cannot be counted may still hold work, so the+ // file is kept and the cleanup rejects as for an unreadable file.+ Err(error) => {+ owner.release().await.map_err(err_js)?;+ return Err(err_js(error));+ }+ // An empty queue or no queue holds no work any build could send.+ Ok(_) => owner,+ }+ }+ // Only one of the pair exists, so it never held a complete database.+ OpenResult::ResetRequired(session) => session.reset().await.map_err(err_js)?,+ };+ owner.remove_and_release().await.map_err(err_js)?;+ to_js(&StaleDatabaseCleanup {+ outcome: "removed",+ queued_mutations: None,+ })+}+/// Recovery-wipes and recreates the cache database for `scope` while holding/// the same exclusive OPFS owner lock used by [`openCache`](open_cache).#[wasm_bindgen(js_name = destroyCache)]@@ -807,6 +1010,9 @@ fn to_js<T: Serialize>(value: &T) -> Result<JsValue, JsValue> {const RESET_REQUIRED_MARKER: &str = "cacheStorageResetRequired";const RESET_REQUIRED_MESSAGE: &str = "cache storage reset required";+const OWNER_LOCK_UNAVAILABLE_MARKER: &str = "cacheOwnerLockUnavailable";+const STORAGE_BUSY_MARKER: &str = "cacheStorageBusy";+const OWNER_LOCK_UNAVAILABLE_MESSAGE: &str = "cache database owner lock is held by another context";/// All rejections surface as real `Error` objects with consistent/// `instanceof Error` and `.message` behavior.@@ -829,6 +1035,32 @@ fn reset_required_js_error() -> JsValue {error.into()}+/// Converts an OPFS adapter error, marking a path another context kept open+/// for the whole busy wait with a `cacheStorageBusy` property of `true`.+fn storage_err_js(error: &OpfsError) -> JsValue {+ let value = err_js(error);+ if error.kind() == OpfsErrorKind::Busy {+ js_sys::Reflect::set(+ &value,+ &JsValue::from_str(STORAGE_BUSY_MARKER),+ &JsValue::TRUE,+ )+ .expect("new JavaScript Error accepts the busy marker");+ }+ value+}++fn owner_lock_unavailable_js_error() -> JsValue {+ let error = js_sys::Error::new(OWNER_LOCK_UNAVAILABLE_MESSAGE);+ js_sys::Reflect::set(+ error.as_ref(),+ &JsValue::from_str(OWNER_LOCK_UNAVAILABLE_MARKER),+ &JsValue::TRUE,+ )+ .expect("new JavaScript Error accepts the unavailable marker");+ error.into()+}+fn engine_error_requires_reset(error: &EngineError<TursoStorageError>) -> bool {matches!(error,diff --git a/crates/client/cache-wasm/src/shell/test.rs b/crates/client/cache-wasm/src/shell/test.rsindex 26ea8cf31f..78eccb5f98 100644--- a/crates/client/cache-wasm/src/shell/test.rs+++ b/crates/client/cache-wasm/src/shell/test.rs@@ -2196,3 +2196,342 @@ async fn calls_serialize_and_owner_lock_excludes_a_second_open() {.expect("close releases owner lock");close_and_destroy(&reopened, SCOPE).await;}++async fn run_js(body: &str, argument: &str) -> JsValue {+ let script = js_sys::Function::new_with_args("name", body);+ let promise = script+ .call1(&JsValue::UNDEFINED, &JsValue::from_str(argument))+ .expect("test script runs")+ .dyn_into::<js_sys::Promise>()+ .expect("test script returns a promise");+ JsFuture::from(promise).await.expect("test script settles")+}++async fn opfs_file_exists(name: &str) -> bool {+ run_js(+ "return (async () => {+ const root = await navigator.storage.getDirectory();+ try { await root.getFileHandle(name); return true; }+ catch (error) { if (error.name === 'NotFoundError') return false; throw error; }+ })()",+ name,+ )+ .await+ .as_bool()+ .expect("existence is boolean")+}++async fn remove_opfs_file(name: &str) {+ run_js(+ "return (async () => {+ const root = await navigator.storage.getDirectory();+ await root.removeEntry(name).catch(() => undefined);+ })()",+ name,+ )+ .await;+}++async fn owner_lock_available(scope: &str) -> bool {+ let identity = database_identity(scope);+ let lock = format!("macro:turso-opfs:v1:{}:{identity}", identity.len());+ run_js(+ "return navigator.locks.request(+ name, { mode: 'exclusive', ifAvailable: true }, (lock) => lock !== null+ )",+ &lock,+ )+ .await+ .as_bool()+ .expect("lock probe is boolean")+}++async fn close_opened(result: &JsValue) {+ let engine = js_sys::Reflect::get(result, &JsValue::from_str("engine")).expect("engine");+ let close = js_sys::Reflect::get(&engine, &JsValue::from_str("close"))+ .expect("close")+ .dyn_into::<js_sys::Function>()+ .expect("close is callable");+ let closing = close+ .call0(&engine)+ .expect("close starts")+ .dyn_into::<js_sys::Promise>()+ .expect("close returns a promise");+ JsFuture::from(closing).await.expect("engine closes");+}++#[wasm_bindgen_test(async)]+async fn outcome_opens_touch_opfs_only_after_their_storage_grant() {+ const SCOPE: &str = "cache-wasm-storage-grant";+ let database = database_identity(SCOPE);+ let wal = format!("{database}-wal");+ for recovery in [false, true] {+ remove_opfs_file(&database).await;+ remove_opfs_file(&wal).await;++ // Refusing the grant releases the owner lock without touching OPFS.+ let refused = js_sys::Function::new_no_args(+ "return Promise.reject(new Error('storage grant refused'))",+ );+ let error = if recovery {+ open_cache_for_recovery_with_outcome(SCOPE.into(), None, Some(refused), None).await+ } else {+ open_cache_with_outcome(SCOPE.into(), None, Some(refused), None).await+ }+ .expect_err("a refused grant fails the open");+ assert_eq!(+ error+ .dyn_into::<js_sys::Error>()+ .expect("error object")+ .message(),+ "storage grant refused"+ );+ assert!(owner_lock_available(SCOPE).await);+ assert!(!opfs_file_exists(&database).await);++ // A granted open finds storage untouched when it asks.+ let observed = Rc::new(RefCell::new(None));+ let observer = observed.clone();+ let file = database.clone();+ let grant = Closure::wrap(Box::new(move || -> js_sys::Promise {+ let observer = observer.clone();+ let file = file.clone();+ future_to_promise(async move {+ *observer.borrow_mut() = Some(opfs_file_exists(&file).await);+ Ok(JsValue::UNDEFINED)+ })+ }) as Box<dyn FnMut() -> js_sys::Promise>)+ .into_js_value()+ .unchecked_into::<js_sys::Function>();+ let opened = if recovery {+ open_cache_for_recovery_with_outcome(SCOPE.into(), None, Some(grant), None).await+ } else {+ open_cache_with_outcome(SCOPE.into(), None, Some(grant), None).await+ }+ .expect("a granted open succeeds");+ assert_eq!(*observed.borrow(), Some(false));+ assert!(opfs_file_exists(&database).await);+ close_opened(&opened).await;+ }+ destroy_cache(SCOPE.into()).await.expect("destroy cache");+}++async fn hold_web_lock(lock: &str) {+ run_js(+ "return new Promise((granted) => {+ const request = navigator.locks.request(name, { mode: 'exclusive' }, () =>+ new Promise((release) => {+ globalThis.cacheWasmHeldLock = { release, request };+ granted(true);+ })+ );+ })",+ lock,+ )+ .await;+}++async fn release_held_web_lock() {+ run_js(+ "return (async () => {+ const held = globalThis.cacheWasmHeldLock;+ delete globalThis.cacheWasmHeldLock;+ held.release();+ await held.request;+ })()",+ "",+ )+ .await;+}++fn owner_lock_for(identity: &str) -> String {+ format!("macro:turso-opfs:v1:{}:{identity}", identity.len())+}++async fn remove_opfs_pair(identity: &str) {+ remove_opfs_file(identity).await;+ remove_opfs_file(&format!("{identity}-wal")).await;+}++async fn create_opfs_file(name: &str) {+ run_js(+ "return (async () => {+ const root = await navigator.storage.getDirectory();+ await root.getFileHandle(name, { create: true });+ })()",+ name,+ )+ .await;+}++/// Renames a closed main/WAL pair, as if another storage version wrote it.+async fn move_opfs_pair(from: &str, to: &str) {+ run_js(+ "return (async () => {+ const [from, to] = JSON.parse(name);+ const root = await navigator.storage.getDirectory();+ for (const suffix of ['', '-wal']) {+ await (await root.getFileHandle(from + suffix)).move(to + suffix);+ }+ })()",+ &serde_json::json!([from, to]).to_string(),+ )+ .await;+}++async fn queue_one_mutation(engine: &CacheEngine) {+ resolved(engine.enqueue_optimistic_mutation(+ None,+ "00000000-0000-4000-8000-000000000011".into(),+ PROPERTY_MUTATION.into(),+ Some("SetEntityProperty".into()),+ js(mutation_variables()),+ js(serde_json::json!({+ "setEntityProperty": { "id": "prop-1", "displayName": "Queued" }+ })),+ JsValue::UNDEFINED,+ JsValue::UNDEFINED,+ 1.0,+ "stale-owner".into(),+ 0.0,+ 100.0,+ ))+ .await;+}++async fn stale_cleanup(scope: &str, identity: &str) -> serde_json::Value {+ from_js(+ remove_stale_cache_database(scope.into(), identity.into())+ .await+ .expect("stale cleanup settles"),+ )+}++#[wasm_bindgen_test]+fn stale_identities_are_only_this_scopes_other_databases() {+ let scope = "scope-a";+ let own = cache_database_identity(scope.into());+ assert_eq!(+ own,+ cache_database_name(scope, cache_turso::STORAGE_SCHEMA_VERSION)+ );+ assert!(!is_stale_identity(scope, &own));+ // The unversioned name is stale only once this build's versions move on.+ assert_eq!(+ is_stale_identity(scope, "graphql-cache:scope-a"),+ own != "graphql-cache:scope-a"+ );+ assert!(is_stale_identity(scope, "graphql-cache:scope-a:s1.v2.t3"));+ let versioned = |storage: u32| {+ format!(+ "graphql-cache:scope-a:s{}.v{}.t{storage}",+ cache_core::codec::CACHE_SCHEMA_COMPATIBILITY_EPOCH,+ cache_core::codec::CACHE_FORMAT_VERSION,+ )+ };+ let current = versioned(cache_turso::STORAGE_SCHEMA_VERSION);+ // Another spelling of this build's own versions is never opened again.+ assert_eq!(is_stale_identity(scope, ¤t), own != current);+ let newer = versioned(cache_turso::STORAGE_SCHEMA_VERSION + 1);+ for other in [+ // A newer build may come back after a rollback, so it keeps its file.+ newer.as_str(),+ "graphql-cache:scope-a:s99.v0.t0",+ "graphql-cache:scope-b:s1.v2.t3",+ "graphql-cache:scope-ab",+ "graphql-cache:scope-a-wal",+ "graphql-cache:scope-a:s1.v2",+ "graphql-cache:scope-a:s1.v2.t3.x",+ "graphql-cache:scope-a:sx.v2.t3",+ "graphql-cache:scope-a:s1.v.t3",+ ] {+ assert!(!is_stale_identity(scope, other), "{other}");+ }+}++#[wasm_bindgen_test(async)]+async fn outcome_opens_can_decline_to_wait_for_a_held_owner_lock() {+ const SCOPE: &str = "cache-wasm-if-available";+ let database = database_identity(SCOPE);+ remove_opfs_pair(&database).await;+ hold_web_lock(&owner_lock_for(&database)).await;+ for recovery in [false, true] {+ let error = if recovery {+ open_cache_for_recovery_with_outcome(SCOPE.into(), None, None, Some(true)).await+ } else {+ open_cache_with_outcome(SCOPE.into(), None, None, Some(true)).await+ }+ .expect_err("a held owner lock declines the open");+ assert_eq!(+ js_sys::Reflect::get(&error, &JsValue::from_str("cacheOwnerLockUnavailable"))+ .expect("marker lookup"),+ JsValue::TRUE+ );+ assert!(!opfs_file_exists(&database).await);+ }+ release_held_web_lock().await;+ let opened = open_cache_with_outcome(SCOPE.into(), None, None, Some(true))+ .await+ .expect("a free owner lock opens");+ close_opened(&opened).await;+ destroy_cache(SCOPE.into()).await.expect("destroy cache");+}++#[wasm_bindgen_test(async)]+async fn stale_databases_are_removed_only_when_unused_and_empty() {+ const SCOPE: &str = "cache-wasm-stale-cleanup";+ let own = database_identity(SCOPE);+ let unversioned = legacy_cache_database_name(SCOPE);+ let queued = format!("{unversioned}:s0.v0.t0");+ let older = format!("{unversioned}:s0.v0.t1");+ let partial = format!("{unversioned}:s0.v0.t2");+ for identity in [&own, &unversioned, &queued, &older, &partial] {+ remove_opfs_pair(identity).await;+ }++ // An older build's database still queues a mutation, so it is kept.+ let engine = fresh_engine(SCOPE).await;+ queue_one_mutation(&engine).await;+ resolved(engine.close()).await;+ move_opfs_pair(&own, &queued).await;+ assert_eq!(+ stale_cleanup(SCOPE, &queued).await,+ serde_json::json!({ "outcome": "queued-mutations", "queuedMutations": 1 })+ );+ assert!(opfs_file_exists(&queued).await);++ // A live engine of that build holds its lock, so nothing is opened.+ hold_web_lock(&owner_lock_for(&queued)).await;+ assert_eq!(+ stale_cleanup(SCOPE, &queued).await,+ serde_json::json!({ "outcome": "in-use" })+ );+ release_held_web_lock().await;++ // An empty older database and a one-sided pair are deleted outright.+ let engine = fresh_engine(SCOPE).await;+ resolved(engine.close()).await;+ move_opfs_pair(&own, &older).await;+ assert_eq!(+ stale_cleanup(SCOPE, &older).await,+ serde_json::json!({ "outcome": "removed" })+ );+ create_opfs_file(&format!("{partial}-wal")).await;+ assert_eq!(+ stale_cleanup(SCOPE, &partial).await,+ serde_json::json!({ "outcome": "removed" })+ );+ for identity in [&older, &partial] {+ assert!(!opfs_file_exists(identity).await);+ assert!(!opfs_file_exists(&format!("{identity}-wal")).await);+ }++ // This build's own database and other scopes' databases are refused.+ for identity in [own.clone(), legacy_cache_database_name("another-scope")] {+ remove_stale_cache_database(SCOPE.into(), identity)+ .await+ .expect_err("not a stale database of this scope");+ }+ remove_opfs_pair(&queued).await;+ remove_opfs_pair(&own).await;+}diff --git a/crates/client/turso-opfs/src/browser.rs b/crates/client/turso-opfs/src/browser.rsindex 44e0dbb5bc..0eb1f2ee0b 100644--- a/crates/client/turso-opfs/src/browser.rs+++ b/crates/client/turso-opfs/src/browser.rs@@ -24,7 +24,7 @@ use wasm_bindgen_futures::JsFuture;use web_sys::{AbortController, DedicatedWorkerGlobalScope, DomException, FileSystemDirectoryHandle,FileSystemFileHandle, FileSystemGetFileOptions, FileSystemReadWriteOptions,- FileSystemRemoveOptions, FileSystemSyncAccessHandle,+ FileSystemRemoveOptions, FileSystemSyncAccessHandle, WorkerGlobalScope,};#[cfg(test)]@@ -79,6 +79,13 @@ pub enum OpfsErrorKind {Poisoned,/// Turso requested a synchronous operation that this adapter forbids.Unsupported,+ /// Another context, usually a predecessor worker still being torn down,+ /// kept a database path open for the whole busy-entry wait. A busy open+ /// changed neither file and hands its owner back. A busy wipe or reset may+ /// already have removed the other path and poisons the owner like any+ /// partial wipe, so the worker must be replaced and the next owner wipes+ /// again; a busy [`OpfsOwner::remove_and_release`] still releases the lock.+ Busy,}/// A payload-free OPFS adapter error.@@ -148,7 +155,11 @@ impl OpfsOwner {pub async fn acquire(database_identity: &str) -> Result<Self, OpfsError> {let paths = approved_paths(database_identity)?;let lock_name = owner_lock_name(database_identity);- let id = acquire_database_lock(database_identity, &lock_name).await?;+ let id = acquire_database_lock(database_identity, &lock_name, false)+ .await?+ .ok_or_else(|| {+ OpfsError::new(OpfsErrorKind::Lock, "exclusive Web Lock was not granted")+ })?;Ok(Self {id,paths,@@ -158,6 +169,24 @@ impl OpfsOwner {})}+ /// Acquires this identity's exclusive Web Lock only if it is free now,+ /// never queueing behind another holder or waiter.+ ///+ /// Returns `None`, having touched no storage, when the lock is taken.+ pub async fn try_acquire(database_identity: &str) -> Result<Option<Self>, OpfsError> {+ let paths = approved_paths(database_identity)?;+ let lock_name = owner_lock_name(database_identity);+ Ok(acquire_database_lock(database_identity, &lock_name, true)+ .await?+ .map(|id| Self {+ id,+ paths,+ lock_name,+ fresh: false,+ armed: true,+ }))+ }+/// Returns the canonical database identity bound into this owner.pub fn database_path(&self) -> &str {self.paths.get(FileRole::Main)@@ -207,6 +236,7 @@ impl OpfsOwner {/// A one-sided pre-existing pair returns [`OpenResult::ResetRequired`], a/// type that exposes no Turso I/O and can only be physically reset.pub async fn open(mut self) -> Result<OpenResult, OpenFailure> {+ let deadline = BusyDeadline::start();let paths = self.paths.clone();let session = match REGISTRY.with(|registry| {registry@@ -236,15 +266,16 @@ impl OpfsOwner {return Err(open_failure_after_cleanup(error, self));}- let (handle, was_created) = match open_sync_handle(&root, paths.get(role)).await {- Ok(value) => value,- Err(error) => {- let adapter_error =- js_error(OpfsErrorKind::Open, "OPFS sync handle open", &error);- cleanup_failed_open(self.id, session, &adapter_error);- return Err(open_failure_after_cleanup(adapter_error, self));- }- };+ let (handle, was_created) =+ match open_sync_handle(&root, paths.get(role), deadline).await {+ Ok(value) => value,+ Err(error) => {+ let adapter_error =+ entry_error(OpfsErrorKind::Open, "OPFS sync handle open", &error);+ cleanup_failed_open(self.id, session, &adapter_error);+ return Err(open_failure_after_cleanup(adapter_error, self));+ }+ };created[role.index()] = was_created;let handle_id = match REGISTRY.with(|registry| {@@ -332,6 +363,29 @@ impl OpfsOwner {guard.disarm();Ok(())}++ /// Deletes both approved paths without recreating them, then releases the+ /// lock. This discards a database that no context uses any more.+ ///+ /// Unlike [`Self::recovery_wipe`], a failed removal does not poison the+ /// registry: the lock is still released, the error is returned, and any+ /// path left behind can be removed by a later attempt.+ pub async fn remove_and_release(self) -> Result<(), OpfsError> {+ REGISTRY+ .with(|registry| {+ registry+ .borrow_mut()+ .machine+ .start_wipe(self.id, self.paths.clone())+ })+ .map_err(state_error)?;+ let removed = remove_paths(&self.paths).await;+ REGISTRY+ .with(|registry| registry.borrow_mut().machine.finish_wipe(self.id))+ .map_err(state_error)?;+ self.release().await?;+ removed+ }}impl Drop for OpfsOwner {@@ -1261,10 +1315,13 @@ impl Drop for PendingAcquireGuard {}}+/// Requests the exclusive lock, queueing unless `if_available` is set. With+/// `if_available`, returns `None` without an owner when the lock is taken.async fn acquire_database_lock(database_identity: &str,lock_name: &str,-) -> Result<OwnerId, OpfsError> {+ if_available: bool,+) -> Result<Option<OwnerId>, OpfsError> {let worker = js_sys::global().dyn_into::<DedicatedWorkerGlobalScope>().map_err(|error| js_error(OpfsErrorKind::Lock, "DedicatedWorker lock scope", &error))?;@@ -1299,12 +1356,23 @@ async fn acquire_database_lock(&JsValue::from_str("exclusive"),).map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock mode", &error))?;- Reflect::set(- options.as_ref(),- &JsValue::from_str("signal"),- controller.signal().as_ref(),- )- .map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock abort signal", &error))?;+ // The Web Locks API rejects an abort signal combined with ifAvailable. An+ // ifAvailable request settles at once, so it needs no cancellation.+ if if_available {+ Reflect::set(+ options.as_ref(),+ &JsValue::from_str("ifAvailable"),+ &JsValue::TRUE,+ )+ .map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock ifAvailable", &error))?;+ } else {+ Reflect::set(+ options.as_ref(),+ &JsValue::from_str("signal"),+ controller.signal().as_ref(),+ )+ .map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock abort signal", &error))?;+ }let mut acquired_resolve = None;let mut acquired_reject = None;@@ -1322,6 +1390,11 @@ async fn acquire_database_lock(lock_name: lock_name.to_owned(),};let grant = Closure::wrap(Box::new(move |lock: JsValue| -> Promise {+ if (lock.is_null() || lock.is_undefined()) && if_available {+ // Another context holds or awaits the lock; nothing was claimed.+ let _ = callback_resolve.call1(&JsValue::UNDEFINED, &JsValue::FALSE);+ return Promise::resolve(&JsValue::UNDEFINED);+ }if lock.is_null() || lock.is_undefined() {let _ = callback_reject.call1(&JsValue::UNDEFINED,@@ -1495,9 +1568,14 @@ async fn acquire_database_lock());}- JsFuture::from(acquired)+ let granted = JsFuture::from(acquired).await.map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock acquisition", &error))?;+ if granted.as_bool() == Some(false) {+ REGISTRY.with(|registry| registry.borrow_mut().pending_lock = None);+ guard.disarm();+ return Ok(None);+ }let owner = REGISTRY.with(|registry| {let mut registry = registry.borrow_mut();@@ -1530,7 +1608,7 @@ async fn acquire_database_lock(Ok(registered.owner)})?;guard.disarm();- Ok(owner)+ Ok(Some(owner))}fn cancel_database_lock_acquire(lock_name: &str) {@@ -1648,9 +1726,86 @@ async fn worker_root() -> Result<FileSystemDirectoryHandle, JsValue> {.dyn_into::<FileSystemDirectoryHandle>()}+/// How long one open or removal waits for a predecessor to release the files.+///+/// Chromium can grant the canonical owner Web Lock to a successor before it+/// releases the file lock of a predecessor that is still being torn down, such+/// as a page's cache worker terminated during `pagehide`. Production logs show+/// that gap outlasting two seconds. Every holder of a sync handle on these+/// paths first holds that Web Lock, so while this worker holds it,+/// `NoModificationAllowedError` from an entry operation is a draining+/// predecessor, not evidence about the files. The wait stays well inside the+/// coordinator's 20 second database-open watchdog.+#[cfg(not(test))]+const BUSY_ENTRY_WAIT_MS: f64 = 10_000.0;+#[cfg(test)]+const BUSY_ENTRY_WAIT_MS: f64 = 400.0;+const BUSY_ENTRY_FIRST_DELAY_MS: i32 = 2;+const BUSY_ENTRY_MAX_DELAY_MS: i32 = 512;++/// The end of one operation's busy-entry wait, shared by both paths.+#[derive(Clone, Copy)]+struct BusyDeadline(f64);++impl BusyDeadline {+ fn start() -> Self {+ Self(performance_now() + BUSY_ENTRY_WAIT_MS)+ }+}++/// Runs an idempotent OPFS entry operation, retrying only a busy entry until+/// `deadline`.+async fn retry_busy_entry(+ deadline: BusyDeadline,+ mut operation: impl FnMut() -> Promise,+) -> Result<JsValue, JsValue> {+ let mut delay = BUSY_ENTRY_FIRST_DELAY_MS;+ loop {+ match JsFuture::from(operation()).await {+ Err(error) if is_entry_busy(&error) => {+ let remaining = deadline.0 - performance_now();+ if remaining <= 0.0 {+ return Err(error);+ }+ sleep_ms(delay.min(remaining.ceil() as i32)).await?;+ delay = (delay * 2).min(BUSY_ENTRY_MAX_DELAY_MS);+ }+ result => return result,+ }+ }+}++fn is_entry_busy(value: &JsValue) -> bool {+ dom_exception_name(value).as_deref() == Some("NoModificationAllowedError")+}++/// Classifies an entry that stayed busy until its deadline as+/// [`OpfsErrorKind::Busy`], keeping the usual message.+fn entry_error(kind: OpfsErrorKind, operation: &'static str, value: &JsValue) -> OpfsError {+ if is_entry_busy(value) {+ OpfsError::new(+ OpfsErrorKind::Busy,+ format!("{operation} failed (NoModificationAllowedError)"),+ )+ } else {+ js_error(kind, operation, value)+ }+}++async fn sleep_ms(delay: i32) -> Result<(), JsValue> {+ let worker = js_sys::global().dyn_into::<WorkerGlobalScope>()?;+ let mut scheduled = Ok(0);+ let timer = Promise::new(&mut |resolve, _reject| {+ scheduled = worker.set_timeout_with_callback_and_timeout_and_arguments_0(&resolve, delay);+ });+ scheduled?;+ JsFuture::from(timer).await.map(|_| ())+}+async fn open_sync_handle(root: &FileSystemDirectoryHandle,path: &str,+ deadline: BusyDeadline,) -> Result<(FileSystemSyncAccessHandle, bool), JsValue> {let (file, created) = match JsFuture::from(root.get_file_handle(path)).await {Ok(file) => (file.dyn_into::<FileSystemFileHandle>()?, false),@@ -1666,7 +1821,7 @@ async fn open_sync_handle(}Err(error) => return Err(error),};- let handle = JsFuture::from(file.create_sync_access_handle())+ let handle = retry_busy_entry(deadline, || file.create_sync_access_handle()).await?.dyn_into::<FileSystemSyncAccessHandle>()?;Ok((handle, created))@@ -1844,11 +1999,12 @@ fn close_sync_handle(handle: &FileSystemSyncAccessHandle) -> Result<(), JsValue>}async fn reset_paths(paths: &Paths, recursive: bool) -> Result<(), OpfsError> {+ let deadline = BusyDeadline::start();let root = worker_root().await.map_err(|error| js_error(OpfsErrorKind::Remove, "OPFS reset root open", &error))?;for role in FileRole::ALL {- remove_if_present(&root, paths.get(role), recursive).await?;+ remove_if_present(&root, paths.get(role), recursive, deadline).await?;#[cfg(test)]if role == FileRole::Main && take_reset_failure(ResetFault::AfterMainRemoval) {return Err(OpfsError::new(@@ -1878,22 +2034,37 @@ async fn reset_paths(paths: &Paths, recursive: bool) -> Result<(), OpfsError> {Ok(())}+/// Removes both approved paths, attempting each even if the other fails.+async fn remove_paths(paths: &Paths) -> Result<(), OpfsError> {+ let deadline = BusyDeadline::start();+ let root = worker_root()+ .await+ .map_err(|error| js_error(OpfsErrorKind::Remove, "OPFS remove root open", &error))?;+ let mut first_error = None;+ for role in FileRole::ALL {+ if let Err(error) = remove_if_present(&root, paths.get(role), true, deadline).await {+ first_error.get_or_insert(error);+ }+ }+ first_error.map_or(Ok(()), Err)+}+async fn remove_if_present(root: &FileSystemDirectoryHandle,path: &str,recursive: bool,+ deadline: BusyDeadline,) -> Result<(), OpfsError> {- let remove = if recursive {- let options = FileSystemRemoveOptions::new();- options.set_recursive(true);- root.remove_entry_with_options(path, &options)- } else {- root.remove_entry(path)- };- match JsFuture::from(remove).await {+ let options = FileSystemRemoveOptions::new();+ options.set_recursive(recursive);+ match retry_busy_entry(deadline, || root.remove_entry_with_options(path, &options)).await {Ok(_) => Ok(()),Err(error) if is_not_found(&error) => Ok(()),- Err(error) => Err(js_error(OpfsErrorKind::Remove, "OPFS path remove", &error)),+ Err(error) => Err(entry_error(+ OpfsErrorKind::Remove,+ "OPFS path remove",+ &error,+ )),}}diff --git a/crates/client/turso-opfs/src/browser/test.rs b/crates/client/turso-opfs/src/browser/test.rsindex 4852c5fcdb..ff1f9bcff9 100644--- a/crates/client/turso-opfs/src/browser/test.rs+++ b/crates/client/turso-opfs/src/browser/test.rs@@ -1049,3 +1049,181 @@ async fn real_worker_opfs_contract_and_consuming_lifecycle() {replace_poisoned_registry_for_test().await;clean_pair(&uncertain_close_db).await;}++/// Closes a raw sync handle after `delay_ms`, standing in for a terminated+/// predecessor whose browser file lock outlives its owner Web Lock.+fn close_after(handle: FileSystemSyncAccessHandle, delay_ms: i32) {+ wasm_bindgen_futures::spawn_local(async move {+ sleep_ms(delay_ms).await.expect("predecessor drain timer");+ close_sync_handle(&handle).expect("close predecessor handle");+ });+}++async fn assert_entry_busy(root: &FileSystemDirectoryHandle, path: &str) {+ let error = JsFuture::from(root.remove_entry(path))+ .await+ .expect_err("predecessor handle locks the entry");+ assert!(is_entry_busy(&error), "unexpected lock error: {error:?}");+}++#[wasm_bindgen_test]+async fn owner_waits_for_a_draining_predecessor_file_lock() {+ let nonce = js_sys::Date::now() as u64;+ let database_name = format!("busy-entry-{nonce}.db");+ let wal_name = format!("{database_name}-wal");+ clean_pair(&database_name).await;+ let root = worker_root().await.expect("worker OPFS root");++ // Recovery wipe removes an entry a predecessor still briefly locks.+ let (predecessor, _) = open_sync_handle(&root, &database_name, BusyDeadline::start())+ .await+ .expect("predecessor main handle");+ assert_entry_busy(&root, &database_name).await;+ close_after(predecessor, 40);+ let owner = OpfsOwner::acquire(&database_name)+ .await+ .expect("owner lock")+ .recovery_wipe()+ .await+ .expect("wipe waits for the draining predecessor");++ // Opening waits the same way for a sync handle on the WAL.+ let (predecessor, _) = open_sync_handle(&root, &wal_name, BusyDeadline::start())+ .await+ .expect("predecessor WAL handle");+ assert_entry_busy(&root, &wal_name).await;+ close_after(predecessor, 40);+ let owner = connect_ready(open_ready(owner).await)+ .try_close()+ .expect("adapter-driven close")+ .preserve()+ .expect("preserve idle owner");++ // An open that outwaits a stuck handle reports it as busy, touches+ // nothing, and keeps the owner usable.+ let (stuck, _) = open_sync_handle(&root, &wal_name, BusyDeadline::start())+ .await+ .expect("stuck WAL handle");+ let started = performance_now();+ let failure = owner+ .open()+ .await+ .expect_err("open gives up on a lock that never drains");+ assert!(performance_now() - started >= BUSY_ENTRY_WAIT_MS);+ assert_eq!(failure.error().kind(), OpfsErrorKind::Busy);+ assert_eq!(+ failure.error().to_string(),+ "OPFS sync handle open failed (NoModificationAllowedError)"+ );+ let owner = failure+ .into_owner()+ .expect("a busy open leaves a recoverable owner");+ assert!(!REGISTRY.with(|registry| registry.borrow().machine.is_poisoned()));+ close_sync_handle(&stuck).expect("close stuck WAL handle");++ // A wipe that outwaits a stuck handle reports it as busy too.+ let (stuck, _) = open_sync_handle(&root, &database_name, BusyDeadline::start())+ .await+ .expect("stuck main handle");+ let started = performance_now();+ let failure = owner+ .recovery_wipe()+ .await+ .expect_err("wipe gives up on a lock that never drains");+ assert!(performance_now() - started >= BUSY_ENTRY_WAIT_MS);+ assert_eq!(failure.error().kind(), OpfsErrorKind::Busy);+ assert_eq!(+ failure.error().to_string(),+ "OPFS path remove failed (NoModificationAllowedError)"+ );+ assert_poisoned();+ close_sync_handle(&stuck).expect("close stuck handle");+ replace_poisoned_registry_for_test().await;+ clean_pair(&database_name).await;+}++async fn entry_exists(root: &FileSystemDirectoryHandle, path: &str) -> bool {+ match JsFuture::from(root.get_file_handle(path)).await {+ Ok(_) => true,+ Err(error) if is_not_found(&error) => false,+ Err(error) => panic!("unexpected OPFS lookup failure: {error:?}"),+ }+}++fn registry_has_no_lock() -> bool {+ REGISTRY.with(|registry| {+ let registry = registry.borrow();+ registry.pending_lock.is_none() && registry.owner_lock.is_none()+ })+}++#[wasm_bindgen_test]+async fn try_acquire_never_queues_and_removal_discards_both_paths() {+ let nonce = js_sys::Date::now() as u64;+ let database_name = format!("try-acquire-{nonce}.db");+ let wal_name = format!("{database_name}-wal");+ let lock_name = owner_lock_name(&database_name);+ clean_pair(&database_name).await;+ let root = worker_root().await.expect("worker OPFS root");++ // Another holder makes the lock unavailable; nothing is claimed or queued.+ let held = acquire_raw_web_lock(&lock_name).await;+ assert!(+ OpfsOwner::try_acquire(&database_name)+ .await+ .expect("unavailable lock is not an error")+ .is_none()+ );+ assert!(registry_has_no_lock());+ assert!(!entry_exists(&root, &database_name).await);+ held.release().await;++ // Once free, the lock is granted and the pair can be created and closed.+ let owner = OpfsOwner::try_acquire(&database_name)+ .await+ .expect("free lock acquisition")+ .expect("free lock is granted");+ assert_eq!(owner.lock_name(), lock_name);+ assert!(!web_lock_available(&lock_name).await);+ let owner = connect_ready(open_ready(owner).await)+ .try_close()+ .expect("adapter-driven close")+ .preserve()+ .expect("preserve idle owner");+ assert!(entry_exists(&root, &database_name).await);+ assert!(entry_exists(&root, &wal_name).await);++ // Removal deletes both paths without recreating them and frees the lock.+ owner+ .remove_and_release()+ .await+ .expect("remove an unused pair");+ assert!(!entry_exists(&root, &database_name).await);+ assert!(!entry_exists(&root, &wal_name).await);+ assert!(web_lock_available(&lock_name).await);+ OpfsOwner::try_acquire(&database_name)+ .await+ .expect("reacquire after removal")+ .expect("lock is free after removal")+ .remove_and_release()+ .await+ .expect("removing absent paths succeeds");++ // A removal that cannot finish still releases without poisoning.+ let (stuck, _) = open_sync_handle(&root, &database_name, BusyDeadline::start())+ .await+ .expect("stuck main handle");+ let failure = OpfsOwner::try_acquire(&database_name)+ .await+ .expect("acquire beside a stuck handle")+ .expect("lock is free beside a stuck handle")+ .remove_and_release()+ .await+ .expect_err("a stuck handle blocks removal");+ assert_eq!(failure.kind(), OpfsErrorKind::Busy);+ assert!(!REGISTRY.with(|registry| registry.borrow().machine.is_poisoned()));+ assert!(registry_has_no_lock());+ assert!(web_lock_available(&lock_name).await);+ close_sync_handle(&stuck).expect("close stuck handle");+ clean_pair(&database_name).await;+}