Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(cache-core): prevent worker disposal on upgrade (#7140)

teo767a999a5f09
diff --git a/Cargo.lock b/Cargo.lock
index 1e8d4e2e75..888ac94906 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -3304,7 +3304,7 @@ dependencies = [
[[package]]
name = "cache-wasm"
-version = "0.6.30"
+version = "0.6.31"
dependencies = [
"async-lock",
"cache-core",
diff --git a/apps/web/docs/graphql-normalized-cache-plan.md b/apps/web/docs/graphql-normalized-cache-plan.md
index f598dda6d2..9251cf12eb 100644
--- a/apps/web/docs/graphql-normalized-cache-plan.md
+++ b/apps/web/docs/graphql-normalized-cache-plan.md
@@ -198,18 +198,90 @@ heartbeat without changing epochs or touching storage. A released lock confirms
silent worker termination and triggers the usual fenced recovery. Read deadlines
still bound callers waiting on an unresponsive engine.
-Protocol v3 separates startup into asset loading and database opening. The engine
-loads/compiles WASM without touching OPFS, reports `engine-assets-ready`, and waits
-for the coordinator's `open-engine` grant. The coordinator records that storage may
-be touched before sending this grant. Pre-grant failures preserve the current
-open/reset requirement; an earlier required wipe is never forgotten. Startup phase
-budgets are relayed to every page, including late joiners: asset loading gets five
-minutes and database opening gets twenty seconds, plus a five-second page response
-grace. Ordinary read deadlines remain ten seconds. Coordinator registration gets
-sixty seconds per attempt, with three attempts; engine recovery retains its bounded
-five retries. Exhausted bootstrap retries preserve the cache scope only when the
-coordinator explicitly proves storage was untouched, not from a page's last-seen
-phase. Missing/broken transports remain conservatively uncertain.
+Protocol v6 separates startup into asset loading, taking the database owner
+lock, and database opening. The engine loads/compiles WASM without touching OPFS
+and reports `engine-assets-ready`. It then takes the owner lock only if it is
+free (`ifAvailable`), never queueing behind another holder: a busy lock is retried
+for about ten seconds with `owner-lock-busy` reports. Once it holds the lock it
+reports `owner-lock-acquired` and waits for the coordinator's `open-engine` grant;
+the coordinator records that storage may be touched before sending this grant.
+Pre-grant failures, including every failure while taking the lock, preserve the
+current open/reset requirement; an earlier required wipe is never forgotten.
+Startup phase budgets are relayed to every page, including late joiners: asset
+loading gets five minutes, taking the lock gets the retry window plus five
+seconds, and database opening gets twenty seconds, plus a five-second page
+response grace. Ordinary read deadlines remain ten seconds. Coordinator
+registration gets sixty seconds per attempt, with three attempts; engine recovery
+retains its bounded five retries. Exhausted bootstrap retries preserve the cache
+scope only when the coordinator explicitly proves storage was untouched, not from
+a page's last-seen phase. Missing/broken transports remain conservatively
+uncertain.
+
+A build's coordinator is usually a separate SharedWorker (its script URL is
+content-hashed), so after a deploy two builds can target one database. The newer
+build takes it over. Pages report their build time (`__APP_BUILD_TIME__`,
+stamped by Vite) when they register. When the lock stays busy while tabs of
+another build are alive (liveness locks of tabs this coordinator never
+registered; a departed tab's lock can outlast it), the coordinator posts a
+takeover request on the cross-build channel
+`graphql-cache-takeover:{database name}` (`coordinator-takeover.ts`, a contract
+every later build must keep). Only builds that open the same database join it,
+and only the coordinator whose engine holds the database answers:
+
+- **Yield**, if the requester is strictly newer. Every tab registered there gets
+ `cache-superseded`. Each page retires its host to the network. The owner's
+ page leaves the way a navigating page does: it terminates its engine, which
+ releases the owner lock without counting as a lost owner. An enqueue that was
+ already sent is reported as uncertain rather than sent again, because it may
+ be in the durable queue that the newer build replays. Nothing is elected
+ again. The requester's engine, still retrying the lock, opens the same
+ database with its data and queued mutations.
+- **Keep**, if the requester is not newer. The requester fails closed as below.
+
+A superseded page offers a persistent reload prompt. It reloads by itself only
+while hidden, online, not holding work (a call or an upload, via
+`holdAutomaticReload`), and not focused on a text field that holds text.
+Otherwise it retries the next time it is hidden or comes back online. A page
+that registers with a superseded coordinator later, for example after a
+rollback or from stale HTML, only runs uncached, because a reload could land on
+the same build again.
+
+A coordinator that holds nothing but gets a request from a newer build sends
+its tabs there too, without answering. When no yield arrives within a second
+(builds from before handover never answer), when the holder keeps the
+database, or when the engine exhausts its retries, the coordinator fails closed:
+it refuses queued requests with `owner-lock-unavailable` and tells every
+registered page `cache-unavailable`. Those pages retire their host for the rest
+of the page session, without quarantine, toast, or error report; only a tab
+that registers later tries again. Nothing waits in line for the lock, and a
+build only ever yields to a newer one. A rollback therefore leaves the rolled-back
+build uncached until the newer tabs close, instead of reloading tabs in a loop.
+
+The browser can also hand the owner lock to the next engine before a departing
+page's terminated worker has let go of the database files. Production logs show
+that gap outlasting two seconds after a reload. Every holder of these files
+first holds the owner lock, so a busy file seen while holding it is a predecessor
+still exiting, not damage. The OPFS adapter waits up to ten seconds for busy
+files within one open or wipe. If they stay busy, the engine reports
+`storage-busy`, and the coordinator fails closed as above. It keeps that
+attempt's open or wipe requirement instead of escalating a failed open to a
+wipe, which could not remove open files anyway.
+
+The physical database name embeds the storage versions,
+`graphql-cache:{scope}:s{epoch}.v{format}.t{storage}`, so builds with different
+compatibility epochs, record formats, or storage schemas use separate files and
+owner locks instead of resetting each other's data. Builds at the versions in
+use when names began to embed them (epoch 3, format 3, storage 11) keep the
+unversioned `graphql-cache:{scope}`, so that change moved no database and the
+next build still replays its queued mutations. Once an owner is active and no
+other build has live tabs, the coordinator lets that page delete stale
+databases of its scope: names for versions no newer than its own that it does
+not open, which includes the unversioned name once the versions move past it.
+A newer version is kept, because its build may come back after a rollback. The page lists OPFS and, only if a stale name exists,
+starts a disposable worker (a file Turso cannot read may poison its OPFS
+registry) that deletes each stale database whose owner lock is free and whose
+`mutation_queue` is empty. Databases still queueing mutations, or whose queue
+cannot be counted, are kept and reported.
Browsers missing the required worker, lock, or OPFS
capabilities use a storage-free no-op cache host. Tauri
@@ -322,8 +394,9 @@ apps/web/src/lib/graphql-cache/ # JS glue
compatibility epoch, record format, and storage schema) before creating the
fixture database and again before fault injection. This matched-build check
is intentionally stricter than normal cache compatibility.
-- Deferred: stale-namespace DB cleanup (browser), `scan_prefix`/
- `approx_size` for GC (hardening phase).
+- Stale-version DB cleanup for the current scope (browser) is done; see §4.2.
+ Deferred: cleanup of other scopes' databases (quarantined or rotated
+ scopes), `scan_prefix`/`approx_size` for GC (hardening phase).
**Phase 3 — hosts + JS glue** *(done)*
- ~~`cache-wasm`~~: wasm-bindgen shell (async-mutex engine, string op-id
diff --git a/apps/web/src/features/channel/Call/CallContext.tsx b/apps/web/src/features/channel/Call/CallContext.tsx
index e22a04bc1e..7b553d414d 100644
--- a/apps/web/src/features/channel/Call/CallContext.tsx
+++ b/apps/web/src/features/channel/Call/CallContext.tsx
@@ -5,6 +5,7 @@ import {
type BackgroundEffect,
backgroundProcessorOptions,
} from '@core/media/background-effect';
+import { holdAutomaticReload } from '@core/util/reloadForNewerBuild';
import type { KrispNoiseFilter } from '@livekit/krisp-noise-filter';
import type { BackgroundProcessorWrapper } from '@livekit/track-processors';
import {
@@ -37,7 +38,7 @@ import {
createCallSessionController,
stopPrejoinTracks,
} from './CallSessionController';
-import { createCallLifecycle } from './call-lifecycle';
+import { type CallLifecycleState, createCallLifecycle } from './call-lifecycle';
import { publishCallResolution } from './call-resolution';
import { createLatestAsyncRequestQueue } from './latest-async-request-queue';
import {
@@ -1630,9 +1631,22 @@ function createCallState() {
const [lifecycleSnapshot, setLifecycleSnapshot] = createSignal(
lifecycle.getState()
);
- const unsubscribeLifecycle = lifecycle.subscribe((state) =>
- setLifecycleSnapshot(state)
- );
+ // Reloading into a newer app build would cut the call off.
+ let releaseReloadHold: (() => void) | undefined;
+ const holdReloadDuringCall = (state: CallLifecycleState) => {
+ const inCall = state.t !== 'idle' && state.t !== 'failed';
+ if (inCall && !releaseReloadHold) {
+ releaseReloadHold = holdAutomaticReload();
+ } else if (!inCall && releaseReloadHold) {
+ releaseReloadHold();
+ releaseReloadHold = undefined;
+ }
+ };
+ holdReloadDuringCall(lifecycle.getState());
+ const unsubscribeLifecycle = lifecycle.subscribe((state) => {
+ setLifecycleSnapshot(state);
+ holdReloadDuringCall(state);
+ });
const unsubscribeNative =
isNativeIosCallKitEnabled() && nativeCall
@@ -1649,6 +1663,7 @@ function createCallState() {
onCleanup(() => {
unsubscribeNative?.();
unsubscribeLifecycle();
+ releaseReloadHold?.();
lifecycle.dispose();
disposed = true;
browserConnectGeneration += 1;
diff --git a/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx b/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx
index 8684176de6..caa8f6dea7 100644
--- a/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx
+++ b/apps/web/src/lib/core/context/quickAccess/QuickAccessSource.tsx
@@ -360,18 +360,25 @@ export function createQuickAccessValue(): QuickAccessContextValue {
equals: equalActivityMaps,
});
const isConnectedSecondaryInbox = useIsConnectedSecondaryInbox();
- const graphqlCacheHost = getGraphqlSoupCacheHost();
- const cacheHost = graphqlCacheHost?.disabled ? undefined : graphqlCacheHost;
+ // Read reactively: if the session abandons its cache mid-session, Quick
+ // Access falls back to the REST channel list instead of the dead host. The
+ // memo keeps the subscription below until the host itself changes.
+ const cacheHost = createMemo(() => {
+ const graphqlCacheHost = getGraphqlSoupCacheHost();
+ return graphqlCacheHost?.disabled ? undefined : graphqlCacheHost;
+ });
const [cacheRevision, setCacheRevision] = createSignal(0);
const cachedChannelsQuery = useCachedGraphqlChannelsQuery(cacheHost);
- if (cacheHost) {
+ createEffect(() => {
+ const host = cacheHost();
+ if (!host) return;
onCleanup(
- subscribeToVisibleCacheChanges(cacheHost, () => {
+ subscribeToVisibleCacheChanges(host, () => {
setCacheRevision((revision) => revision + 1);
return cachedChannelsQuery.refetch({ cancelRefetch: false });
})
);
- }
+ });
const instructionsIdQuery = useInstructionsMdIdQuery();
const { query: crmCompaniesQuery, companies: crmCompaniesAccessor } =
useQuickAccessCrmCompaniesQuery();
@@ -490,7 +497,8 @@ export function createQuickAccessValue(): QuickAccessContextValue {
// The GraphQL cache is authoritative while enabled. Otherwise preserve the
// existing channel-list source unchanged.
- const channelData = cacheHost
+ const usesCache = cacheHost() !== undefined;
+ const channelData = usesCache
? queryReadyGate(cachedChannelsQuery)
? cachedChannelsQuery.data
: []
@@ -946,10 +954,12 @@ export function createQuickAccessValue(): QuickAccessContextValue {
? searchQuickAccessItems(baseList(), options.searchTerm?.() ?? '')
: baseList()
);
+ // A retired host degrades this list to local items; later lists skip it.
+ const projectionHost = options ? cacheHost() : undefined;
const projected =
- options && cacheHost
+ options && projectionHost
? createProjectedList<QuickAccessItem>({
- host: cacheHost,
+ host: projectionHost,
get buckets() {
return projectedBuckets();
},
@@ -968,9 +978,9 @@ export function createQuickAccessValue(): QuickAccessContextValue {
);
const [historyItems, cachedChannelItems, cachedCompanies] =
await Promise.all([
- materializeCachedGraphqlHistoryItems(cacheHost, missing),
- materializeCachedGraphqlChannels(cacheHost, missing),
- materializeCachedGraphqlCrmCompanies(cacheHost, missing),
+ materializeCachedGraphqlHistoryItems(projectionHost, missing),
+ materializeCachedGraphqlChannels(projectionHost, missing),
+ materializeCachedGraphqlCrmCompanies(projectionHost, missing),
]);
const historyById = new Map(
historyItems.map((item) => [item.id, item])
@@ -1066,7 +1076,7 @@ export function createQuickAccessValue(): QuickAccessContextValue {
Boolean(
projected?.isLoading() ||
historyQuery.isLoading ||
- (cacheHost ? cachedChannelsQuery.isLoading : channelsLoading())
+ (cacheHost() ? cachedChannelsQuery.isLoading : channelsLoading())
),
isLoadingMore: () => projected?.isLoadingMore() ?? false,
loadMore: async () => {
@@ -1079,10 +1089,10 @@ export function createQuickAccessValue(): QuickAccessContextValue {
// resolves rather than gating quick access on a slower/failing CRM fetch.
const isLoading = () =>
historyQuery.isLoading ||
- (cacheHost ? cachedChannelsQuery.isLoading : channelsLoading());
+ (cacheHost() ? cachedChannelsQuery.isLoading : channelsLoading());
const refresh = () => {
- if (cacheHost) {
+ if (cacheHost()) {
setCacheRevision((revision) => revision + 1);
void cachedChannelsQuery.refetch();
}
@@ -1096,7 +1106,7 @@ export function createQuickAccessValue(): QuickAccessContextValue {
return {
useList,
usesRecordSelection: () => false,
- usesSearchProjection: () => cacheHost !== undefined,
+ usesSearchProjection: () => cacheHost() !== undefined,
isLoading,
refresh,
getById,
diff --git a/apps/web/src/lib/core/util/reloadForNewerBuild.test.ts b/apps/web/src/lib/core/util/reloadForNewerBuild.test.ts
new file mode 100644
index 0000000000..9e8d8b787d
--- /dev/null
+++ b/apps/web/src/lib/core/util/reloadForNewerBuild.test.ts
@@ -0,0 +1,138 @@
+import { beforeEach, describe, expect, it, vi } from 'vitest';
+
+vi.mock('@core/component/Toast/Toast', () => ({
+ toast: { custom: vi.fn() },
+}));
+
+import {
+ holdAutomaticReload,
+ type ReloadPage,
+ reloadForNewerBuild,
+ resetReloadForNewerBuildForTests,
+} from './reloadForNewerBuild';
+
+function fakePage(state: {
+ hidden: boolean;
+ offline: boolean;
+ editingText?: boolean;
+}) {
+ const listeners: Array<() => void> = [];
+ const prompts: Array<() => void> = [];
+ const page: ReloadPage & { reload: ReturnType<typeof vi.fn> } = {
+ isHidden: () => state.hidden,
+ isOffline: () => state.offline,
+ isEditingText: () => state.editingText ?? false,
+ reload: vi.fn<() => void>(),
+ onHiddenOrOnline: (callback) => listeners.push(callback),
+ promptReload: (reload) => prompts.push(reload),
+ };
+ const notify = () => {
+ for (const listener of listeners) listener();
+ };
+ return {
+ page,
+ prompts,
+ state,
+ hide() {
+ state.hidden = true;
+ notify();
+ },
+ show() {
+ state.hidden = false;
+ },
+ goOnline() {
+ state.offline = false;
+ notify();
+ },
+ };
+}
+
+describe('reloadForNewerBuild', () => {
+ beforeEach(() => {
+ resetReloadForNewerBuildForTests();
+ });
+
+ it('reloads a hidden tab right away and still offers the reload', () => {
+ const { page, prompts } = fakePage({ hidden: true, offline: false });
+
+ reloadForNewerBuild(page);
+
+ expect(page.reload).toHaveBeenCalledOnce();
+ expect(prompts).toHaveLength(1);
+ });
+
+ it('prompts on the visible tab and reloads when the user switches away', () => {
+ const { page, prompts, hide } = fakePage({ hidden: false, offline: false });
+
+ reloadForNewerBuild(page);
+
+ expect(page.reload).not.toHaveBeenCalled();
+ expect(prompts).toHaveLength(1);
+ hide();
+ expect(page.reload).toHaveBeenCalledOnce();
+ });
+
+ it('reloads from the prompt without waiting to be hidden', () => {
+ const { page, prompts } = fakePage({ hidden: false, offline: false });
+
+ reloadForNewerBuild(page);
+ prompts[0]?.();
+
+ expect(page.reload).toHaveBeenCalledOnce();
+ });
+
+ it('waits until the device is online again, and only while hidden', () => {
+ const hidden = fakePage({ hidden: true, offline: true });
+ reloadForNewerBuild(hidden.page);
+ expect(hidden.page.reload).not.toHaveBeenCalled();
+ hidden.goOnline();
+ expect(hidden.page.reload).toHaveBeenCalledOnce();
+
+ resetReloadForNewerBuildForTests();
+ const woken = fakePage({ hidden: true, offline: true });
+ reloadForNewerBuild(woken.page);
+ // The device wakes up with the tab in front: the user may be typing.
+ woken.show();
+ woken.goOnline();
+ expect(woken.page.reload).not.toHaveBeenCalled();
+ woken.hide();
+ expect(woken.page.reload).toHaveBeenCalledOnce();
+ });
+
+ it('never reloads by itself during held work or while text is written', () => {
+ const release = holdAutomaticReload();
+ const { page, prompts, state, hide, show } = fakePage({
+ hidden: true,
+ offline: false,
+ editingText: true,
+ });
+
+ reloadForNewerBuild(page);
+ expect(page.reload).not.toHaveBeenCalled();
+
+ release();
+ show();
+ hide();
+ expect(page.reload).not.toHaveBeenCalled();
+
+ state.editingText = false;
+ show();
+ hide();
+ expect(page.reload).toHaveBeenCalledOnce();
+ // The prompt still reloads whenever the user asks.
+ prompts[0]?.();
+ expect(page.reload).toHaveBeenCalledTimes(2);
+ });
+
+ it('schedules one reload however many times the cache asks', () => {
+ const first = fakePage({ hidden: false, offline: false });
+ const second = fakePage({ hidden: true, offline: false });
+
+ reloadForNewerBuild(first.page);
+ reloadForNewerBuild(second.page);
+
+ expect(first.prompts).toHaveLength(1);
+ expect(second.prompts).toHaveLength(0);
+ expect(second.page.reload).not.toHaveBeenCalled();
+ });
+});
diff --git a/apps/web/src/lib/core/util/reloadForNewerBuild.ts b/apps/web/src/lib/core/util/reloadForNewerBuild.ts
new file mode 100644
index 0000000000..685c2b4374
--- /dev/null
+++ b/apps/web/src/lib/core/util/reloadForNewerBuild.ts
@@ -0,0 +1,98 @@
+import { toast } from '@core/component/Toast/Toast';
+import { deviceLooksOffline } from './connectivity';
+import { isEditableInput } from './isEditableInput';
+
+/** What the reload decision needs from the page; replaced in tests. */
+export interface ReloadPage {
+ isHidden(): boolean;
+ isOffline(): boolean;
+ /** Whether focus is in a text field that holds text a reload would lose. */
+ isEditingText(): boolean;
+ reload(): void;
+ /** Runs `callback` each time the page is hidden or comes back online. */
+ onHiddenOrOnline(callback: () => void): void;
+ /** Offers the user a reload now. */
+ promptReload(reload: () => void): void;
+}
+
+const browserPage: ReloadPage = {
+ isHidden: () => document.visibilityState === 'hidden',
+ isOffline: deviceLooksOffline,
+ isEditingText() {
+ const focused = document.activeElement;
+ if (!isEditableInput(focused)) return false;
+ const text =
+ focused instanceof HTMLInputElement ||
+ focused instanceof HTMLTextAreaElement
+ ? focused.value
+ : focused?.textContent;
+ return Boolean(text?.trim());
+ },
+ reload: () => window.location.reload(),
+ onHiddenOrOnline(callback) {
+ // Run a task later: a page being unloaded also turns hidden, and its
+ // timers never run, so leaving the app never becomes a reload.
+ const later = () => setTimeout(callback, 0);
+ document.addEventListener('visibilitychange', () => {
+ if (document.visibilityState === 'hidden') later();
+ });
+ window.addEventListener('online', later);
+ },
+ promptReload(reload) {
+ toast.custom(
+ {
+ title: 'A new version of Macro is ready',
+ actions: [{ label: 'Reload', onClick: reload }],
+ },
+ { persistent: true }
+ );
+ },
+};
+
+const holds = new Set<symbol>();
+let scheduled = false;
+
+/**
+ * Keeps this tab from reloading by itself into a newer build while work a
+ * reload would cut off is running, such as a call or an upload. Returns the
+ * release. The user can still reload from the prompt.
+ */
+export function holdAutomaticReload(): () => void {
+ const hold = Symbol('automatic reload hold');
+ holds.add(hold);
+ return () => {
+ holds.delete(hold);
+ };
+}
+
+/**
+ * A newer build of the app took over this tab's local cache, so the tab keeps
+ * working without it and should move to that build. It offers a reload, and
+ * reloads by itself only while it is in the background and the reload loses
+ * nothing: the device is online, nothing holds the tab, and no text is being
+ * written. Otherwise it tries again the next time it is hidden or comes back
+ * online.
+ */
+export function reloadForNewerBuild(page: ReloadPage = browserPage): void {
+ if (scheduled) return;
+ scheduled = true;
+ const reloadIfIdle = () => {
+ if (
+ page.isHidden() &&
+ !page.isOffline() &&
+ holds.size === 0 &&
+ !page.isEditingText()
+ ) {
+ page.reload();
+ }
+ };
+ page.onHiddenOrOnline(reloadIfIdle);
+ page.promptReload(() => page.reload());
+ reloadIfIdle();
+}
+
+/** Test seam: forget the scheduled reload and every hold. */
+export function resetReloadForNewerBuildForTests(): void {
+ scheduled = false;
+ holds.clear();
+}
diff --git a/apps/web/src/lib/core/util/upload.ts b/apps/web/src/lib/core/util/upload.ts
index e45688bfd3..e9a99b7461 100644
--- a/apps/web/src/lib/core/util/upload.ts
+++ b/apps/web/src/lib/core/util/upload.ts
@@ -21,6 +21,7 @@ import {
blockAcceptsFileExtension,
} from '@core/constant/allBlocks';
import { heicConversionService } from '@core/heic/service';
+import { holdAutomaticReload } from '@core/util/reloadForNewerBuild';
import {
createStaticUploadFile,
createUploadFile,
@@ -411,6 +412,8 @@ export async function uploadFile(
dssOptions: DssUploadFileOptions = {}
): Promise<MaybeUploadResult> {
const uploadSource = createUploadFile(file);
+ // Reloading into a newer app build would cut the upload off.
+ const releaseReloadHold = holdAutomaticReload();
try {
validateFileSize(uploadSource);
@@ -448,6 +451,8 @@ export async function uploadFile(
: new UploadError(uploadSource, 'dss', new Error(String(error))),
name,
};
+ } finally {
+ releaseReloadHold();
}
}
diff --git a/apps/web/src/lib/graphql-cache/app-build.ts b/apps/web/src/lib/graphql-cache/app-build.ts
new file mode 100644
index 0000000000..395531cbfc
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/app-build.ts
@@ -0,0 +1,6 @@
+/**
+ * When this app build was made, in ms since the epoch, or 0 where the bundler
+ * did not stamp it (tests and harnesses). A newer build takes the local cache
+ * database over from tabs of an older one.
+ */
+export const APP_BUILD_TIME: number = import.meta.env.__APP_BUILD_TIME__ ?? 0;
diff --git a/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts b/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts
index 4b9c3cf5f7..a239ce1daa 100644
--- a/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/inspection.test.ts
@@ -129,7 +129,7 @@ describe('typed generated query inspection', () => {
it('returns no selections through the no-op host', async () => {
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
try {
- const host = createNoopCacheHost('test');
+ const host = createNoopCacheHost();
const selection = selectAll(GroupSoupMembershipDocument)
.field('user')
.field('groupSoup');
diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts
index 17c65a4955..b90a22cff0 100644
--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.test.ts
@@ -33,6 +33,7 @@ import {
type EnqueueOptimisticMutationResult,
INITIAL_CACHE_REVISION,
type MutationClaim,
+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE,
type ReadResult,
type WriteResult,
} from '../protocol';
@@ -3067,6 +3068,63 @@ describe('normalizedCacheExchange', () => {
expect(host.commits).toHaveLength(1);
});
+ it('sends a patched mutation directly while another context owns the database', async () => {
+ const base = makeMutationOp(1, optimistic);
+ const op = makeOperation(base.kind, base, {
+ ...base.context,
+ normalizedCacheOptimistic: {
+ uuid: crypto.randomUUID(),
+ optimisticResponse: optimistic,
+ linkPatches: [
+ {
+ query:
+ 'query Group { user { groupSoup { bins { items { id } } } } }',
+ operationName: 'Group',
+ variablesJson: '{}',
+ path: [
+ { field: 'user' },
+ { field: 'groupSoup' },
+ { field: 'bins' },
+ ],
+ operation: {
+ kind: 'remove',
+ entityKey: 'GraphqlSoupItem:task-1',
+ },
+ },
+ ],
+ revalidations: [],
+ },
+ });
+ host.enqueueOptimisticMutation = vi.fn().mockRejectedValue(
+ Object.assign(new Error('owner lock is held by another build'), {
+ errorCode: OWNER_LOCK_UNAVAILABLE_ERROR_CODE,
+ })
+ );
+ const warn = vi
+ .spyOn(console, 'warn')
+ .mockImplementation(() => undefined);
+ const onCacheError = vi.fn();
+ const { ops, results, forwarded } = harness(host, undefined, {
+ onCacheError,
+ });
+ try {
+ ops.next(op);
+ await tick();
+
+ // No link-patch degradation retry: the refusal says nothing about
+ // the patches, and nothing was admitted to a durable queue.
+ expect(host.enqueueOptimisticMutation).toHaveBeenCalledOnce();
+ expect(forwarded.map((forwardedOp) => forwardedOp.kind)).toEqual([
+ 'mutation',
+ ]);
+ expect(results[0]?.data).toEqual({ from: 'network' });
+ expect(onCacheError).not.toHaveBeenCalled();
+ expect(warn).not.toHaveBeenCalled();
+ } finally {
+ warn.mockRestore();
+ }
+ });
+
it('uses only explicit recovery queries when a record-rooted parent is missing', async () => {
const base = makeMutationOp(1, optimistic);
const recovery = {
diff --git a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts
index 242f06b850..ed291dca1a 100644
--- a/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts
+++ b/apps/web/src/lib/graphql-cache/exchange/normalized-cache-exchange.ts
@@ -72,6 +72,7 @@ import {
isAdmittedEnqueueUncertainError,
isCacheRevision,
isOwnerEpochLostError,
+ isOwnerLockUnavailableError,
type QueryRevalidationWire,
} from '../protocol';
import { createDeferredQueryRereads } from './deferred-query-rereads';
@@ -1101,6 +1102,12 @@ export function normalizedCacheExchange(
// unsafe to forward or retry without a coordinator fence.
return uncertainEnqueueResult(op, error);
}
+ // Another context holds the database, so the enqueue never reached
+ // an engine. Send it as a disabled cache would.
+ if (isOwnerLockUnavailableError(error)) {
+ enqueueForward(op);
+ return undefined;
+ }
// A cached bin/page may disappear between inspect and enqueue. Do
// not expose a partial relation move: retain entity optimism and
// the post-success revalidation descriptors instead.
diff --git a/apps/web/src/lib/graphql-cache/host/noop-host.ts b/apps/web/src/lib/graphql-cache/host/noop-host.ts
index ff0a047e50..9f217f71dd 100644
--- a/apps/web/src/lib/graphql-cache/host/noop-host.ts
+++ b/apps/web/src/lib/graphql-cache/host/noop-host.ts
@@ -16,13 +16,12 @@ const emptyWriteResult = (): WriteResult => ({
});
/**
- * CacheHost used when the platform cannot run the shared cache engine.
- * It never stores data. Its disabled marker makes the exchange bypass
- * optimistic persistence and forward mutations directly.
+ * CacheHost used when the platform cannot run the shared cache engine, or
+ * after a page stops using it. It never stores data. Its disabled marker
+ * makes the exchange bypass optimistic persistence and forward mutations
+ * directly.
*/
-export function createNoopCacheHost(reason: string): CacheHost {
- console.warn(`[graphql-cache] disabled: ${reason}`);
-
+export function createNoopCacheHost(): CacheHost {
return {
clientId: 'noop',
disabled: true,
diff --git a/apps/web/src/lib/graphql-cache/host/retirable-host.test.ts b/apps/web/src/lib/graphql-cache/host/retirable-host.test.ts
new file mode 100644
index 0000000000..fc0c01d13b
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/host/retirable-host.test.ts
@@ -0,0 +1,87 @@
+import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
+import { INITIAL_CACHE_REVISION } from '../protocol';
+import { createNoopCacheHost } from './noop-host';
+import { createRetirableCacheHost } from './retirable-host';
+import type { CacheHost } from './types';
+
+const QUERY = { query: 'query Q { user { id } }', operationName: 'Q' };
+
+/** A host whose every call fails like a disposed worker host. */
+function disposedLikeHost(): CacheHost {
+ const rejecting = createNoopCacheHost();
+ const disposed = () =>
+ Promise.reject(new Error('cache worker host was disposed'));
+ return {
+ ...rejecting,
+ clientId: 'inner-client',
+ disabled: false,
+ currentStorageGeneration: vi.fn(async () => 'generation-1'),
+ readQuery: vi.fn(disposed),
+ writeQuery: vi.fn(disposed),
+ search: vi.fn(disposed),
+ teardown: vi.fn(disposed),
+ dispose: vi.fn(),
+ };
+}
+
+describe('createRetirableCacheHost', () => {
+ beforeEach(() => {
+ vi.spyOn(console, 'warn');
+ });
+
+ afterEach(() => {
+ vi.restoreAllMocks();
+ });
+
+ it('delegates to the wrapped host until it is retired', async () => {
+ const inner = disposedLikeHost();
+ const host = createRetirableCacheHost(inner);
+
+ expect(host.clientId).toBe('inner-client');
+ expect(host.disabled).toBe(false);
+ await expect(host.currentStorageGeneration()).resolves.toBe('generation-1');
+ await expect(host.readQuery(QUERY)).rejects.toThrow(
+ 'cache worker host was disposed'
+ );
+ expect(inner.readQuery).toHaveBeenCalledOnce();
+ });
+
+ it('turns every captured reference into a disabled cache on dispose', async () => {
+ const inner = disposedLikeHost();
+ const host = createRetirableCacheHost(inner);
+
+ host.dispose();
+
+ expect(inner.dispose).toHaveBeenCalledOnce();
+ // The owner already logged why it retired the cache.
+ expect(console.warn).not.toHaveBeenCalled();
+ expect(host.disabled).toBe(true);
+ expect(host.clientId).toBe('inner-client');
+ // Durable checkpoints must not trust a generation from a retired cache.
+ await expect(host.currentStorageGeneration()).rejects.toThrow(
+ 'normalized GraphQL cache is unavailable'
+ );
+ await expect(host.readQuery(QUERY)).resolves.toEqual({ kind: 'miss' });
+ await expect(
+ host.writeQuery({ ...QUERY, data: { user: { id: 'u' } } })
+ ).resolves.toMatchObject({ revision: INITIAL_CACHE_REVISION });
+ await expect(
+ host.search({ profile: 'quick-access-v1', query: '', limit: 10 })
+ ).resolves.toEqual({ documents: [], nextCursor: null });
+ await expect(host.teardown(1)).resolves.toBeUndefined();
+ expect(inner.readQuery).not.toHaveBeenCalled();
+ expect(inner.writeQuery).not.toHaveBeenCalled();
+ expect(inner.search).not.toHaveBeenCalled();
+ expect(inner.teardown).not.toHaveBeenCalled();
+ });
+
+ it('retires once', () => {
+ const inner = disposedLikeHost();
+ const host = createRetirableCacheHost(inner);
+
+ host.dispose();
+ host.dispose();
+
+ expect(inner.dispose).toHaveBeenCalledOnce();
+ });
+});
diff --git a/apps/web/src/lib/graphql-cache/host/retirable-host.ts b/apps/web/src/lib/graphql-cache/host/retirable-host.ts
new file mode 100644
index 0000000000..f0c1946248
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/host/retirable-host.ts
@@ -0,0 +1,58 @@
+import { createNoopCacheHost } from './noop-host';
+import type { CacheHost } from './types';
+
+/**
+ * Wraps a host that its owner may abandon mid-session. `dispose()` releases
+ * the wrapped host and routes every later call to the storage-free no-op
+ * host. References captured before retirement (the replaced urql client's
+ * exchange, query closures) then see a disabled cache: reads miss to the
+ * network, writes are dropped, and mutations bypass the durable queue,
+ * instead of every call rejecting with "cache worker host was disposed".
+ */
+export function createRetirableCacheHost(inner: CacheHost): CacheHost {
+ let current = inner;
+ return {
+ clientId: inner.clientId,
+ get disabled() {
+ return current.disabled;
+ },
+ currentRevision: () => current.currentRevision(),
+ currentStorageGeneration: () => current.currentStorageGeneration(),
+ readQuery: (args) => current.readQuery(args),
+ readRecordsByKeys: (args) => current.readRecordsByKeys(args),
+ search: (args) => current.search(args),
+ entityFilter: (args) => current.entityFilter(args),
+ writeQuery: (args) => current.writeQuery(args),
+ hydrateQuery: (args) => current.hydrateQuery(args),
+ enqueueOptimisticMutation: (args, claim) =>
+ current.enqueueOptimisticMutation(args, claim),
+ inspectQueryVariants: (args) => current.inspectQueryVariants(args),
+ inspectQuery: (args) => current.inspectQuery(args),
+ claimNextMutation: (owner, nowMs, leaseExpiresAtMs) =>
+ current.claimNextMutation(owner, nowMs, leaseExpiresAtMs),
+ deferOptimisticWrite: (transactionId, claim, nextAttemptAtMs, error) =>
+ current.deferOptimisticWrite(
+ transactionId,
+ claim,
+ nextAttemptAtMs,
+ error
+ ),
+ commitOptimisticWrite: (transactionId, claim, args) =>
+ current.commitOptimisticWrite(transactionId, claim, args),
+ rollbackOptimisticWrite: (transactionId, claim, error) =>
+ current.rollbackOptimisticWrite(transactionId, claim, error),
+ invalidate: (keys) => current.invalidate(keys),
+ deleteRecords: (keys) => current.deleteRecords(keys),
+ teardown: (opKey) => current.teardown(opKey),
+ clear: () => current.clear(),
+ onOpsAffected: (cb) => current.onOpsAffected(cb),
+ onCacheChanged: (cb, options) => current.onCacheChanged(cb, options),
+ onCacheGenerationChanged: (cb) => current.onCacheGenerationChanged(cb),
+ onMutationSettled: (cb) => current.onMutationSettled(cb),
+ dispose() {
+ if (current !== inner) return;
+ current = createNoopCacheHost();
+ inner.dispose();
+ },
+ };
+}
diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts b/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts
index efab6bed68..95f1477d19 100644
--- a/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts
+++ b/apps/web/src/lib/graphql-cache/host/worker-host.integration.test.ts
@@ -132,7 +132,8 @@ class IntegrationCore {
private readonly logs: RuntimeLog[],
private readonly activationGates: Map<number, Promise<void>>,
private readonly hostInitGates: Map<number, Promise<void>>,
- private readonly enqueueGates: Map<string, Promise<void>>
+ private readonly enqueueGates: Map<string, Promise<void>>,
+ private readonly onOwnerLockAcquired?: () => Promise<void>
) {}
addPort(): void {}
@@ -148,6 +149,8 @@ class IntegrationCore {
query: 'query' in request ? request.query : undefined,
});
if (request.kind === 'init') {
+ // Like WASM: hold the owner lock and get the storage grant, then open.
+ if (request.id === 0) await this.onOwnerLockAcquired?.();
const gate =
request.id === 0
? this.activationGates.get(this.epoch)
@@ -208,13 +211,14 @@ class IntegrationDedicatedWorker {
) {
installCacheEngineWorker({
scope: this.scope,
- createCore: () =>
+ createCore: (options) =>
new IntegrationCore(
epoch,
logs,
activationGates,
hostInitGates,
- enqueueGates
+ enqueueGates,
+ options.onOwnerLockAcquired
),
ownerLockIsHeld: async () => true,
});
diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.test.ts b/apps/web/src/lib/graphql-cache/host/worker-host.test.ts
index 629bddefcb..13e24df94f 100644
--- a/apps/web/src/lib/graphql-cache/host/worker-host.test.ts
+++ b/apps/web/src/lib/graphql-cache/host/worker-host.test.ts
@@ -138,6 +138,14 @@ class FakePageAdapter {
this.options.onTerminalError?.(error);
}
+ unavailable(reason: string): void {
+ this.options.onCacheUnavailable?.(reason);
+ }
+
+ superseded(reason: string): void {
+ this.options.onCacheSuperseded?.(reason);
+ }
+
private emit(message: WorkerMessage): void {
this.onmessage?.({ data: message } as MessageEvent<WorkerMessage>);
}
@@ -225,6 +233,18 @@ describe('createWorkerCacheHost', () => {
).resolves.toEqual(EMPTY_WRITE);
});
+ it("refuses a scope whose database would read as another scope's WAL file", () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+
+ const host = createWorkerCacheHost({ scope: 'scope-1-wal' });
+
+ expect(warn).toHaveBeenCalledWith(
+ '[graphql-cache] disabled: cache scope must not end with -wal'
+ );
+ expect(adapterFactory).not.toHaveBeenCalled();
+ expect(host.disabled).toBe(true);
+ });
+
it('routes hydration through the payload-projecting RPC', async () => {
const host = createWorkerCacheHost({ scope: 'scope-1' });
@@ -1287,6 +1307,139 @@ describe('createWorkerCacheHost', () => {
);
});
+ it('stops using the cache without quarantine when another context holds the database', async () => {
+ configureAdapter = (fake) => {
+ fake.ignoredKinds.add('init');
+ };
+ localStorage.setItem('graphql-cache:scope', 'scope-1');
+ const onInitializationError = vi.fn();
+ const host = createWorkerCacheHost({
+ scope: 'scope-1',
+ onInitializationError,
+ });
+ const read = host.readQuery({ query: 'query Q { q }' });
+ const unavailable = expect(read).rejects.toMatchObject({
+ errorCode: 'owner-lock-unavailable',
+ });
+ await vi.waitFor(() =>
+ expect(requireAdapter().requests.map(({ kind }) => kind)).toContain(
+ 'init'
+ )
+ );
+ const adapter = requireAdapter();
+
+ adapter.unavailable('cache database owner lock is held by another context');
+
+ await unavailable;
+ expect(onInitializationError).toHaveBeenCalledExactlyOnceWith(
+ expect.objectContaining({ errorCode: 'owner-lock-unavailable' })
+ );
+ expect(adapter.dispose).toHaveBeenCalledWith({ graceful: false });
+ // Storage was never touched, so the scope stays reusable next load.
+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');
+ await expect(
+ host.readQuery({ query: 'query Q { q }' })
+ ).rejects.toMatchObject({ errorCode: 'owner-lock-unavailable' });
+ adapter.unavailable('duplicate notice');
+ expect(onInitializationError).toHaveBeenCalledOnce();
+ });
+
+ it('retires a ready host once a request is refused because the database is unavailable', async () => {
+ localStorage.setItem('graphql-cache:scope', 'scope-1');
+ const onInitializationError = vi.fn();
+ const host = createWorkerCacheHost({
+ scope: 'scope-1',
+ onInitializationError,
+ });
+ await host.clear();
+ const adapter = requireAdapter();
+ adapter.ignoredKinds.add('read');
+ const read = host.readQuery({ query: 'query Q { q }' });
+ await vi.waitFor(() => expect(adapter.requests.at(-1)?.kind).toBe('read'));
+
+ adapter.reject(
+ adapter.requests.at(-1)?.id ?? -1,
+ 'cache database owner lock is held by another context',
+ 'owner-lock-unavailable'
+ );
+
+ await expect(read).rejects.toMatchObject({
+ errorCode: 'owner-lock-unavailable',
+ });
+ expect(onInitializationError).toHaveBeenCalledOnce();
+ expect(adapter.dispose).toHaveBeenCalledWith({ graceful: false });
+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');
+ });
+
+ it('retires quietly and asks the app to reload when a newer build takes over', async () => {
+ localStorage.setItem('graphql-cache:scope', 'scope-1');
+ const onInitializationError = vi.fn();
+ const onSuperseded = vi.fn();
+ const host = createWorkerCacheHost({
+ scope: 'scope-1',
+ onInitializationError,
+ onSuperseded,
+ });
+ await host.clear();
+ const adapter = requireAdapter();
+
+ adapter.superseded('a newer version of the app took over the local cache');
+ adapter.superseded('duplicate notice');
+
+ expect(onSuperseded).toHaveBeenCalledOnce();
+ expect(onInitializationError).toHaveBeenCalledExactlyOnceWith(
+ expect.objectContaining({ errorCode: 'owner-lock-unavailable' })
+ );
+ // Leaving like a navigating page stops the engine without counting as a
+ // lost owner, so the newer build opens the same database.
+ expect(adapter.dispose).toHaveBeenCalledWith({
+ graceful: false,
+ preserveDatabase: true,
+ });
+ // The newer build opens the same database, so the scope is kept.
+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');
+ await expect(
+ host.readQuery({ query: 'query Q { q }' })
+ ).rejects.toMatchObject({ errorCode: 'owner-lock-unavailable' });
+ });
+
+ it('reports an enqueue in flight at the handover as uncertain, never unsent', async () => {
+ localStorage.setItem('graphql-cache:scope', 'scope-1');
+ const host = createWorkerCacheHost({ scope: 'scope-1' });
+ await host.clear();
+ const adapter = requireAdapter();
+ adapter.ignoredKinds.add('enqueue-optimistic-mutation');
+ adapter.ignoredKinds.add('read');
+ const mutation = host.enqueueOptimisticMutation(
+ {
+ uuid: '00000000-0000-4000-8000-000000000101',
+ query: 'mutation Update { update }',
+ data: { update: true },
+ },
+ { owner: 'runner', nowMs: 1, leaseExpiresAtMs: 101 }
+ );
+ const read = host.readQuery({ query: 'query Q { q }' });
+ // The newer build replays the queued row, so the page must not send the
+ // mutation itself; the read never touched durable state.
+ const mutationRejected = expect(mutation).rejects.toMatchObject({
+ errorCode: 'admitted-enqueue-uncertain',
+ });
+ const readRejected = expect(read).rejects.toMatchObject({
+ errorCode: 'owner-lock-unavailable',
+ });
+ await vi.waitFor(() =>
+ expect(adapter.requests.map(({ kind }) => kind)).toEqual(
+ expect.arrayContaining(['enqueue-optimistic-mutation', 'read'])
+ )
+ );
+
+ adapter.superseded('a newer version of the app took over the local cache');
+
+ await mutationRejected;
+ await readRejected;
+ expect(localStorage.getItem('graphql-cache:scope')).toBe('scope-1');
+ });
+
it('quarantines transport scope before invoking the product failure callback', async () => {
let runLock!: () => Promise<void>;
vi.stubGlobal('navigator', {
diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.ts b/apps/web/src/lib/graphql-cache/host/worker-host.ts
index 9b3f535908..56672fb7fd 100644
--- a/apps/web/src/lib/graphql-cache/host/worker-host.ts
+++ b/apps/web/src/lib/graphql-cache/host/worker-host.ts
@@ -25,6 +25,7 @@ import {
type MutationClaim,
type MutationSettlement,
OWNER_EPOCH_LOST_ERROR_CODE,
+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE,
parseStorageGeneration,
type ReadRecordsByKeysArgs,
type ReadRecordsByKeysResult,
@@ -52,6 +53,7 @@ import {
createCacheCoordinatorPageAdapter,
} from '../worker/coordinator-page-adapter';
import type { EngineOpenOutcome } from '../worker/coordinator-protocol';
+import { isUnambiguousCacheScope } from '../worker/stale-databases';
import {
CacheBootstrapExhaustedError,
COORDINATOR_CONNECT_ATTEMPTS,
@@ -114,6 +116,9 @@ export interface WorkerHostOptions {
initializationTimeoutMs?: number;
/** Reports terminal initialization or coordinator-transport failure. */
onInitializationError?: (error: Error) => void;
+ /** A newer app build took the local cache over. The host has already
+ * retired to the network; the app should reload the page into that build. */
+ onSuperseded?: () => void;
/** Allowlisted rollout cohort attached to browser-cache telemetry. */
rolloutCohort?: CacheRolloutCohort;
/** Injectable recorder for deterministic tests or alternate exporters. */
@@ -177,6 +182,10 @@ const isOwnerEpochLoss = (error: unknown): error is CacheResponseError =>
error instanceof CacheResponseError &&
error.errorCode === OWNER_EPOCH_LOST_ERROR_CODE;
+const isOwnerLockUnavailable = (error: unknown): error is CacheResponseError =>
+ error instanceof CacheResponseError &&
+ error.errorCode === OWNER_LOCK_UNAVAILABLE_ERROR_CODE;
+
export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
let pageTelemetry = options.telemetry
? undefined
@@ -188,7 +197,11 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
options.telemetry ?? pageTelemetry?.recorder
);
const now = (): number => globalThis.performance?.now() ?? Date.now();
- const unsupportedReason = unsupportedBrowserReason();
+ const unsupportedReason =
+ unsupportedBrowserReason() ??
+ (isUnambiguousCacheScope(options.scope)
+ ? undefined
+ : 'cache scope must not end with -wal');
if (unsupportedReason) {
telemetry?.record({
name: 'graphql_cache.host_ready',
@@ -197,7 +210,8 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
errorCode: 'unsupported',
});
telemetry?.flush();
- return createNoopCacheHost(unsupportedReason);
+ console.warn(`[graphql-cache] disabled: ${unsupportedReason}`);
+ return createNoopCacheHost();
}
const clientId = crypto.randomUUID();
@@ -233,6 +247,7 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
let latestReplacementEpoch = 0;
let failureReported = false;
let terminalFailureHandled = false;
+ let superseded = false;
let adapter: CacheCoordinatorPageAdapter | undefined;
let registeredAdapter: CacheCoordinatorPageAdapter | undefined;
let adapterDisposePromise: Promise<void> | undefined;
@@ -318,6 +333,7 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
if (entry.timer !== undefined) clearTimeout(entry.timer);
recordRequestOutcome(entry, 'error', error);
entry.reject(error);
+ if (isOwnerLockUnavailable(error)) retireUnavailable(msg.error);
finishGracefulDisposeIfDrained();
return;
}
@@ -450,6 +466,15 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
if (adapter === created && !registrationInProgress && !suspension)
failTransport(error);
},
+ onCacheUnavailable: (reason) => {
+ if (adapter === created) retireUnavailable(reason);
+ },
+ onCacheSuperseded: (reason) => {
+ if (adapter !== created || superseded) return;
+ superseded = true;
+ retireUnavailable(reason, true);
+ options.onSuperseded?.();
+ },
telemetry,
});
created.onmessage = (event) => {
@@ -605,13 +630,66 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
durationMs:
initializationStartedAt > 0 ? now() - initializationStartedAt : 0,
});
+ stopForSession(error);
+ reportFailure(error);
+ }
+
+ /**
+ * Fails this host for the rest of the page session: pending requests reject
+ * (an admitted enqueue as uncertain when `uncertain`), subscribers are
+ * dropped, and the coordinator connection closes.
+ */
+ function stopForSession(
+ error: Error,
+ { uncertain = false, preserveDatabase = false } = {}
+ ): void {
state = 'failed';
initialization = undefined;
initializationError = error;
- rejectPending(error);
+ rejectPending(error, uncertain);
clearSubscribers();
unregisterPagehide();
- void disposeAdapter(false).then(finishTelemetry);
+ void disposeAdapter(false, preserveDatabase).then(finishTelemetry);
+ }
+
+ /**
+ * Another context, usually a tab on another deployed build, holds the
+ * database, and the coordinator touched no storage. Nothing is quarantined:
+ * this page stops using the cache until it reloads, and a later page load
+ * tries again. A page handing the database to a newer build leaves like a
+ * navigating page: its engine stops without counting as a lost owner, and
+ * an enqueue already sent may have reached the durable queue, so it is
+ * reported uncertain rather than sent again.
+ */
+ function retireUnavailable(reason: string, leaveForNewerBuild = false): void {
+ if (
+ terminalFailureHandled ||
+ state === 'failed' ||
+ state === 'suspended' ||
+ state === 'disposing' ||
+ state === 'disposed'
+ )
+ return;
+ terminalFailureHandled = true;
+ const error = new CacheResponseError(
+ reason,
+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE
+ );
+ if (state === 'initializing') {
+ telemetry?.record({
+ name: 'graphql_cache.host_ready',
+ operationCategory: 'initialization',
+ outcome: 'error',
+ errorCode: 'lock',
+ durationMs:
+ initializationStartedAt > 0 ? now() - initializationStartedAt : 0,
+ });
+ }
+ stopStorageHealthSampling();
+ stopForSession(error, {
+ uncertain: leaveForNewerBuild,
+ preserveDatabase: leaveForNewerBuild,
+ });
reportFailure(error);
}
@@ -630,13 +708,7 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
);
if (state === 'disposed' && !admittedWorkIsUncertain) return;
terminalFailureHandled = true;
- state = 'failed';
- initialization = undefined;
- initializationError = error;
- rejectPending(error, true);
- clearSubscribers();
- unregisterPagehide();
- void disposeAdapter(false).then(finishTelemetry);
+ stopForSession(error, { uncertain: true });
// Product failure handling may immediately construct another host. Make
// the matching old scope unreachable before invoking that callback.
const quarantine = storageWasUntouched
diff --git a/apps/web/src/lib/graphql-cache/protocol.ts b/apps/web/src/lib/graphql-cache/protocol.ts
index 3aeea6820a..af6d6026e9 100644
--- a/apps/web/src/lib/graphql-cache/protocol.ts
+++ b/apps/web/src/lib/graphql-cache/protocol.ts
@@ -585,12 +585,21 @@ export type CacheRequest = { id: number } & (
/** Stable machine-readable cache RPC rejection codes. */
export type CacheResponseErrorCode =
| 'owner-epoch-lost'
+ | 'owner-lock-unavailable'
| 'admitted-enqueue-uncertain';
/** Old-owner work was rejected after fenced engine loss and was not replayed. */
export const OWNER_EPOCH_LOST_ERROR_CODE: CacheResponseErrorCode =
'owner-epoch-lost';
+/**
+ * Another context holds the database, so this page's cache stays off until
+ * reload: typically another deployed build keeps its owner lock, or a closing
+ * tab's worker still has its files open. The request never reached an engine.
+ */
+export const OWNER_LOCK_UNAVAILABLE_ERROR_CODE: CacheResponseErrorCode =
+ 'owner-lock-unavailable';
+
/** An enqueue send was admitted before its unfenced transport became uncertain. */
export const ADMITTED_ENQUEUE_UNCERTAIN_ERROR_CODE: CacheResponseErrorCode =
'admitted-enqueue-uncertain';
@@ -640,6 +649,7 @@ export const isCacheResponseErrorCode = (
value: unknown
): value is CacheResponseErrorCode =>
value === OWNER_EPOCH_LOST_ERROR_CODE ||
+ value === OWNER_LOCK_UNAVAILABLE_ERROR_CODE ||
value === ADMITTED_ENQUEUE_UNCERTAIN_ERROR_CODE;
/** Identifies a coordinator-fenced rejection from a lost owner epoch. */
@@ -650,6 +660,14 @@ export const isOwnerEpochLostError = (
'errorCode' in value &&
value.errorCode === OWNER_EPOCH_LOST_ERROR_CODE;
+/** Identifies a request refused because another context holds the database. */
+export const isOwnerLockUnavailableError = (
+ value: unknown
+): value is Error & { errorCode: 'owner-lock-unavailable' } =>
+ value instanceof Error &&
+ 'errorCode' in value &&
+ value.errorCode === OWNER_LOCK_UNAVAILABLE_ERROR_CODE;
+
/** Identifies the host-only uncertainty result for an admitted enqueue send. */
export const isAdmittedEnqueueUncertainError = (
value: unknown
diff --git a/apps/web/src/lib/graphql-cache/telemetry.ts b/apps/web/src/lib/graphql-cache/telemetry.ts
index 1edd8436bd..36c26e6693 100644
--- a/apps/web/src/lib/graphql-cache/telemetry.ts
+++ b/apps/web/src/lib/graphql-cache/telemetry.ts
@@ -133,6 +133,12 @@ export const CACHE_OWNER_EVENTS = [
'abrupt-loss',
'replacement',
'multiple-owner-detected',
+ 'owner-lock-unavailable',
+ 'storage-busy',
+ 'takeover-granted',
+ 'superseded',
+ 'stale-databases-removed',
+ 'stale-database-kept',
] as const;
export type CacheOwnerEvent = (typeof CACHE_OWNER_EVENTS)[number];
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts
index 4285e09996..138577934e 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/coordinator.browser.e2e.ts
@@ -299,12 +299,14 @@ test('production cache-wasm Turso engine preserves graceful data and atomically
realTursoDataPreservedGracefully: boolean;
gracefulCloseReleasedOwnerLock: boolean;
gracefulReplacementWaitedForPhysicalLock: boolean;
- gracefulPendingOwnerLockRequests: number;
+ gracefulQueuedOwnerLockRequests: number;
+ gracefulRetriedHeldOwnerLock: boolean;
abruptInflightRejected: boolean;
abruptRequestReplayCount: number;
abruptOwnerPageStayedAlive: boolean;
recoveryReplacementWaitedForPhysicalLock: boolean;
- recoveryPendingOwnerLockRequests: number;
+ recoveryQueuedOwnerLockRequests: number;
+ recoveryRetriedHeldOwnerLock: boolean;
atomicRecoveryOpenWipedToMiss: boolean;
recoveryDatabaseAction: string;
ownerEpochs: number[];
@@ -316,12 +318,14 @@ test('production cache-wasm Turso engine preserves graceful data and atomically
realTursoDataPreservedGracefully: true,
gracefulCloseReleasedOwnerLock: true,
gracefulReplacementWaitedForPhysicalLock: true,
- gracefulPendingOwnerLockRequests: 1,
+ gracefulQueuedOwnerLockRequests: 0,
+ gracefulRetriedHeldOwnerLock: true,
abruptInflightRejected: true,
abruptRequestReplayCount: 1,
abruptOwnerPageStayedAlive: true,
recoveryReplacementWaitedForPhysicalLock: true,
- recoveryPendingOwnerLockRequests: 1,
+ recoveryQueuedOwnerLockRequests: 0,
+ recoveryRetriedHeldOwnerLock: true,
atomicRecoveryOpenWipedToMiss: true,
recoveryDatabaseAction: 'wipe-before-open',
ownerEpochs: [1, 2, 3],
@@ -329,3 +333,88 @@ test('production cache-wasm Turso engine preserves graceful data and atomically
});
expect(browserErrors).toEqual([]);
});
+
+test('production engine waits out a departing owner that still holds the files, and never wipes them', async ({
+ context,
+ page,
+}, testInfo) => {
+ const browserErrors: string[] = [];
+ const watch = (candidate: typeof page): void => {
+ candidate.on('console', (message) => {
+ if (message.type() === 'error') browserErrors.push(message.text());
+ });
+ candidate.on('pageerror', (error) => browserErrors.push(error.message));
+ };
+ watch(page);
+ context.on('page', watch);
+
+ await page.goto(harnessPath(testInfo.project.name, 'production-busy.html'));
+ const result = page.locator('#result');
+ await expect(result).toHaveAttribute('data-status', 'passed', {
+ timeout: 80_000,
+ });
+ const report = JSON.parse((await result.textContent()) ?? '') as Record<
+ string,
+ unknown
+ >;
+
+ expect(report).toEqual({
+ passed: true,
+ firstReplacementDatabaseAction: 'open-existing',
+ firstReplacementWaitedForFiles: true,
+ firstReplacementKeptData: true,
+ firstRecoveryAttempts: 0,
+ busyReplacementDatabaseAction: 'open-existing',
+ busyReplacementOpened: false,
+ busyReplacementTerminated: true,
+ survivorToldFilesAreOpen: true,
+ survivorReadFellBack: true,
+ retriedWhileBusy: false,
+ laterTabDatabaseAction: 'open-existing',
+ laterTabKeptData: true,
+ protocolErrors: [],
+ });
+ expect(browserErrors).toEqual([]);
+});
+
+test('a newer build takes the production database over and older builds are turned away', async ({
+ context,
+ page,
+}, testInfo) => {
+ const browserErrors: string[] = [];
+ const watch = (candidate: typeof page): void => {
+ candidate.on('console', (message) => {
+ if (message.type() === 'error') browserErrors.push(message.text());
+ });
+ candidate.on('pageerror', (error) => browserErrors.push(error.message));
+ };
+ watch(page);
+ context.on('page', watch);
+
+ await page.goto(
+ harnessPath(testInfo.project.name, 'production-takeover.html')
+ );
+ const result = page.locator('#result');
+ await expect(result).toHaveAttribute('data-status', 'passed', {
+ timeout: 80_000,
+ });
+ const report = JSON.parse((await result.textContent()) ?? '') as Record<
+ string,
+ unknown
+ >;
+
+ expect(report).toEqual({
+ passed: true,
+ newBuildDatabaseAction: 'open-existing',
+ newBuildKeptData: true,
+ oldEngineStoppedLikeNavigation: true,
+ oldTabsSentOn: true,
+ olderBuildTurnedAway: true,
+ olderBuildSentOn: false,
+ lateOldTabRunsUncached: true,
+ lateOldTabSentOn: false,
+ newBuildSentOn: false,
+ protocolErrors: [],
+ });
+ expect(browserErrors).toEqual([]);
+});
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts
index 0afe763de6..202beb638d 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/fake-cache.engine-worker.ts
@@ -13,10 +13,11 @@ import {
validateCoordinatorToEngineEnvelope,
validatePageToEngineEnvelope,
} from '../coordinator-protocol';
+import { OWNER_LOCK_RETRY_DELAYS_MS } from '../startup';
declare const self: DedicatedWorkerGlobalScope;
-const withVersion = <T extends { coordinatorVersion: 5 }>(
+const withVersion = <T extends { coordinatorVersion: 6 }>(
value: T extends unknown ? Omit<T, 'coordinatorVersion'> : never
): T =>
({
@@ -157,7 +158,9 @@ async function activate(
activationValue = activation;
controlPort = port;
telemetry = new BroadcastChannel(`graphql-cache-wp08:${activation.scope}`);
- await new Promise<void>((resolve) => {
+ // Like the production engine: announce assets, take the owner lock, then
+ // touch storage only after the coordinator's `open-engine` grant.
+ const storageGrant = new Promise<void>((resolve) => {
port.onmessage = ({ data }) => {
const parsed = validateCoordinatorToEngineEnvelope(
Array.isArray(data) ? data[1] : data
@@ -169,180 +172,218 @@ async function activate(
)
resolve();
};
- port.start();
- port.postMessage([0]);
- sendEngine(
- withVersion<EngineToCoordinatorEnvelope>({
- kind: 'engine-assets-ready',
- tabId: activation.tabId,
- ownerEpoch: activation.ownerEpoch,
- })
- );
});
- await navigator.locks.request(
- activation.ownerLockName,
- { mode: 'exclusive' },
- async (lock) => {
- if (!lock) throw new Error('owner lock was not acquired');
- telemetry?.postMessage({
- kind: 'lock-acquired',
- tabId: activation.tabId,
- ownerEpoch: activation.ownerEpoch,
- });
- let database: IDBDatabase | undefined;
- const engineState: FakeEngineState = { revision: 0n };
- let queue = Promise.resolve();
- let draining = false;
- let requestShutdown: (() => void) | undefined;
- const shutdown = new Promise<void>((resolve) => {
- requestShutdown = resolve;
- });
- try {
- if (activation.databaseAction === 'wipe-before-open') {
- telemetry?.postMessage({
- kind: 'wipe-started',
- ownerEpoch: activation.ownerEpoch,
- });
- await deleteDatabase(activation.scope);
- telemetry?.postMessage({
- kind: 'wipe-completed',
- ownerEpoch: activation.ownerEpoch,
- });
+ port.start();
+ port.postMessage([0]);
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'engine-assets-ready',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ );
+ // Never queue for the owner lock: retry it briefly, then give up without
+ // touching storage, exactly like the production engine.
+ for (let attempt = 1; ; attempt += 1) {
+ let lockWasBusy = false;
+ await navigator.locks.request(
+ activation.ownerLockName,
+ { mode: 'exclusive', ifAvailable: true },
+ async (lock) => {
+ if (!lock) {
+ lockWasBusy = true;
+ return;
}
- database = await openDatabase(activation.scope);
- const activeDatabase = database;
- port.onmessage = (event: MessageEvent<unknown>) => {
- if (Array.isArray(event.data) && event.data[0] === 1) {
- requestShutdown?.();
- return;
+ telemetry?.postMessage({
+ kind: 'lock-acquired',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ });
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'owner-lock-acquired',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ );
+ await storageGrant;
+ let database: IDBDatabase | undefined;
+ const engineState: FakeEngineState = { revision: 0n };
+ let queue = Promise.resolve();
+ let draining = false;
+ let requestShutdown: (() => void) | undefined;
+ const shutdown = new Promise<void>((resolve) => {
+ requestShutdown = resolve;
+ });
+ try {
+ if (activation.databaseAction === 'wipe-before-open') {
+ telemetry?.postMessage({
+ kind: 'wipe-started',
+ ownerEpoch: activation.ownerEpoch,
+ });
+ await deleteDatabase(activation.scope);
+ telemetry?.postMessage({
+ kind: 'wipe-completed',
+ ownerEpoch: activation.ownerEpoch,
+ });
}
- const payload =
- Array.isArray(event.data) && event.data[0] === 0
- ? event.data[1]
- : event.data;
- const parsed = validateCoordinatorToEngineEnvelope(payload);
- if (!parsed.ok) return;
- const message: CoordinatorToEngineEnvelope = parsed.value;
- if (message.ownerEpoch !== activation.ownerEpoch) return;
- switch (message.kind) {
- case 'heartbeat':
- if (!ignoreHeartbeats) {
- sendEngine(
- withVersion<EngineToCoordinatorEnvelope>({
- kind: 'heartbeat-ack',
- ownerEpoch: activation.ownerEpoch,
- heartbeatId: message.heartbeatId,
- })
- );
- }
- break;
- case 'drain-engine':
- if (draining) return;
- draining = true;
- void queue.finally(() => requestShutdown?.());
- break;
- case 'engine-request':
- if (draining) return;
- telemetry?.postMessage({
- kind: 'request-started',
- tabId: activation.tabId,
- ownerEpoch: activation.ownerEpoch,
- routeId: message.routeId,
- requestKind: message.request.kind,
- requestId: message.request.id,
- slow:
- message.request.kind === 'read' &&
- message.request.query.includes('Slow'),
- });
- queue = queue.then(async () => {
- let response: CacheResponse;
- try {
- response = await execute(
- activeDatabase,
- message.request,
- engineState
+ database = await openDatabase(activation.scope);
+ const activeDatabase = database;
+ port.onmessage = (event: MessageEvent<unknown>) => {
+ if (Array.isArray(event.data) && event.data[0] === 1) {
+ requestShutdown?.();
+ return;
+ }
+ const payload =
+ Array.isArray(event.data) && event.data[0] === 0
+ ? event.data[1]
+ : event.data;
+ const parsed = validateCoordinatorToEngineEnvelope(payload);
+ if (!parsed.ok) return;
+ const message: CoordinatorToEngineEnvelope = parsed.value;
+ if (message.ownerEpoch !== activation.ownerEpoch) return;
+ switch (message.kind) {
+ case 'heartbeat':
+ if (!ignoreHeartbeats) {
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'heartbeat-ack',
+ ownerEpoch: activation.ownerEpoch,
+ heartbeatId: message.heartbeatId,
+ })
);
- } catch (error) {
- response = {
- id: message.request.id,
- ok: false,
- error:
- error instanceof Error ? error.message : String(error),
- };
}
- sendEngine(
- withVersion<EngineToCoordinatorEnvelope>({
- kind: 'engine-response',
- ownerEpoch: activation.ownerEpoch,
- routeId: message.routeId,
- response,
- })
- );
- if (
- response.ok &&
- (message.request.kind === 'write' ||
- message.request.kind === 'clear')
- ) {
+ break;
+ case 'drain-engine':
+ if (draining) return;
+ draining = true;
+ void queue.finally(() => requestShutdown?.());
+ break;
+ case 'engine-request':
+ if (draining) return;
+ telemetry?.postMessage({
+ kind: 'request-started',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ routeId: message.routeId,
+ requestKind: message.request.kind,
+ requestId: message.request.id,
+ slow:
+ message.request.kind === 'read' &&
+ message.request.query.includes('Slow'),
+ });
+ queue = queue.then(async () => {
+ let response: CacheResponse;
+ try {
+ response = await execute(
+ activeDatabase,
+ message.request,
+ engineState
+ );
+ } catch (error) {
+ response = {
+ id: message.request.id,
+ ok: false,
+ error:
+ error instanceof Error ? error.message : String(error),
+ };
+ }
sendEngine(
withVersion<EngineToCoordinatorEnvelope>({
- kind: 'engine-push',
+ kind: 'engine-response',
ownerEpoch: activation.ownerEpoch,
- push: {
- kind: 'cache-changed',
- revision:
- engineState.revision.toString() as typeof INITIAL_CACHE_REVISION,
- },
+ routeId: message.routeId,
+ response,
})
);
- }
- });
- break;
- }
- };
- sendEngine(
- withVersion<EngineToCoordinatorEnvelope>({
- kind: 'engine-ready',
+ if (
+ response.ok &&
+ (message.request.kind === 'write' ||
+ message.request.kind === 'clear')
+ ) {
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'engine-push',
+ ownerEpoch: activation.ownerEpoch,
+ push: {
+ kind: 'cache-changed',
+ revision:
+ engineState.revision.toString() as typeof INITIAL_CACHE_REVISION,
+ },
+ })
+ );
+ }
+ });
+ break;
+ }
+ };
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'engine-ready',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ ownerLockName: activation.ownerLockName,
+ ownerLockHeld: true,
+ databaseActionProof:
+ activation.databaseAction === 'wipe-before-open'
+ ? 'wiped-before-open'
+ : 'opened-existing',
+ openOutcome:
+ activation.databaseAction === 'wipe-before-open'
+ ? 'reset-storage-uncertain'
+ : 'opened-existing',
+ })
+ );
+ telemetry?.postMessage({
+ kind: 'ready',
tabId: activation.tabId,
ownerEpoch: activation.ownerEpoch,
- ownerLockName: activation.ownerLockName,
- ownerLockHeld: true,
- databaseActionProof:
- activation.databaseAction === 'wipe-before-open'
- ? 'wiped-before-open'
- : 'opened-existing',
- openOutcome:
- activation.databaseAction === 'wipe-before-open'
- ? 'reset-storage-uncertain'
- : 'opened-existing',
- })
- );
- telemetry?.postMessage({
- kind: 'ready',
- tabId: activation.tabId,
- ownerEpoch: activation.ownerEpoch,
- });
+ });
- await shutdown;
- await queue;
- database.close();
- database = undefined;
- sendEngine(
- withVersion<EngineToCoordinatorEnvelope>({
- kind: 'engine-drained',
+ await shutdown;
+ await queue;
+ database.close();
+ database = undefined;
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'engine-drained',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ );
+ } finally {
+ database?.close();
+ telemetry?.postMessage({
+ kind: 'lock-releasing',
tabId: activation.tabId,
ownerEpoch: activation.ownerEpoch,
- })
- );
- } finally {
- database?.close();
- telemetry?.postMessage({
- kind: 'lock-releasing',
+ });
+ }
+ }
+ );
+ if (!lockWasBusy) break;
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'owner-lock-busy',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ );
+ const delayMs = OWNER_LOCK_RETRY_DELAYS_MS[attempt - 1];
+ if (delayMs === undefined) {
+ sendEngine(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'owner-lock-unavailable',
tabId: activation.tabId,
ownerEpoch: activation.ownerEpoch,
- });
- }
+ })
+ );
+ return;
}
- );
+ await new Promise<void>((resolve) => {
+ setTimeout(resolve, delayMs);
+ });
+ }
port.close();
telemetry?.close();
self.close();
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts
index bce3d68b6a..3a2e7ef1f9 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-browser-wire.ts
@@ -22,7 +22,9 @@ export type ProductionHarnessCommand =
commandId: string;
request: ProductionCacheRequestWithoutId;
}
- | { kind: 'terminate-worker'; commandId: string };
+ | { kind: 'terminate-worker'; commandId: string }
+ /** Leaves the page the way `pagehide` does on navigation or reload. */
+ | { kind: 'navigate-away'; commandId: string };
export type ProductionHarnessEnvelope =
| {
@@ -49,5 +51,7 @@ export type ProductionHarnessEnvelope =
ok: false;
error: string;
}
- | { kind: 'protocol-error'; error: string };
+ | { kind: 'protocol-error'; error: string }
+ | { kind: 'cache-unavailable'; reason: string }
+ | { kind: 'cache-superseded'; reason: string };
};
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy-harness.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy-harness.ts
new file mode 100644
index 0000000000..d1b743c4d9
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy-harness.ts
@@ -0,0 +1,344 @@
+import {
+ cacheDatabaseIdentity,
+ databaseOwnerLockName,
+} from '../coordinator-protocol';
+import type {
+ ProductionHarnessCommand,
+ ProductionHarnessEnvelope,
+} from './production-browser-wire';
+import type {
+ FileHolderRequest,
+ FileHolderResponse,
+} from './production-cache.file-holder-worker';
+
+// A page that leaves while it owns the cache terminates its engine worker. The
+// browser can hand the owner Web Lock to the next engine before that worker
+// lets go of the database files. This harness recreates that gap with a worker
+// that keeps the files open without the lock.
+
+const resultElement = document.querySelector<HTMLElement>('#result');
+if (!resultElement) throw new Error('missing result element');
+
+type CommandWithoutId = ProductionHarnessCommand extends infer Command
+ ? Command extends ProductionHarnessCommand
+ ? Omit<Command, 'commandId'>
+ : never
+ : never;
+
+type RuntimeTelemetry = {
+ kind: string;
+ tabId: string;
+ ownerEpoch: number;
+ databaseAction: 'open-existing' | 'wipe-before-open';
+ receivedAt: number;
+};
+
+/** Longer than both the retry this change replaced and the gap seen in
+ * production, and well inside the engine's busy-file wait. */
+const HOLD_AFTER_OWNER_LOCK_MS = 3_000;
+
+const assert: (condition: unknown, message: string) => asserts condition = (
+ condition,
+ message
+) => {
+ if (!condition) throw new Error(message);
+};
+
+const sleep = (ms: number) =>
+ new Promise<void>((resolve) => setTimeout(resolve, ms));
+
+const waitUntil = async (
+ description: string,
+ predicate: () => boolean | Promise<boolean>,
+ timeoutMs = 30_000
+): Promise<void> => {
+ const started = performance.now();
+ while (!(await predicate())) {
+ if (performance.now() - started > timeoutMs) {
+ throw new Error(`timed out waiting for ${description}`);
+ }
+ await sleep(20);
+ }
+};
+
+const run = async (): Promise<Record<string, unknown>> => {
+ const scope = `busy-files-${crypto.randomUUID()}`;
+ const identity = cacheDatabaseIdentity(scope);
+ const ownerLockName = databaseOwnerLockName(scope);
+ const tabChannel = new BroadcastChannel(
+ `graphql-cache-wp08-production-tabs:${scope}`
+ );
+ const telemetryChannel = new BroadcastChannel(
+ `graphql-cache-wp08-production:${scope}`
+ );
+ const popups = new Map<string, Window>();
+ const registered = new Set<string>();
+ const workers: Array<{ tabId: string; ownerEpoch: number }> = [];
+ const terminated: Array<{ ownerEpoch: number; reason: string }> = [];
+ const unavailable = new Map<string, string>();
+ const telemetry: RuntimeTelemetry[] = [];
+ const protocolErrors: string[] = [];
+ const pendingCommands = new Map<
+ string,
+ {
+ resolve: (value: unknown) => void;
+ reject: (error: Error) => void;
+ timer: ReturnType<typeof setTimeout>;
+ }
+ >();
+
+ tabChannel.onmessage = (event: MessageEvent<ProductionHarnessEnvelope>) => {
+ const message = event.data;
+ if (message.source !== 'tab') return;
+ switch (message.event.kind) {
+ case 'registered':
+ registered.add(message.tabId);
+ break;
+ case 'worker-created':
+ workers.push({
+ tabId: message.tabId,
+ ownerEpoch: message.event.ownerEpoch,
+ });
+ break;
+ case 'worker-terminated':
+ terminated.push({
+ ownerEpoch: message.event.ownerEpoch,
+ reason: message.event.reason,
+ });
+ break;
+ case 'cache-unavailable':
+ unavailable.set(message.tabId, message.event.reason);
+ break;
+ case 'protocol-error':
+ protocolErrors.push(message.event.error);
+ break;
+ case 'command-result': {
+ const pending = pendingCommands.get(message.event.commandId);
+ if (!pending) return;
+ pendingCommands.delete(message.event.commandId);
+ clearTimeout(pending.timer);
+ if (message.event.ok) pending.resolve(message.event.result);
+ else pending.reject(new Error(message.event.error));
+ break;
+ }
+ }
+ };
+ telemetryChannel.onmessage = (
+ event: MessageEvent<Omit<RuntimeTelemetry, 'receivedAt'>>
+ ) => {
+ telemetry.push({ ...event.data, receivedAt: performance.now() });
+ };
+
+ const command = async (
+ targetTabId: string,
+ value: CommandWithoutId
+ ): Promise<unknown> => {
+ const commandId = crypto.randomUUID();
+ const result = new Promise<unknown>((resolve, reject) => {
+ const timer = setTimeout(() => {
+ pendingCommands.delete(commandId);
+ reject(new Error(`busy-file command timed out: ${value.kind}`));
+ }, 30_000);
+ pendingCommands.set(commandId, { resolve, reject, timer });
+ });
+ tabChannel.postMessage({
+ source: 'harness',
+ targetTabId,
+ command: { ...value, commandId } as ProductionHarnessCommand,
+ } satisfies ProductionHarnessEnvelope);
+ return await result;
+ };
+
+ const openTab = (tabId: string): void => {
+ const url = new URL('./production-tab.html', location.href);
+ url.searchParams.set('scope', scope);
+ url.searchParams.set('tabId', tabId);
+ const popup = window.open(url, `${scope}:${tabId}`);
+ if (!popup) throw new Error(`popup blocked for ${tabId}`);
+ popups.set(tabId, popup);
+ };
+
+ const ownerOf = (ownerEpoch: number) => {
+ const owner = workers.find((worker) => worker.ownerEpoch === ownerEpoch);
+ assert(owner, `missing owner for epoch ${ownerEpoch}`);
+ return owner;
+ };
+
+ const event = (kind: string, ownerEpoch: number) =>
+ telemetry.find(
+ (candidate) =>
+ candidate.kind === kind && candidate.ownerEpoch === ownerEpoch
+ );
+
+ const heldOwnerLocks = async (): Promise<number> =>
+ (await navigator.locks.query()).held?.filter(
+ (lock) => lock.name === ownerLockName
+ ).length ?? 0;
+
+ const holder = new Worker(
+ new URL('./production-cache.file-holder-worker.ts', import.meta.url),
+ { type: 'module', name: `busy-file-holder:${scope}` }
+ );
+ const askHolder = (request: FileHolderRequest): Promise<void> =>
+ new Promise((resolve, reject) => {
+ holder.onmessage = (message: MessageEvent<FileHolderResponse>) => {
+ if (message.data.kind === 'error')
+ reject(new Error(message.data.error));
+ else resolve();
+ };
+ holder.postMessage(request);
+ });
+
+ /**
+ * The owner leaves like a navigating page. While the harness holds the
+ * owner lock the holder opens both files, then the next engine may take the
+ * lock and finds the files busy. Resolves when that engine holds the lock.
+ */
+ const leaveWhileFilesStayOpen = async (
+ ownerEpoch: number
+ ): Promise<number> => {
+ let releaseLock!: () => void;
+ let markAcquired!: () => void;
+ const acquired = new Promise<void>((resolve) => {
+ markAcquired = resolve;
+ });
+ const lockReleased = navigator.locks.request(
+ ownerLockName,
+ { mode: 'exclusive' },
+ async () => {
+ markAcquired();
+ await new Promise<void>((resolve) => {
+ releaseLock = resolve;
+ });
+ }
+ );
+ await command(ownerOf(ownerEpoch).tabId, { kind: 'navigate-away' });
+ await acquired;
+ await askHolder({ kind: 'hold', paths: [identity, `${identity}-wal`] });
+ releaseLock();
+ await lockReleased;
+ await waitUntil(
+ `epoch ${ownerEpoch + 1} to take the owner lock`,
+ async () => (await heldOwnerLocks()) === 1
+ );
+ return performance.now();
+ };
+
+ for (const tabId of ['busy-a', 'busy-b', 'busy-c']) openTab(tabId);
+ await waitUntil('three registrations', () => registered.size === 3);
+ await waitUntil('epoch 1 ready', () => event('ready', 1) !== undefined);
+ const standby = [...registered].find((tabId) => tabId !== ownerOf(1).tabId);
+ assert(standby, 'missing standby');
+ await command(standby, { kind: 'write', value: 'busy-preserved' });
+
+ // Phase 1: the files stay open for 3 seconds after the next engine takes
+ // the owner lock. It must wait and reopen the same database.
+ const firstLockedAt = await leaveWhileFilesStayOpen(1);
+ assert(
+ event('activation-started', 2)?.databaseAction === 'open-existing',
+ 'navigation replacement did not start with open-existing'
+ );
+ await sleep(HOLD_AFTER_OWNER_LOCK_MS);
+ assert(!event('ready', 2), 'epoch 2 opened while its files were busy');
+ await askHolder({ kind: 'release' });
+ const firstReleasedAt = performance.now();
+ await waitUntil('epoch 2 ready', () => event('ready', 2) !== undefined);
+ const secondOwner = ownerOf(2).tabId;
+ expectHit(await command(secondOwner, { kind: 'read' }), 'busy-preserved');
+ const firstRecoveryAttempts = telemetry.filter(
+ (candidate) =>
+ candidate.kind === 'activation-started' && candidate.ownerEpoch > 2
+ ).length;
+
+ // Phase 2: the files stay open past the engine's whole wait. The cache
+ // must turn off quietly without wiping, and a later tab reopens the data.
+ const survivor = [...registered].find(
+ (tabId) => tabId !== ownerOf(1).tabId && tabId !== secondOwner
+ );
+ assert(survivor, 'missing surviving tab');
+ await leaveWhileFilesStayOpen(2);
+ await waitUntil(
+ 'the surviving tab to be told the cache is unavailable',
+ () => unavailable.has(survivor),
+ 30_000
+ );
+ let survivorRead = 'hit';
+ try {
+ await command(survivor, { kind: 'read' });
+ } catch (error) {
+ survivorRead = error instanceof Error ? error.message : String(error);
+ }
+ await sleep(1_000);
+ const retriedWhileBusy = telemetry.some(
+ (candidate) =>
+ candidate.kind === 'activation-started' && candidate.ownerEpoch > 3
+ );
+ await askHolder({ kind: 'release' });
+ openTab('busy-d');
+ await waitUntil('epoch 4 ready', () => event('ready', 4) !== undefined);
+ expectHit(
+ await command(ownerOf(4).tabId, { kind: 'read' }),
+ 'busy-preserved'
+ );
+
+ const result = {
+ passed: true,
+ firstReplacementDatabaseAction: event('activation-started', 2)
+ ?.databaseAction,
+ firstReplacementWaitedForFiles:
+ (event('ready', 2)?.receivedAt ?? 0) >= firstReleasedAt &&
+ firstReleasedAt - firstLockedAt >= HOLD_AFTER_OWNER_LOCK_MS,
+ firstReplacementKeptData: true,
+ firstRecoveryAttempts,
+ busyReplacementDatabaseAction: event('activation-started', 3)
+ ?.databaseAction,
+ busyReplacementOpened: event('ready', 3) !== undefined,
+ busyReplacementTerminated: terminated.some(
+ (candidate) =>
+ candidate.ownerEpoch === 3 &&
+ candidate.reason.includes('files stayed busy')
+ ),
+ survivorToldFilesAreOpen:
+ unavailable.get(survivor)?.includes('files are still open') ?? false,
+ survivorReadFellBack: survivorRead.includes('files are still open'),
+ retriedWhileBusy,
+ laterTabDatabaseAction: event('activation-started', 4)?.databaseAction,
+ laterTabKeptData: true,
+ protocolErrors,
+ };
+
+ holder.terminate();
+ for (const popup of popups.values()) popup.close();
+ tabChannel.close();
+ telemetryChannel.close();
+ return result;
+};
+
+function expectHit(value: unknown, expected: string): void {
+ assert(
+ typeof value === 'object' &&
+ value !== null &&
+ (value as { kind?: unknown }).kind === 'hit' &&
+ (
+ value as {
+ data?: { user?: { soup?: { items?: Array<{ id?: unknown }> } } };
+ }
+ ).data?.user?.soup?.items?.[0]?.id === expected,
+ `expected cache hit ${expected}`
+ );
+}
+
+void (async () => {
+ try {
+ const result = await run();
+ resultElement.dataset.status = 'passed';
+ resultElement.textContent = JSON.stringify(result, null, 2);
+ } catch (error) {
+ resultElement.dataset.status = 'failed';
+ // Firefox stacks leave out the message.
+ resultElement.textContent =
+ error instanceof Error
+ ? `${error.message}\n${error.stack ?? ''}`
+ : String(error);
+ }
+})();
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy.html b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy.html
new file mode 100644
index 0000000000..e82d0f2e1b
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-busy.html
@@ -0,0 +1,12 @@
+<!doctype html>
+<html lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <link rel="icon" href="data:," />
+ <title>Production Turso busy-file harness</title>
+ </head>
+ <body>
+ <pre id="result" data-status="running">running</pre>
+ <script type="module" src="./production-busy-harness.ts"></script>
+ </body>
+</html>
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts
index 65298ffbf3..0432637d84 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.engine-worker.ts
@@ -50,6 +50,7 @@ const report = (event: CacheEngineRuntimeEvent): void => {
event.request.kind === 'read' &&
event.request.query.includes('CacheAdmissionBarrier'),
reason: event.kind === 'fatal' ? event.reason : undefined,
+ attempt: event.kind === 'owner-lock-busy' ? event.attempt : undefined,
fatalCode: event.kind === 'fatal' ? event.fatalCode : undefined,
performanceMemoryAvailable:
event.kind === 'ready' && workerPerformance.memory !== undefined,
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.file-holder-worker.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.file-holder-worker.ts
new file mode 100644
index 0000000000..2d7d666a2d
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-cache.file-holder-worker.ts
@@ -0,0 +1,64 @@
+/// <reference lib="webworker" />
+
+/**
+ * Keeps database files open without their owner Web Lock, standing in for a
+ * page's cache worker that is still being torn down after the page left.
+ */
+export type FileHolderRequest =
+ | { kind: 'hold'; paths: string[] }
+ | { kind: 'release' };
+
+export type FileHolderResponse =
+ | { kind: 'holding' }
+ | { kind: 'released' }
+ | { kind: 'error'; error: string };
+
+const worker = self as unknown as DedicatedWorkerGlobalScope;
+let handles: FileSystemSyncAccessHandle[] = [];
+
+const isBusy = (error: unknown): boolean =>
+ error instanceof DOMException && error.name === 'NoModificationAllowedError';
+
+/** The terminated owner may still be letting go of the file. */
+const openWhenFree = async (
+ root: FileSystemDirectoryHandle,
+ path: string
+): Promise<FileSystemSyncAccessHandle> => {
+ const file = await root.getFileHandle(path);
+ for (let attempt = 1; ; attempt += 1) {
+ try {
+ return await file.createSyncAccessHandle();
+ } catch (error) {
+ if (!isBusy(error) || attempt >= 500) throw error;
+ await new Promise((resolve) => setTimeout(resolve, 10));
+ }
+ }
+};
+
+const handle = async (
+ request: FileHolderRequest
+): Promise<FileHolderResponse> => {
+ if (request.kind === 'hold') {
+ const root = await navigator.storage.getDirectory();
+ for (const path of request.paths) {
+ handles.push(await openWhenFree(root, path));
+ }
+ return { kind: 'holding' };
+ }
+ for (const held of handles) held.close();
+ handles = [];
+ return { kind: 'released' };
+};
+
+worker.onmessage = async (event: MessageEvent<FileHolderRequest>) => {
+ let response: FileHolderResponse;
+ try {
+ response = await handle(event.data);
+ } catch (error) {
+ response = {
+ kind: 'error',
+ error: error instanceof Error ? error.message : String(error),
+ };
+ }
+ worker.postMessage(response);
+};
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts
index bd9af4ece7..6b2bfd993b 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-harness.ts
@@ -155,10 +155,12 @@ const run = async (): Promise<Record<string, unknown>> => {
);
return { acquired, release, completed };
};
- const waitForPendingReplacementLock = async (
+ // The replacement retries the contender's lock without queueing behind it.
+ const waitForRetriedReplacementLock = async (
+ ownerEpoch: number,
description: string
- ): Promise<{ held: number; pending: number }> => {
- let evidence = { held: 0, pending: 0 };
+ ): Promise<{ held: number; pending: number; busyAttempts: number }> => {
+ let evidence = { held: 0, pending: 0, busyAttempts: 0 };
await waitUntil(description, async () => {
const snapshot = await navigator.locks.query();
evidence = {
@@ -168,8 +170,16 @@ const run = async (): Promise<Record<string, unknown>> => {
pending:
snapshot.pending?.filter((lock) => lock.name === ownerLockName)
.length ?? 0,
+ busyAttempts: telemetry.filter(
+ (event) =>
+ event.kind === 'owner-lock-busy' && event.ownerEpoch === ownerEpoch
+ ).length,
};
- return evidence.held === 1 && evidence.pending === 1;
+ return (
+ evidence.held === 1 &&
+ evidence.pending === 0 &&
+ evidence.busyAttempts > 0
+ );
});
return evidence;
};
@@ -215,8 +225,9 @@ const run = async (): Promise<Record<string, unknown>> => {
(event) => event.kind === 'activation-started' && event.ownerEpoch === 2
)
);
- const gracefulLockEvidence = await waitForPendingReplacementLock(
- 'epoch 2 exact owner lock request to become pending'
+ const gracefulLockEvidence = await waitForRetriedReplacementLock(
+ 2,
+ 'epoch 2 to retry the held owner lock'
);
const gracefulWaitedForPhysicalLock = !telemetry.some(
(event) => event.kind === 'ready' && event.ownerEpoch === 2
@@ -277,8 +288,9 @@ const run = async (): Promise<Record<string, unknown>> => {
(event) => event.kind === 'activation-started' && event.ownerEpoch === 3
)
);
- const recoveryLockEvidence = await waitForPendingReplacementLock(
- 'epoch 3 exact owner lock request to become pending'
+ const recoveryLockEvidence = await waitForRetriedReplacementLock(
+ 3,
+ 'epoch 3 to retry the held owner lock'
);
const recoveryWaitedForPhysicalLock = !telemetry.some(
(event) => event.kind === 'ready' && event.ownerEpoch === 3
@@ -318,12 +330,14 @@ const run = async (): Promise<Record<string, unknown>> => {
(event) => event.kind === 'drained' && event.ownerEpoch === 1
),
gracefulReplacementWaitedForPhysicalLock: gracefulWaitedForPhysicalLock,
- gracefulPendingOwnerLockRequests: gracefulLockEvidence.pending,
+ gracefulQueuedOwnerLockRequests: gracefulLockEvidence.pending,
+ gracefulRetriedHeldOwnerLock: gracefulLockEvidence.busyAttempts > 0,
abruptInflightRejected: abruptError.includes('owner epoch 2 was lost'),
abruptRequestReplayCount: slowAdmissions.length,
abruptOwnerPageStayedAlive: !secondPage.closed,
recoveryReplacementWaitedForPhysicalLock: recoveryWaitedForPhysicalLock,
- recoveryPendingOwnerLockRequests: recoveryLockEvidence.pending,
+ recoveryQueuedOwnerLockRequests: recoveryLockEvidence.pending,
+ recoveryRetriedHeldOwnerLock: recoveryLockEvidence.busyAttempts > 0,
atomicRecoveryOpenWipedToMiss: true,
recoveryDatabaseAction: telemetry.find(
(event) => event.kind === 'ready' && event.ownerEpoch === 3
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts
index 1b90e367d5..5e81d4520d 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-tab.ts
@@ -13,6 +13,10 @@ const parameters = new URLSearchParams(location.search);
const tabId = parameters.get('tabId') ?? '';
const scope = parameters.get('scope') ?? '';
if (!tabId || !scope) throw new Error('missing production harness parameters');
+// A named build gets its own coordinator, as each deployed build does, and
+// reports its build time so the newer build can take the database over.
+const build = parameters.get('build');
+const buildTime = Number(parameters.get('buildTime') ?? 0);
const QUERY = `query Soup($input: SoupInput!) {
user {
@@ -49,6 +53,19 @@ const pending = new Map<number, (response: CacheResponse) => void>();
const adapter = createCacheCoordinatorPageAdapter({
scope,
tabId,
+ buildTime,
+ ...(build
+ ? {
+ createSharedWorker: (workerScope: string) =>
+ new SharedWorker(
+ new URL('../cache.coordinator.shared-worker.ts', import.meta.url),
+ {
+ type: 'module',
+ name: `graphql-cache-coordinator:${workerScope}:build-${build}`,
+ }
+ ),
+ }
+ : {}),
createDedicatedWorker: (_workerScope, ownerEpoch) =>
new Worker(
new URL('./production-cache.engine-worker.ts', import.meta.url),
@@ -68,6 +85,15 @@ const adapter = createCacheCoordinatorPageAdapter({
onProtocolError: (error) => {
report({ kind: 'protocol-error', error: error.message });
},
+ onCacheUnavailable: (reason) => {
+ report({ kind: 'cache-unavailable', reason });
+ },
+ onCacheSuperseded: (reason) => {
+ report({ kind: 'cache-superseded', reason });
+ // What the app's host does: leave like a navigating page, which stops
+ // this tab's engine so the newer build can open the database.
+ void adapter.dispose({ graceful: false, preserveDatabase: true });
+ },
});
adapter.onmessage = (event) => {
@@ -203,6 +229,17 @@ const handleCommand = (command: ProductionHarnessCommand): void => {
currentWorker.terminate();
report({ kind: 'command-result', commandId: value.commandId, ok: true });
})
+ .with({ kind: 'navigate-away' }, (value) => {
+ void (async () => {
+ await adapter.dispose({ graceful: false, preserveDatabase: true });
+ report({
+ kind: 'command-result',
+ commandId: value.commandId,
+ ok: true,
+ });
+ setTimeout(() => window.close());
+ })();
+ })
.exhaustive();
};
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover-harness.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover-harness.ts
new file mode 100644
index 0000000000..ec173c4ac9
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover-harness.ts
@@ -0,0 +1,239 @@
+import type {
+ ProductionHarnessCommand,
+ ProductionHarnessEnvelope,
+} from './production-browser-wire';
+
+// Tabs of three simulated builds share one scope. Each build runs its own
+// coordinator over the one database, as deployed builds do. The newer build
+// must take the database over without wiping it, and send the older build's
+// tabs on; an even older build must be turned away.
+
+const resultElement = document.querySelector<HTMLElement>('#result');
+if (!resultElement) throw new Error('missing result element');
+
+type CommandWithoutId = ProductionHarnessCommand extends infer Command
+ ? Command extends ProductionHarnessCommand
+ ? Omit<Command, 'commandId'>
+ : never
+ : never;
+
+type RuntimeTelemetry = {
+ kind: string;
+ tabId: string;
+ ownerEpoch: number;
+ databaseAction: 'open-existing' | 'wipe-before-open';
+};
+
+const BUILDS = { older: 500, old: 1_000, new: 2_000 } as const;
+
+const assert: (condition: unknown, message: string) => asserts condition = (
+ condition,
+ message
+) => {
+ if (!condition) throw new Error(message);
+};
+
+const waitUntil = async (
+ description: string,
+ predicate: () => boolean,
+ timeoutMs = 30_000
+): Promise<void> => {
+ const started = performance.now();
+ while (!predicate()) {
+ if (performance.now() - started > timeoutMs) {
+ throw new Error(`timed out waiting for ${description}`);
+ }
+ await new Promise<void>((resolve) => setTimeout(resolve, 20));
+ }
+};
+
+const run = async (): Promise<Record<string, unknown>> => {
+ const scope = `takeover-${crypto.randomUUID()}`;
+ const tabChannel = new BroadcastChannel(
+ `graphql-cache-wp08-production-tabs:${scope}`
+ );
+ const telemetryChannel = new BroadcastChannel(
+ `graphql-cache-wp08-production:${scope}`
+ );
+ const popups = new Map<string, Window>();
+ const registered = new Set<string>();
+ const superseded = new Map<string, string>();
+ const unavailable = new Map<string, string>();
+ const terminated: Array<{ tabId: string; reason: string }> = [];
+ const telemetry: RuntimeTelemetry[] = [];
+ const protocolErrors: string[] = [];
+ const pendingCommands = new Map<
+ string,
+ {
+ resolve: (value: unknown) => void;
+ reject: (error: Error) => void;
+ timer: ReturnType<typeof setTimeout>;
+ }
+ >();
+
+ tabChannel.onmessage = (event: MessageEvent<ProductionHarnessEnvelope>) => {
+ const message = event.data;
+ if (message.source !== 'tab') return;
+ switch (message.event.kind) {
+ case 'registered':
+ registered.add(message.tabId);
+ break;
+ case 'cache-superseded':
+ superseded.set(message.tabId, message.event.reason);
+ break;
+ case 'cache-unavailable':
+ unavailable.set(message.tabId, message.event.reason);
+ break;
+ case 'worker-terminated':
+ terminated.push({
+ tabId: message.tabId,
+ reason: message.event.reason,
+ });
+ break;
+ case 'protocol-error':
+ protocolErrors.push(message.event.error);
+ break;
+ case 'command-result': {
+ const pending = pendingCommands.get(message.event.commandId);
+ if (!pending) return;
+ pendingCommands.delete(message.event.commandId);
+ clearTimeout(pending.timer);
+ if (message.event.ok) pending.resolve(message.event.result);
+ else pending.reject(new Error(message.event.error));
+ break;
+ }
+ }
+ };
+ telemetryChannel.onmessage = (event: MessageEvent<RuntimeTelemetry>) => {
+ telemetry.push(event.data);
+ };
+
+ const command = async (
+ targetTabId: string,
+ value: CommandWithoutId
+ ): Promise<unknown> => {
+ const commandId = crypto.randomUUID();
+ const result = new Promise<unknown>((resolve, reject) => {
+ const timer = setTimeout(() => {
+ pendingCommands.delete(commandId);
+ reject(new Error(`takeover command timed out: ${value.kind}`));
+ }, 30_000);
+ pendingCommands.set(commandId, { resolve, reject, timer });
+ });
+ tabChannel.postMessage({
+ source: 'harness',
+ targetTabId,
+ command: { ...value, commandId } as ProductionHarnessCommand,
+ } satisfies ProductionHarnessEnvelope);
+ return await result;
+ };
+
+ const openTab = (tabId: string, build: keyof typeof BUILDS): void => {
+ const url = new URL('./production-tab.html', location.href);
+ url.searchParams.set('scope', scope);
+ url.searchParams.set('tabId', tabId);
+ url.searchParams.set('build', build);
+ url.searchParams.set('buildTime', String(BUILDS[build]));
+ const popup = window.open(url, `${scope}:${tabId}`);
+ if (!popup) throw new Error(`popup blocked for ${tabId}`);
+ popups.set(tabId, popup);
+ };
+
+ const readyFor = (tabId: string) =>
+ telemetry.find((event) => event.kind === 'ready' && event.tabId === tabId);
+
+ // The old build owns the database and holds data.
+ openTab('old-a', 'old');
+ openTab('old-b', 'old');
+ await waitUntil('both old-build tabs to register', () =>
+ ['old-a', 'old-b'].every((tabId) => registered.has(tabId))
+ );
+ await waitUntil('the old build to open the database', () =>
+ ['old-a', 'old-b'].some((tabId) => readyFor(tabId))
+ );
+ const oldOwner = ['old-a', 'old-b'].find((tabId) => readyFor(tabId));
+ assert(oldOwner, 'missing old-build owner');
+ await command('old-b', { kind: 'write', value: 'takeover-preserved' });
+
+ // A tab of the newer build opens and takes the database over.
+ openTab('new-a', 'new');
+ await waitUntil('the new build to open the database', () =>
+ Boolean(readyFor('new-a'))
+ );
+ expectHit(await command('new-a', { kind: 'read' }), 'takeover-preserved');
+ await waitUntil('both old-build tabs to be sent on', () =>
+ ['old-a', 'old-b'].every((tabId) => superseded.has(tabId))
+ );
+
+ // A tab of an even older build is turned away; the new build keeps it.
+ openTab('older-a', 'older');
+ await waitUntil('the older-build tab to be told to run uncached', () =>
+ unavailable.has('older-a')
+ );
+ const olderBuildSentOn = superseded.has('older-a');
+ // A page of the old build that loads after the handover, as after a
+ // rollback, runs uncached rather than reloading, which could land on the
+ // old build again. Its coordinator either was superseded or, once the old
+ // tabs let it go, starts fresh and is turned away by the newer build.
+ openTab('old-c', 'old');
+ await waitUntil('the late old-build tab to be told to run uncached', () =>
+ unavailable.has('old-c')
+ );
+ expectHit(await command('new-a', { kind: 'read' }), 'takeover-preserved');
+
+ const result = {
+ passed: true,
+ newBuildDatabaseAction: telemetry.find(
+ (event) => event.kind === 'activation-started' && event.tabId === 'new-a'
+ )?.databaseAction,
+ newBuildKeptData: true,
+ // The old owner stopped its engine the way a navigating page does.
+ oldEngineStoppedLikeNavigation:
+ terminated.find((event) => event.tabId === oldOwner)?.reason ===
+ 'page navigation',
+ oldTabsSentOn: ['old-a', 'old-b'].every((tabId) =>
+ superseded.get(tabId)?.includes('newer version of the app')
+ ),
+ olderBuildTurnedAway:
+ unavailable.get('older-a')?.includes('owner lock is held') ?? false,
+ olderBuildSentOn,
+ lateOldTabRunsUncached: unavailable.has('old-c'),
+ lateOldTabSentOn: superseded.has('old-c'),
+ newBuildSentOn: superseded.has('new-a'),
+ protocolErrors,
+ };
+
+ for (const popup of popups.values()) popup.close();
+ tabChannel.close();
+ telemetryChannel.close();
+ return result;
+};
+
+function expectHit(value: unknown, expected: string): void {
+ assert(
+ typeof value === 'object' &&
+ value !== null &&
+ (value as { kind?: unknown }).kind === 'hit' &&
+ (
+ value as {
+ data?: { user?: { soup?: { items?: Array<{ id?: unknown }> } } };
+ }
+ ).data?.user?.soup?.items?.[0]?.id === expected,
+ `expected cache hit ${expected}`
+ );
+}
+
+void (async () => {
+ try {
+ const result = await run();
+ resultElement.dataset.status = 'passed';
+ resultElement.textContent = JSON.stringify(result, null, 2);
+ } catch (error) {
+ resultElement.dataset.status = 'failed';
+ // Firefox stacks leave out the message.
+ resultElement.textContent =
+ error instanceof Error
+ ? `${error.message}\n${error.stack ?? ''}`
+ : String(error);
+ }
+})();
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover.html b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover.html
new file mode 100644
index 0000000000..61a563b1c9
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/production-takeover.html
@@ -0,0 +1,12 @@
+<!doctype html>
+<html lang="en">
+ <head>
+ <meta charset="UTF-8" />
+ <link rel="icon" href="data:," />
+ <title>Production Turso takeover harness</title>
+ </head>
+ <body>
+ <pre id="result" data-status="running">running</pre>
+ <script type="module" src="./production-takeover-harness.ts"></script>
+ </body>
+</html>
diff --git a/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts b/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts
index 1a616ded5d..e4df0f10fd 100644
--- a/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts
+++ b/apps/web/src/lib/graphql-cache/worker/browser-test/vite.config.ts
@@ -42,6 +42,8 @@ export default defineConfig(({ command }) => ({
'host.html',
'cutover.html',
'production.html',
+ 'production-busy.html',
+ 'production-takeover.html',
'production-tab.html',
'tab.html',
'performance.html',
diff --git a/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts b/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts
index bd6ff68338..f362664c75 100644
--- a/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/cache-coordinator-runtime.test.ts
@@ -44,7 +44,7 @@ class FakeEndpoint extends EventTarget {
}
const message = {
- coordinatorVersion: 5,
+ coordinatorVersion: 6,
kind: 'disconnect-tab',
tabId: 'tab-a',
reason: 'test',
@@ -90,7 +90,7 @@ describe('cache coordinator runtime', () => {
expect(() =>
ports[0]?.postMessage({
- coordinatorVersion: 5,
+ coordinatorVersion: 6,
kind: 'protocol-error',
error: 'test',
} satisfies CoordinatorToTabEnvelope)
diff --git a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts
index 119f8169d6..e84f3f9046 100644
--- a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.test.ts
@@ -3,10 +3,11 @@ import type { CacheRequest } from '../protocol';
import { INITIAL_CACHE_REVISION } from '../protocol';
import {
type CacheEngineRuntimeOptions,
- installCacheEngineWorker,
+ installCacheEngineWorker as installRuntime,
} from './cache-engine-runtime';
import {
CACHE_COORDINATOR_PROTOCOL_VERSION,
+ cacheDatabaseIdentity,
databaseOwnerLockName,
type PageToEngineEnvelope,
} from './coordinator-protocol';
@@ -18,6 +19,8 @@ import {
EFFECT_WORKER_REQUEST_TAG,
EFFECT_WORKER_RESPONSE_TAG,
} from './effect-worker-transport';
+import { OWNER_LOCK_RETRY_DELAYS_MS } from './startup';
+import type { CacheWasmModule } from './wasm-module';
import type { CacheWorkerCoreOptions } from './worker-core';
class FakePort extends EventTarget {
@@ -33,7 +36,7 @@ class FakePort extends EventTarget {
const payload = effectPayload(message) as
| { kind?: string; ownerEpoch?: number }
| undefined;
- if (this.autoOpen && payload?.kind === 'engine-assets-ready') {
+ if (this.autoOpen && payload?.kind === 'owner-lock-acquired') {
queueMicrotask(() =>
this.receive({
...version,
@@ -83,6 +86,33 @@ const version = {
coordinatorVersion: CACHE_COORDINATOR_PROTOCOL_VERSION,
} as const;
+type RuntimeCore = ReturnType<
+ NonNullable<CacheEngineRuntimeOptions['createCore']>
+>;
+
+/** Like CacheWorkerCore over WASM: initialization holds the owner lock and
+ * asks for the storage grant before its first storage access. */
+const installCacheEngineWorker = (options: CacheEngineRuntimeOptions): void => {
+ const createCore = options.createCore;
+ installRuntime({
+ ...options,
+ createCore:
+ createCore &&
+ ((coreOptions: CacheWorkerCoreOptions): RuntimeCore => {
+ const core = createCore(coreOptions);
+ return {
+ ...core,
+ handleRequest: async (port, request) => {
+ if (request.kind === 'init') {
+ await coreOptions.onOwnerLockAcquired?.();
+ }
+ return await core.handleRequest(port, request);
+ },
+ };
+ }),
+ });
+};
+
const activation = (
databaseAction: 'open-existing' | 'wipe-before-open' = 'open-existing'
): PageToEngineEnvelope => ({
@@ -122,6 +152,7 @@ const registerTab = async (
scope: 'scope',
tabId,
livenessLockName: `graphql-cache-tab:scope:${tabId}`,
+ buildTime: 0,
});
};
@@ -184,7 +215,7 @@ const cacheResponses = (port: FakePort) =>
);
describe('cache engine worker runtime', () => {
- it('does not touch storage until assets load and the coordinator grants opening', async () => {
+ it('does not touch storage until it holds the owner lock and the coordinator grants opening', async () => {
const scope = new FakeWorkerScope();
const direct = new FakePort();
direct.autoOpen = false;
@@ -214,8 +245,9 @@ describe('cache engine worker runtime', () => {
expect(messagesOfKind(direct, 'engine-assets-ready')).toHaveLength(0);
finishLoading();
await vi.waitFor(() =>
- expect(messagesOfKind(direct, 'engine-assets-ready')).toHaveLength(1)
+ expect(messagesOfKind(direct, 'owner-lock-acquired')).toHaveLength(1)
);
+ expect(messagesOfKind(direct, 'engine-assets-ready')).toHaveLength(1);
expect(handleRequest).not.toHaveBeenCalled();
direct.receive({ ...version, kind: 'open-engine', ownerEpoch: 7 });
await vi.waitFor(() =>
@@ -224,6 +256,31 @@ describe('cache engine worker runtime', () => {
expect(handleRequest).toHaveBeenCalledOnce();
});
+ it('fatals on a storage grant before it holds the owner lock', async () => {
+ const scope = new FakeWorkerScope();
+ const direct = new FakePort();
+ direct.autoOpen = false;
+ const handleRequest = vi.fn();
+ installCacheEngineWorker({
+ scope,
+ createCore: () => ({
+ prepare: () => new Promise<void>(() => undefined),
+ handleRequest,
+ addPort: vi.fn(),
+ drain: vi.fn(),
+ }),
+ });
+ scope.activate(activation(), direct);
+ direct.receive({ ...version, kind: 'open-engine', ownerEpoch: 7 });
+
+ await vi.waitFor(() =>
+ expect(messagesOfKind(direct, 'engine-fatal')).toEqual([
+ expect.objectContaining({ reason: 'unexpected database-open grant' }),
+ ])
+ );
+ expect(handleRequest).not.toHaveBeenCalled();
+ });
+
it('reports failed asset loading without initializing or opening storage', async () => {
const scope = new FakeWorkerScope();
const direct = new FakePort();
@@ -248,6 +305,171 @@ describe('cache engine worker runtime', () => {
expect(messagesOfKind(direct, 'engine-ready')).toHaveLength(0);
});
+ it('retries a busy owner lock, reporting each attempt, and opens once it frees', async () => {
+ const scope = new FakeWorkerScope();
+ const direct = new FakePort();
+ const sleep = vi.fn(async (_delayMs: number) => undefined);
+ installRuntime({
+ scope,
+ sleep,
+ ownerLockIsHeld: async () => true,
+ createCore: (coreOptions) => ({
+ handleRequest: async (port, request) => {
+ if (request.kind === 'init') {
+ for (const attempt of [1, 2]) {
+ await coreOptions.onOwnerLockBusy?.(attempt);
+ }
+ await coreOptions.onOwnerLockAcquired?.();
+ }
+ port.postMessage({ id: request.id, ok: true, result: null });
+ },
+ addPort: vi.fn(),
+ drain: vi.fn(),
+ }),
+ });
+ scope.activate(activation(), direct);
+
+ await vi.waitFor(() =>
+ expect(messagesOfKind(direct, 'engine-ready')).toHaveLength(1)
+ );
+ expect(messagesOfKind(direct, 'owner-lock-busy')).toEqual([
+ expect.objectContaining({ tabId: 'tab-a', ownerEpoch: 7 }),
+ expect.objectContaining({ tabId: 'tab-a', ownerEpoch: 7 }),
+ ]);
+ expect(sleep.mock.calls.map(([delayMs]) => delayMs)).toEqual(
+ OWNER_LOCK_RETRY_DELAYS_MS.slice(0, 2)
+ );
+ });
+
+ it('gives up on an owner lock that stays busy without touching storage', async () => {
+ const scope = new FakeWorkerScope();
+ const direct = new FakePort();
+ installRuntime({
+ scope,
+ sleep: async () => undefined,
+ createCore: (coreOptions) => ({
+ handleRequest: async (port, request) => {
+ try {
+ for (let attempt = 1; ; attempt += 1) {
+ await coreOptions.onOwnerLockBusy?.(attempt);
+ }
+ } catch (error) {
+ port.postMessage({
+ id: request.id,
+ ok: false,
+ error: (error as Error).message,
+ });
+ }
+ },
+ addPort: vi.fn(),
+ drain: vi.fn(),
+ }),
+ });
+ scope.activate(activation(), direct);
+
+ await vi.waitFor(() =>
+ expect(messagesOfKind(direct, 'owner-lock-unavailable')).toEqual([
+ {
+ ...version,
+ kind: 'owner-lock-unavailable',
+ tabId: 'tab-a',
+ ownerEpoch: 7,
+ },
+ ])
+ );
+ expect(messagesOfKind(direct, 'owner-lock-busy')).toHaveLength(
+ OWNER_LOCK_RETRY_DELAYS_MS.length + 1
+ );
+ expect(messagesOfKind(direct, 'activation-failed')).toHaveLength(0);
+ expect(messagesOfKind(direct, 'owner-lock-acquired')).toHaveLength(0);
+ });
+
+ it('reports database files another context kept open as storage-busy', async () => {
+ for (const databaseAction of [
+ 'open-existing',
+ 'wipe-before-open',
+ ] as const) {
+ const scope = new FakeWorkerScope();
+ const direct = new FakePort();
+ installRuntime({
+ scope,
+ ownerLockIsHeld: async () => true,
+ createCore: (coreOptions) => ({
+ handleRequest: async (port, request) => {
+ await coreOptions.onOwnerLockAcquired?.();
+ coreOptions.onStorageBusy?.();
+ port.postMessage({
+ id: request.id,
+ ok: false,
+ error:
+ 'OPFS sync handle open failed (NoModificationAllowedError)',
+ });
+ },
+ addPort: vi.fn(),
+ drain: vi.fn(),
+ }),
+ });
+ scope.activate(activation(databaseAction), direct);
+
+ await vi.waitFor(() =>
+ expect(messagesOfKind(direct, 'activation-failed')).toEqual([
+ expect.objectContaining({
+ tabId: 'tab-a',
+ ownerEpoch: 7,
+ reason: 'OPFS sync handle open failed (NoModificationAllowedError)',
+ failureCode: 'storage-busy',
+ }),
+ ])
+ );
+ expect(messagesOfKind(direct, 'engine-ready')).toHaveLength(0);
+ }
+ });
+
+ it('deletes stale databases when started as a cleanup worker', async () => {
+ const scope = new FakeWorkerScope();
+ const reply = new FakePort();
+ const removeStaleCacheDatabase = vi.fn(
+ async (_scope: string, identity: string) =>
+ identity === 'graphql-cache:scope:s0.v0.t1'
+ ? { outcome: 'removed' as const }
+ : { outcome: 'queued-mutations' as const, queuedMutations: 2 }
+ );
+ installRuntime({
+ scope,
+ loadWasm: async () =>
+ ({ removeStaleCacheDatabase }) as unknown as CacheWasmModule,
+ listOpfsRootNames: async () => [
+ 'graphql-cache:scope:s0.v0.t1',
+ 'graphql-cache:scope:s0.v0.t1-wal',
+ 'graphql-cache:scope:s0.v0.t0',
+ cacheDatabaseIdentity('scope'),
+ `${cacheDatabaseIdentity('scope')}-wal`,
+ ],
+ });
+
+ scope.onmessage?.({
+ data: { ...version, kind: 'remove-stale-databases', scope: 'scope' },
+ ports: [reply],
+ } as unknown as MessageEvent);
+
+ await vi.waitFor(() => expect(scope.closed).toBe(true));
+ expect(removeStaleCacheDatabase.mock.calls).toEqual([
+ ['scope', 'graphql-cache:scope:s0.v0.t1'],
+ ['scope', 'graphql-cache:scope:s0.v0.t0'],
+ ]);
+ expect(reply.messages).toEqual([
+ {
+ ...version,
+ kind: 'stale-databases-removed',
+ removed: 1,
+ inUse: 0,
+ keptWithQueuedMutations: 1,
+ failed: false,
+ },
+ ]);
+ expect(reply.closed).toBe(true);
+ });
+
it('selects atomic recovery-open and only proves wipe after initialization', async () => {
const scope = new FakeWorkerScope();
const direct = new FakePort();
diff --git a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts
index 7b0f8e2dae..c5c98238d3 100644
--- a/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts
+++ b/apps/web/src/lib/graphql-cache/worker/cache-engine-runtime.ts
@@ -15,21 +15,35 @@ import {
import { workerCacheTelemetry } from '../telemetry-relay';
import {
CACHE_COORDINATOR_PROTOCOL_VERSION,
+ type CleanupToPageEnvelope,
type CoordinatorToEngineEnvelope,
type EngineFatalCode,
type EngineOpenOutcome,
type EngineToCoordinatorEnvelope,
isCachePush,
isCacheResponse,
+ type PageToCleanupEnvelope,
type PageToEngineEnvelope,
validateCoordinatorToEngineEnvelope,
+ validatePageToCleanupEnvelope,
validatePageToEngineEnvelope,
} from './coordinator-protocol';
import {
createEffectWorkerRunnerTransport,
type EffectWorkerRunnerTransport,
} from './effect-worker-transport';
-import { cacheWasmLinearMemoryBytes } from './wasm-module';
+import {
+ listOpfsRootNames,
+ removeStaleCacheDatabases,
+ type StaleDatabaseCleanupTally,
+ staleCacheDatabaseIdentities,
+} from './stale-databases';
+import { OWNER_LOCK_RETRY_DELAYS_MS } from './startup';
+import {
+ type CacheWasmModule,
+ cacheWasmLinearMemoryBytes,
+ loadCacheWasm,
+} from './wasm-module';
import { CacheWorkerCore, type CacheWorkerCoreOptions } from './worker-core';
export type CacheEngineRuntimeEvent =
@@ -39,6 +53,11 @@ export type CacheEngineRuntimeEvent =
activation: PageToEngineEnvelope;
request: CacheRequest;
}
+ | {
+ kind: 'owner-lock-busy';
+ activation: PageToEngineEnvelope;
+ attempt: number;
+ }
| { kind: 'ready'; activation: PageToEngineEnvelope }
| { kind: 'drained'; activation: PageToEngineEnvelope }
| {
@@ -80,11 +99,14 @@ export interface CacheEngineRuntimeOptions {
telemetry?: CacheTelemetryRecorderLike;
/** Injectable clock/memory source for throttling tests. */
now?: () => number;
+ sleep?: (delayMs: number) => Promise<void>;
+ loadWasm?: () => Promise<CacheWasmModule>;
+ listOpfsRootNames?: () => Promise<string[]>;
readLinearMemoryBytes?: () => number;
memoryTelemetryIntervalMs?: number;
}
-const withVersion = <T extends { coordinatorVersion: 5 }>(
+const withVersion = <T extends { coordinatorVersion: 6 }>(
value: T extends unknown ? Omit<T, 'coordinatorVersion'> : never
): T =>
({
@@ -95,6 +117,11 @@ const withVersion = <T extends { coordinatorVersion: 5 }>(
const errorMessage = (error: unknown): string =>
error instanceof Error ? error.message : String(error);
+const sleep = (delayMs: number): Promise<void> =>
+ new Promise((resolve) => {
+ setTimeout(resolve, delayMs);
+ });
+
async function initializeCore(
core: CacheWorkerCoreLike,
activation: PageToEngineEnvelope
@@ -181,7 +208,11 @@ async function activate(
};
emitEvent({ kind: 'activation-started', activation });
let failed = false;
- let assetsReady = false;
+ let ownerLockAcquired = false;
+ // Set when retries for a busy owner lock ran out; storage stayed untouched.
+ let ownerLockGaveUp = false;
+ // Set when opening outwaited database files another context kept open.
+ let storageBusy = false;
let openGranted = false;
let resolveOpen!: () => void;
const openPermission = new Promise<void>((resolve) => {
@@ -246,9 +277,50 @@ async function activate(
((coreOptions: CacheWorkerCoreOptions) => new CacheWorkerCore(coreOptions));
const core = createCore({
recoveryOpen: activation.databaseAction === 'wipe-before-open',
+ // WASM holds the owner lock but has not touched OPFS yet. The coordinator
+ // records that this epoch may change storage before granting the open.
+ onOwnerLockAcquired: async () => {
+ if (failed || ownerLockAcquired) {
+ throw new Error('cache engine was not waiting for a storage grant');
+ }
+ ownerLockAcquired = true;
+ post(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'owner-lock-acquired',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ );
+ await openPermission;
+ if (failed)
+ throw new Error('cache engine failed before its storage grant');
+ },
+ // Never queue for the lock. The coordinator decides from these reports
+ // whether another build holds it and may end this activation early.
+ onOwnerLockBusy: async (attempt) => {
+ if (failed) throw new Error('cache engine failed while waiting to open');
+ emitEvent({ kind: 'owner-lock-busy', activation, attempt });
+ post(
+ withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'owner-lock-busy',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ );
+ const delayMs = OWNER_LOCK_RETRY_DELAYS_MS[attempt - 1];
+ if (delayMs === undefined) {
+ ownerLockGaveUp = true;
+ throw new Error('cache database owner lock stayed unavailable');
+ }
+ await (options.sleep ?? sleep)(delayMs);
+ if (failed) throw new Error('cache engine failed while waiting to open');
+ },
onStorageResetRequired: () => {
fatal('cache storage requested physical reset', 'storage-reset-required');
},
+ onStorageBusy: () => {
+ storageBusy = true;
+ },
onInitializationOutcome: (outcome) => {
initializationOpenOutcome = outcome;
},
@@ -322,7 +394,7 @@ async function activate(
}
match(message)
.with({ kind: 'open-engine' }, () => {
- if (!assetsReady || openGranted) {
+ if (!ownerLockAcquired || openGranted) {
fatal('unexpected database-open grant');
return;
}
@@ -397,7 +469,6 @@ async function activate(
try {
await core.prepare?.();
if (failed) return;
- assetsReady = true;
post(
withVersion<EngineToCoordinatorEnvelope>({
kind: 'engine-assets-ready',
@@ -405,10 +476,8 @@ async function activate(
ownerEpoch: activation.ownerEpoch,
})
);
- // Sending readiness is not permission: the coordinator must first record
- // that this epoch may touch storage before the first OPFS operation.
- await openPermission;
- if (failed) return;
+ // Opening first takes the database owner lock, retrying briefly while
+ // another context holds it. Storage is touched only after `open-engine`.
await initializeCore(core, activation);
if (failed) return;
const ownerLockIsHeld = options.ownerLockIsHeld ?? defaultOwnerLockIsHeld;
@@ -447,19 +516,72 @@ async function activate(
durationMs: now() - activationStartedAt,
});
telemetry.flush();
- post(
- withVersion<EngineToCoordinatorEnvelope>({
- kind: 'activation-failed',
- tabId: activation.tabId,
- ownerEpoch: activation.ownerEpoch,
- reason: errorMessage(error),
- failureCode:
- activation.databaseAction === 'wipe-before-open'
- ? 'recovery-open-failed'
- : 'initialization-failed',
+ // A fatal report already ended this activation; the open only unwound.
+ if (!failed) {
+ post(
+ ownerLockGaveUp
+ ? withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'owner-lock-unavailable',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ })
+ : withVersion<EngineToCoordinatorEnvelope>({
+ kind: 'activation-failed',
+ tabId: activation.tabId,
+ ownerEpoch: activation.ownerEpoch,
+ reason: errorMessage(error),
+ failureCode: storageBusy
+ ? 'storage-busy'
+ : activation.databaseAction === 'wipe-before-open'
+ ? 'recovery-open-failed'
+ : 'initialization-failed',
+ })
+ );
+ }
+ await closeLifecycle();
+ }
+}
+
+/** Deletes this scope's stale databases in a worker used for nothing else,
+ * because a file Turso cannot read can poison the worker's OPFS registry. */
+async function removeStaleDatabases(
+ request: PageToCleanupEnvelope,
+ port: MessagePort,
+ workerScope: DedicatedWorkerScopeLike,
+ options: CacheEngineRuntimeOptions
+): Promise<void> {
+ const failed: StaleDatabaseCleanupTally = {
+ removed: 0,
+ inUse: 0,
+ keptWithQueuedMutations: 0,
+ failed: true,
+ };
+ let tally = failed;
+ try {
+ const wasm = await (options.loadWasm ?? loadCacheWasm)();
+ const removeOne = wasm.removeStaleCacheDatabase;
+ if (removeOne) {
+ const identities = staleCacheDatabaseIdentities(
+ request.scope,
+ await (options.listOpfsRootNames ?? listOpfsRootNames)()
+ );
+ tally = await removeStaleCacheDatabases(identities, (identity) =>
+ removeOne(request.scope, identity)
+ );
+ }
+ } catch {
+ tally = failed;
+ }
+ try {
+ port.postMessage(
+ withVersion<CleanupToPageEnvelope>({
+ kind: 'stale-databases-removed',
+ ...tally,
})
);
- await closeLifecycle();
+ } finally {
+ port.close();
+ workerScope.close();
}
}
@@ -471,12 +593,27 @@ export function installCacheEngineWorker(
options.scope ?? (self as unknown as DedicatedWorkerScopeLike);
let activated = false;
workerScope.onmessage = (event: MessageEvent<unknown>) => {
- const parsed = validatePageToEngineEnvelope(event.data);
const directPort = event.ports[0];
- if (!parsed.ok || event.ports.length !== 1 || !directPort || activated) {
+ if (event.ports.length !== 1 || !directPort || activated) {
for (const port of event.ports) port.close();
return;
}
+ const cleanup = validatePageToCleanupEnvelope(event.data);
+ if (cleanup.ok) {
+ activated = true;
+ void removeStaleDatabases(
+ cleanup.value,
+ directPort,
+ workerScope,
+ options
+ );
+ return;
+ }
+ const parsed = validatePageToEngineEnvelope(event.data);
+ if (!parsed.ok) {
+ directPort.close();
+ return;
+ }
activated = true;
void activate(parsed.value, directPort, workerScope, options);
};
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts
index 4546351c80..df121af9a6 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts
@@ -14,13 +14,23 @@ const action = <K extends CoordinatorAction['kind']>(
return found as Extract<CoordinatorAction, { kind: K }>;
};
+/** Assets loaded, owner lock acquired, storage granted. */
+const grantStorage = (
+ core: CoordinatorCore,
+ tabId: string,
+ ownerEpoch: number
+): void => {
+ core.beginOwnerLockWait(tabId, ownerEpoch);
+ core.beginEngineOpen(tabId, ownerEpoch);
+};
+
const ready = (
core: CoordinatorCore,
tabId: string,
ownerEpoch: number,
databaseActionProof: DatabaseActionProof
): CoordinatorAction[] => {
- core.beginEngineOpen(tabId, ownerEpoch);
+ grantStorage(core, tabId, ownerEpoch);
return core.engineReady({
tabId,
ownerEpoch,
@@ -40,6 +50,226 @@ const init = (id: number) => ({ id, kind: 'init', scope: 'scope' }) as const;
const clear = (id: number) => ({ id, kind: 'clear' }) as const;
describe('CoordinatorCore', () => {
+ it('preserves storage when the engine is lost while trying the owner lock', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ expect(core.beginOwnerLockWait('tab-a', 1)).toBe(true);
+
+ core.ownerLost('tab-a', 1, 'owner tab closed while the lock was busy');
+
+ // Trying to take the lock cannot have changed OPFS.
+ expect(action(core.resumeAfterLoss(), 'elect-owner').databaseAction).toBe(
+ 'open-existing'
+ );
+ });
+
+ it('fails closed when another context keeps the owner lock', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ core.registerTab('tab-b');
+ expect(core.request('tab-a', init(1))).toEqual([]);
+ expect(core.request('tab-b', clear(2))).toEqual([]);
+ core.beginOwnerLockWait('tab-a', 1);
+
+ const actions = core.ownerLockUnavailable('tab-a', 1);
+
+ const unavailable = {
+ kind: 'reject-request',
+ error: expect.stringContaining('owner lock is held'),
+ errorCode: 'owner-lock-unavailable',
+ };
+ expect(actions).toEqual([
+ { ...unavailable, tabId: 'tab-a', requestId: 1 },
+ { ...unavailable, tabId: 'tab-b', requestId: 2 },
+ { kind: 'close-engine-route', tabId: 'tab-a', ownerEpoch: 1 },
+ {
+ kind: 'broadcast-cache-unavailable',
+ reason: expect.stringContaining('another app version'),
+ },
+ ]);
+ expect(core.state).toEqual({
+ kind: 'waiting-for-tab',
+ nextDatabaseAction: 'open-existing',
+ });
+ // Tabs told to stay uncached are refused and never elected again.
+ expect(core.request('tab-b', init(3))).toEqual([
+ { ...unavailable, tabId: 'tab-b', requestId: 3 },
+ ]);
+ expect(core.ownerLockUnavailable('tab-a', 1)).toEqual([]);
+ expect(action(core.registerTab('tab-c'), 'elect-owner')).toEqual({
+ kind: 'elect-owner',
+ tabId: 'tab-c',
+ ownerEpoch: 2,
+ databaseAction: 'open-existing',
+ });
+ expect(core.request('tab-c', init(4))).toEqual([]);
+ });
+
+ it('never fails closed once storage may have been touched', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ grantStorage(core, 'tab-a', 1);
+
+ expect(core.ownerLockUnavailable('tab-a', 1)).toEqual([]);
+ expect(core.state).toMatchObject({
+ kind: 'activating',
+ phase: 'opening-database',
+ });
+ });
+
+ it('keeps a pending recovery wipe for the attempt after failing closed', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ core.registerTab('tab-b');
+ ready(core, 'tab-a', 1, 'opened-existing');
+ core.tabLost('tab-a');
+ expect(action(core.resumeAfterLoss(), 'elect-owner')).toMatchObject({
+ tabId: 'tab-b',
+ databaseAction: 'wipe-before-open',
+ });
+ core.beginOwnerLockWait('tab-b', 2);
+
+ core.ownerLockUnavailable('tab-b', 2);
+
+ expect(action(core.registerTab('tab-c'), 'elect-owner')).toMatchObject({
+ tabId: 'tab-c',
+ databaseAction: 'wipe-before-open',
+ });
+ });
+
+ it('fails closed without a wipe when the database files stay busy', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ core.registerTab('tab-b');
+ expect(core.request('tab-b', init(1))).toEqual([]);
+ // Only an engine granted storage can find the files busy.
+ expect(core.storageBusy('tab-a', 1)).toEqual([]);
+ grantStorage(core, 'tab-a', 1);
+
+ const actions = core.storageBusy('tab-a', 1);
+
+ const unavailable = {
+ kind: 'reject-request',
+ error: expect.stringContaining('files are still open'),
+ errorCode: 'owner-lock-unavailable',
+ };
+ expect(actions).toEqual([
+ { ...unavailable, tabId: 'tab-b', requestId: 1 },
+ { kind: 'close-engine-route', tabId: 'tab-a', ownerEpoch: 1 },
+ {
+ kind: 'broadcast-cache-unavailable',
+ reason: expect.stringContaining('files are still open'),
+ },
+ ]);
+ // The open changed nothing, so no attempt is scheduled to repair it.
+ expect(core.state).toEqual({
+ kind: 'waiting-for-tab',
+ nextDatabaseAction: 'open-existing',
+ });
+ expect(core.request('tab-a', init(2))).toEqual([
+ { ...unavailable, tabId: 'tab-a', requestId: 2 },
+ ]);
+ expect(core.storageBusy('tab-a', 1)).toEqual([]);
+ expect(action(core.registerTab('tab-c'), 'elect-owner')).toEqual({
+ kind: 'elect-owner',
+ tabId: 'tab-c',
+ ownerEpoch: 2,
+ databaseAction: 'open-existing',
+ });
+ });
+
+ it('hands the database to a newer build and never elects again', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ core.registerTab('tab-b');
+ ready(core, 'tab-a', 1, 'opened-existing');
+ expect(core.request('tab-b', init(1))).toMatchObject([
+ { kind: 'route-request', ownerEpoch: 1 },
+ ]);
+
+ expect(core.supersede()).toEqual([
+ {
+ kind: 'broadcast-cache-superseded',
+ reason: expect.stringContaining('newer version of the app'),
+ },
+ ]);
+ expect(core.superseded).toBe(true);
+ expect(core.supersede()).toEqual([]);
+ // Routed work still gets its answer; new work is refused quietly.
+ expect(core.state).toMatchObject({ kind: 'active', tabId: 'tab-a' });
+ expect(core.request('tab-b', clear(2))).toEqual([
+ expect.objectContaining({
+ kind: 'reject-request',
+ requestId: 2,
+ errorCode: 'owner-lock-unavailable',
+ }),
+ ]);
+
+ // The owner's page then leaves like a navigation, freeing the database.
+ const departure = core.departForNavigation(
+ 'tab-a',
+ 1,
+ 'a newer build took over'
+ );
+ expect(departure).toContainEqual({
+ kind: 'close-engine-route',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ expect(departure.some((action) => action.kind === 'elect-owner')).toBe(
+ false
+ );
+ expect(core.state.kind).toBe('waiting-for-tab');
+ // A page of this build that loads afterwards could reload into this
+ // build again, so it runs uncached instead of reloading.
+ expect(core.registerTab('tab-c')).toEqual([
+ {
+ kind: 'notify-cache-unavailable',
+ tabId: 'tab-c',
+ reason: expect.stringContaining('newer version of the app'),
+ },
+ ]);
+ });
+
+ it('hands over only while no engine is changing the files', () => {
+ const activating = new CoordinatorCore('scope');
+ activating.registerTab('tab-a');
+ grantStorage(activating, 'tab-a', 1);
+ expect(activating.supersede()).toEqual([]);
+
+ // An idle coordinator's tabs, already uncached, move on too.
+ const idle = new CoordinatorCore('scope');
+ idle.registerTab('tab-a');
+ idle.beginOwnerLockWait('tab-a', 1);
+ idle.ownerLockUnavailable('tab-a', 1);
+ expect(idle.supersede()).toEqual([
+ {
+ kind: 'broadcast-cache-superseded',
+ reason: expect.stringContaining('newer version of the app'),
+ },
+ ]);
+ });
+
+ it('keeps a pending recovery wipe when its files stay busy', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ core.registerTab('tab-b');
+ ready(core, 'tab-a', 1, 'opened-existing');
+ core.tabLost('tab-a');
+ expect(action(core.resumeAfterLoss(), 'elect-owner')).toMatchObject({
+ tabId: 'tab-b',
+ databaseAction: 'wipe-before-open',
+ });
+ grantStorage(core, 'tab-b', 2);
+
+ core.storageBusy('tab-b', 2);
+
+ expect(action(core.registerTab('tab-c'), 'elect-owner')).toMatchObject({
+ tabId: 'tab-c',
+ databaseAction: 'wipe-before-open',
+ });
+ });
+
it('preserves storage after bootstrap loss but requires recovery once opening was granted', () => {
const core = new CoordinatorCore('scope');
core.registerTab('tab-a');
@@ -47,7 +277,10 @@ describe('CoordinatorCore', () => {
expect(action(core.resumeAfterLoss(), 'elect-owner').databaseAction).toBe(
'open-existing'
);
- expect(core.beginEngineOpen('tab-a', 1)).toBe(false);
+ expect(core.beginOwnerLockWait('tab-a', 1)).toBe(false);
+ // Storage is granted only once the engine holds the owner lock.
+ expect(core.beginEngineOpen('tab-a', 2)).toBe(false);
+ expect(core.beginOwnerLockWait('tab-a', 2)).toBe(true);
expect(core.beginEngineOpen('tab-a', 2)).toBe(true);
expect(core.beginEngineOpen('tab-a', 2)).toBe(false);
core.ownerLost('tab-a', 2, 'database open timeout');
@@ -150,7 +383,7 @@ describe('CoordinatorCore', () => {
expect(action(handoff, 'elect-owner').databaseAction).toBe(
'open-existing'
);
- core.beginEngineOpen('tab-b', 2);
+ grantStorage(core, 'tab-b', 2);
const actions = core.engineReady({
tabId: 'tab-b',
ownerEpoch: 2,
@@ -455,7 +688,7 @@ describe('CoordinatorCore', () => {
core.registerTab('tab-a');
core.registerTab('tab-b');
- core.beginEngineOpen('tab-a', 1);
+ grantStorage(core, 'tab-a', 1);
const wrongLock = core.engineReady({
tabId: 'tab-a',
ownerEpoch: 1,
@@ -469,7 +702,7 @@ describe('CoordinatorCore', () => {
'wrong physical owner lock'
);
core.resumeAfterLoss();
- core.beginEngineOpen('tab-b', 2);
+ grantStorage(core, 'tab-b', 2);
const wrongProof = core.engineReady({
tabId: 'tab-b',
ownerEpoch: 2,
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts
index bb11c8ec2e..f7f507b555 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts
@@ -4,6 +4,7 @@ import {
type CacheResponse,
type CacheResponseErrorCode,
OWNER_EPOCH_LOST_ERROR_CODE,
+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE,
} from '../protocol';
import type {
DatabaseAction,
@@ -86,6 +87,9 @@ export type CoordinatorAction =
| { kind: 'drop-tab'; tabId: string }
| { kind: 'retire-tab'; tabId: string; ownerEpoch: OwnerEpoch }
| { kind: 'schedule-reset-activation' }
+ | { kind: 'broadcast-cache-unavailable'; reason: string }
+ | { kind: 'broadcast-cache-superseded'; reason: string }
+ | { kind: 'notify-cache-unavailable'; tabId: string; reason: string }
| {
kind: 'broadcast-engine-replaced';
ownerEpoch: OwnerEpoch;
@@ -113,6 +117,19 @@ export type EngineReady = {
const proofFor = (outcome: EngineOpenOutcome): DatabaseActionProof =>
outcome.startsWith('reset-') ? 'wiped-before-open' : 'opened-existing';
+/** Only `opening-database` follows the storage grant; earlier phases, including
+ * waiting for the owner lock, cannot have changed OPFS. */
+export const startupPhaseMayTouchStorage = (
+ phase: EngineStartupPhase
+): boolean => phase === 'opening-database';
+
+const OWNER_LOCK_UNAVAILABLE_REASON =
+ 'cache database owner lock is held by another context, such as another app version';
+const STORAGE_BUSY_REASON =
+ 'cache database files are still open in another context, such as a closing tab';
+const SUPERSEDED_REASON =
+ 'a newer version of the app took over the local cache';
+
/** Pure election, epoch, routing, drain, and abrupt-loss state machine. */
export class CoordinatorCore {
private stateValue: CoordinatorState = {
@@ -121,6 +138,11 @@ export class CoordinatorCore {
};
private readonly tabs: string[] = [];
private readonly retiringTabs = new Set<string>();
+ /** Tabs told the database is unavailable, with the reason they were given;
+ * their pages stay uncached. */
+ private readonly unavailableTabs = new Map<string, string>();
+ /** Set once a newer app build took the database over; never cleared. */
+ private supersededReason: string | undefined;
private readonly queuedRequests: QueuedRequest[] = [];
private readonly inFlight = new Map<RouteId, InFlightRequest>();
private currentEpoch = 0;
@@ -150,9 +172,27 @@ export class CoordinatorCore {
};
}
+ /** Whether a newer app build took the database over. */
+ get superseded(): boolean {
+ return this.supersededReason !== undefined;
+ }
+
registerTab(tabId: string): CoordinatorAction[] {
if (this.tabs.includes(tabId)) return [];
this.tabs.push(tabId);
+ if (this.supersededReason !== undefined) {
+ // A page of this build loaded after the handover, as after a rollback
+ // or from stale HTML. Reloading could land on this build again, so it
+ // only runs uncached.
+ this.unavailableTabs.set(tabId, this.supersededReason);
+ return [
+ {
+ kind: 'notify-cache-unavailable',
+ tabId,
+ reason: this.supersededReason,
+ },
+ ];
+ }
if (this.stateValue.kind === 'failed') {
return [
{
@@ -180,6 +220,10 @@ export class CoordinatorCore {
if (this.stateValue.kind === 'failed') {
return [this.reject(tabId, request.id, this.stateValue.reason)];
}
+ const unavailable = this.unavailableTabs.get(tabId);
+ if (unavailable !== undefined) {
+ return [this.rejectUnavailable(tabId, request.id, unavailable)];
+ }
const queued = { tabId, request };
if (this.stateValue.kind !== 'active') {
@@ -189,8 +233,75 @@ export class CoordinatorCore {
return [this.route(queued, this.stateValue)];
}
- /** Grant storage access only after assets have loaded. Before this grant an
- * abandoned worker cannot have changed OPFS, so its successor preserves it.
+ /** The engine loaded its assets and tries to take the database owner lock.
+ * Trying grants no storage access. */
+ beginOwnerLockWait(tabId: string, ownerEpoch: OwnerEpoch): boolean {
+ const state = this.stateValue;
+ if (
+ state.kind !== 'activating' ||
+ state.tabId !== tabId ||
+ state.ownerEpoch !== ownerEpoch ||
+ state.phase !== 'loading-assets'
+ )
+ return false;
+ this.stateValue = { ...state, phase: 'awaiting-owner-lock' };
+ return true;
+ }
+
+ /**
+ * Another context, usually another deployed build, keeps the database owner
+ * lock. Stop without touching storage: refuse every queued request and tell
+ * every tab registered now to stay uncached for the rest of its page
+ * session. Only a tab that registers later starts a new attempt.
+ */
+ ownerLockUnavailable(
+ tabId: string,
+ ownerEpoch: OwnerEpoch
+ ): CoordinatorAction[] {
+ const state = this.stateValue;
+ if (
+ state.kind !== 'activating' ||
+ state.tabId !== tabId ||
+ state.ownerEpoch !== ownerEpoch ||
+ startupPhaseMayTouchStorage(state.phase)
+ )
+ return [];
+ return this.failClosed(
+ tabId,
+ ownerEpoch,
+ state.databaseAction,
+ OWNER_LOCK_UNAVAILABLE_REASON
+ );
+ }
+
+ /**
+ * The engine held the owner lock, but another context kept the database
+ * files open through the engine's bounded wait, usually a predecessor worker
+ * that is still being torn down. Wiping cannot remove files that are open,
+ * and they need no repair, so fail closed like an unavailable owner lock.
+ * The attempt's database action carries over: an open changed nothing, and
+ * a wipe that removed one file must still finish.
+ */
+ storageBusy(tabId: string, ownerEpoch: OwnerEpoch): CoordinatorAction[] {
+ const state = this.stateValue;
+ if (
+ state.kind !== 'activating' ||
+ state.tabId !== tabId ||
+ state.ownerEpoch !== ownerEpoch ||
+ state.phase !== 'opening-database'
+ )
+ return [];
+ return this.failClosed(
+ tabId,
+ ownerEpoch,
+ state.databaseAction,
+ STORAGE_BUSY_REASON
+ );
+ }
+
+ /** Grant storage access only once the engine holds the owner lock. Before
+ * this grant an abandoned worker cannot have changed OPFS, so its successor
+ * preserves it.
*/
beginEngineOpen(tabId: string, ownerEpoch: OwnerEpoch): boolean {
const state = this.stateValue;
@@ -198,7 +309,7 @@ export class CoordinatorCore {
state.kind !== 'activating' ||
state.tabId !== tabId ||
state.ownerEpoch !== ownerEpoch ||
- state.phase !== 'loading-assets'
+ state.phase !== 'awaiting-owner-lock'
)
return false;
this.stateValue = { ...state, phase: 'opening-database' };
@@ -344,6 +455,35 @@ export class CoordinatorCore {
return this.transitionToAbruptLoss(tabId, ownerEpoch, reason);
}
+ /**
+ * A newer app build takes the database over. Every tab registered now stops
+ * using the cache and moves to that build; a tab that registers later only
+ * runs uncached. The owner's page leaves the way a navigating page does,
+ * which stops its engine and frees the database, and nothing is elected
+ * afterwards. Refused while an engine is starting, recovering, or draining,
+ * because its files may be changing.
+ */
+ supersede(): CoordinatorAction[] {
+ const { kind } = this.stateValue;
+ if (
+ this.supersededReason !== undefined ||
+ (kind !== 'active' && kind !== 'waiting-for-tab' && kind !== 'failed')
+ ) {
+ return [];
+ }
+ const reason = SUPERSEDED_REASON;
+ this.supersededReason = reason;
+ const actions: CoordinatorAction[] = this.queuedRequests
+ .splice(0)
+ .map(({ tabId, request }) =>
+ this.rejectUnavailable(tabId, request.id, reason)
+ );
+ for (const tabId of this.tabs) this.unavailableTabs.set(tabId, reason);
+ actions.push({ kind: 'broadcast-cache-superseded', reason });
+ this.assertInvariants();
+ return actions;
+ }
+
/** Stops owner recovery and fails every current or future page connection. */
terminalFailure(reason: string): CoordinatorAction[] {
if (this.stateValue.kind === 'failed') return [];
@@ -499,7 +639,7 @@ export class CoordinatorCore {
previousEpoch: ownerEpoch,
nextEpoch: this.currentEpoch + 1,
databaseAction:
- state.kind === 'activating' && state.phase === 'loading-assets'
+ state.kind === 'activating' && !startupPhaseMayTouchStorage(state.phase)
? state.databaseAction
: 'wipe-before-open',
reason,
@@ -552,7 +692,10 @@ export class CoordinatorCore {
}
private chooseCandidate(avoid?: string): string | undefined {
- const eligible = this.tabs.filter((tabId) => !this.retiringTabs.has(tabId));
+ const eligible = this.tabs.filter(
+ (tabId) =>
+ !this.retiringTabs.has(tabId) && !this.unavailableTabs.has(tabId)
+ );
return eligible.find((tabId) => tabId !== avoid) ?? eligible[0];
}
@@ -582,6 +725,7 @@ export class CoordinatorCore {
private removeTabRecord(tabId: string): void {
const index = this.tabs.indexOf(tabId);
if (index >= 0) this.tabs.splice(index, 1);
+ this.unavailableTabs.delete(tabId);
for (let index = this.queuedRequests.length - 1; index >= 0; index -= 1) {
if (this.queuedRequests[index]?.tabId === tabId) {
this.queuedRequests.splice(index, 1);
@@ -615,6 +759,44 @@ export class CoordinatorCore {
};
}
+ /** Refuses queued work, keeps every registered tab uncached, and waits for
+ * a tab that registers later to try again. */
+ private failClosed(
+ tabId: string,
+ ownerEpoch: OwnerEpoch,
+ nextDatabaseAction: DatabaseAction,
+ reason: string
+ ): CoordinatorAction[] {
+ const actions: CoordinatorAction[] = this.queuedRequests
+ .splice(0)
+ .map(({ tabId: requester, request }) =>
+ this.rejectUnavailable(requester, request.id, reason)
+ );
+ for (const registered of this.tabs) {
+ this.unavailableTabs.set(registered, reason);
+ }
+ actions.push(
+ { kind: 'close-engine-route', tabId, ownerEpoch },
+ { kind: 'broadcast-cache-unavailable', reason }
+ );
+ this.stateValue = { kind: 'waiting-for-tab', nextDatabaseAction };
+ this.assertInvariants();
+ return actions;
+ }
+
+ private rejectUnavailable(
+ tabId: string,
+ requestId: number,
+ reason: string
+ ): Extract<CoordinatorAction, { kind: 'reject-request' }> {
+ return this.reject(
+ tabId,
+ requestId,
+ reason,
+ OWNER_LOCK_UNAVAILABLE_ERROR_CODE
+ );
+ }
+
private dropStale(
ownerEpoch: OwnerEpoch,
routeId: RouteId | undefined,
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts
index 82614459b2..fc96cbc709 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.test.ts
@@ -4,7 +4,10 @@ import {
type DedicatedWorkerLike,
type SharedWorkerLike,
} from './coordinator-page-adapter';
-import { CACHE_COORDINATOR_PROTOCOL_VERSION } from './coordinator-protocol';
+import {
+ CACHE_COORDINATOR_PROTOCOL_VERSION,
+ cacheDatabaseIdentity,
+} from './coordinator-protocol';
class FakeCoordinatorPort extends EventTarget {
readonly messages: Array<{ message: unknown; transfer: Transferable[] }> = [];
@@ -208,6 +211,205 @@ describe('CacheCoordinatorPageAdapter', () => {
expect(port.closed).toBe(true);
});
+ it('forwards the owner-lock phases in order and rejects stepping back', async () => {
+ const port = new FakeCoordinatorPort();
+ const progress = vi.fn();
+ const error = vi.fn();
+ const adapter = createCacheCoordinatorPageAdapter({
+ scope: 'scope',
+ tabId: 'tab-a',
+ lockManager: heldLockManager(),
+ createSharedWorker: () => ({ port: port as unknown as MessagePort }),
+ createDedicatedWorker: () => new FakeWorker(),
+ onStartupProgress: progress,
+ onTerminalError: error,
+ });
+ const started = adapter.start();
+ await vi.waitFor(() => expect(port.messages).toHaveLength(1));
+ port.receive({ ...version, kind: 'registered', tabId: 'tab-a' });
+ await started;
+ const phase = (name: string, timeoutMs: number) => ({
+ ...version,
+ kind: 'engine-startup',
+ ownerEpoch: 1,
+ phase: name,
+ databaseAction: 'open-existing',
+ timeoutMs,
+ });
+ port.receive(phase('loading-assets', 300_000));
+ port.receive(phase('awaiting-owner-lock', 15_000));
+ port.receive(phase('awaiting-owner-lock', 15_000));
+ port.receive(phase('opening-database', 20_000));
+ expect(progress.mock.calls.map(([message]) => message.phase)).toEqual([
+ 'loading-assets',
+ 'awaiting-owner-lock',
+ 'opening-database',
+ ]);
+ expect(error).not.toHaveBeenCalled();
+
+ port.receive(phase('awaiting-owner-lock', 15_000));
+
+ expect(error).toHaveBeenCalledOnce();
+ expect(port.closed).toBe(true);
+ });
+
+ const registeredAdapter = async (
+ options: Partial<Parameters<typeof createCacheCoordinatorPageAdapter>[0]>
+ ) => {
+ const port = new FakeCoordinatorPort();
+ const adapter = createCacheCoordinatorPageAdapter({
+ scope: 'scope',
+ tabId: 'tab-a',
+ lockManager: heldLockManager(),
+ createSharedWorker: () => ({ port: port as unknown as MessagePort }),
+ createDedicatedWorker: () => new FakeWorker(),
+ ...options,
+ });
+ const started = adapter.start();
+ await vi.waitFor(() => expect(port.messages).toHaveLength(1));
+ port.receive({ ...version, kind: 'registered', tabId: 'tab-a' });
+ await started;
+ return port;
+ };
+
+ const removeStale = (ownerEpoch: number) => ({
+ ...version,
+ kind: 'remove-stale-databases',
+ tabId: 'tab-a',
+ ownerEpoch,
+ });
+
+ it('relays that another context holds the database', async () => {
+ const onCacheUnavailable = vi.fn();
+ const port = await registeredAdapter({ onCacheUnavailable });
+
+ port.receive({
+ ...version,
+ kind: 'cache-unavailable',
+ reason: 'another app version holds the database',
+ });
+
+ expect(onCacheUnavailable).toHaveBeenCalledExactlyOnceWith(
+ 'another app version holds the database'
+ );
+ });
+
+ it('registers with its build time and relays a newer build taking over', async () => {
+ const onCacheSuperseded = vi.fn();
+ const port = await registeredAdapter({
+ buildTime: 1_790_000_000_000,
+ onCacheSuperseded,
+ });
+ expect(port.messages[0]?.message).toMatchObject({
+ kind: 'register-tab',
+ buildTime: 1_790_000_000_000,
+ });
+
+ port.receive({
+ ...version,
+ kind: 'cache-superseded',
+ reason: 'a newer version of the app took over the local cache',
+ });
+
+ expect(onCacheSuperseded).toHaveBeenCalledExactlyOnceWith(
+ 'a newer version of the app took over the local cache'
+ );
+ });
+
+ it('registers as an unstamped build outside app bundles', async () => {
+ const port = await registeredAdapter({});
+
+ expect(port.messages[0]?.message).toMatchObject({
+ kind: 'register-tab',
+ buildTime: 0,
+ });
+ });
+
+ it('deletes stale databases in a disposable worker, once, for its own engine', async () => {
+ const cleanupWorker = new FakeWorker();
+ const createCleanupWorker = vi.fn(() => cleanupWorker);
+ const listOpfsRootNames = vi.fn(async () => [
+ 'graphql-cache:scope:s1.v1.t1',
+ 'graphql-cache:scope:s1.v1.t1-wal',
+ cacheDatabaseIdentity('scope'),
+ 'unrelated.db',
+ ]);
+ const observations: Array<Record<string, unknown>> = [];
+ const port = await registeredAdapter({
+ createCleanupWorker,
+ listOpfsRootNames,
+ telemetry: {
+ record: (observation) => observations.push(observation),
+ flush: vi.fn(),
+ },
+ });
+
+ // Only the engine's current owner cleans up.
+ port.receive(removeStale(1));
+ await Promise.resolve();
+ expect(listOpfsRootNames).not.toHaveBeenCalled();
+ port.receive(election(1));
+ port.receive(removeStale(1));
+
+ await vi.waitFor(() => expect(cleanupWorker.messages).toHaveLength(1));
+ expect(cleanupWorker.messages[0]?.message).toEqual({
+ ...version,
+ kind: 'remove-stale-databases',
+ scope: 'scope',
+ });
+ const reply = FakeMessageChannel.instances.at(-1)?.port1 as unknown as {
+ onmessage: (event: MessageEvent<unknown>) => void;
+ };
+ reply.onmessage(
+ new MessageEvent('message', {
+ data: {
+ ...version,
+ kind: 'stale-databases-removed',
+ removed: 1,
+ inUse: 0,
+ keptWithQueuedMutations: 1,
+ failed: false,
+ },
+ })
+ );
+ await vi.waitFor(() => expect(cleanupWorker.terminated).toBe(true));
+ expect(
+ observations.filter((observation) =>
+ String(observation.ownerEvent).startsWith('stale-')
+ )
+ ).toEqual([
+ expect.objectContaining({
+ ownerEvent: 'stale-databases-removed',
+ outcome: 'success',
+ count: 1,
+ }),
+ expect.objectContaining({ ownerEvent: 'stale-database-kept', count: 1 }),
+ ]);
+
+ port.receive(removeStale(1));
+ await Promise.resolve();
+ expect(createCleanupWorker).toHaveBeenCalledOnce();
+ });
+
+ it('starts no cleanup worker when only its own database exists', async () => {
+ const createCleanupWorker = vi.fn(() => new FakeWorker());
+ const listOpfsRootNames = vi.fn(async () => [
+ cacheDatabaseIdentity('scope'),
+ `${cacheDatabaseIdentity('scope')}-wal`,
+ ]);
+ const port = await registeredAdapter({
+ createCleanupWorker,
+ listOpfsRootNames,
+ });
+ port.receive(election(1));
+
+ port.receive(removeStale(1));
+
+ await vi.waitFor(() => expect(listOpfsRootNames).toHaveBeenCalledOnce());
+ await Promise.resolve();
+ expect(createCleanupWorker).not.toHaveBeenCalled();
+ });
+
it('terminates and clears a failed worker before reporting owner loss', async () => {
const order: string[] = [];
const coordinatorPort = new FakeCoordinatorPort();
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts
index dc9aa58280..e37235d745 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-page-adapter.ts
@@ -1,4 +1,5 @@
import * as Effect from 'effect/Effect';
+import { APP_BUILD_TIME } from '../app-build';
import type { CacheRequest, WorkerMessage } from '../protocol';
import {
type CacheTelemetryRecorderLike,
@@ -7,18 +8,26 @@ import {
} from '../telemetry';
import {
CACHE_COORDINATOR_PROTOCOL_VERSION,
+ type CleanupToPageEnvelope,
type CoordinatorToTabEnvelope,
+ ENGINE_STARTUP_PHASES,
type EngineOpenOutcome,
isCacheRequest,
+ type PageToCleanupEnvelope,
type PageToEngineEnvelope,
type TabToCoordinatorEnvelope,
tabLivenessLockName,
+ validateCleanupToPageEnvelope,
validateCoordinatorToTabEnvelope,
} from './coordinator-protocol';
import {
createEffectWorkerTransport,
type EffectWorkerTransport,
} from './effect-worker-transport';
+import {
+ listOpfsRootNames,
+ staleCacheDatabaseIdentities,
+} from './stale-databases';
import { CacheBootstrapExhaustedError } from './startup';
export interface SharedWorkerLike {
@@ -51,6 +60,17 @@ export interface CacheCoordinatorPageAdapterOptions {
progress: Extract<CoordinatorToTabEnvelope, { kind: 'engine-startup' }>
) => void;
onOwnerChanged?: (ownerEpoch: number | undefined) => void;
+ /** Another context holds the database, so the cache stays off until reload. */
+ onCacheUnavailable?: (reason: string) => void;
+ /** A newer app build took the database over; this page should reload. */
+ onCacheSuperseded?: (reason: string) => void;
+ /** When this page's app build was made, in ms since the epoch. Defaults to
+ * the bundler's stamp; a newer build takes the database over. */
+ buildTime?: number;
+ /** Disposable worker, running the engine script, that deletes stale databases. */
+ createCleanupWorker?: (scope: string) => DedicatedWorkerLike;
+ listOpfsRootNames?: () => Promise<string[]>;
+ cleanupTimeoutMs?: number;
onWorkerCreated?: (worker: DedicatedWorkerLike, ownerEpoch: number) => void;
onWorkerTerminated?: (ownerEpoch: number, reason: string) => void;
/** Advisory coordinator diagnostic after successful registration. */
@@ -74,8 +94,9 @@ interface CoordinatorConnection {
}
const DEFAULT_GRACEFUL_TIMEOUT_MS = 10_000;
+const DEFAULT_CLEANUP_TIMEOUT_MS = 30_000;
-const withVersion = <T extends { coordinatorVersion: 5 }>(
+const withVersion = <T extends { coordinatorVersion: 6 }>(
value: T extends unknown ? Omit<T, 'coordinatorVersion'> : never
): T =>
({
@@ -98,6 +119,12 @@ const defaultDedicatedWorkerFactory = (
name: `graphql-cache-engine:${scope}:${ownerEpoch}`,
});
+const defaultCleanupWorkerFactory = (scope: string): DedicatedWorkerLike =>
+ new Worker(new URL('./cache.engine-worker.ts', import.meta.url), {
+ type: 'module',
+ name: `graphql-cache-cleanup:${scope}`,
+ });
+
/**
* Import-safe page endpoint for the browser coordinator topology.
* SharedWorker creation starts on first use; DedicatedWorker creation starts
@@ -136,6 +163,8 @@ export class CacheCoordinatorPageAdapter {
| Extract<CoordinatorToTabEnvelope, { kind: 'engine-startup' }>
| undefined;
private closed = false;
+ private staleCleanupStarted = false;
+ private cleanupWorker: DedicatedWorkerLike | undefined;
private readonly telemetry: CacheTelemetryRecorderLike;
private readonly now = (): number =>
globalThis.performance?.now() ?? Date.now();
@@ -461,6 +490,7 @@ export class CacheCoordinatorPageAdapter {
tabId: this.tabId,
livenessLockName,
hotCapacity: this.options.hotCapacity,
+ buildTime: this.options.buildTime ?? APP_BUILD_TIME,
})
);
}
@@ -561,7 +591,8 @@ export class CacheCoordinatorPageAdapter {
return;
if (
previous?.ownerEpoch === message.ownerEpoch &&
- previous.phase === 'opening-database'
+ ENGINE_STARTUP_PHASES.indexOf(message.phase) <
+ ENGINE_STARTUP_PHASES.indexOf(previous.phase)
) {
this.failTerminal(new Error('engine startup phase moved backwards'));
return;
@@ -595,6 +626,125 @@ export class CacheCoordinatorPageAdapter {
: new Error(message.error)
);
break;
+ case 'cache-unavailable':
+ this.options.onCacheUnavailable?.(message.reason);
+ break;
+ case 'cache-superseded':
+ this.options.onCacheSuperseded?.(message.reason);
+ break;
+ case 'remove-stale-databases':
+ // Only the engine's current owner cleans up, once per page.
+ if (
+ message.tabId === this.tabId &&
+ message.ownerEpoch === this.ownerEpoch
+ ) {
+ void this.removeStaleDatabases();
+ }
+ break;
+ }
+ }
+
+ /** Lists OPFS first so the second WASM instance starts only when needed. */
+ private async removeStaleDatabases(): Promise<void> {
+ if (this.staleCleanupStarted || this.closed) return;
+ this.staleCleanupStarted = true;
+ const scope = this.options.scope;
+ let stale: string[];
+ try {
+ stale = staleCacheDatabaseIdentities(
+ scope,
+ await (this.options.listOpfsRootNames ?? listOpfsRootNames)()
+ );
+ } catch {
+ return;
+ }
+ if (stale.length === 0 || this.closed) return;
+ let worker: DedicatedWorkerLike;
+ try {
+ worker = (
+ this.options.createCleanupWorker ?? defaultCleanupWorkerFactory
+ )(scope);
+ } catch {
+ this.recordStaleCleanup(undefined);
+ return;
+ }
+ this.cleanupWorker = worker;
+ const channel = new MessageChannel();
+ const result = await new Promise<CleanupToPageEnvelope | undefined>(
+ (resolve) => {
+ const timer = setTimeout(
+ () => resolve(undefined),
+ this.options.cleanupTimeoutMs ?? DEFAULT_CLEANUP_TIMEOUT_MS
+ );
+ const settle = (value: CleanupToPageEnvelope | undefined) => {
+ clearTimeout(timer);
+ resolve(value);
+ };
+ channel.port1.onmessage = (event) => {
+ const parsed = validateCleanupToPageEnvelope(event.data);
+ settle(parsed.ok ? parsed.value : undefined);
+ };
+ worker.onerror = (event) => {
+ event.preventDefault();
+ settle(undefined);
+ };
+ try {
+ worker.postMessage(
+ withVersion<PageToCleanupEnvelope>({
+ kind: 'remove-stale-databases',
+ scope,
+ }),
+ [channel.port2]
+ );
+ } catch {
+ settle(undefined);
+ }
+ }
+ );
+ this.closePort(channel.port1);
+ this.terminateCleanupWorker();
+ this.recordStaleCleanup(result);
+ }
+
+ private recordStaleCleanup(result: CleanupToPageEnvelope | undefined): void {
+ if (!result || result.failed) {
+ this.telemetry.record({
+ name: 'graphql_cache.owner',
+ operationCategory: 'storage',
+ outcome: 'error',
+ ownerEvent: 'stale-databases-removed',
+ errorCode: result ? 'opfs-io' : 'timeout',
+ count: result?.removed ?? 0,
+ });
+ } else if (result.removed > 0) {
+ this.telemetry.record({
+ name: 'graphql_cache.owner',
+ operationCategory: 'storage',
+ outcome: 'success',
+ ownerEvent: 'stale-databases-removed',
+ count: result.removed,
+ });
+ }
+ if (result && result.keptWithQueuedMutations > 0) {
+ this.telemetry.record({
+ name: 'graphql_cache.owner',
+ operationCategory: 'storage',
+ outcome: 'success',
+ ownerEvent: 'stale-database-kept',
+ count: result.keptWithQueuedMutations,
+ });
+ }
+ }
+
+ private terminateCleanupWorker(): void {
+ const worker = this.cleanupWorker;
+ this.cleanupWorker = undefined;
+ if (!worker) return;
+ try {
+ worker.onerror = null;
+ worker.terminate();
+ } catch {
+ // A failed cleanup worker only delays deletion to a later owner.
}
}
@@ -773,6 +923,7 @@ export class CacheCoordinatorPageAdapter {
this.terminalErrorReported = true;
this.closed = true;
this.registered = false;
+ this.terminateCleanupWorker();
this.clearGracefulTimeout();
if (this.ownerEpoch !== undefined) {
this.terminateEngine(this.ownerEpoch, error.message, false);
@@ -791,6 +942,7 @@ export class CacheCoordinatorPageAdapter {
private finishDispose(): void {
this.closed = true;
this.registered = false;
+ this.terminateCleanupWorker();
this.clearGracefulTimeout();
this.closeCoordinatorPort();
this.releaseLiveness();
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts
index bac80c4977..439282674a 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.test.ts
@@ -1,14 +1,25 @@
+import { readFileSync } from 'node:fs';
+import { dirname, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
import { describe, expect, it } from 'vitest';
import { INITIAL_CACHE_REVISION } from '../protocol';
import {
CACHE_COORDINATOR_PROTOCOL_VERSION,
+ CACHE_STORAGE_VERSION,
+ cacheDatabaseIdentity,
databaseOwnerLockName,
isCachePush,
isCacheRequest,
isCacheResponse,
+ isStaleCacheDatabaseIdentity,
+ tabIdFromLivenessLockName,
+ tabLivenessLockName,
+ UNVERSIONED_STORAGE_VERSION,
+ validateCleanupToPageEnvelope,
validateCoordinatorToEngineEnvelope,
validateCoordinatorToTabEnvelope,
validateEngineToCoordinatorEnvelope,
+ validatePageToCleanupEnvelope,
validatePageToEngineEnvelope,
validateTabToCoordinatorEnvelope,
} from './coordinator-protocol';
@@ -75,6 +86,108 @@ describe('coordinator runtime protocol', () => {
validateCoordinatorToEngineEnvelope({ ...grant, coordinatorVersion: 2 })
.ok
).toBe(false);
+ // Protocol 5 and earlier granted storage right after assets loaded,
+ // before the lock.
+ expect(
+ validateCoordinatorToEngineEnvelope({ ...grant, coordinatorVersion: 5 })
+ .ok
+ ).toBe(false);
+ });
+
+ it('validates owner-lock retries, unavailability, and stale cleanup envelopes', () => {
+ const waiting = {
+ ...version,
+ kind: 'engine-startup',
+ ownerEpoch: 1,
+ phase: 'awaiting-owner-lock',
+ databaseAction: 'open-existing',
+ timeoutMs: 15_000,
+ };
+ expect(validateCoordinatorToTabEnvelope(waiting).ok).toBe(true);
+ for (const invalid of [
+ { ...waiting, timeoutMs: undefined },
+ { ...waiting, phase: 'owner-lock-blocked' },
+ ]) {
+ expect(validateCoordinatorToTabEnvelope(invalid).ok).toBe(false);
+ }
+ const acquired = {
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId: 'tab',
+ ownerEpoch: 1,
+ };
+ const busy = { ...acquired, kind: 'owner-lock-busy' };
+ const unavailable = { ...acquired, kind: 'owner-lock-unavailable' };
+ for (const valid of [acquired, busy, unavailable]) {
+ expect(validateEngineToCoordinatorEnvelope(valid).ok).toBe(true);
+ }
+ for (const invalid of [
+ { ...acquired, tabId: '' },
+ { ...acquired, ownerEpoch: 0 },
+ { ...acquired, extra: true },
+ { ...busy, attempt: 2 },
+ { ...unavailable, reason: 'extra' },
+ ]) {
+ expect(validateEngineToCoordinatorEnvelope(invalid).ok).toBe(false);
+ }
+
+ const cacheUnavailable = {
+ ...version,
+ kind: 'cache-unavailable',
+ reason: 'another app version holds the database',
+ };
+ const superseded = {
+ ...version,
+ kind: 'cache-superseded',
+ reason: 'a newer version of the app took over the local cache',
+ };
+ const removeStale = {
+ ...version,
+ kind: 'remove-stale-databases',
+ tabId: 'tab',
+ ownerEpoch: 2,
+ };
+ for (const valid of [cacheUnavailable, superseded, removeStale]) {
+ expect(validateCoordinatorToTabEnvelope(valid).ok).toBe(true);
+ }
+ for (const invalid of [
+ { ...cacheUnavailable, reason: '' },
+ { ...superseded, reason: '' },
+ { ...superseded, buildTime: 1 },
+ { ...removeStale, ownerEpoch: 0 },
+ ]) {
+ expect(validateCoordinatorToTabEnvelope(invalid).ok).toBe(false);
+ }
+
+ const cleanup = { ...version, kind: 'remove-stale-databases', scope: 's' };
+ expect(validatePageToCleanupEnvelope(cleanup).ok).toBe(true);
+ expect(validatePageToCleanupEnvelope({ ...cleanup, scope: '' }).ok).toBe(
+ false
+ );
+ const tally = {
+ ...version,
+ kind: 'stale-databases-removed',
+ removed: 1,
+ inUse: 0,
+ keptWithQueuedMutations: 2,
+ failed: false,
+ };
+ expect(validateCleanupToPageEnvelope(tally).ok).toBe(true);
+ for (const invalid of [
+ { ...tally, removed: -1 },
+ { ...tally, failed: 'no' },
+ { ...tally, extra: true },
+ ]) {
+ expect(validateCleanupToPageEnvelope(invalid).ok).toBe(false);
+ }
+ expect(
+ isCacheResponse({
+ id: 3,
+ ok: false,
+ error: 'owner lock is held elsewhere',
+ errorCode: 'owner-lock-unavailable',
+ })
+ ).toBe(true);
});
it('validates bounded reconciliation evidence without changing exact requests', () => {
@@ -265,6 +378,7 @@ describe('coordinator runtime protocol', () => {
scope: 'scope',
tabId: 'tab',
livenessLockName: 'graphql-cache-tab:scope:tab',
+ buildTime: 1_790_000_000_000,
},
{
...version,
@@ -473,6 +587,26 @@ describe('coordinator runtime protocol', () => {
failureCode: 'recovery-open-failed',
}).ok
).toBe(true);
+ expect(
+ validateEngineToCoordinatorEnvelope({
+ ...version,
+ kind: 'activation-failed',
+ tabId: 'tab',
+ ownerEpoch: 1,
+ reason: 'OPFS sync handle open failed (NoModificationAllowedError)',
+ failureCode: 'storage-busy',
+ }).ok
+ ).toBe(true);
+ expect(
+ validateEngineToCoordinatorEnvelope({
+ ...version,
+ kind: 'activation-failed',
+ tabId: 'tab',
+ ownerEpoch: 1,
+ reason: 'open failed',
+ failureCode: 'busy',
+ }).ok
+ ).toBe(false);
expect(
validateEngineToCoordinatorEnvelope({
...version,
@@ -496,6 +630,17 @@ describe('coordinator runtime protocol', () => {
scope: 'scope',
tabId: 'tab',
livenessLockName: 'lock',
+ buildTime: 0,
+ }).ok
+ ).toBe(false);
+ // Every page reports its build, which decides database takeovers.
+ expect(
+ validateTabToCoordinatorEnvelope({
+ ...version,
+ kind: 'register-tab',
+ scope: 'scope',
+ tabId: 'tab',
+ livenessLockName: 'lock',
}).ok
).toBe(false);
expect(
@@ -533,6 +678,8 @@ describe('coordinator runtime protocol', () => {
});
it('derives the exact UTF-8 canonical turso-opfs lock name', () => {
+ // The current versions keep the unversioned name, and so the lock that
+ // builds before versioned names hold.
expect(databaseOwnerLockName('scope')).toBe(
'macro:turso-opfs:v1:19:graphql-cache:scope'
);
@@ -540,4 +687,87 @@ describe('coordinator runtime protocol', () => {
'macro:turso-opfs:v1:16:graphql-cache:é'
);
});
+
+ it('mirrors the storage versions the Rust crates embed in database names', () => {
+ const repositoryRoot = resolve(
+ dirname(fileURLToPath(import.meta.url)),
+ '../../../../../..'
+ );
+ const rustSource = (path: string): string =>
+ readFileSync(resolve(repositoryRoot, path), 'utf8');
+ const rustConstant = (path: string, name: string): number => {
+ const found = new RegExp(`pub const ${name}: u32 = (\\d+);`).exec(
+ rustSource(path)
+ );
+ if (!found?.[1]) throw new Error(`${name} not found in ${path}`);
+ return Number(found[1]);
+ };
+ const unversioned =
+ /pub const UNVERSIONED_STORAGE_VERSIONS: \(u32, u32, u32\) = \((\d+), (\d+), (\d+)\);/.exec(
+ rustSource('crates/client/cache-core/src/codec.rs')
+ );
+ if (!unversioned) throw new Error('UNVERSIONED_STORAGE_VERSIONS not found');
+ expect(UNVERSIONED_STORAGE_VERSION).toEqual({
+ schemaCompatibilityEpoch: Number(unversioned[1]),
+ formatVersion: Number(unversioned[2]),
+ storageSchemaVersion: Number(unversioned[3]),
+ });
+ expect(CACHE_STORAGE_VERSION).toEqual({
+ schemaCompatibilityEpoch: rustConstant(
+ 'crates/client/cache-core/src/codec.rs',
+ 'CACHE_SCHEMA_COMPATIBILITY_EPOCH'
+ ),
+ formatVersion: rustConstant(
+ 'crates/client/cache-core/src/codec.rs',
+ 'CACHE_FORMAT_VERSION'
+ ),
+ storageSchemaVersion: rustConstant(
+ 'crates/client/cache-turso/src/storage.rs',
+ 'STORAGE_SCHEMA_VERSION'
+ ),
+ });
+ });
+
+ it("recognizes only this scope's older databases as stale", () => {
+ const { schemaCompatibilityEpoch, formatVersion, storageSchemaVersion } =
+ CACHE_STORAGE_VERSION;
+ const own = cacheDatabaseIdentity('a');
+ const versioned = (storage: number): string =>
+ `graphql-cache:a:s${schemaCompatibilityEpoch}.v${formatVersion}.t${storage}`;
+ // The current versions still open the unversioned name.
+ expect(own).toBe('graphql-cache:a');
+ expect(isStaleCacheDatabaseIdentity('a', own)).toBe(false);
+ // Another spelling of this build's own versions is never opened again.
+ expect(
+ isStaleCacheDatabaseIdentity('a', versioned(storageSchemaVersion))
+ ).toBe(true);
+ expect(isStaleCacheDatabaseIdentity('a', 'graphql-cache:a:s1.v2.t3')).toBe(
+ true
+ );
+ expect(
+ isStaleCacheDatabaseIdentity('a', versioned(storageSchemaVersion - 1))
+ ).toBe(true);
+ for (const other of [
+ // A newer build may come back after a rollback, so it keeps its file.
+ versioned(storageSchemaVersion + 1),
+ 'graphql-cache:a:s99.v0.t0',
+ 'graphql-cache:b:s1.v2.t3',
+ 'graphql-cache:ab',
+ 'graphql-cache:a-wal',
+ 'graphql-cache:a:s1.v2',
+ 'graphql-cache:a:s1.v2.t3.x',
+ 'graphql-cache:quarantine:a',
+ ]) {
+ expect(isStaleCacheDatabaseIdentity('a', other)).toBe(false);
+ }
+ expect(
+ tabIdFromLivenessLockName('a', tabLivenessLockName('a', 'tab-1'))
+ ).toBe('tab-1');
+ expect(
+ tabIdFromLivenessLockName('a', tabLivenessLockName('ab', 'tab-1'))
+ ).toBeUndefined();
+ expect(
+ tabIdFromLivenessLockName('a', 'graphql-cache-tab:a:')
+ ).toBeUndefined();
+ });
});
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts
index 08a108bf00..084873809e 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-protocol.ts
@@ -20,8 +20,23 @@ import {
export { isCachePush, isCacheResponse, isWorkerMessage };
/** Version of the topology envelope and routed cache RPC surface. */
-export const CACHE_COORDINATOR_PROTOCOL_VERSION = 5 as const;
-export type EngineStartupPhase = 'loading-assets' | 'opening-database';
+export const CACHE_COORDINATOR_PROTOCOL_VERSION = 6 as const;
+/**
+ * Startup phases in order. The engine may touch storage only in
+ * `opening-database`, which begins once it holds the database owner lock. It
+ * never queues for that lock: while another context holds it, the engine
+ * retries briefly and then gives up without touching storage.
+ */
+export type EngineStartupPhase =
+ | 'loading-assets'
+ | 'awaiting-owner-lock'
+ | 'opening-database';
+
+export const ENGINE_STARTUP_PHASES: readonly EngineStartupPhase[] = [
+ 'loading-assets',
+ 'awaiting-owner-lock',
+ 'opening-database',
+];
export type OwnerEpoch = number;
export type RouteId = number;
@@ -34,54 +49,61 @@ export type EngineOpenOutcome =
| 'reset-corrupt'
| 'reset-storage-uncertain';
export type EngineFatalCode = 'storage-reset-required' | 'runtime-failure';
+/** `storage-busy`: another context kept the database files open through the
+ * engine's bounded wait. An open changed nothing; a wipe may have removed one
+ * file, so the coordinator keeps the attempt's database action. */
export type ActivationFailureCode =
| 'initialization-failed'
- | 'recovery-open-failed';
+ | 'recovery-open-failed'
+ | 'storage-busy';
export type TabToCoordinatorEnvelope =
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'register-tab';
scope: string;
tabId: string;
livenessLockName: string;
hotCapacity?: number;
+ /** When the page's app build was made, in ms since the epoch; 0 if
+ * unstamped. A newer build takes the database over from an older one. */
+ buildTime: number;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'cache-request';
tabId: string;
request: CacheRequest;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'attach-engine-port';
tabId: string;
ownerEpoch: OwnerEpoch;
enginePort: MessagePort;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'graceful-departure';
tabId: string;
ownerEpoch: OwnerEpoch;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'navigation-departure';
tabId: string;
ownerEpoch: OwnerEpoch;
reason: string;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-lost';
tabId: string;
ownerEpoch: OwnerEpoch;
reason: string;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'disconnect-tab';
tabId: string;
reason: string;
@@ -89,12 +111,12 @@ export type TabToCoordinatorEnvelope =
export type CoordinatorToTabEnvelope =
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'registered';
tabId: string;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'become-owner';
scope: string;
tabId: string;
@@ -104,25 +126,25 @@ export type CoordinatorToTabEnvelope =
hotCapacity?: number;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'cache-message';
message: WorkerMessage;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'terminate-engine';
tabId: string;
ownerEpoch: OwnerEpoch;
reason: string;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'retire-complete';
tabId: string;
ownerEpoch: OwnerEpoch;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-startup';
ownerEpoch: OwnerEpoch;
phase: EngineStartupPhase;
@@ -130,26 +152,46 @@ export type CoordinatorToTabEnvelope =
timeoutMs: number;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-replaced';
ownerEpoch: OwnerEpoch;
/** Whether this engine reopened durable data or created/reset it. */
openOutcome: EngineOpenOutcome;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'protocol-error';
error: string;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'terminal-error';
error: string;
storageUntouched?: true;
+ }
+ | {
+ /** Another context holds the database; stop using the cache until reload. */
+ coordinatorVersion: 6;
+ kind: 'cache-unavailable';
+ reason: string;
+ }
+ | {
+ /** A newer app build took the database over. The page stops using the
+ * cache and reloads into that build. */
+ coordinatorVersion: 6;
+ kind: 'cache-superseded';
+ reason: string;
+ }
+ | {
+ /** No other build has live tabs, so the owner may delete stale databases. */
+ coordinatorVersion: 6;
+ kind: 'remove-stale-databases';
+ tabId: string;
+ ownerEpoch: OwnerEpoch;
};
export type PageToEngineEnvelope = {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'activate-engine';
scope: string;
tabId: string;
@@ -161,24 +203,25 @@ export type PageToEngineEnvelope = {
export type CoordinatorToEngineEnvelope =
| {
- coordinatorVersion: 5;
+ /** Storage grant, sent only after `owner-lock-acquired`. */
+ coordinatorVersion: 6;
kind: 'open-engine';
ownerEpoch: OwnerEpoch;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-request';
ownerEpoch: OwnerEpoch;
routeId: RouteId;
request: CacheRequest;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'drain-engine';
ownerEpoch: OwnerEpoch;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'heartbeat';
ownerEpoch: OwnerEpoch;
heartbeatId: number;
@@ -186,13 +229,34 @@ export type CoordinatorToEngineEnvelope =
export type EngineToCoordinatorEnvelope =
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-assets-ready';
tabId: string;
ownerEpoch: OwnerEpoch;
}
| {
- coordinatorVersion: 5;
+ /** Holds the owner lock, has not touched storage, awaits `open-engine`. */
+ coordinatorVersion: 6;
+ kind: 'owner-lock-acquired';
+ tabId: string;
+ ownerEpoch: OwnerEpoch;
+ }
+ | {
+ /** The owner lock was held elsewhere on this attempt; retrying. */
+ coordinatorVersion: 6;
+ kind: 'owner-lock-busy';
+ tabId: string;
+ ownerEpoch: OwnerEpoch;
+ }
+ | {
+ /** Gave up on the owner lock without touching storage. */
+ coordinatorVersion: 6;
+ kind: 'owner-lock-unavailable';
+ tabId: string;
+ ownerEpoch: OwnerEpoch;
+ }
+ | {
+ coordinatorVersion: 6;
kind: 'engine-ready';
tabId: string;
ownerEpoch: OwnerEpoch;
@@ -202,26 +266,26 @@ export type EngineToCoordinatorEnvelope =
openOutcome: EngineOpenOutcome;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-response';
ownerEpoch: OwnerEpoch;
routeId: RouteId;
response: CacheResponse;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-push';
ownerEpoch: OwnerEpoch;
push: CachePush;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-drained';
tabId: string;
ownerEpoch: OwnerEpoch;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'engine-fatal';
tabId: string;
ownerEpoch: OwnerEpoch;
@@ -229,7 +293,7 @@ export type EngineToCoordinatorEnvelope =
fatalCode: EngineFatalCode;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'activation-failed';
tabId: string;
ownerEpoch: OwnerEpoch;
@@ -237,12 +301,30 @@ export type EngineToCoordinatorEnvelope =
failureCode: ActivationFailureCode;
}
| {
- coordinatorVersion: 5;
+ coordinatorVersion: 6;
kind: 'heartbeat-ack';
ownerEpoch: OwnerEpoch;
heartbeatId: number;
};
+/** Starts a disposable worker that deletes this scope's stale databases. */
+export type PageToCleanupEnvelope = {
+ coordinatorVersion: 6;
+ kind: 'remove-stale-databases';
+ scope: string;
+};
+
+/** Tally from a stale-database cleanup worker. */
+export type CleanupToPageEnvelope = {
+ coordinatorVersion: 6;
+ kind: 'stale-databases-removed';
+ removed: number;
+ inUse: number;
+ keptWithQueuedMutations: number;
+ /** Cleanup stopped at an error; remaining databases wait for a later run. */
+ failed: boolean;
+};
+
export type EnvelopeValidation<T> =
| { ok: true; value: T }
| { ok: false; error: string };
@@ -296,6 +378,9 @@ const hasVersion = (record: UnknownRecord): boolean =>
const isDatabaseAction = (value: unknown): value is DatabaseAction =>
value === 'open-existing' || value === 'wipe-before-open';
+const isEngineStartupPhase = (value: unknown): value is EngineStartupPhase =>
+ ENGINE_STARTUP_PHASES.includes(value as EngineStartupPhase);
+
const isDatabaseActionProof = (value: unknown): value is DatabaseActionProof =>
value === 'opened-existing' || value === 'wiped-before-open';
@@ -315,7 +400,9 @@ const isEngineFatalCode = (value: unknown): value is EngineFatalCode =>
const isActivationFailureCode = (
value: unknown
): value is ActivationFailureCode =>
- value === 'initialization-failed' || value === 'recovery-open-failed';
+ value === 'initialization-failed' ||
+ value === 'recovery-open-failed' ||
+ value === 'storage-busy';
const isOptionalRecord = (
value: unknown
@@ -675,10 +762,12 @@ export function validateTabToCoordinatorEnvelope(
'tabId',
'livenessLockName',
'hotCapacity',
+ 'buildTime',
]) &&
isNonEmptyString(value.scope) &&
isNonEmptyString(value.livenessLockName) &&
- isOptionalPositiveInteger(value.hotCapacity)
+ isOptionalPositiveInteger(value.hotCapacity) &&
+ isSafeNonNegativeInteger(value.buildTime)
) {
return pass(value as TabToCoordinatorEnvelope);
}
@@ -853,8 +942,7 @@ export function validateCoordinatorToTabEnvelope(
'timeoutMs',
]) &&
isPositiveInteger(value.ownerEpoch) &&
- (value.phase === 'loading-assets' ||
- value.phase === 'opening-database') &&
+ isEngineStartupPhase(value.phase) &&
isDatabaseAction(value.databaseAction) &&
isPositiveInteger(value.timeoutMs)
) {
@@ -897,10 +985,75 @@ export function validateCoordinatorToTabEnvelope(
return pass(value as CoordinatorToTabEnvelope);
}
break;
+ case 'cache-unavailable':
+ case 'cache-superseded':
+ if (
+ hasOnlyKeys(value, ['coordinatorVersion', 'kind', 'reason']) &&
+ isNonEmptyString(value.reason)
+ ) {
+ return pass(value as CoordinatorToTabEnvelope);
+ }
+ break;
+ case 'remove-stale-databases':
+ if (
+ hasOnlyKeys(value, [
+ 'coordinatorVersion',
+ 'kind',
+ 'tabId',
+ 'ownerEpoch',
+ ]) &&
+ isNonEmptyString(value.tabId) &&
+ isPositiveInteger(value.ownerEpoch)
+ ) {
+ return pass(value as CoordinatorToTabEnvelope);
+ }
+ break;
}
return fail(`invalid ${value.kind} coordinator envelope`);
}
+/** Validates the page's message starting a stale-database cleanup worker. */
+export function validatePageToCleanupEnvelope(
+ value: unknown
+): EnvelopeValidation<PageToCleanupEnvelope> {
+ if (
+ isRecord(value) &&
+ hasVersion(value) &&
+ value.kind === 'remove-stale-databases' &&
+ hasOnlyKeys(value, ['coordinatorVersion', 'kind', 'scope']) &&
+ isNonEmptyString(value.scope)
+ ) {
+ return pass(value as PageToCleanupEnvelope);
+ }
+ return fail('invalid remove-stale-databases envelope');
+}
+
+/** Validates a cleanup worker's tally. */
+export function validateCleanupToPageEnvelope(
+ value: unknown
+): EnvelopeValidation<CleanupToPageEnvelope> {
+ if (
+ isRecord(value) &&
+ hasVersion(value) &&
+ value.kind === 'stale-databases-removed' &&
+ hasOnlyKeys(value, [
+ 'coordinatorVersion',
+ 'kind',
+ 'removed',
+ 'inUse',
+ 'keptWithQueuedMutations',
+ 'failed',
+ ]) &&
+ isSafeNonNegativeInteger(value.removed) &&
+ isSafeNonNegativeInteger(value.inUse) &&
+ isSafeNonNegativeInteger(value.keptWithQueuedMutations) &&
+ typeof value.failed === 'boolean'
+ ) {
+ return pass(value as CleanupToPageEnvelope);
+ }
+ return fail('invalid stale-databases-removed envelope');
+}
+
/** Validates the page's one-time DedicatedWorker activation message. */
export function validatePageToEngineEnvelope(
value: unknown
@@ -1046,6 +1199,9 @@ export function validateEngineToCoordinatorEnvelope(
}
break;
case 'engine-assets-ready':
+ case 'owner-lock-acquired':
+ case 'owner-lock-busy':
+ case 'owner-lock-unavailable':
case 'engine-drained':
if (
hasOnlyKeys(value, [
@@ -1114,12 +1270,123 @@ export function validateEngineToCoordinatorEnvelope(
return fail(`invalid ${value.kind} coordinator envelope`);
}
+const TAB_LIVENESS_LOCK_PREFIX = 'graphql-cache-tab:';
+
export const tabLivenessLockName = (scope: string, tabId: string): string =>
- `graphql-cache-tab:${scope}:${tabId}`;
+ `${TAB_LIVENESS_LOCK_PREFIX}${scope}:${tabId}`;
+
+/** The tab id in a liveness lock name for `scope`, if it is one. */
+export function tabIdFromLivenessLockName(
+ scope: string,
+ lockName: string
+): string | undefined {
+ const prefix = `${TAB_LIVENESS_LOCK_PREFIX}${scope}:`;
+ if (!lockName.startsWith(prefix)) return;
+ const tabId = lockName.slice(prefix.length);
+ return tabId.length > 0 ? tabId : undefined;
+}
+
+/**
+ * Storage versions embedded in the physical database name. Mirrors
+ * `CACHE_SCHEMA_COMPATIBILITY_EPOCH`, `CACHE_FORMAT_VERSION` and
+ * `STORAGE_SCHEMA_VERSION` in the Rust crates: a parity test compares them,
+ * and the engine checks its WASM before touching storage.
+ */
+export const CACHE_STORAGE_VERSION = {
+ schemaCompatibilityEpoch: 3,
+ formatVersion: 3,
+ storageSchemaVersion: 11,
+} as const;
+
+/**
+ * Storage versions in use when database names began to embed versions; a
+ * build at exactly these keeps the unversioned name. Mirrors
+ * `UNVERSIONED_STORAGE_VERSIONS` in cache-core.
+ */
+export const UNVERSIONED_STORAGE_VERSION = {
+ schemaCompatibilityEpoch: 3,
+ formatVersion: 3,
+ storageSchemaVersion: 11,
+} as const;
+
+const DATABASE_PREFIX = 'graphql-cache:';
+
+type StorageVersionTuple = readonly [number, number, number];
+
+const versionTuple = (
+ version: typeof CACHE_STORAGE_VERSION | typeof UNVERSIONED_STORAGE_VERSION
+): StorageVersionTuple => [
+ version.schemaCompatibilityEpoch,
+ version.formatVersion,
+ version.storageSchemaVersion,
+];
+
+/** Compared as a tuple, like the WASM: the first version that differs decides. */
+const compareVersions = (
+ left: StorageVersionTuple,
+ right: StorageVersionTuple
+): number => {
+ const index = left.findIndex((version, at) => version !== right[at]);
+ return index < 0 ? 0 : (left[index] ?? 0) - (right[index] ?? 0);
+};
+
+/** Mirrors `cache_database_name`: the physical database this build opens. */
+export function cacheDatabaseIdentity(scope: string): string {
+ const unversioned = `${DATABASE_PREFIX}${scope}`;
+ const own = versionTuple(CACHE_STORAGE_VERSION);
+ if (compareVersions(own, versionTuple(UNVERSIONED_STORAGE_VERSION)) === 0) {
+ return unversioned;
+ }
+ return `${unversioned}:s${own[0]}.v${own[1]}.t${own[2]}`;
+}
+
+/**
+ * Whether `identity` is one of this scope's databases that this build does
+ * not open and that no newer build could want: the unversioned name once this
+ * build's versions have moved past it, or another name for versions no newer
+ * than this build's. A newer version belongs to a build that may come back,
+ * for example after a rollback. Mirrors the WASM's own check.
+ */
+export function isStaleCacheDatabaseIdentity(
+ scope: string,
+ identity: string
+): boolean {
+ const unversioned = `${DATABASE_PREFIX}${scope}`;
+ const version =
+ identity === unversioned
+ ? versionTuple(UNVERSIONED_STORAGE_VERSION)
+ : storageVersion(unversioned, identity);
+ return (
+ identity !== cacheDatabaseIdentity(scope) &&
+ version !== undefined &&
+ compareVersions(version, versionTuple(CACHE_STORAGE_VERSION)) <= 0
+ );
+}
+
+/** The `[epoch, format, storage]` versions embedded in a database name. */
+function storageVersion(
+ unversioned: string,
+ identity: string
+): StorageVersionTuple | undefined {
+ if (!identity.startsWith(unversioned)) return;
+ const match = /^:s(\d+)\.v(\d+)\.t(\d+)$/.exec(
+ identity.slice(unversioned.length)
+ );
+ if (!match) return;
+ const [epoch, format, storage] = match.slice(1).map(Number);
+ if (
+ !Number.isSafeInteger(epoch) ||
+ !Number.isSafeInteger(format) ||
+ !Number.isSafeInteger(storage)
+ ) {
+ return;
+ }
+ return [epoch ?? 0, format ?? 0, storage ?? 0];
+}
/** Mirrors turso-opfs's canonical lock derivation without exposing a new lock. */
export function databaseOwnerLockName(scope: string): string {
- const databaseIdentity = `graphql-cache:${scope}`;
+ const databaseIdentity = cacheDatabaseIdentity(scope);
const byteLength = new TextEncoder().encode(databaseIdentity).byteLength;
return `macro:turso-opfs:v1:${byteLength}:${databaseIdentity}`;
}
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts
index d9839d0a4b..71cc76246c 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts
@@ -8,9 +8,11 @@ import {
type CoordinatorMessagePort,
CoordinatorRouter,
} from './coordinator-router';
+import type { CacheTakeoverMessage } from './coordinator-takeover';
class FakePort extends EventTarget {
readonly messages: unknown[] = [];
+ readonly received: unknown[] = [];
readonly events: string[] = [];
closed = false;
started = false;
@@ -38,6 +40,7 @@ class FakePort extends EventTarget {
}
receive(message: unknown): void {
+ this.received.push(message);
if (this.effectProtocol) {
this.dispatchEvent(new MessageEvent('message', { data: [1, message] }));
return;
@@ -60,7 +63,8 @@ const version = {
const register = async (
router: CoordinatorRouter,
port: FakePort,
- tabId: string
+ tabId: string,
+ buildTime = 0
): Promise<void> => {
await router.handleTabMessage(port as CoordinatorMessagePort, {
...version,
@@ -68,6 +72,7 @@ const register = async (
scope: 'scope',
tabId,
livenessLockName: `graphql-cache-tab:scope:${tabId}`,
+ buildTime,
});
};
@@ -94,7 +99,11 @@ const ready = (
ownerEpoch: number,
proof: 'opened-existing' | 'wiped-before-open'
): void => {
- if (messagesOfKind(enginePort, 'open-engine').length === 0) {
+ const sent = (kind: string) =>
+ enginePort.received.some(
+ (message) => (message as { kind?: unknown }).kind === kind
+ );
+ if (!sent('engine-assets-ready')) {
enginePort.receive({
...version,
kind: 'engine-assets-ready',
@@ -102,6 +111,14 @@ const ready = (
ownerEpoch,
});
}
+ if (!sent('owner-lock-acquired')) {
+ enginePort.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId,
+ ownerEpoch,
+ });
+ }
enginePort.receive({
...version,
kind: 'engine-ready',
@@ -234,6 +251,12 @@ describe('CoordinatorRouter', () => {
tabId: 'tab-a',
ownerEpoch: 1,
});
+ initialEngine.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
initialEngine.receive({
...version,
kind: 'activation-failed',
@@ -265,6 +288,12 @@ describe('CoordinatorRouter', () => {
tabId,
ownerEpoch: state.ownerEpoch,
});
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId,
+ ownerEpoch: state.ownerEpoch,
+ });
engine.receive({
...version,
kind: 'activation-failed',
@@ -345,6 +374,7 @@ describe('CoordinatorRouter', () => {
scope: 'scope',
tabId: 'stale-tab',
livenessLockName: 'graphql-cache-tab:scope:stale-tab',
+ buildTime: 0,
});
stalePort.onmessageerror?.();
@@ -691,10 +721,712 @@ describe('CoordinatorRouter', () => {
);
});
+ const heldLocks = (names: () => string[]) => async (): Promise<string[]> =>
+ names();
+
+ const livenessLocks = (...tabIds: string[]) =>
+ tabIds.map((tabId) => `graphql-cache-tab:scope:${tabId}`);
+
+ const busy = (engine: FakePort): void => {
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-busy',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ };
+
+ const fakeTakeoverChannel = () => {
+ const posted: CacheTakeoverMessage[] = [];
+ let deliver: ((message: CacheTakeoverMessage) => void) | undefined;
+ return {
+ posted,
+ open: (
+ _scope: string,
+ onMessage: (message: CacheTakeoverMessage) => void
+ ) => {
+ deliver = onMessage;
+ return {
+ post: (message: CacheTakeoverMessage) => posted.push(message),
+ close: () => {},
+ };
+ },
+ deliver: (message: CacheTakeoverMessage) => deliver?.(message),
+ };
+ };
+
+ /** Two tabs of this build queue work while another build's tab is alive
+ * and the engine finds the owner lock busy. */
+ const askForAnotherBuildsDatabase = async () => {
+ const observations: Array<{ name: string; ownerEvent?: string }> = [];
+ const channel = fakeTakeoverChannel();
+ const router = new CoordinatorRouter({
+ takeoverReplyTimeoutMs: 100,
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ queryHeldLockNames: heldLocks(() =>
+ livenessLocks('tab-a', 'tab-b', 'other-build-tab')
+ ),
+ openTakeoverChannel: channel.open,
+ telemetry: {
+ record: (observation) => observations.push(observation),
+ flush: vi.fn(),
+ },
+ });
+ const tabA = new FakePort();
+ const tabB = new FakePort();
+ await register(router, tabA, 'tab-a', 2_000);
+ await register(router, tabB, 'tab-b', 1_900);
+ for (const [tab, tabId, request] of [
+ [tabA, 'tab-a', { id: 1, kind: 'init', scope: 'scope' }],
+ [tabB, 'tab-b', { id: 2, kind: 'clear' }],
+ ] as const) {
+ await router.handleTabMessage(tab as CoordinatorMessagePort, {
+ ...version,
+ kind: 'cache-request',
+ tabId,
+ request,
+ });
+ }
+ const engine = new FakePort();
+ await attach(router, tabA, 'tab-a', 1, engine);
+ engine.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ busy(engine);
+ await vi.advanceTimersByTimeAsync(0);
+ return { router, channel, observations, tabA, tabB, engine };
+ };
+
+ const expectFailedClosed = (
+ tabA: FakePort,
+ tabB: FakePort,
+ reason: string
+ ): void => {
+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([
+ expect.objectContaining({
+ ownerEpoch: 1,
+ reason: expect.stringContaining(reason),
+ }),
+ ]);
+ for (const [tab, requestId] of [
+ [tabA, 1],
+ [tabB, 2],
+ ] as const) {
+ expect(messagesOfKind(tab, 'cache-message')).toEqual([
+ expect.objectContaining({
+ message: expect.objectContaining({
+ id: requestId,
+ ok: false,
+ errorCode: 'owner-lock-unavailable',
+ }),
+ }),
+ ]);
+ expect(messagesOfKind(tab, 'cache-unavailable')).toHaveLength(1);
+ }
+ };
+
+ it('asks another build for its database and fails closed if nobody answers', async () => {
+ vi.useFakeTimers();
+ const { router, channel, observations, tabA, tabB, engine } =
+ await askForAnotherBuildsDatabase();
+
+ // The request carries the newest build among this coordinator's tabs.
+ expect(channel.posted).toEqual([
+ {
+ takeover: 1,
+ kind: 'request',
+ scope: 'scope',
+ requestId: expect.stringMatching(/^1:/),
+ buildTime: 2_000,
+ },
+ ]);
+ busy(engine);
+ await vi.advanceTimersByTimeAsync(99);
+ expect(channel.posted).toHaveLength(1);
+ expect(messagesOfKind(tabB, 'cache-unavailable')).toHaveLength(0);
+
+ // Builds from before handover never answer.
+ await vi.advanceTimersByTimeAsync(1);
+ expectFailedClosed(tabA, tabB, 'another build keeps the database');
+ expect(
+ observations.filter(
+ (observation) => observation.ownerEvent === 'owner-lock-unavailable'
+ )
+ ).toHaveLength(1);
+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');
+
+ // Nothing waits in line for the lock; a new tab starts a fresh attempt.
+ await vi.advanceTimersByTimeAsync(10 * 60_000);
+ const tabC = new FakePort();
+ await register(router, tabC, 'tab-c', 2_000);
+ expect(messagesOfKind(tabC, 'become-owner')).toEqual([
+ expect.objectContaining({
+ ownerEpoch: 2,
+ databaseAction: 'open-existing',
+ }),
+ ]);
+ });
+
+ it('keeps waiting when the unknown lock was a tab of this build still registering', async () => {
+ vi.useFakeTimers();
+ const channel = fakeTakeoverChannel();
+ const router = new CoordinatorRouter({
+ takeoverReplyTimeoutMs: 100,
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ // tab-late holds its liveness lock before it sends register-tab.
+ queryHeldLockNames: heldLocks(() => livenessLocks('tab-a', 'tab-late')),
+ openTakeoverChannel: channel.open,
+ });
+ const tab = new FakePort();
+ await register(router, tab, 'tab-a', 2_000);
+ const engine = new FakePort();
+ await attach(router, tab, 'tab-a', 1, engine);
+ engine.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ busy(engine);
+ await vi.advanceTimersByTimeAsync(0);
+ expect(channel.posted).toHaveLength(1);
+
+ await register(router, new FakePort(), 'tab-late', 2_000);
+ await vi.advanceTimersByTimeAsync(100);
+
+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([]);
+ expect(router.snapshot()?.state).toMatchObject({
+ kind: 'activating',
+ phase: 'awaiting-owner-lock',
+ });
+ // Later busy reports find only this build's tabs and ask nobody.
+ busy(engine);
+ await vi.advanceTimersByTimeAsync(5_000);
+ expect(channel.posted).toHaveLength(1);
+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([]);
+ });
+
+ it('fails closed at once when the holding build keeps its database', async () => {
+ vi.useFakeTimers();
+ const { channel, tabA, tabB } = await askForAnotherBuildsDatabase();
+ const [request] = channel.posted;
+ if (request?.kind !== 'request') throw new Error('missing request');
+
+ channel.deliver({
+ takeover: 1,
+ kind: 'reply',
+ scope: 'scope',
+ requestId: request.requestId,
+ decision: 'keep',
+ });
+
+ expectFailedClosed(tabA, tabB, 'another build keeps the database');
+ });
+
+ it('keeps retrying the lock while the holding build hands it over', async () => {
+ vi.useFakeTimers();
+ const { router, channel, observations, tabA, tabB, engine } =
+ await askForAnotherBuildsDatabase();
+ const [request] = channel.posted;
+ if (request?.kind !== 'request') throw new Error('missing request');
+
+ // A reply to some other request changes nothing.
+ channel.deliver({
+ takeover: 1,
+ kind: 'reply',
+ scope: 'scope',
+ requestId: 'another-request',
+ decision: 'keep',
+ });
+ channel.deliver({
+ takeover: 1,
+ kind: 'reply',
+ scope: 'scope',
+ requestId: request.requestId,
+ decision: 'yield',
+ });
+ for (let attempt = 2; attempt <= 4; attempt += 1) busy(engine);
+ await vi.advanceTimersByTimeAsync(5_000);
+ expect(messagesOfKind(tabB, 'cache-unavailable')).toHaveLength(0);
+ expect(channel.posted).toHaveLength(1);
+ expect(
+ observations.filter(
+ (observation) => observation.ownerEvent === 'takeover-granted'
+ )
+ ).toHaveLength(1);
+
+ ready(engine, 'tab-a', 1, 'opened-existing');
+ expect(router.snapshot()?.state).toMatchObject({
+ kind: 'active',
+ tabId: 'tab-a',
+ });
+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([]);
+ });
+
+ it('asks for nothing once the engine took the lock during the lock query', async () => {
+ vi.useFakeTimers();
+ const channel = fakeTakeoverChannel();
+ let answerQuery: ((names: string[]) => void) | undefined;
+ const router = new CoordinatorRouter({
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ queryHeldLockNames: () =>
+ new Promise<string[]>((resolve) => {
+ answerQuery = resolve;
+ }),
+ openTakeoverChannel: channel.open,
+ });
+ const tab = new FakePort();
+ await register(router, tab, 'tab-a', 2_000);
+ const engine = new FakePort();
+ await attach(router, tab, 'tab-a', 1, engine);
+ engine.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+
+ busy(engine);
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ answerQuery?.(livenessLocks('tab-a', 'other-build-tab'));
+ await vi.advanceTimersByTimeAsync(5_000);
+
+ expect(channel.posted).toEqual([]);
+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([]);
+ expect(router.snapshot()?.state).toMatchObject({
+ kind: 'activating',
+ phase: 'opening-database',
+ });
+ });
+
+ it("never takes a departed owner's lingering liveness lock for another build", async () => {
+ vi.useFakeTimers();
+ const channel = fakeTakeoverChannel();
+ const router = new CoordinatorRouter({
+ takeoverReplyTimeoutMs: 100,
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ // The departed page's lock is still held while its document unloads.
+ queryHeldLockNames: heldLocks(() => livenessLocks('tab-a', 'tab-b')),
+ openTakeoverChannel: channel.open,
+ });
+ const tabA = new FakePort();
+ const tabB = new FakePort();
+ await register(router, tabA, 'tab-a', 1_000);
+ await register(router, tabB, 'tab-b', 1_000);
+ const first = new FakePort();
+ await attach(router, tabA, 'tab-a', 1, first);
+ ready(first, 'tab-a', 1, 'opened-existing');
+ await router.handleTabMessage(tabA as CoordinatorMessagePort, {
+ ...version,
+ kind: 'navigation-departure',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ reason: 'page navigation',
+ });
+ const next = new FakePort();
+ await attach(router, tabB, 'tab-b', 2, next);
+ next.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-b',
+ ownerEpoch: 2,
+ });
+
+ // The departing engine still holds the lock for a moment.
+ next.receive({
+ ...version,
+ kind: 'owner-lock-busy',
+ tabId: 'tab-b',
+ ownerEpoch: 2,
+ });
+ await vi.advanceTimersByTimeAsync(5_000);
+
+ expect(channel.posted).toEqual([]);
+ expect(messagesOfKind(tabB, 'cache-unavailable')).toEqual([]);
+ expect(router.snapshot()?.state).toMatchObject({
+ kind: 'activating',
+ tabId: 'tab-b',
+ phase: 'awaiting-owner-lock',
+ });
+ });
+
+ /** An active engine of an older build, plus a second tab of that build. */
+ const holdTheDatabase = async (buildTime: number) => {
+ vi.useFakeTimers();
+ const observations: Array<{ name: string; ownerEvent?: string }> = [];
+ const channel = fakeTakeoverChannel();
+ const router = new CoordinatorRouter({
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ openTakeoverChannel: channel.open,
+ telemetry: {
+ record: (observation) => observations.push(observation),
+ flush: vi.fn(),
+ },
+ });
+ const tabA = new FakePort();
+ const tabB = new FakePort();
+ await register(router, tabA, 'tab-a', buildTime);
+ await register(router, tabB, 'tab-b', buildTime);
+ const engine = new FakePort();
+ await attach(router, tabA, 'tab-a', 1, engine);
+ ready(engine, 'tab-a', 1, 'opened-existing');
+ const ask = (requestBuildTime: number) =>
+ channel.deliver({
+ takeover: 1,
+ kind: 'request',
+ scope: 'scope',
+ requestId: 'request-1',
+ buildTime: requestBuildTime,
+ });
+ return { router, channel, observations, tabA, tabB, engine, ask };
+ };
+
+ it('keeps its database for a build that is not newer', async () => {
+ const { router, channel, tabA, tabB, engine, ask } =
+ await holdTheDatabase(2_000);
+
+ ask(2_000);
+ ask(1_000);
+
+ expect(channel.posted).toEqual([
+ expect.objectContaining({ kind: 'reply', decision: 'keep' }),
+ expect.objectContaining({ kind: 'reply', decision: 'keep' }),
+ ]);
+ expect(router.snapshot()?.state.kind).toBe('active');
+ expect(messagesOfKind(engine, 'drain-engine')).toEqual([]);
+ for (const tab of [tabA, tabB]) {
+ expect(messagesOfKind(tab, 'cache-superseded')).toEqual([]);
+ }
+ });
+
+ it('hands its database to a newer build and sends every tab there', async () => {
+ const { router, channel, observations, tabA, tabB, engine, ask } =
+ await holdTheDatabase(1_000);
+
+ ask(2_000);
+
+ expect(channel.posted).toEqual([
+ {
+ takeover: 1,
+ kind: 'reply',
+ scope: 'scope',
+ requestId: 'request-1',
+ decision: 'yield',
+ },
+ ]);
+ for (const tab of [tabA, tabB]) {
+ expect(messagesOfKind(tab, 'cache-superseded')).toHaveLength(1);
+ }
+ // The owner's page stops its engine itself when it leaves.
+ expect(messagesOfKind(engine, 'drain-engine')).toEqual([]);
+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([]);
+ expect(
+ observations.filter(
+ (observation) => observation.ownerEvent === 'superseded'
+ )
+ ).toHaveLength(1);
+ // Superseded tabs are refused quietly while they wait to reload.
+ await router.handleTabMessage(tabB as CoordinatorMessagePort, {
+ ...version,
+ kind: 'cache-request',
+ tabId: 'tab-b',
+ request: { id: 5, kind: 'clear' },
+ });
+ expect(messagesOfKind(tabB, 'cache-message')).toEqual([
+ expect.objectContaining({
+ message: expect.objectContaining({
+ id: 5,
+ ok: false,
+ errorCode: 'owner-lock-unavailable',
+ }),
+ }),
+ ]);
+
+ await router.handleTabMessage(tabA as CoordinatorMessagePort, {
+ ...version,
+ kind: 'navigation-departure',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ reason: 'page navigation',
+ });
+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');
+ expect(
+ observations.filter(
+ (observation) => observation.ownerEvent === 'abrupt-loss'
+ )
+ ).toEqual([]);
+
+ // Nothing is elected again. A late tab of this build runs uncached rather
+ // than reloading, which could land on this build again, and a second
+ // request finds nothing left to hand over.
+ await vi.advanceTimersByTimeAsync(10 * 60_000);
+ const tabC = new FakePort();
+ await register(router, tabC, 'tab-c', 1_000);
+ for (const tab of [tabA, tabB, tabC]) {
+ expect(messagesOfKind(tab, 'become-owner')).toHaveLength(
+ tab === tabA ? 1 : 0
+ );
+ }
+ expect(messagesOfKind(tabC, 'cache-superseded')).toEqual([]);
+ expect(messagesOfKind(tabC, 'cache-unavailable')).toHaveLength(1);
+ ask(3_000);
+ expect(channel.posted).toHaveLength(1);
+ });
+
+ it('sends the tabs of an idle older build to the newer one without answering', async () => {
+ vi.useFakeTimers();
+ const channel = fakeTakeoverChannel();
+ const router = new CoordinatorRouter({
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ openTakeoverChannel: channel.open,
+ });
+ const tab = new FakePort();
+ await register(router, tab, 'tab-a', 1_000);
+ const engine = new FakePort();
+ await attach(router, tab, 'tab-a', 1, engine);
+ engine.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-unavailable',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');
+
+ channel.deliver({
+ takeover: 1,
+ kind: 'request',
+ scope: 'scope',
+ requestId: 'request-1',
+ buildTime: 2_000,
+ });
+
+ expect(channel.posted).toEqual([]);
+ expect(messagesOfKind(tab, 'cache-superseded')).toHaveLength(1);
+ });
+
+ it('retries a busy lock within one build until the engine gives up', async () => {
+ vi.useFakeTimers();
+ const router = new CoordinatorRouter({
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ queryHeldLockNames: heldLocks(() => livenessLocks('tab-a')),
+ });
+ const tab = new FakePort();
+ await register(router, tab, 'tab-a');
+ const engine = new FakePort();
+ await attach(router, tab, 'tab-a', 1, engine);
+ engine.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+
+ busy(engine);
+ await vi.advanceTimersByTimeAsync(0);
+ expect(router.snapshot()?.state).toMatchObject({
+ kind: 'activating',
+ phase: 'awaiting-owner-lock',
+ });
+
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-unavailable',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ expect(messagesOfKind(tab, 'cache-unavailable')).toHaveLength(1);
+ expect(messagesOfKind(tab, 'terminate-engine')).toEqual([
+ expect.objectContaining({
+ reason: expect.stringContaining('the engine stopped retrying'),
+ }),
+ ]);
+ });
+
+ it('fails closed without a wipe when an open finds the files still busy', async () => {
+ vi.useFakeTimers();
+ const observations: Array<{ name: string; ownerEvent?: string }> = [];
+ const router = new CoordinatorRouter({
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ telemetry: {
+ record: (observation) => observations.push(observation),
+ flush: vi.fn(),
+ },
+ });
+ const tabA = new FakePort();
+ const tabB = new FakePort();
+ await register(router, tabA, 'tab-a');
+ await register(router, tabB, 'tab-b');
+ await router.handleTabMessage(tabB as CoordinatorMessagePort, {
+ ...version,
+ kind: 'cache-request',
+ tabId: 'tab-b',
+ request: { id: 1, kind: 'clear' },
+ });
+ const engine = new FakePort();
+ await attach(router, tabA, 'tab-a', 1, engine);
+ for (const kind of [
+ 'engine-assets-ready',
+ 'owner-lock-acquired',
+ ] as const) {
+ engine.receive({ ...version, kind, tabId: 'tab-a', ownerEpoch: 1 });
+ }
+
+ // A predecessor, such as a reloading tab's worker, kept the files open.
+ engine.receive({
+ ...version,
+ kind: 'activation-failed',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ reason: 'OPFS sync handle open failed (NoModificationAllowedError)',
+ failureCode: 'storage-busy',
+ });
+
+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([
+ expect.objectContaining({
+ ownerEpoch: 1,
+ reason: expect.stringContaining('files stayed busy'),
+ }),
+ ]);
+ expect(messagesOfKind(tabB, 'cache-message')).toEqual([
+ expect.objectContaining({
+ message: expect.objectContaining({
+ id: 1,
+ ok: false,
+ errorCode: 'owner-lock-unavailable',
+ }),
+ }),
+ ]);
+ for (const tab of [tabA, tabB]) {
+ expect(messagesOfKind(tab, 'cache-unavailable')).toEqual([
+ expect.objectContaining({
+ reason: expect.stringContaining('files are still open'),
+ }),
+ ]);
+ expect(messagesOfKind(tab, 'terminal-error')).toEqual([]);
+ }
+ expect(
+ observations.filter(
+ (observation) => observation.ownerEvent === 'storage-busy'
+ )
+ ).toHaveLength(1);
+ expect(
+ observations.filter(
+ (observation) =>
+ observation.name === 'graphql_cache.reset_wipe' ||
+ observation.name === 'graphql_cache.storage_reset_required'
+ )
+ ).toEqual([]);
+
+ // Nothing retries in the background. A tab that registers later opens
+ // the existing database instead of wiping it.
+ await vi.advanceTimersByTimeAsync(10 * 60_000);
+ expect(messagesOfKind(tabB, 'become-owner')).toEqual([]);
+ const tabC = new FakePort();
+ await register(router, tabC, 'tab-c');
+ expect(messagesOfKind(tabC, 'become-owner')).toEqual([
+ expect.objectContaining({
+ ownerEpoch: 2,
+ databaseAction: 'open-existing',
+ }),
+ ]);
+ });
+
+ it('lets the owner delete stale databases only when no other build has tabs', async () => {
+ vi.useFakeTimers();
+ let locks = livenessLocks('tab-a', 'old-build-tab');
+ const setup = async () => {
+ const router = new CoordinatorRouter({
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ queryHeldLockNames: heldLocks(() => locks),
+ });
+ const tab = new FakePort();
+ await register(router, tab, 'tab-a');
+ const engine = new FakePort();
+ await attach(router, tab, 'tab-a', 1, engine);
+ ready(engine, 'tab-a', 1, 'opened-existing');
+ await vi.advanceTimersByTimeAsync(0);
+ return tab;
+ };
+
+ expect(
+ messagesOfKind(await setup(), 'remove-stale-databases')
+ ).toHaveLength(0);
+
+ locks = livenessLocks('tab-a');
+ expect(messagesOfKind(await setup(), 'remove-stale-databases')).toEqual([
+ {
+ ...version,
+ kind: 'remove-stale-databases',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ },
+ ]);
+ });
+
+ it('replaces an engine that stops reporting while it tries the owner lock', async () => {
+ vi.useFakeTimers();
+ const router = new CoordinatorRouter({
+ ownerLockWaitTimeoutMs: 30,
+ verifyTabLockHeld: async () => true,
+ watchTabLock: () => () => {},
+ });
+ const tabA = new FakePort();
+ const tabB = new FakePort();
+ await register(router, tabA, 'tab-a');
+ await register(router, tabB, 'tab-b');
+ const engine = new FakePort();
+ await attach(router, tabA, 'tab-a', 1, engine);
+ engine.receive({
+ ...version,
+ kind: 'engine-assets-ready',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+
+ await vi.advanceTimersByTimeAsync(30);
+
+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([
+ expect.objectContaining({
+ reason: 'engine owner lock watchdog timed out',
+ }),
+ ]);
+ // It never had the storage grant, so its replacement keeps the database.
+ expect(messagesOfKind(tabB, 'become-owner')).toEqual([
+ expect.objectContaining({
+ ownerEpoch: 2,
+ databaseAction: 'open-existing',
+ }),
+ ]);
+ });
+
it('allows slow asset loading and reports phase budgets to owners and late joiners', async () => {
vi.useFakeTimers();
const router = new CoordinatorRouter({
assetLoadTimeoutMs: 100,
+ ownerLockWaitTimeoutMs: 30,
activationTimeoutMs: 10,
verifyTabLockHeld: async () => true,
watchTabLock: () => () => {},
@@ -721,6 +1453,18 @@ describe('CoordinatorRouter', () => {
tabId: 'tab-a',
ownerEpoch: 1,
});
+ // Loading assets grants no storage access; the lock wait comes first.
+ expect(messagesOfKind(engine, 'open-engine')).toHaveLength(0);
+ expect(messagesOfKind(tabB, 'engine-startup').at(-1)).toMatchObject({
+ phase: 'awaiting-owner-lock',
+ timeoutMs: 30,
+ });
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
expect(messagesOfKind(engine, 'open-engine')).toHaveLength(1);
expect(messagesOfKind(tabB, 'engine-startup').at(-1)).toMatchObject({
phase: 'opening-database',
@@ -968,6 +1712,12 @@ describe('CoordinatorRouter', () => {
tabId: 'tab-a',
ownerEpoch: 1,
});
+ openingEngine.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
await vi.advanceTimersByTimeAsync(11);
expect(messagesOfKind(tabA, 'terminate-engine')).toContainEqual(
expect.objectContaining({
@@ -1156,6 +1906,7 @@ describe('CoordinatorRouter', () => {
scope: 'scope',
tabId: 'tab-a',
livenessLockName: 'graphql-cache-tab:scope:tab-a',
+ buildTime: 0,
});
await vi.waitFor(() =>
expect(messagesOfKind(tabA, 'registered')).toHaveLength(1)
@@ -1335,6 +2086,12 @@ describe('CoordinatorRouter', () => {
tabId: 'tab-a',
ownerEpoch: 1,
});
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-acquired',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
engine.receive({ kind: 'engine-ready', ownerEpoch: 1 });
await Promise.resolve();
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts
index 3032126f72..decae5f253 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts
@@ -16,6 +16,7 @@ import {
type CoordinatorAction,
CoordinatorCore,
type CoordinatorSnapshot,
+ startupPhaseMayTouchStorage,
} from './coordinator-core';
import {
type ActivationFailureCode,
@@ -27,10 +28,18 @@ import {
type EngineOpenOutcome,
type EngineToCoordinatorEnvelope,
type TabToCoordinatorEnvelope,
+ tabIdFromLivenessLockName,
tabLivenessLockName,
validateEngineToCoordinatorEnvelope,
validateTabToCoordinatorEnvelope,
} from './coordinator-protocol';
+import {
+ CACHE_TAKEOVER_VERSION,
+ type CacheTakeoverChannel,
+ type CacheTakeoverMessage,
+ type CacheTakeoverRequest,
+ openCacheTakeoverChannel,
+} from './coordinator-takeover';
import {
createEffectWorkerTransport,
type EffectWorkerTransport,
@@ -38,6 +47,8 @@ import {
import {
ENGINE_ASSET_LOAD_TIMEOUT_MS,
ENGINE_DATABASE_OPEN_TIMEOUT_MS,
+ OWNER_LOCK_WAIT_TIMEOUT_MS,
+ TAKEOVER_REPLY_TIMEOUT_MS,
} from './startup';
export interface CoordinatorMessagePort {
@@ -52,11 +63,20 @@ export type CancelLivenessWatch = () => void;
export interface CoordinatorRouterOptions {
assetLoadTimeoutMs?: number;
+ ownerLockWaitTimeoutMs?: number;
+ /** How long a takeover request waits for the holding build to agree. */
+ takeoverReplyTimeoutMs?: number;
+ openTakeoverChannel?: (
+ scope: string,
+ onMessage: (message: CacheTakeoverMessage) => void
+ ) => CacheTakeoverChannel | undefined;
activationTimeoutMs?: number;
heartbeatIntervalMs?: number;
heartbeatTimeoutMs?: number;
verifyTabLockHeld?: (lockName: string) => Promise<boolean>;
verifyOwnerLockHeld?: (lockName: string) => Promise<boolean>;
+ /** Names of every Web Lock currently held in this storage partition. */
+ queryHeldLockNames?: () => Promise<string[]>;
watchTabLock?: (
lockName: string,
onReleased: () => void
@@ -72,7 +92,16 @@ type TabConnection = {
cancelLivenessWatch: CancelLivenessWatch;
};
-type PendingRegistration = { cancelled: boolean };
+type PendingRegistration = { cancelled: boolean; tabId: string };
+
+/** This coordinator's request for another build's database, per activation. */
+type PendingTakeover = {
+ tabId: string;
+ ownerEpoch: number;
+ requestId: string;
+ granted: boolean;
+ timer: ReturnType<typeof setTimeout> | undefined;
+};
type EngineRoute = {
tabId: string;
@@ -82,6 +111,8 @@ type EngineRoute = {
const DEFAULT_HEARTBEAT_INTERVAL_MS = 2_000;
const DEFAULT_HEARTBEAT_TIMEOUT_MS = 5_000;
+/** How many departed tab ids a long-lived coordinator remembers. */
+const DEPARTED_TAB_MEMORY = 256;
const RECOVERY_RETRY_LIMIT = 5;
const RECOVERY_RETRY_BASE_DELAY_MS = 100;
@@ -122,7 +153,7 @@ type WithoutVersion<T> = T extends unknown
? Omit<T, 'coordinatorVersion'>
: never;
-const envelope = <T extends { coordinatorVersion: 5 }>(
+const envelope = <T extends { coordinatorVersion: 6 }>(
value: WithoutVersion<T>
): T =>
({
@@ -141,6 +172,13 @@ export async function verifyExclusiveLockHeld(
);
}
+export async function queryHeldLockNames(): Promise<string[]> {
+ const snapshot = await navigator.locks.query();
+ return (snapshot.held ?? []).flatMap((lock) =>
+ lock.name === undefined ? [] : [lock.name]
+ );
+}
+
/** Waits for the page's liveness lock to be released or abandoned. */
export function watchTabLivenessLock(
lockName: string,
@@ -171,6 +209,9 @@ export class CoordinatorRouter {
private coreValue: CoordinatorCore | undefined;
private hotCapacity: number | undefined;
private readonly tabs = new Map<string, TabConnection>();
+ /** Recently departed tabs, whose liveness locks can outlast their
+ * departure but never belong to another build. */
+ private readonly departedTabIds = new Set<string>();
private readonly portTabs = new Map<CoordinatorMessagePort, string>();
private readonly pendingRegistrations = new Map<
CoordinatorMessagePort,
@@ -186,11 +227,21 @@ export class CoordinatorRouter {
| undefined;
private readonly assetLoadTimeoutMs: number;
+ private readonly ownerLockWaitTimeoutMs: number;
+ private readonly takeoverReplyTimeoutMs: number;
+ private readonly openTakeoverChannel: NonNullable<
+ CoordinatorRouterOptions['openTakeoverChannel']
+ >;
+ private takeoverChannel: CacheTakeoverChannel | undefined;
+ private takeover: PendingTakeover | undefined;
+ /** The newest app build among this coordinator's registered pages. */
+ private buildTime = 0;
private readonly activationTimeoutMs: number;
private readonly heartbeatIntervalMs: number;
private readonly heartbeatTimeoutMs: number;
private readonly verifyTabLockHeld: (lockName: string) => Promise<boolean>;
private readonly verifyOwnerLockHeld: (lockName: string) => Promise<boolean>;
+ private readonly queryHeldLockNames: () => Promise<string[]>;
private readonly watchTabLock: (
lockName: string,
onReleased: () => void
@@ -222,6 +273,12 @@ export class CoordinatorRouter {
constructor(options: CoordinatorRouterOptions = {}) {
this.assetLoadTimeoutMs =
options.assetLoadTimeoutMs ?? ENGINE_ASSET_LOAD_TIMEOUT_MS;
+ this.ownerLockWaitTimeoutMs =
+ options.ownerLockWaitTimeoutMs ?? OWNER_LOCK_WAIT_TIMEOUT_MS;
+ this.takeoverReplyTimeoutMs =
+ options.takeoverReplyTimeoutMs ?? TAKEOVER_REPLY_TIMEOUT_MS;
+ this.openTakeoverChannel =
+ options.openTakeoverChannel ?? openCacheTakeoverChannel;
this.activationTimeoutMs =
options.activationTimeoutMs ?? ENGINE_DATABASE_OPEN_TIMEOUT_MS;
this.heartbeatIntervalMs =
@@ -232,6 +289,7 @@ export class CoordinatorRouter {
options.verifyTabLockHeld ?? verifyExclusiveLockHeld;
this.verifyOwnerLockHeld =
options.verifyOwnerLockHeld ?? verifyExclusiveLockHeld;
+ this.queryHeldLockNames = options.queryHeldLockNames ?? queryHeldLockNames;
this.watchTabLock = options.watchTabLock ?? watchTabLivenessLock;
this.setTimeoutFn =
options.setTimeout ?? globalThis.setTimeout.bind(globalThis);
@@ -374,7 +432,10 @@ export class CoordinatorRouter {
return;
}
- const registration: PendingRegistration = { cancelled: false };
+ const registration: PendingRegistration = {
+ cancelled: false,
+ tabId: message.tabId,
+ };
this.pendingRegistrations.set(port, registration);
let lockHeld = false;
try {
@@ -426,7 +487,12 @@ export class CoordinatorRouter {
if (!this.coreValue) {
this.coreValue = new CoordinatorCore(message.scope);
this.hotCapacity = message.hotCapacity;
+ this.takeoverChannel = this.openTakeoverChannel(
+ message.scope,
+ (takeover) => this.handleTakeoverMessage(takeover)
+ );
}
+ this.buildTime = Math.max(this.buildTime, message.buildTime);
let connection: TabConnection;
const cancelLivenessWatch = this.watchTabLock(
message.livenessLockName,
@@ -544,7 +610,9 @@ export class CoordinatorRouter {
switch (message.kind) {
case 'engine-assets-ready': {
- if (!core.beginEngineOpen(message.tabId, message.ownerEpoch)) {
+ // The engine now tries to take the owner lock without touching
+ // storage, and gives up by itself if another context keeps it.
+ if (!core.beginOwnerLockWait(message.tabId, message.ownerEpoch)) {
this.failOwner(
message.tabId,
message.ownerEpoch,
@@ -553,6 +621,19 @@ export class CoordinatorRouter {
break;
}
this.armStartupWatchdog(message.tabId, message.ownerEpoch);
+ break;
+ }
+ case 'owner-lock-acquired': {
+ if (!core.beginEngineOpen(message.tabId, message.ownerEpoch)) {
+ this.failOwner(
+ message.tabId,
+ message.ownerEpoch,
+ 'unexpected owner lock acquisition'
+ );
+ break;
+ }
+ this.clearTakeover();
+ this.armStartupWatchdog(message.tabId, message.ownerEpoch);
this.sendToEngine(
route,
envelope<CoordinatorToEngineEnvelope>({
@@ -562,6 +643,16 @@ export class CoordinatorRouter {
);
break;
}
+ case 'owner-lock-busy':
+ void this.checkBusyOwnerLock(route);
+ break;
+ case 'owner-lock-unavailable':
+ this.closeUnavailable(
+ route.tabId,
+ route.ownerEpoch,
+ 'the engine stopped retrying'
+ );
+ break;
case 'engine-ready': {
const actions = core.engineReady({
...message,
@@ -626,6 +717,7 @@ export class CoordinatorRouter {
this.clearActivationTimer();
this.resetRecoveryRetries();
this.scheduleHeartbeat(message.ownerEpoch);
+ void this.allowStaleDatabaseRemoval(route);
}
break;
}
@@ -684,6 +776,15 @@ export class CoordinatorRouter {
);
break;
case 'activation-failed':
+ if (message.failureCode === 'storage-busy') {
+ this.closeUnavailable(
+ route.tabId,
+ route.ownerEpoch,
+ message.reason,
+ 'storage-busy'
+ );
+ break;
+ }
this.failOwner(
message.tabId,
message.ownerEpoch,
@@ -844,6 +945,31 @@ export class CoordinatorRouter {
case 'schedule-reset-activation':
this.scheduleResetActivation();
break;
+ case 'broadcast-cache-unavailable':
+ this.broadcast(
+ envelope<CoordinatorToTabEnvelope>({
+ kind: 'cache-unavailable',
+ reason: action.reason,
+ })
+ );
+ break;
+ case 'broadcast-cache-superseded':
+ this.broadcast(
+ envelope<CoordinatorToTabEnvelope>({
+ kind: 'cache-superseded',
+ reason: action.reason,
+ })
+ );
+ break;
+ case 'notify-cache-unavailable':
+ this.postToTab(
+ action.tabId,
+ envelope<CoordinatorToTabEnvelope>({
+ kind: 'cache-unavailable',
+ reason: action.reason,
+ })
+ );
+ break;
case 'broadcast-engine-replaced':
this.telemetry.record({
name: 'graphql_cache.owner',
@@ -912,10 +1038,11 @@ export class CoordinatorRouter {
ownerEpoch: state.ownerEpoch,
phase: state.phase,
databaseAction: state.databaseAction,
- timeoutMs:
- state.phase === 'loading-assets'
- ? this.assetLoadTimeoutMs
- : this.activationTimeoutMs,
+ timeoutMs: match(state.phase)
+ .with('loading-assets', () => this.assetLoadTimeoutMs)
+ .with('awaiting-owner-lock', () => this.ownerLockWaitTimeoutMs)
+ .with('opening-database', () => this.activationTimeoutMs)
+ .exhaustive(),
});
}
@@ -924,15 +1051,138 @@ export class CoordinatorRouter {
const progress = this.startupProgress();
if (!progress) return;
this.broadcast(progress);
+ const { phase, timeoutMs } = progress;
this.activationTimer = this.setTimeoutFn(() => {
this.failOwner(
tabId,
ownerEpoch,
- progress.phase === 'loading-assets'
- ? 'engine asset loading watchdog timed out'
- : 'engine database opening watchdog timed out'
+ match(phase)
+ .with(
+ 'loading-assets',
+ () => 'engine asset loading watchdog timed out'
+ )
+ .with(
+ 'awaiting-owner-lock',
+ () => 'engine owner lock watchdog timed out'
+ )
+ .with(
+ 'opening-database',
+ () => 'engine database opening watchdog timed out'
+ )
+ .exhaustive()
);
- }, progress.timeoutMs);
+ }, timeoutMs);
+ }
+
+ /** Tab ids holding this scope's liveness locks without registering here,
+ * so they belong to another coordinator: another deployed build. */
+ private async otherBuildTabIds(): Promise<string[] | undefined> {
+ const core = this.coreValue;
+ if (!core) return;
+ let names: string[];
+ try {
+ names = await this.queryHeldLockNames();
+ } catch {
+ return;
+ }
+ const known = new Set([...this.tabs.keys(), ...this.departedTabIds]);
+ for (const registration of this.pendingRegistrations.values()) {
+ known.add(registration.tabId);
+ }
+ return names.flatMap((name) => {
+ const tabId = tabIdFromLivenessLockName(core.scope, name);
+ return tabId === undefined || known.has(tabId) ? [] : [tabId];
+ });
+ }
+
+ /** This build's previous owner frees the lock within milliseconds. While
+ * another build has live tabs, the lock is likely theirs, so ask for it. */
+ private async checkBusyOwnerLock(route: EngineRoute): Promise<void> {
+ if (this.takeover?.ownerEpoch === route.ownerEpoch) return;
+ const others = await this.otherBuildTabIds();
+ const state = this.coreValue?.state;
+ if (
+ !others?.length ||
+ this.engineRoute !== route ||
+ state?.kind !== 'activating' ||
+ state.ownerEpoch !== route.ownerEpoch ||
+ state.phase !== 'awaiting-owner-lock'
+ ) {
+ return;
+ }
+ this.requestTakeover(route);
+ }
+
+ /** Fails closed without touching storage again: the owner lock stayed with
+ * another context, or it kept the database files open (`storage-busy`). */
+ private closeUnavailable(
+ tabId: string,
+ ownerEpoch: number,
+ detail: string,
+ cause: 'owner-lock' | 'storage-busy' = 'owner-lock'
+ ): void {
+ const core = this.coreValue;
+ if (!core) return;
+ const state = core.state;
+ const actions =
+ cause === 'storage-busy'
+ ? core.storageBusy(tabId, ownerEpoch)
+ : core.ownerLockUnavailable(tabId, ownerEpoch);
+ if (actions.length === 0) return;
+ this.activationStarted.delete(ownerEpoch);
+ // Storage was not changed, or a busy wipe removed only part of it, so a
+ // pending recovery wipe carries over to the next attempt exactly as after
+ // a navigation departure.
+ const interruptedResetReason =
+ state.kind === 'activating' && state.databaseAction === 'wipe-before-open'
+ ? this.pendingRecoveryResetEpochs.get(ownerEpoch)
+ : undefined;
+ if (interruptedResetReason !== undefined) {
+ this.pendingRecoveryResetEpochs.delete(ownerEpoch);
+ this.nextRecoveryResetReason = interruptedResetReason;
+ }
+ this.telemetry.record({
+ name: 'graphql_cache.owner',
+ operationCategory: 'lifecycle',
+ outcome: 'error',
+ ownerEvent:
+ cause === 'storage-busy' ? 'storage-busy' : 'owner-lock-unavailable',
+ errorCode: cause === 'storage-busy' ? 'opfs-io' : 'lock',
+ });
+ this.postTerminateEngine(
+ tabId,
+ ownerEpoch,
+ cause === 'storage-busy'
+ ? `cache database files stayed busy: ${detail}`
+ : `cache database owner lock is unavailable: ${detail}`
+ );
+ this.clearEngineWatchdogs();
+ this.resetRecoveryRetries();
+ this.applyActions(actions);
+ }
+
+ /** Stale databases are deleted only once no other build could still use
+ * them; the cleanup itself also skips any database whose lock is held. */
+ private async allowStaleDatabaseRemoval(route: EngineRoute): Promise<void> {
+ const others = await this.otherBuildTabIds();
+ const state = this.coreValue?.state;
+ if (
+ others === undefined ||
+ others.length > 0 ||
+ this.engineRoute !== route ||
+ state?.kind !== 'active' ||
+ state.ownerEpoch !== route.ownerEpoch
+ ) {
+ return;
+ }
+ this.postToTab(
+ route.tabId,
+ envelope<CoordinatorToTabEnvelope>({
+ kind: 'remove-stale-databases',
+ tabId: route.tabId,
+ ownerEpoch: route.ownerEpoch,
+ })
+ );
}
private scheduleResetActivation(): void {
@@ -1091,7 +1341,7 @@ export class CoordinatorRouter {
this.activationStarted.delete(state.ownerEpoch);
if (
state.kind !== 'activating' ||
- state.phase !== 'loading-assets' ||
+ startupPhaseMayTouchStorage(state.phase) ||
state.databaseAction === 'wipe-before-open'
) {
const pendingReason = this.recordPendingResetFailure(
@@ -1258,6 +1508,12 @@ export class CoordinatorRouter {
const connection = this.tabs.get(tabId);
if (!connection) return;
this.tabs.delete(tabId);
+ // Tab ids are fresh per page load, so the oldest can be forgotten.
+ this.departedTabIds.add(tabId);
+ if (this.departedTabIds.size > DEPARTED_TAB_MEMORY) {
+ const [oldest] = this.departedTabIds;
+ if (oldest !== undefined) this.departedTabIds.delete(oldest);
+ }
this.portTabs.delete(connection.port);
connection.cancelLivenessWatch();
connection.port.close();
@@ -1390,5 +1646,137 @@ export class CoordinatorRouter {
private clearEngineWatchdogs(): void {
this.clearActivationTimer();
this.clearHeartbeatTimers();
+ this.clearTakeover();
+ }
+
+ /**
+ * Asks the build that holds the database to hand it over. The engine keeps
+ * retrying the lock meanwhile. Unless the holder agrees promptly, or the
+ * unknown tabs turn out to be this build's, this activation fails closed.
+ */
+ private requestTakeover(route: EngineRoute): void {
+ const core = this.coreValue;
+ if (!core || this.takeover?.ownerEpoch === route.ownerEpoch) return;
+ const channel = this.takeoverChannel;
+ if (!channel) {
+ this.closeUnavailable(
+ route.tabId,
+ route.ownerEpoch,
+ 'another build has live tabs'
+ );
+ return;
+ }
+ const requestId = `${route.ownerEpoch}:${crypto.randomUUID()}`;
+ const takeover: PendingTakeover = {
+ tabId: route.tabId,
+ ownerEpoch: route.ownerEpoch,
+ requestId,
+ granted: false,
+ timer: undefined,
+ };
+ takeover.timer = this.setTimeoutFn(() => {
+ takeover.timer = undefined;
+ void this.expireTakeover(takeover);
+ }, this.takeoverReplyTimeoutMs);
+ this.takeover = takeover;
+ channel.post({
+ takeover: CACHE_TAKEOVER_VERSION,
+ kind: 'request',
+ scope: core.scope,
+ requestId,
+ buildTime: this.buildTime,
+ });
+ }
+
+ /**
+ * Nobody answered. The unknown liveness lock that prompted the request may
+ * belong to a tab of this build that had not registered yet, so look again:
+ * once every live tab is known, keep waiting for the engine's own retries.
+ */
+ private async expireTakeover(takeover: PendingTakeover): Promise<void> {
+ if (this.takeover !== takeover || takeover.granted) return;
+ const others = await this.otherBuildTabIds();
+ if (this.takeover !== takeover || takeover.granted) return;
+ if (others?.length === 0) {
+ this.clearTakeover();
+ return;
+ }
+ this.closeUnavailable(
+ takeover.tabId,
+ takeover.ownerEpoch,
+ 'another build keeps the database'
+ );
+ }
+
+ private handleTakeoverMessage(message: CacheTakeoverMessage): void {
+ if (message.kind === 'request') {
+ this.answerTakeover(message);
+ return;
+ }
+ const takeover = this.takeover;
+ if (
+ !takeover ||
+ takeover.requestId !== message.requestId ||
+ takeover.granted
+ ) {
+ return;
+ }
+ if (takeover.timer !== undefined) this.clearTimeoutFn(takeover.timer);
+ takeover.timer = undefined;
+ if (message.decision === 'yield') {
+ takeover.granted = true;
+ this.telemetry.record({
+ name: 'graphql_cache.owner',
+ operationCategory: 'lifecycle',
+ outcome: 'success',
+ ownerEvent: 'takeover-granted',
+ });
+ return;
+ }
+ this.closeUnavailable(
+ takeover.tabId,
+ takeover.ownerEpoch,
+ 'another build keeps the database'
+ );
+ }
+
+ /**
+ * Only the coordinator whose engine holds the database answers, and it
+ * yields only to a strictly newer build, so builds never trade the database
+ * back and forth and a reload never lands on an older build. An idle
+ * coordinator's tabs already run on the network; they move on silently.
+ */
+ private answerTakeover(request: CacheTakeoverRequest): void {
+ const core = this.coreValue;
+ if (!core || core.superseded) return;
+ const newer = request.buildTime > this.buildTime;
+ const holdsDatabase = core.state.kind === 'active';
+ if (holdsDatabase) {
+ this.takeoverChannel?.post({
+ takeover: CACHE_TAKEOVER_VERSION,
+ kind: 'reply',
+ scope: core.scope,
+ requestId: request.requestId,
+ decision: newer ? 'yield' : 'keep',
+ });
+ }
+ if (!newer) return;
+ const actions = core.supersede();
+ if (actions.length === 0) return;
+ if (holdsDatabase) {
+ this.telemetry.record({
+ name: 'graphql_cache.owner',
+ operationCategory: 'lifecycle',
+ outcome: 'graceful',
+ ownerEvent: 'superseded',
+ });
+ }
+ this.applyActions(actions);
+ }
+
+ private clearTakeover(): void {
+ const takeover = this.takeover;
+ this.takeover = undefined;
+ if (takeover?.timer !== undefined) this.clearTimeoutFn(takeover.timer);
}
}
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.test.ts
new file mode 100644
index 0000000000..305600b3a1
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.test.ts
@@ -0,0 +1,61 @@
+import { describe, expect, it } from 'vitest';
+import { cacheDatabaseIdentity } from './coordinator-protocol';
+import {
+ cacheTakeoverChannelName,
+ parseCacheTakeoverMessage,
+} from './coordinator-takeover';
+
+describe('cache takeover messages', () => {
+ const request = {
+ takeover: 1,
+ kind: 'request',
+ scope: 'scope',
+ requestId: 'r1',
+ buildTime: 1_790_000_000_000,
+ };
+ const reply = {
+ takeover: 1,
+ kind: 'reply',
+ scope: 'scope',
+ requestId: 'r1',
+ decision: 'yield',
+ };
+
+ it('names one channel per database, shared only by builds that open it', () => {
+ expect(cacheTakeoverChannelName('scope')).toBe(
+ `graphql-cache-takeover:${cacheDatabaseIdentity('scope')}`
+ );
+ expect(cacheTakeoverChannelName('scope')).toBe(
+ 'graphql-cache-takeover:graphql-cache:scope'
+ );
+ });
+
+ it('reads requests and replies from any build, ignoring added fields', () => {
+ expect(parseCacheTakeoverMessage(request)).toEqual(request);
+ expect(parseCacheTakeoverMessage(reply)).toEqual(reply);
+ expect(
+ parseCacheTakeoverMessage({ ...request, addedByALaterBuild: true })
+ ).toEqual(request);
+ expect(parseCacheTakeoverMessage({ ...reply, decision: 'keep' })).toEqual({
+ ...reply,
+ decision: 'keep',
+ });
+ });
+
+ it('ignores anything it cannot act on', () => {
+ for (const invalid of [
+ undefined,
+ 'request',
+ [request],
+ { ...request, takeover: 2 },
+ { ...request, scope: '' },
+ { ...request, requestId: '' },
+ { ...request, buildTime: -1 },
+ { ...request, buildTime: 1.5 },
+ { ...request, kind: 'announce' },
+ { ...reply, decision: 'maybe' },
+ ]) {
+ expect(parseCacheTakeoverMessage(invalid)).toBeUndefined();
+ }
+ });
+});
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.ts
new file mode 100644
index 0000000000..3c4cc7eccd
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-takeover.ts
@@ -0,0 +1,110 @@
+import { cacheDatabaseIdentity } from './coordinator-protocol';
+
+/**
+ * Handover of the cache database between app builds.
+ *
+ * A build usually runs its own coordinator, because the coordinator's script
+ * URL is content-hashed, yet builds with the same storage version open one
+ * database. When a newer build finds that database held by an older one, it
+ * asks on this channel, which only builds opening the same database join.
+ * The holder either yields, closing the database and telling its tabs to
+ * reload into the newer build, or keeps it.
+ *
+ * Builds answer each other across releases, so the channel name and these
+ * message shapes must never change. Extend them only with fields that older
+ * builds may ignore, or add a new `takeover` version beside this one.
+ */
+export const CACHE_TAKEOVER_VERSION = 1;
+
+export const cacheTakeoverChannelName = (scope: string): string =>
+ `graphql-cache-takeover:${cacheDatabaseIdentity(scope)}`;
+
+export type CacheTakeoverRequest = {
+ takeover: typeof CACHE_TAKEOVER_VERSION;
+ kind: 'request';
+ scope: string;
+ requestId: string;
+ /** The requesting build's time; only a strictly newer build is yielded to. */
+ buildTime: number;
+};
+
+export type CacheTakeoverReply = {
+ takeover: typeof CACHE_TAKEOVER_VERSION;
+ kind: 'reply';
+ scope: string;
+ requestId: string;
+ /** Sent only by the build whose engine holds the database. */
+ decision: 'yield' | 'keep';
+};
+
+export type CacheTakeoverMessage = CacheTakeoverRequest | CacheTakeoverReply;
+
+const isRecord = (value: unknown): value is Record<string, unknown> =>
+ typeof value === 'object' && value !== null && !Array.isArray(value);
+
+const isNonEmptyString = (value: unknown): value is string =>
+ typeof value === 'string' && value.length > 0;
+
+/** Reads a message from any build. Unknown fields are ignored, not rejected,
+ * so later builds can add them. */
+export function parseCacheTakeoverMessage(
+ value: unknown
+): CacheTakeoverMessage | undefined {
+ if (
+ !isRecord(value) ||
+ value.takeover !== CACHE_TAKEOVER_VERSION ||
+ !isNonEmptyString(value.scope) ||
+ !isNonEmptyString(value.requestId)
+ ) {
+ return;
+ }
+ const { scope, requestId } = value;
+ if (
+ value.kind === 'request' &&
+ Number.isSafeInteger(value.buildTime) &&
+ (value.buildTime as number) >= 0
+ ) {
+ return {
+ takeover: CACHE_TAKEOVER_VERSION,
+ kind: 'request',
+ scope,
+ requestId,
+ buildTime: value.buildTime as number,
+ };
+ }
+ if (
+ value.kind === 'reply' &&
+ (value.decision === 'yield' || value.decision === 'keep')
+ ) {
+ return {
+ takeover: CACHE_TAKEOVER_VERSION,
+ kind: 'reply',
+ scope,
+ requestId,
+ decision: value.decision,
+ };
+ }
+}
+
+export interface CacheTakeoverChannel {
+ post(message: CacheTakeoverMessage): void;
+ close(): void;
+}
+
+/** Opens the scope's takeover channel, or returns undefined where the
+ * browser has no BroadcastChannel; builds then never hand over. */
+export function openCacheTakeoverChannel(
+ scope: string,
+ onMessage: (message: CacheTakeoverMessage) => void
+): CacheTakeoverChannel | undefined {
+ if (typeof BroadcastChannel !== 'function') return;
+ const channel = new BroadcastChannel(cacheTakeoverChannelName(scope));
+ channel.onmessage = (event: MessageEvent<unknown>) => {
+ const message = parseCacheTakeoverMessage(event.data);
+ if (message?.scope === scope) onMessage(message);
+ };
+ return {
+ post: (message) => channel.postMessage(message),
+ close: () => channel.close(),
+ };
+}
diff --git a/apps/web/src/lib/graphql-cache/worker/stale-databases.test.ts b/apps/web/src/lib/graphql-cache/worker/stale-databases.test.ts
new file mode 100644
index 0000000000..c24a9ca328
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/stale-databases.test.ts
@@ -0,0 +1,54 @@
+import { describe, expect, it, vi } from 'vitest';
+import { cacheDatabaseIdentity } from './coordinator-protocol';
+import {
+ isUnambiguousCacheScope,
+ removeStaleCacheDatabases,
+ staleCacheDatabaseIdentities,
+} from './stale-databases';
+
+describe('stale cache databases', () => {
+ it('lists each stale database of the scope once, whichever file remains', () => {
+ const own = cacheDatabaseIdentity('scope');
+ expect(
+ staleCacheDatabaseIdentities('scope', [
+ own,
+ `${own}-wal`,
+ 'graphql-cache:scope:s1.v1.t0',
+ 'graphql-cache:scope:s1.v1.t0-wal',
+ 'graphql-cache:scope:s1.v1.t1-wal',
+ 'graphql-cache:other-scope',
+ 'unrelated.db',
+ ])
+ ).toEqual(['graphql-cache:scope:s1.v1.t0', 'graphql-cache:scope:s1.v1.t1']);
+ });
+
+ it('refuses scopes whose database would read as another WAL file', () => {
+ expect(
+ isUnambiguousCacheScope('4f7c9d1e-0b2a-4c8e-9f31-7a6d5e4c3b2a')
+ ).toBe(true);
+ expect(isUnambiguousCacheScope('quarantine:scope')).toBe(true);
+ expect(isUnambiguousCacheScope('scope-wal')).toBe(false);
+ });
+
+ it('tallies each outcome and stops at the first failure', async () => {
+ const removeOne = vi
+ .fn()
+ .mockResolvedValueOnce({ outcome: 'removed' })
+ .mockResolvedValueOnce({ outcome: 'in-use' })
+ .mockResolvedValueOnce({
+ outcome: 'queued-mutations',
+ queuedMutations: 3,
+ })
+ .mockRejectedValueOnce(new Error('unreadable database'));
+
+ expect(
+ await removeStaleCacheDatabases(['a', 'b', 'c', 'd', 'e'], removeOne)
+ ).toEqual({
+ removed: 1,
+ inUse: 1,
+ keptWithQueuedMutations: 1,
+ failed: true,
+ });
+ expect(removeOne).toHaveBeenCalledTimes(4);
+ });
+});
diff --git a/apps/web/src/lib/graphql-cache/worker/stale-databases.ts b/apps/web/src/lib/graphql-cache/worker/stale-databases.ts
new file mode 100644
index 0000000000..30576c71ba
--- /dev/null
+++ b/apps/web/src/lib/graphql-cache/worker/stale-databases.ts
@@ -0,0 +1,76 @@
+import { isStaleCacheDatabaseIdentity } from './coordinator-protocol';
+
+const WAL_SUFFIX = '-wal';
+
+/**
+ * A database's WAL file is its name plus `-wal`, so the database of a scope
+ * ending in `-wal` would read as another scope's WAL file and could be
+ * removed with it. App scopes are UUIDs; a host refuses any other such scope.
+ */
+export const isUnambiguousCacheScope = (scope: string): boolean =>
+ !scope.endsWith(WAL_SUFFIX);
+
+/** This scope's databases this build does not open, from OPFS entry names. */
+export function staleCacheDatabaseIdentities(
+ scope: string,
+ entryNames: Iterable<string>
+): string[] {
+ const identities = new Set<string>();
+ for (const name of entryNames) {
+ const identity = name.endsWith(WAL_SUFFIX)
+ ? name.slice(0, -WAL_SUFFIX.length)
+ : name;
+ if (isStaleCacheDatabaseIdentity(scope, identity)) identities.add(identity);
+ }
+ return [...identities];
+}
+
+/** Lists the origin-private root without opening any file. */
+export async function listOpfsRootNames(): Promise<string[]> {
+ const root = await navigator.storage.getDirectory();
+ const names: string[] = [];
+ for await (const name of root.keys()) names.push(name);
+ return names;
+}
+
+export type StaleDatabaseRemovalOutcome = {
+ outcome: 'removed' | 'in-use' | 'queued-mutations';
+ queuedMutations?: number;
+};
+
+export type StaleDatabaseCleanupTally = {
+ removed: number;
+ inUse: number;
+ keptWithQueuedMutations: number;
+ failed: boolean;
+};
+
+/**
+ * Deletes each stale database whose owner lock is free and whose mutation
+ * queue is empty. Stops at the first failure: in the disposable cleanup worker
+ * it can leave the worker-local OPFS registry poisoned.
+ */
+export async function removeStaleCacheDatabases(
+ identities: readonly string[],
+ removeOne: (identity: string) => Promise<StaleDatabaseRemovalOutcome>
+): Promise<StaleDatabaseCleanupTally> {
+ const tally: StaleDatabaseCleanupTally = {
+ removed: 0,
+ inUse: 0,
+ keptWithQueuedMutations: 0,
+ failed: false,
+ };
+ for (const identity of identities) {
+ let result: StaleDatabaseRemovalOutcome;
+ try {
+ result = await removeOne(identity);
+ } catch {
+ tally.failed = true;
+ break;
+ }
+ if (result.outcome === 'removed') tally.removed += 1;
+ else if (result.outcome === 'in-use') tally.inUse += 1;
+ else tally.keptWithQueuedMutations += 1;
+ }
+ return tally;
+}
diff --git a/apps/web/src/lib/graphql-cache/worker/startup.ts b/apps/web/src/lib/graphql-cache/worker/startup.ts
index 59f842a172..bba181a6af 100644
--- a/apps/web/src/lib/graphql-cache/worker/startup.ts
+++ b/apps/web/src/lib/graphql-cache/worker/startup.ts
@@ -4,6 +4,21 @@
export const COORDINATOR_CONNECT_TIMEOUT_MS = 60_000;
export const ENGINE_ASSET_LOAD_TIMEOUT_MS = 5 * 60_000;
export const ENGINE_DATABASE_OPEN_TIMEOUT_MS = 20_000;
+/** Delays between attempts to take a busy database owner lock. The engine
+ * never queues for it: a handoff between owners of one build frees it within
+ * milliseconds, and after these ~10s the engine gives up without touching
+ * storage, so pages use the network until they reload.
+ */
+export const OWNER_LOCK_RETRY_DELAYS_MS: readonly number[] = [
+ 25, 50, 100, 200, 400, 800, 1_600, 3_200, 3_200,
+];
+/** When another build has live tabs, a busy lock is almost certainly theirs.
+ * The coordinator asks that build to hand the database over and gives up
+ * unless the holder agrees this soon; builds from before handover never do. */
+export const TAKEOVER_REPLY_TIMEOUT_MS = 1_000;
+/** Coordinator backstop for an engine that stops reporting while it waits. */
+export const OWNER_LOCK_WAIT_TIMEOUT_MS =
+ OWNER_LOCK_RETRY_DELAYS_MS.reduce((total, delay) => total + delay, 0) + 5_000;
/** Lets the coordinator's phase timeout/retry reach a page before its guard fires. */
export const STARTUP_RESPONSE_GRACE_MS = 5_000;
export const COORDINATOR_CONNECT_ATTEMPTS = 3;
diff --git a/apps/web/src/lib/graphql-cache/worker/wasm-module.ts b/apps/web/src/lib/graphql-cache/worker/wasm-module.ts
index f4c86c70eb..1e39dc0c77 100644
--- a/apps/web/src/lib/graphql-cache/worker/wasm-module.ts
+++ b/apps/web/src/lib/graphql-cache/worker/wasm-module.ts
@@ -31,6 +31,7 @@ import type {
WriteResult,
} from '../protocol';
import { workerCacheTelemetry } from '../telemetry-relay';
+import type { StaleDatabaseRemovalOutcome } from './stale-databases';
/** Stable, payload-free marker latched on reset-required WASM errors. */
export interface CacheStorageResetRequiredError extends Error {
@@ -176,22 +177,45 @@ export interface CacheEngine {
export interface CacheWasmModule {
default: (input?: { module_or_path?: unknown }) => Promise<unknown>;
openCache(scope: string, hotCapacity?: number): Promise<CacheEngine>;
- /** Additive open API with a coarse, payload-free recovery outcome. */
+ /**
+ * Additive open API with a coarse, payload-free recovery outcome. When
+ * given, `onOwnerLockAcquired` runs after the owner lock is held and before
+ * any OPFS access; opening continues once its promise resolves. With
+ * `ifAvailable`, a held owner lock rejects at once, touching nothing, with an
+ * error whose `cacheOwnerLockUnavailable` property is `true`. Files another
+ * context keeps open through a bounded wait reject with an error whose
+ * `cacheStorageBusy` property is `true`; a plain open has then changed
+ * neither file.
+ */
openCacheWithOutcome?(
scope: string,
- hotCapacity?: number
+ hotCapacity?: number,
+ onOwnerLockAcquired?: () => Promise<void>,
+ ifAvailable?: boolean
): Promise<CacheOpenResult>;
/** Atomically wipes before Turso open while retaining one OPFS owner lock. */
openCacheForRecovery(
scope: string,
hotCapacity?: number
): Promise<CacheEngine>;
- /** Additive recovery-open API with its coarse wipe outcome. */
+ /** Additive recovery-open API with its coarse wipe outcome; see above for
+ * `onOwnerLockAcquired` and `ifAvailable`, which precede the recovery wipe. */
openCacheForRecoveryWithOutcome?(
scope: string,
- hotCapacity?: number
+ hotCapacity?: number,
+ onOwnerLockAcquired?: () => Promise<void>,
+ ifAvailable?: boolean
): Promise<CacheOpenResult>;
destroyCache(scope: string): Promise<void>;
+ /** Physical database this build opens for `scope`; absent before databases
+ * were named by storage version. */
+ cacheDatabaseIdentity?(scope: string): string;
+ /** Deletes one stale database of `scope` if unused and without queued
+ * mutations; see `staleCacheDatabaseIdentities`. */
+ removeStaleCacheDatabase?(
+ scope: string,
+ identity: string
+ ): Promise<StaleDatabaseRemovalOutcome>;
/** Optional while older cached WASM artifacts are still in circulation. */
setSlowQueryCallback?(
callback: (
diff --git a/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts b/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts
index 1d46812905..456b1dfb59 100644
--- a/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/worker-core.test.ts
@@ -5,6 +5,7 @@ const loadCacheWasmMock = vi.hoisted(() => vi.fn());
vi.mock('./wasm-module', () => ({ loadCacheWasm: loadCacheWasmMock }));
+import { cacheDatabaseIdentity } from './coordinator-protocol';
import { CacheWorkerCore } from './worker-core';
describe('CacheWorkerCore', () => {
@@ -1132,6 +1133,176 @@ describe('CacheWorkerCore', () => {
expect(order).toEqual(['read:start', 'read:done', 'response', 'close']);
});
+ it.each([
+ [false, 'openCacheWithOutcome'],
+ [true, 'openCacheForRecoveryWithOutcome'],
+ ] as const)(
+ 'asks for the storage grant from inside the WASM open (recovery: %s)',
+ async (recoveryOpen, openName) => {
+ const order: string[] = [];
+ const open = vi.fn(
+ async (
+ _scope: string,
+ _hotCapacity: number | undefined,
+ grant?: () => Promise<void>
+ ) => {
+ order.push('owner-lock');
+ await grant?.();
+ order.push('storage');
+ return { engine: {}, outcome: 'opened-existing' };
+ }
+ );
+ loadCacheWasmMock.mockResolvedValue({ [openName]: open });
+ const messages: unknown[] = [];
+ const core = new CacheWorkerCore({
+ recoveryOpen,
+ onOwnerLockAcquired: async () => {
+ order.push('grant');
+ },
+ });
+
+ await core.handleRequest(
+ { postMessage: (message: unknown) => messages.push(message) },
+ { id: 1, kind: 'init', scope: 'scope-1' }
+ );
+
+ expect(order).toEqual(['owner-lock', 'grant', 'storage']);
+ expect(messages).toEqual([{ id: 1, ok: true, result: null }]);
+ }
+ );
+
+ it('refuses a WASM open that never asked for the storage grant', async () => {
+ loadCacheWasmMock.mockResolvedValue({
+ openCacheWithOutcome: vi.fn(async () => ({
+ engine: {},
+ outcome: 'opened-existing',
+ })),
+ });
+ const messages: unknown[] = [];
+ const core = new CacheWorkerCore({
+ onOwnerLockAcquired: async () => undefined,
+ });
+
+ await core.handleRequest(
+ { postMessage: (message: unknown) => messages.push(message) },
+ { id: 1, kind: 'init', scope: 'scope-1' }
+ );
+
+ expect(messages).toEqual([
+ {
+ id: 1,
+ ok: false,
+ error: 'cache WASM does not support the owner-lock grant',
+ },
+ ]);
+ });
+
+ const busyOwnerLock = () =>
+ Object.assign(new Error('owner lock is held by another context'), {
+ cacheOwnerLockUnavailable: true,
+ });
+
+ const initMessages = async (core: CacheWorkerCore): Promise<unknown[]> => {
+ const messages: unknown[] = [];
+ await core.handleRequest(
+ { postMessage: (message: unknown) => messages.push(message) },
+ { id: 1, kind: 'init', scope: 'scope-1' }
+ );
+ return messages;
+ };
+
+ it.each([
+ [false, 'openCacheWithOutcome'],
+ [true, 'openCacheForRecoveryWithOutcome'],
+ ] as const)(
+ 'opens only a free owner lock, asking before each retry (recovery: %s)',
+ async (recoveryOpen, openName) => {
+ const open = vi
+ .fn()
+ .mockRejectedValueOnce(busyOwnerLock())
+ .mockRejectedValueOnce(busyOwnerLock())
+ .mockImplementation(
+ async (
+ _scope: string,
+ _hotCapacity: number | undefined,
+ grant?: () => Promise<void>
+ ) => {
+ await grant?.();
+ return { engine: {}, outcome: 'opened-existing' };
+ }
+ );
+ loadCacheWasmMock.mockResolvedValue({
+ [openName]: open,
+ cacheDatabaseIdentity,
+ });
+ const onOwnerLockBusy = vi.fn(async (_attempt: number) => undefined);
+ const core = new CacheWorkerCore({
+ recoveryOpen,
+ onOwnerLockAcquired: async () => undefined,
+ onOwnerLockBusy,
+ });
+
+ expect(await initMessages(core)).toEqual([
+ { id: 1, ok: true, result: null },
+ ]);
+ expect(onOwnerLockBusy.mock.calls).toEqual([[1], [2]]);
+ // Every attempt declines to queue behind another holder.
+ expect(open.mock.calls.map((call) => call[3])).toEqual([
+ true,
+ true,
+ true,
+ ]);
+ }
+ );
+
+ it('gives up on a busy owner lock when told to stop retrying', async () => {
+ const open = vi.fn().mockRejectedValue(busyOwnerLock());
+ loadCacheWasmMock.mockResolvedValue({
+ openCacheWithOutcome: open,
+ cacheDatabaseIdentity,
+ });
+ const core = new CacheWorkerCore({
+ onOwnerLockAcquired: async () => undefined,
+ onOwnerLockBusy: async () => {
+ throw new Error('owner lock stayed unavailable');
+ },
+ });
+
+ expect(await initMessages(core)).toEqual([
+ { id: 1, ok: false, error: 'owner lock stayed unavailable' },
+ ]);
+ expect(open).toHaveBeenCalledOnce();
+ });
+
+ it.each([
+ [
+ 'predates storage versions',
+ {},
+ 'cache WASM predates storage-versioned databases; rebuild it',
+ ],
+ [
+ 'names another version',
+ { cacheDatabaseIdentity: () => 'graphql-cache:scope-1:s0.v0.t0' },
+ 'cache WASM storage version does not match this build',
+ ],
+ ])(
+ 'refuses a WASM build that %s before opening anything',
+ async (_label, identity, error) => {
+ const open = vi.fn();
+ loadCacheWasmMock.mockResolvedValue({
+ openCacheWithOutcome: open,
+ ...identity,
+ });
+ const core = new CacheWorkerCore({
+ onOwnerLockAcquired: async () => undefined,
+ onOwnerLockBusy: async () => undefined,
+ });
+
+ expect(await initMessages(core)).toEqual([{ id: 1, ok: false, error }]);
+ expect(open).not.toHaveBeenCalled();
+ }
+ );
+
it('uses atomic recovery-open instead of opening before a reset', async () => {
const openCache = vi.fn();
const openCacheForRecovery = vi.fn().mockResolvedValue({});
@@ -1700,4 +1871,33 @@ describe('CacheWorkerCore', () => {
{ id: 3, ok: false, error: 'cache storage reset required' },
]);
});
+
+ it('reports an open that gave up on busy database files', async () => {
+ const busyError = Object.assign(
+ new Error('OPFS sync handle open failed (NoModificationAllowedError)'),
+ { cacheStorageBusy: true as const }
+ );
+ loadCacheWasmMock.mockResolvedValue({
+ openCache: vi.fn().mockRejectedValue(busyError),
+ });
+ const onStorageBusy = vi.fn();
+ const onStorageResetRequired = vi.fn();
+ const messages: unknown[] = [];
+ const core = new CacheWorkerCore({ onStorageBusy, onStorageResetRequired });
+
+ await core.handleRequest(
+ { postMessage: (message: unknown) => messages.push(message) },
+ { id: 1, kind: 'init', scope: 'scope-1' }
+ );
+
+ expect(onStorageBusy).toHaveBeenCalledOnce();
+ expect(onStorageResetRequired).not.toHaveBeenCalled();
+ expect(messages).toEqual([
+ {
+ id: 1,
+ ok: false,
+ error: 'OPFS sync handle open failed (NoModificationAllowedError)',
+ },
+ ]);
+ });
});
diff --git a/apps/web/src/lib/graphql-cache/worker/worker-core.ts b/apps/web/src/lib/graphql-cache/worker/worker-core.ts
index cb6b66ef25..3b8c643b7b 100644
--- a/apps/web/src/lib/graphql-cache/worker/worker-core.ts
+++ b/apps/web/src/lib/graphql-cache/worker/worker-core.ts
@@ -26,12 +26,27 @@ import {
isStorageTransactionRequest,
operationCategoryForRequest,
} from '../telemetry';
+import { cacheDatabaseIdentity } from './coordinator-protocol';
import {
type CacheEngine,
type CacheOpenOutcome,
+ type CacheOpenResult,
loadCacheWasm,
} from './wasm-module';
+const isOwnerLockUnavailable = (error: unknown): boolean =>
+ typeof error === 'object' &&
+ error !== null &&
+ 'cacheOwnerLockUnavailable' in error &&
+ error.cacheOwnerLockUnavailable === true;
+
+/** WASM gave up on database files another context kept open. */
+const isStorageBusy = (error: unknown): boolean =>
+ typeof error === 'object' &&
+ error !== null &&
+ 'cacheStorageBusy' in error &&
+ error.cacheStorageBusy === true;
+
type PortLike = {
postMessage(msg: unknown): void;
};
@@ -53,8 +68,22 @@ export interface CacheWorkerCoreOptions {
recoveryOpen?: boolean;
/** Called once when WASM latches a reset-required storage failure. */
onStorageResetRequired?: (error: Error) => void;
+ /** Called when opening gave up on database files another context kept
+ * open. The failed request's response carries only the message. */
+ onStorageBusy?: () => void;
/** Reports the bounded open outcome to the coordinator transport. */
onInitializationOutcome?: (outcome: CacheOpenOutcome) => void;
+ /**
+ * Runs once WASM holds the database owner lock and before it touches OPFS;
+ * opening proceeds when the returned promise resolves.
+ */
+ onOwnerLockAcquired?: () => Promise<void>;
+ /**
+ * With `onOwnerLockAcquired`, opening never queues for the owner lock. This
+ * runs with the 1-based attempt that found it held elsewhere: resolve to
+ * try again, or reject to give up without touching storage.
+ */
+ onOwnerLockBusy?: (attempt: number) => Promise<void>;
telemetry?: CacheTelemetryRecorderLike;
/** Injectable clocks and cadence for payload-free diagnostics tests. */
monotonicNow?: () => number;
@@ -288,6 +317,7 @@ export class CacheWorkerCore {
});
}
this.reportResetRequired(error);
+ if (isStorageBusy(error)) this.options.onStorageBusy?.();
respond({
id: request.id,
ok: false,
@@ -858,14 +888,50 @@ export class CacheWorkerCore {
this.hotCapacity = hotCapacity;
this.initPromise = (async () => {
const wasm = await loadCacheWasm();
- const schemaStartedAt = this.now();
+ let schemaStartedAt = this.now();
+ const onOwnerLockAcquired = this.options.onOwnerLockAcquired;
+ const onOwnerLockBusy = this.options.onOwnerLockBusy;
+ let storageGrantRequested = false;
+ const awaitStorageGrant = onOwnerLockAcquired
+ ? async () => {
+ storageGrantRequested = true;
+ // Time the open itself, not the wait for another owner.
+ schemaStartedAt = this.now();
+ await onOwnerLockAcquired();
+ }
+ : undefined;
try {
let openOutcome: CacheOpenOutcome;
- if (this.options.recoveryOpen) {
+ if (awaitStorageGrant && onOwnerLockBusy) {
+ const open = this.options.recoveryOpen
+ ? wasm.openCacheForRecoveryWithOutcome
+ : wasm.openCacheWithOutcome;
+ // Checked before any open: an older artifact would queue for the
+ // lock and touch storage without the coordinator's grant.
+ if (!open || !wasm.cacheDatabaseIdentity) {
+ throw new Error(
+ 'cache WASM predates storage-versioned databases; rebuild it'
+ );
+ }
+ if (
+ wasm.cacheDatabaseIdentity(scope) !== cacheDatabaseIdentity(scope)
+ ) {
+ throw new Error(
+ 'cache WASM storage version does not match this build'
+ );
+ }
+ const opened = await this.openWhenOwnerLockIsFree(
+ () => open(scope, hotCapacity, awaitStorageGrant, true),
+ onOwnerLockBusy
+ );
+ this.engine = opened.engine;
+ openOutcome = opened.outcome;
+ } else if (this.options.recoveryOpen) {
if (wasm.openCacheForRecoveryWithOutcome) {
const opened = await wasm.openCacheForRecoveryWithOutcome(
scope,
- hotCapacity
+ hotCapacity,
+ awaitStorageGrant
);
this.engine = opened.engine;
openOutcome = opened.outcome;
@@ -874,13 +940,21 @@ export class CacheWorkerCore {
openOutcome = 'reset-storage-uncertain';
}
} else if (wasm.openCacheWithOutcome) {
- const opened = await wasm.openCacheWithOutcome(scope, hotCapacity);
+ const opened = await wasm.openCacheWithOutcome(
+ scope,
+ hotCapacity,
+ awaitStorageGrant
+ );
this.engine = opened.engine;
openOutcome = opened.outcome;
} else {
this.engine = await wasm.openCache(scope, hotCapacity);
openOutcome = 'opened-existing';
}
+ if (awaitStorageGrant && !storageGrantRequested) {
+ // A stale artifact opened storage without asking the coordinator.
+ throw new Error('cache WASM does not support the owner-lock grant');
+ }
this.options.onInitializationOutcome?.(openOutcome);
this.telemetry.record({
name: 'graphql_cache.schema_init',
@@ -905,6 +979,20 @@ export class CacheWorkerCore {
await this.initPromise;
}
+ private async openWhenOwnerLockIsFree(
+ open: () => Promise<CacheOpenResult>,
+ onOwnerLockBusy: (attempt: number) => Promise<void>
+ ): Promise<CacheOpenResult> {
+ for (let attempt = 1; ; attempt += 1) {
+ try {
+ return await open();
+ } catch (error) {
+ if (!isOwnerLockUnavailable(error)) throw error;
+ await onOwnerLockBusy(attempt);
+ }
+ }
+ }
+
private reportResetRequired(error: unknown): void {
if (
this.resetRequiredReported ||
diff --git a/apps/web/src/lib/queries/channel/channels.ts b/apps/web/src/lib/queries/channel/channels.ts
index 69b1bebcaf..d2b8903936 100644
--- a/apps/web/src/lib/queries/channel/channels.ts
+++ b/apps/web/src/lib/queries/channel/channels.ts
@@ -14,6 +14,7 @@ import type { CreateChannelRequest } from '@service-storage/generated/schemas/cr
import type { CreateChannelResponse } from '@service-storage/generated/schemas/createChannelResponse';
import type { PatchChannelRequest } from '@service-storage/generated/schemas/patchChannelRequest';
import { useMutation, useQuery } from '@tanstack/solid-query';
+import type { Accessor } from 'solid-js';
import { invalidateChannelParticipants } from './channel-participants';
import {
type CachedGraphqlChannel,
@@ -52,9 +53,12 @@ export function useListChannelsQuery() {
}
/** Reads the recent Quick Access channel list from the normalized GraphQL cache. */
-export function useCachedGraphqlChannelsQuery(cacheHost?: CacheHost) {
+export function useCachedGraphqlChannelsQuery(
+ cacheHost: Accessor<CacheHost | undefined>
+) {
return useQuery(() => {
- const enabledCacheHost = cacheHost?.disabled ? undefined : cacheHost;
+ const host = cacheHost();
+ const enabledCacheHost = host?.disabled ? undefined : host;
return {
queryKey: channelKeys.quickAccessGraphql(
enabledCacheHost?.clientId ?? 'disabled'
diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts
index bfeb217e59..363b7b265c 100644
--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts
+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.test.ts
@@ -1,4 +1,5 @@
import type { NormalizedCacheExchangeOptions } from '@graphql-cache/exchange/normalized-cache-exchange';
+import type { CacheHost } from '@graphql-cache/host/types';
import type { BrowserTursoCacheRolloutDecision } from '@graphql-cache/rollout-policy';
import type { Operation } from '@urql/core';
import { parse } from 'graphql';
@@ -117,6 +118,7 @@ const mocks = vi.hoisted(() => {
});
let queuedMutationCount = 0;
let initializationErrorHandler: ((error: Error) => void) | undefined;
+ let supersededHandler: (() => void) | undefined;
const cleanupOrder: string[] = [];
const host = {
disabled: false,
@@ -174,6 +176,7 @@ const mocks = vi.hoisted(() => {
queuedMutationCount = 0;
cleanupOrder.length = 0;
initializationErrorHandler = undefined;
+ supersededHandler = undefined;
},
queueDepth: () => queuedMutationCount,
recordSubscriptionDisposal: () => cleanupOrder.push('subscriptions'),
@@ -181,6 +184,8 @@ const mocks = vi.hoisted(() => {
failInitialization: (
error = new Error('injected initialization failure')
) => initializationErrorHandler?.(error),
+ supersede: () => supersededHandler?.(),
+ reloadForNewerBuild: vi.fn(),
plainClient,
realtimeClient,
replaceSubscriptions,
@@ -194,8 +199,12 @@ const mocks = vi.hoisted(() => {
})
),
createWorkerCacheHost: vi.fn(
- (options: { onInitializationError?: (error: Error) => void }) => {
+ (options: {
+ onInitializationError?: (error: Error) => void;
+ onSuperseded?: () => void;
+ }) => {
initializationErrorHandler = options.onInitializationError;
+ supersededHandler = options.onSuperseded;
return host;
}
),
@@ -220,6 +229,9 @@ vi.mock('@core/constant/servers', () => ({
SERVER_HOSTS: { 'document-storage-service': 'http://dss.test' },
}));
vi.mock('@core/util/fetchWithToken', () => ({ fetchToken: vi.fn() }));
+vi.mock('@core/util/reloadForNewerBuild', () => ({
+ reloadForNewerBuild: mocks.reloadForNewerBuild,
+}));
vi.mock('@core/util/platform', () => ({ isTauri: () => mocks.tauri }));
vi.mock('@core/util/platformFetch', () => ({
platformFetch: mocks.platformFetch,
@@ -707,6 +719,12 @@ describe('GraphQL Soup browser cache session gate', () => {
'@graphql-cache/host/navigation-error'
);
report?.(new CacheNavigationError(), operation);
+ report?.(
+ Object.assign(new Error('owner lock is held by another context'), {
+ errorCode: 'owner-lock-unavailable',
+ }),
+ operation
+ );
expect(mocks.telemetryError).not.toHaveBeenCalled();
report?.(error, operation);
@@ -818,6 +836,84 @@ describe('GraphQL Soup browser cache session gate', () => {
);
});
+ it('retires a failed host so the replaced client degrades instead of rejecting', async () => {
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const soup = await import('./graphql-soup');
+ soup.getGraphqlSoupClient();
+ const exchangeHost = mocks.normalizedCacheExchange.mock.calls[0]?.[0] as
+ | CacheHost
+ | undefined;
+ expect(exchangeHost?.disabled).toBe(false);
+
+ mocks.failInitialization();
+
+ // Operations still running on the old client, and callers that captured
+ // the host, now miss to the network rather than "host was disposed".
+ expect(mocks.host.dispose).toHaveBeenCalledOnce();
+ expect(exchangeHost?.disabled).toBe(true);
+ await expect(
+ exchangeHost?.readQuery({ query: 'query Q { user { id } }' })
+ ).resolves.toEqual({ kind: 'miss' });
+ await expect(
+ exchangeHost?.search({ profile: 'quick-access-v1', limit: 10 })
+ ).resolves.toEqual({ documents: [], nextCursor: null });
+ });
+
+ it('moves reactive cache readers off a host abandoned mid-session', async () => {
+ vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const soup = await import('./graphql-soup');
+ soup.getGraphqlSoupClient();
+ const { createMemo, createRoot } = await import('solid-js');
+ const readers = createRoot((dispose) => ({
+ dispose,
+ enabled: createMemo(() => soup.graphqlCacheEnabled()),
+ host: createMemo(() => soup.getGraphqlSoupCacheHost()),
+ }));
+ expect(readers.enabled()).toBe(true);
+ expect(readers.host()).toBeDefined();
+
+ mocks.failInitialization();
+
+ expect(readers.enabled()).toBe(false);
+ expect(readers.host()).toBeUndefined();
+ readers.dispose();
+ });
+
+ it('falls back quietly while another context holds the database', async () => {
+ const info = vi.spyOn(console, 'info').mockImplementation(() => {});
+ const soup = await import('./graphql-soup');
+ const cachedClient = soup.getGraphqlSoupClient();
+
+ mocks.failInitialization(
+ Object.assign(new Error('owner lock is held by another context'), {
+ errorCode: 'owner-lock-unavailable',
+ })
+ );
+
+ // Expected after a deploy or while a closing tab lets go: no toast and no
+ // error report, but the cache is off for this page session and its host
+ // is retired.
+ expect(mocks.toastFailure).not.toHaveBeenCalled();
+ expect(mocks.telemetryError).not.toHaveBeenCalled();
+ expect(info).toHaveBeenCalledExactlyOnceWith(
+ '[graphql-cache] owner lock is held by another context; using the network until reload'
+ );
+ expect(mocks.host.dispose).toHaveBeenCalledOnce();
+ expect(soup.graphqlCacheEnabled()).toBe(false);
+ expect(soup.getGraphqlSoupClient()).not.toBe(cachedClient);
+ });
+
+ it('moves the page to a newer build that took the database over', async () => {
+ const soup = await import('./graphql-soup');
+ soup.getGraphqlSoupClient();
+ expect(mocks.reloadForNewerBuild).not.toHaveBeenCalled();
+
+ mocks.supersede();
+
+ expect(mocks.reloadForNewerBuild).toHaveBeenCalledOnce();
+ expect(mocks.toastFailure).not.toHaveBeenCalled();
+ });
+
it('imports and uses the native path without constructing browser workers', async () => {
const WorkerConstructor = vi.fn(() => {
throw new Error('browser worker must not be constructed on Tauri');
diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
index 72c4eec347..aa83b7cf9c 100644
--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
@@ -8,11 +8,13 @@ import { SERVER_HOSTS } from '@core/constant/servers';
import { fetchToken } from '@core/util/fetchWithToken';
import { isTauri } from '@core/util/platform';
import { platformFetch } from '@core/util/platformFetch';
+import { reloadForNewerBuild } from '@core/util/reloadForNewerBuild';
import {
HYDRATE_ONLY_CONTEXT_KEY,
normalizedCacheExchange,
} from '@graphql-cache/exchange/normalized-cache-exchange';
import { CacheNavigationError } from '@graphql-cache/host/navigation-error';
+import { createRetirableCacheHost } from '@graphql-cache/host/retirable-host';
import type { CacheHost } from '@graphql-cache/host/types';
import {
createTauriCacheHost,
@@ -20,6 +22,7 @@ import {
entityFromArgument,
} from '@graphql-cache/index';
import { registerCacheHost } from '@graphql-cache/lifecycle';
+import { isOwnerLockUnavailableError } from '@graphql-cache/protocol';
import { getBrowserTursoCacheRolloutDecision } from '@graphql-cache/rollout';
import { getOrCreateCacheScope } from '@graphql-cache/scope';
import { Telemetry } from '@macro-inc/observability';
@@ -50,6 +53,7 @@ import {
createClient as createGraphqlWsClient,
type Client as GraphqlWsClient,
} from 'graphql-ws';
+import { createSignal } from 'solid-js';
import { match } from 'ts-pattern';
import { delegateChannelNotificationRefresh } from '../../queries/channel/notification-refresh';
import type { SoupApiItem } from './generated/schemas/soupApiItem';
@@ -348,6 +352,9 @@ function getUncachedRealtimeClient(): Client {
}
let cacheInitializationFailed = false;
+// Changes when the session abandons its cache so reactive readers of the
+// availability getters below (query options, effects) move to network paths.
+const [cacheAvailability, setCacheAvailability] = createSignal(0);
/**
* Whether the normalized cache is active for soup GraphQL queries.
@@ -355,6 +362,7 @@ let cacheInitializationFailed = false;
* process (graphql_cache_plugin).
*/
export function graphqlCacheEnabled(): boolean {
+ cacheAvailability();
if (cacheInitializationFailed) return false;
if (!isTauri() && browserCacheClientActivated) return true;
return getBrowserTursoCacheRolloutDecision().enabled;
@@ -379,10 +387,12 @@ function fallbackAfterInitializationFailure(): void {
} catch {
// Initialization-failure cleanup cannot alter GraphQL transport fallback.
}
+ setCacheAvailability((version) => version + 1);
}
/** Returns the persistent normalized-cache host after client initialization. */
export function getGraphqlCacheHost(): CacheHost | undefined {
+ cacheAvailability();
return cachedCacheHost?.disabled ? undefined : cachedCacheHost;
}
@@ -416,9 +426,14 @@ export function getGraphqlSoupClient(): Client {
operationKind?: Operation['kind']
) => {
try {
- // Navigation deliberately rejects outstanding reads. Do not turn that
- // expected shutdown into an error; unexpected disposal still reports.
- if (error instanceof CacheNavigationError) return;
+ // Navigation deliberately rejects outstanding reads, and a tab on
+ // another deployed build can own the database. Neither is an error;
+ // unexpected disposal still reports.
+ if (
+ error instanceof CacheNavigationError ||
+ isOwnerLockUnavailableError(error)
+ )
+ return;
// Caught cache failures never reach window.unhandledrejection. Report
// them through the Datadog-bound exporter without query/variable data.
Telemetry.error(error, {
@@ -442,11 +457,23 @@ export function getGraphqlSoupClient(): Client {
// Unsubscribing emits urql teardown operations; keep the cache host
// available until those best-effort registration removals are issued.
subscriptionsLifecycle.dispose();
+ // Retires the host: operations still running on this client, and any
+ // caller that captured the host, fall back to the network from here.
host?.dispose();
if (websocketClient) void websocketClient.dispose();
};
const onInitializationError = (error: Error) => {
if (!host || cachedCacheHost !== host) return;
+ // Another context holds the database: a tab on another deployed build,
+ // or a closing tab whose worker still has the files open. This page
+ // simply runs uncached until it reloads.
+ if (isOwnerLockUnavailableError(error)) {
+ fallbackAfterInitializationFailure();
+ console.info(
+ `[graphql-cache] ${error.message}; using the network until reload`
+ );
+ return;
+ }
reportCacheError(error, 'initialization');
fallbackAfterInitializationFailure();
toast.failure('Local cache unavailable', {
@@ -459,13 +486,17 @@ export function getGraphqlSoupClient(): Client {
};
try {
const scope = getOrCreateCacheScope();
- host = native
- ? createTauriCacheHost({ scope, onInitializationError })
- : createWorkerCacheHost({
- scope,
- onInitializationError,
- rolloutCohort: rollout.cohort,
- });
+ host = createRetirableCacheHost(
+ native
+ ? createTauriCacheHost({ scope, onInitializationError })
+ : createWorkerCacheHost({
+ scope,
+ onInitializationError,
+ rolloutCohort: rollout.cohort,
+ // A newer deploy took the local cache over: move to it.
+ onSuperseded: () => reloadForNewerBuild(),
+ })
+ );
const graphqlWsClient = createGraphqlSoupWebSocketClient(
subscriptionsLifecycle.connected
);
@@ -519,6 +550,7 @@ export function getGraphqlSoupClient(): Client {
cachedCacheHost = undefined;
cachedCacheCleanup = undefined;
cacheInitializationFailed = true;
+ setCacheAvailability((version) => version + 1);
console.warn('graphql cache init failed; using uncached client', error);
return isFeatureEnabled(enableGraphqlSoup)
? getUncachedRealtimeClient()
diff --git a/apps/web/src/vite-env.d.ts b/apps/web/src/vite-env.d.ts
index 9a4afc616c..7d089cb8da 100644
--- a/apps/web/src/vite-env.d.ts
+++ b/apps/web/src/vite-env.d.ts
@@ -1,5 +1,7 @@
interface ImportMetaEnv {
readonly __APP_VERSION__: string;
+ /** When this build was made, in ms since the epoch. Unset outside app builds. */
+ readonly __APP_BUILD_TIME__?: number;
readonly __LOCAL_JWT__: string;
readonly __GIT_BRANCH__: string;
diff --git a/apps/web/vite.base.ts b/apps/web/vite.base.ts
index d732e6ddb1..a4e550f571 100644
--- a/apps/web/vite.base.ts
+++ b/apps/web/vite.base.ts
@@ -25,6 +25,8 @@ function readShortSha(): string {
const shortSha = readShortSha();
const appVersion = `${version}+${shortSha}`;
+/** Orders builds: a newer build takes the local cache over from older tabs. */
+const appBuildTime = Date.now();
function readGitBranch(): string {
try {
@@ -252,6 +254,7 @@ function defineEnv(mode: string, command: string) {
});
return {
'import.meta.env.__APP_VERSION__': JSON.stringify(appVersion),
+ 'import.meta.env.__APP_BUILD_TIME__': JSON.stringify(appBuildTime),
'import.meta.env.ASSETS_PATH': JSON.stringify(getAssetsPath(mode, command)),
'import.meta.env.__LOCAL_DOCKER__': process.env.LOCAL_DOCKER === 'true',
'import.meta.env.__LOCAL_JWT__': JSON.stringify(process.env.LOCAL_JWT),
diff --git a/crates/client/cache-core/src/codec.rs b/crates/client/cache-core/src/codec.rs
index b1eeb58079..e1c3db1609 100644
--- a/crates/client/cache-core/src/codec.rs
+++ b/crates/client/cache-core/src/codec.rs
@@ -1,10 +1,11 @@
//! Binary record codec shared by all persistent Turso backends (browser and
//! native hosts). Records are stored as postcard bytes; the logical namespace
-//! embeds [`CACHE_FORMAT_VERSION`] and [`CACHE_SCHEMA_COMPATIBILITY_EPOCH`].
-//! A healthy, graceful browser reopen with matching versions preserves records,
-//! queued mutations, and optimistic layers. A compatibility/format mismatch or
-//! abrupt/uncertain browser owner loss physically resets the database and
-//! discards all three.
+//! embeds [`CACHE_FORMAT_VERSION`] and [`CACHE_SCHEMA_COMPATIBILITY_EPOCH`],
+//! and so does the browser's physical database name after
+//! [`UNVERSIONED_STORAGE_VERSIONS`]. A healthy, graceful
+//! browser reopen with matching versions preserves records, queued mutations,
+//! and optimistic layers. A compatibility/format mismatch or abrupt/uncertain
+//! browser owner loss physically resets the database and discards all three.
use crate::normalize::RecordUpdates;
use crate::queue::{PersistedOptimisticLayer, StoredMutation};
@@ -86,15 +87,41 @@ pub fn cache_namespace(scope: &str) -> String {
format!("graphql-cache:{scope}:s{CACHE_SCHEMA_COMPATIBILITY_EPOCH}:v{CACHE_FORMAT_VERSION}")
}
-/// Stable physical database name for a cache scope.
+/// Storage versions `(schema compatibility epoch, cache format, storage
+/// schema)` in use when database names began to embed versions. A build at
+/// exactly these versions keeps the unversioned name, so that change moved no
+/// database: the next build opens the same file, queued mutations included.
+pub const UNVERSIONED_STORAGE_VERSIONS: (u32, u32, u32) = (3, 3, 11);
+
+/// Physical database name for a cache scope at one storage version.
///
-/// Unlike [`cache_namespace`], this deliberately excludes the schema
-/// compatibility epoch and cache format version so the browser can acquire the
-/// same main/WAL paths and validate their metadata. Only a healthy, graceful
-/// close and compatible reopen preserves those files. A compatibility/format
-/// mismatch or abrupt/uncertain owner loss physically resets both files,
-/// discarding every row, including queued mutations and optimistic layers.
-pub fn cache_database_name(scope: &str) -> String {
+/// After [`UNVERSIONED_STORAGE_VERSIONS`], it embeds the schema compatibility
+/// epoch, the cache format version, and the backend's
+/// `storage_schema_version`, so builds that cannot share records use separate
+/// files and owner locks instead of resetting each other's database while tabs
+/// of both stay open after a deploy. Builds with matching versions share one
+/// database: only a healthy, graceful close and reopen preserves its files.
+/// Metadata is still validated on open, and a mismatch or an abrupt/uncertain
+/// owner loss physically resets both files, discarding every row, including
+/// queued mutations and optimistic layers.
+pub fn cache_database_name(scope: &str, storage_schema_version: u32) -> String {
+ let unversioned = legacy_cache_database_name(scope);
+ if (
+ CACHE_SCHEMA_COMPATIBILITY_EPOCH,
+ CACHE_FORMAT_VERSION,
+ storage_schema_version,
+ ) == UNVERSIONED_STORAGE_VERSIONS
+ {
+ return unversioned;
+ }
+ format!(
+ "{unversioned}:s{CACHE_SCHEMA_COMPATIBILITY_EPOCH}.v{CACHE_FORMAT_VERSION}.t{storage_schema_version}"
+ )
+}
+
+/// Physical name every storage version shared before names embedded versions,
+/// and still the name at [`UNVERSIONED_STORAGE_VERSIONS`].
+pub fn legacy_cache_database_name(scope: &str) -> String {
format!("graphql-cache:{scope}")
}
@@ -145,4 +172,21 @@ mod tests {
"graphql-cache:client-token-1:s3:v3"
);
}
+
+ #[test]
+ fn database_name_separates_storage_versions() {
+ // The versions of the last unversioned builds keep their file.
+ assert_eq!(
+ cache_database_name("client-token-1", 11),
+ "graphql-cache:client-token-1"
+ );
+ assert_eq!(
+ cache_database_name("client-token-1", 12),
+ "graphql-cache:client-token-1:s3.v3.t12"
+ );
+ assert_eq!(
+ legacy_cache_database_name("client-token-1"),
+ "graphql-cache:client-token-1"
+ );
+ }
}
diff --git a/crates/client/cache-turso/src/lib.rs b/crates/client/cache-turso/src/lib.rs
index 4bb4b3f4cf..9786d83561 100644
--- a/crates/client/cache-turso/src/lib.rs
+++ b/crates/client/cache-turso/src/lib.rs
@@ -17,13 +17,14 @@ mod storage;
pub use error::{PhysicalResetReason, TursoStorageError};
pub use storage::{
- STORAGE_SCHEMA_VERSION, TursoStorage, TursoStorageCloseOutcome, TursoStorageOpenOutcome,
+ QueuedMutationCount, STORAGE_SCHEMA_VERSION, TursoStorage, TursoStorageCloseOutcome,
+ TursoStorageOpenOutcome,
};
#[cfg(target_arch = "wasm32")]
pub use storage::{
HealthyTursoStorageClosed, ResetRequiredTursoStorageClosed, TursoStorageCloseFailure,
- TursoStorageOpenFailure, TursoStorageResetFailure,
+ TursoStorageOpenFailure, TursoStorageResetFailure, count_queued_mutations,
};
#[cfg(not(target_arch = "wasm32"))]
pub use storage::{TursoFileDatabase, TursoMemoryDatabase};
diff --git a/crates/client/cache-turso/src/storage.rs b/crates/client/cache-turso/src/storage.rs
index c7052841a7..d8bbb2c1c5 100644
--- a/crates/client/cache-turso/src/storage.rs
+++ b/crates/client/cache-turso/src/storage.rs
@@ -331,6 +331,42 @@ impl TursoStorage {
}
}
+/// Mutations still queued in a database of any storage schema version.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum QueuedMutationCount {
+ /// The file has a mutation queue holding this many rows.
+ Queued(u64),
+ /// The file has no mutation queue, so it never held queued work.
+ NoQueue,
+}
+
+/// Counts queued mutations without validating or changing the metadata.
+///
+/// Every storage schema version keeps its queue in `mutation_queue` and
+/// deletes a row once the mutation settles, so this is safe to call on a
+/// database written by an older build before deciding whether to delete it.
+#[cfg(target_arch = "wasm32")]
+pub fn count_queued_mutations(
+ session: &ConnectedOpfsSession,
+) -> Result<QueuedMutationCount, TursoStorageError> {
+ let connection = session.connection();
+ let count = |sql: &str| -> Result<u64, TursoStorageError> {
+ let rows = driver::query(&connection, sql, Vec::new())?;
+ let [row] = rows.as_slice() else {
+ return Err(invariant());
+ };
+ u64::try_from(required_i64(row, 0)?).map_err(|_| invariant())
+ };
+ if count("SELECT COUNT(*) FROM sqlite_schema WHERE type = 'table' AND name = 'mutation_queue'")?
+ == 0
+ {
+ return Ok(QueuedMutationCount::NoQueue);
+ }
+ Ok(QueuedMutationCount::Queued(count(
+ "SELECT COUNT(*) FROM mutation_queue",
+ )?))
+}
+
/// Result of consuming and gracefully closing a browser Turso storage.
#[cfg(target_arch = "wasm32")]
#[derive(Debug)]
diff --git a/crates/client/cache-wasm/Cargo.toml b/crates/client/cache-wasm/Cargo.toml
index d3b9f736cd..bb9290d17e 100644
--- a/crates/client/cache-wasm/Cargo.toml
+++ b/crates/client/cache-wasm/Cargo.toml
@@ -2,8 +2,10 @@
edition = "2024"
name = "cache-wasm"
publish = false
-# Rebuild for entity-only Soup hydration and bounded page retention.
-version = "0.6.30"
+# Rebuild for entity-only Soup hydration, bounded page retention,
+# storage-version database names, stale-database cleanup, and waiting out
+# files a departing worker holds.
+version = "0.6.31"
[features]
default = []
diff --git a/crates/client/cache-wasm/src/shell.rs b/crates/client/cache-wasm/src/shell.rs
index ed6abc3bf7..dc026e5d59 100644
--- a/crates/client/cache-wasm/src/shell.rs
+++ b/crates/client/cache-wasm/src/shell.rs
@@ -1,5 +1,8 @@
use async_lock::Mutex;
-use cache_core::codec::cache_database_name;
+use cache_core::codec::{
+ CACHE_FORMAT_VERSION, CACHE_SCHEMA_COMPATIBILITY_EPOCH, UNVERSIONED_STORAGE_VERSIONS,
+ cache_database_name, legacy_cache_database_name,
+};
use cache_core::deps::OpId;
use cache_core::engine::{
BeginOptimisticWrite, CommitOptimisticWriteResult, DeferOptimisticWriteResult, Engine,
@@ -18,8 +21,8 @@ use cache_core::search::SearchRequest;
use cache_core::store::QueueDiagnosticsAvailability;
use cache_core::value::EntityKey;
use cache_turso::{
- PhysicalResetReason, TursoStorage, TursoStorageCloseOutcome, TursoStorageError,
- TursoStorageOpenOutcome,
+ PhysicalResetReason, QueuedMutationCount, TursoStorage, TursoStorageCloseOutcome,
+ TursoStorageError, TursoStorageOpenOutcome, count_queued_mutations,
};
use predicate_index::RecordKey;
use serde::{Deserialize, Serialize};
@@ -33,9 +36,9 @@ use soup_filter_cache_adapter::{
use std::cell::RefCell;
use std::collections::HashMap;
use std::rc::Rc;
-use turso_opfs::{OpenResult, OpfsOwner};
+use turso_opfs::{OpenResult, OpfsError, OpfsErrorKind, OpfsOwner};
use wasm_bindgen::prelude::*;
-use wasm_bindgen_futures::future_to_promise;
+use wasm_bindgen_futures::{JsFuture, future_to_promise};
#[cfg(test)]
mod test;
@@ -492,7 +495,45 @@ pub struct CacheEngine {
}
fn database_identity(scope: &str) -> String {
- cache_database_name(scope)
+ cache_database_name(scope, cache_turso::STORAGE_SCHEMA_VERSION)
+}
+
+/// Whether `identity` names one of this scope's databases that this build does
+/// not open and that no newer build could want: the unversioned name once this
+/// build's versions have moved past it, or another name for versions no newer
+/// than this build's. A newer version belongs to a build that may come back,
+/// for example after a rollback, so it is never stale.
+fn is_stale_identity(scope: &str, identity: &str) -> bool {
+ let unversioned = legacy_cache_database_name(scope);
+ let version = if identity == unversioned {
+ Some(UNVERSIONED_STORAGE_VERSIONS)
+ } else {
+ storage_version(&unversioned, identity)
+ };
+ let own = (
+ CACHE_SCHEMA_COMPATIBILITY_EPOCH,
+ CACHE_FORMAT_VERSION,
+ cache_turso::STORAGE_SCHEMA_VERSION,
+ );
+ identity != database_identity(scope) && version.is_some_and(|version| version <= own)
+}
+
+/// The `(epoch, format, storage)` versions embedded in a scope's database name.
+fn storage_version(unversioned: &str, identity: &str) -> Option<(u32, u32, u32)> {
+ let mut parts = identity
+ .strip_prefix(unversioned)?
+ .strip_prefix(":s")?
+ .split('.');
+ let number = |part: &str| -> Option<u32> {
+ if part.is_empty() || !part.bytes().all(|byte| byte.is_ascii_digit()) {
+ return None;
+ }
+ part.parse().ok()
+ };
+ let epoch = number(parts.next()?)?;
+ let format = number(parts.next()?.strip_prefix('v')?)?;
+ let storage = number(parts.next()?.strip_prefix('t')?)?;
+ parts.next().is_none().then_some((epoch, format, storage))
}
fn build_engine(storage: TursoStorage, hot_capacity: Option<u32>) -> BrowserEngine {
@@ -515,7 +556,7 @@ async fn open_owner(owner: OpfsOwner) -> Result<OpenResult, JsValue> {
match owner.open().await {
Ok(opened) => Ok(opened),
Err(failure) => {
- let error = err_js(&failure);
+ let error = storage_err_js(failure.error());
if let Some(owner) = failure.into_owner() {
owner.release().await.map_err(err_js)?;
}
@@ -552,12 +593,21 @@ async fn open_storage(scope: &str, owner: OpfsOwner) -> Result<OpenedStorage, Js
.physical_reset_reason()
.expect("reset-required error has a reset reason"),
);
- (failure.reset().await.map_err(err_js)?, outcome)
+ (
+ failure
+ .reset()
+ .await
+ .map_err(|failure| storage_err_js(failure.error()))?,
+ outcome,
+ )
}
}
}
OpenResult::ResetRequired(session) => (
- session.reset().await.map_err(err_js)?,
+ session
+ .reset()
+ .await
+ .map_err(|failure| storage_err_js(failure.error()))?,
CacheOpenOutcome::ResetStorageUncertain,
),
};
@@ -572,7 +622,10 @@ async fn open_storage(scope: &str, owner: OpfsOwner) -> Result<OpenedStorage, Js
}),
Err(failure) => {
let reset_required = failure.error().requires_physical_reset();
- let owner = failure.reset().await.map_err(err_js)?;
+ let owner = failure
+ .reset()
+ .await
+ .map_err(|failure| storage_err_js(failure.error()))?;
owner.release().await.map_err(err_js)?;
Err(if reset_required {
reset_required_js_error()
@@ -583,19 +636,60 @@ async fn open_storage(scope: &str, owner: OpfsOwner) -> Result<OpenedStorage, Js
}
}
OpenResult::ResetRequired(session) => {
- let owner = session.reset().await.map_err(err_js)?;
+ let owner = session
+ .reset()
+ .await
+ .map_err(|failure| storage_err_js(failure.error()))?;
owner.release().await.map_err(err_js)?;
Err(reset_required_js_error())
}
}
}
-async fn acquire_storage(scope: &str, recovery_wipe: bool) -> Result<OpenedStorage, JsValue> {
- let owner = OpfsOwner::acquire(&database_identity(scope))
- .await
- .map_err(err_js)?;
+/// Waits until the caller may touch storage while this owner holds the lock.
+/// Holding the lock only proves no other engine uses the files; the browser
+/// coordinator must also record that this engine's epoch may change them.
+async fn await_storage_grant(
+ owner: OpfsOwner,
+ grant: &js_sys::Function,
+) -> Result<OpfsOwner, JsValue> {
+ let granted = match grant.call0(&JsValue::UNDEFINED) {
+ Ok(value) => JsFuture::from(js_sys::Promise::resolve(&value)).await,
+ Err(error) => Err(error),
+ };
+ match granted {
+ Ok(_) => Ok(owner),
+ Err(error) => {
+ owner.release().await.map_err(err_js)?;
+ Err(error)
+ }
+ }
+}
+
+async fn acquire_storage(
+ scope: &str,
+ recovery_wipe: bool,
+ storage_grant: Option<&js_sys::Function>,
+ if_available: bool,
+) -> Result<OpenedStorage, JsValue> {
+ let identity = database_identity(scope);
+ let owner = if if_available {
+ OpfsOwner::try_acquire(&identity)
+ .await
+ .map_err(err_js)?
+ .ok_or_else(owner_lock_unavailable_js_error)?
+ } else {
+ OpfsOwner::acquire(&identity).await.map_err(err_js)?
+ };
+ let owner = match storage_grant {
+ Some(grant) => await_storage_grant(owner, grant).await?,
+ None => owner,
+ };
let owner = if recovery_wipe {
- owner.recovery_wipe().await.map_err(err_js)?
+ owner
+ .recovery_wipe()
+ .await
+ .map_err(|failure| storage_err_js(failure.error()))?
} else {
owner
};
@@ -610,9 +704,12 @@ async fn open_cache_inner(
scope: String,
hot_capacity: Option<u32>,
recovery_wipe: bool,
+ storage_grant: Option<js_sys::Function>,
+ if_available: bool,
) -> Result<(CacheEngine, CacheOpenOutcome), JsValue> {
validate_hot_capacity(hot_capacity)?;
- let OpenedStorage { storage, outcome } = acquire_storage(&scope, recovery_wipe).await?;
+ let OpenedStorage { storage, outcome } =
+ acquire_storage(&scope, recovery_wipe, storage_grant.as_ref(), if_available).await?;
Ok((
CacheEngine {
state: Rc::new(Mutex::new(CacheState {
@@ -648,18 +745,40 @@ fn cache_open_result(engine: CacheEngine, outcome: CacheOpenOutcome) -> Result<J
/// incomplete or incompatible files are reset and reopened before returning.
#[wasm_bindgen(js_name = openCache)]
pub async fn open_cache(scope: String, hot_capacity: Option<u32>) -> Result<CacheEngine, JsValue> {
- open_cache_inner(scope, hot_capacity, false)
+ open_cache_inner(scope, hot_capacity, false, None, false)
.await
.map(|(engine, _)| engine)
}
/// Additive open API returning the engine and payload-free recovery outcome.
+///
+/// When given, `on_owner_lock_acquired` is called once the owner lock is held
+/// and before any OPFS access; opening continues after its promise resolves.
+/// A rejection releases the lock without touching storage.
+///
+/// With `if_available`, opening never waits behind another holder of the
+/// owner lock: it rejects at once, touching nothing, with an error whose
+/// `cacheOwnerLockUnavailable` property is `true`.
+///
+/// When another context keeps the database files open through the adapter's
+/// bounded wait, opening rejects with an error whose `cacheStorageBusy`
+/// property is `true`. A plain open has then changed neither file; a recovery
+/// wipe may have removed one of them.
#[wasm_bindgen(js_name = openCacheWithOutcome)]
pub async fn open_cache_with_outcome(
scope: String,
hot_capacity: Option<u32>,
+ on_owner_lock_acquired: Option<js_sys::Function>,
+ if_available: Option<bool>,
) -> Result<JsValue, JsValue> {
- let (engine, outcome) = open_cache_inner(scope, hot_capacity, false).await?;
+ let (engine, outcome) = open_cache_inner(
+ scope,
+ hot_capacity,
+ false,
+ on_owner_lock_acquired,
+ if_available.unwrap_or(false),
+ )
+ .await?;
cache_open_result(engine, outcome)
}
@@ -670,21 +789,105 @@ pub async fn open_cache_for_recovery(
scope: String,
hot_capacity: Option<u32>,
) -> Result<CacheEngine, JsValue> {
- open_cache_inner(scope, hot_capacity, true)
+ open_cache_inner(scope, hot_capacity, true, None, false)
.await
.map(|(engine, _)| engine)
}
/// Additive recovery-open API returning the engine and coarse wipe outcome.
+/// `on_owner_lock_acquired` and `if_available` behave as in
+/// [`openCacheWithOutcome`](open_cache_with_outcome); both precede the wipe.
#[wasm_bindgen(js_name = openCacheForRecoveryWithOutcome)]
pub async fn open_cache_for_recovery_with_outcome(
scope: String,
hot_capacity: Option<u32>,
+ on_owner_lock_acquired: Option<js_sys::Function>,
+ if_available: Option<bool>,
) -> Result<JsValue, JsValue> {
- let (engine, outcome) = open_cache_inner(scope, hot_capacity, true).await?;
+ let (engine, outcome) = open_cache_inner(
+ scope,
+ hot_capacity,
+ true,
+ on_owner_lock_acquired,
+ if_available.unwrap_or(false),
+ )
+ .await?;
cache_open_result(engine, outcome)
}
+/// Physical database identity this build opens for `scope`; it also names the
+/// OPFS files and derives the owner Web Lock name.
+#[wasm_bindgen(js_name = cacheDatabaseIdentity)]
+pub fn cache_database_identity(scope: String) -> String {
+ database_identity(&scope)
+}
+
+#[derive(Serialize)]
+#[serde(rename_all = "camelCase")]
+struct StaleDatabaseCleanup {
+ outcome: &'static str,
+ #[serde(skip_serializing_if = "Option::is_none")]
+ queued_mutations: Option<u64>,
+}
+
+/// Deletes one of this scope's databases that an older build left behind (see
+/// [`cacheDatabaseIdentity`](cache_database_identity)), unless another context
+/// holds its owner lock (`in-use`) or it still queues mutations
+/// (`queued-mutations`). Never waits for the lock. A file Turso cannot read,
+/// or whose queue cannot be counted, is kept and rejects; the caller should
+/// use a disposable worker, because such a failure can poison the
+/// worker-local OPFS registry.
+#[wasm_bindgen(js_name = removeStaleCacheDatabase)]
+pub async fn remove_stale_cache_database(
+ scope: String,
+ identity: String,
+) -> Result<JsValue, JsValue> {
+ if !is_stale_identity(&scope, &identity) {
+ return Err(err_js("not a stale cache database for this scope"));
+ }
+ let Some(owner) = OpfsOwner::try_acquire(&identity).await.map_err(err_js)? else {
+ return to_js(&StaleDatabaseCleanup {
+ outcome: "in-use",
+ queued_mutations: None,
+ });
+ };
+ let owner = match open_owner(owner).await? {
+ OpenResult::Ready(session) => {
+ let connected = session.connect().map_err(err_js)?;
+ let counted = count_queued_mutations(&connected);
+ let owner = connected
+ .try_close()
+ .map_err(err_js)?
+ .preserve()
+ .map_err(err_js)?;
+ match counted {
+ Ok(QueuedMutationCount::Queued(queued)) if queued > 0 => {
+ owner.release().await.map_err(err_js)?;
+ return to_js(&StaleDatabaseCleanup {
+ outcome: "queued-mutations",
+ queued_mutations: Some(queued),
+ });
+ }
+ // A queue that cannot be counted may still hold work, so the
+ // file is kept and the cleanup rejects as for an unreadable file.
+ Err(error) => {
+ owner.release().await.map_err(err_js)?;
+ return Err(err_js(error));
+ }
+ // An empty queue or no queue holds no work any build could send.
+ Ok(_) => owner,
+ }
+ }
+ // Only one of the pair exists, so it never held a complete database.
+ OpenResult::ResetRequired(session) => session.reset().await.map_err(err_js)?,
+ };
+ owner.remove_and_release().await.map_err(err_js)?;
+ to_js(&StaleDatabaseCleanup {
+ outcome: "removed",
+ queued_mutations: None,
+ })
+}
+
/// Recovery-wipes and recreates the cache database for `scope` while holding
/// the same exclusive OPFS owner lock used by [`openCache`](open_cache).
#[wasm_bindgen(js_name = destroyCache)]
@@ -807,6 +1010,9 @@ fn to_js<T: Serialize>(value: &T) -> Result<JsValue, JsValue> {
const RESET_REQUIRED_MARKER: &str = "cacheStorageResetRequired";
const RESET_REQUIRED_MESSAGE: &str = "cache storage reset required";
+const OWNER_LOCK_UNAVAILABLE_MARKER: &str = "cacheOwnerLockUnavailable";
+const STORAGE_BUSY_MARKER: &str = "cacheStorageBusy";
+const OWNER_LOCK_UNAVAILABLE_MESSAGE: &str = "cache database owner lock is held by another context";
/// All rejections surface as real `Error` objects with consistent
/// `instanceof Error` and `.message` behavior.
@@ -829,6 +1035,32 @@ fn reset_required_js_error() -> JsValue {
error.into()
}
+/// Converts an OPFS adapter error, marking a path another context kept open
+/// for the whole busy wait with a `cacheStorageBusy` property of `true`.
+fn storage_err_js(error: &OpfsError) -> JsValue {
+ let value = err_js(error);
+ if error.kind() == OpfsErrorKind::Busy {
+ js_sys::Reflect::set(
+ &value,
+ &JsValue::from_str(STORAGE_BUSY_MARKER),
+ &JsValue::TRUE,
+ )
+ .expect("new JavaScript Error accepts the busy marker");
+ }
+ value
+}
+
+fn owner_lock_unavailable_js_error() -> JsValue {
+ let error = js_sys::Error::new(OWNER_LOCK_UNAVAILABLE_MESSAGE);
+ js_sys::Reflect::set(
+ error.as_ref(),
+ &JsValue::from_str(OWNER_LOCK_UNAVAILABLE_MARKER),
+ &JsValue::TRUE,
+ )
+ .expect("new JavaScript Error accepts the unavailable marker");
+ error.into()
+}
+
fn engine_error_requires_reset(error: &EngineError<TursoStorageError>) -> bool {
matches!(
error,
diff --git a/crates/client/cache-wasm/src/shell/test.rs b/crates/client/cache-wasm/src/shell/test.rs
index 26ea8cf31f..78eccb5f98 100644
--- a/crates/client/cache-wasm/src/shell/test.rs
+++ b/crates/client/cache-wasm/src/shell/test.rs
@@ -2196,3 +2196,342 @@ async fn calls_serialize_and_owner_lock_excludes_a_second_open() {
.expect("close releases owner lock");
close_and_destroy(&reopened, SCOPE).await;
}
+
+async fn run_js(body: &str, argument: &str) -> JsValue {
+ let script = js_sys::Function::new_with_args("name", body);
+ let promise = script
+ .call1(&JsValue::UNDEFINED, &JsValue::from_str(argument))
+ .expect("test script runs")
+ .dyn_into::<js_sys::Promise>()
+ .expect("test script returns a promise");
+ JsFuture::from(promise).await.expect("test script settles")
+}
+
+async fn opfs_file_exists(name: &str) -> bool {
+ run_js(
+ "return (async () => {
+ const root = await navigator.storage.getDirectory();
+ try { await root.getFileHandle(name); return true; }
+ catch (error) { if (error.name === 'NotFoundError') return false; throw error; }
+ })()",
+ name,
+ )
+ .await
+ .as_bool()
+ .expect("existence is boolean")
+}
+
+async fn remove_opfs_file(name: &str) {
+ run_js(
+ "return (async () => {
+ const root = await navigator.storage.getDirectory();
+ await root.removeEntry(name).catch(() => undefined);
+ })()",
+ name,
+ )
+ .await;
+}
+
+async fn owner_lock_available(scope: &str) -> bool {
+ let identity = database_identity(scope);
+ let lock = format!("macro:turso-opfs:v1:{}:{identity}", identity.len());
+ run_js(
+ "return navigator.locks.request(
+ name, { mode: 'exclusive', ifAvailable: true }, (lock) => lock !== null
+ )",
+ &lock,
+ )
+ .await
+ .as_bool()
+ .expect("lock probe is boolean")
+}
+
+async fn close_opened(result: &JsValue) {
+ let engine = js_sys::Reflect::get(result, &JsValue::from_str("engine")).expect("engine");
+ let close = js_sys::Reflect::get(&engine, &JsValue::from_str("close"))
+ .expect("close")
+ .dyn_into::<js_sys::Function>()
+ .expect("close is callable");
+ let closing = close
+ .call0(&engine)
+ .expect("close starts")
+ .dyn_into::<js_sys::Promise>()
+ .expect("close returns a promise");
+ JsFuture::from(closing).await.expect("engine closes");
+}
+
+#[wasm_bindgen_test(async)]
+async fn outcome_opens_touch_opfs_only_after_their_storage_grant() {
+ const SCOPE: &str = "cache-wasm-storage-grant";
+ let database = database_identity(SCOPE);
+ let wal = format!("{database}-wal");
+ for recovery in [false, true] {
+ remove_opfs_file(&database).await;
+ remove_opfs_file(&wal).await;
+
+ // Refusing the grant releases the owner lock without touching OPFS.
+ let refused = js_sys::Function::new_no_args(
+ "return Promise.reject(new Error('storage grant refused'))",
+ );
+ let error = if recovery {
+ open_cache_for_recovery_with_outcome(SCOPE.into(), None, Some(refused), None).await
+ } else {
+ open_cache_with_outcome(SCOPE.into(), None, Some(refused), None).await
+ }
+ .expect_err("a refused grant fails the open");
+ assert_eq!(
+ error
+ .dyn_into::<js_sys::Error>()
+ .expect("error object")
+ .message(),
+ "storage grant refused"
+ );
+ assert!(owner_lock_available(SCOPE).await);
+ assert!(!opfs_file_exists(&database).await);
+
+ // A granted open finds storage untouched when it asks.
+ let observed = Rc::new(RefCell::new(None));
+ let observer = observed.clone();
+ let file = database.clone();
+ let grant = Closure::wrap(Box::new(move || -> js_sys::Promise {
+ let observer = observer.clone();
+ let file = file.clone();
+ future_to_promise(async move {
+ *observer.borrow_mut() = Some(opfs_file_exists(&file).await);
+ Ok(JsValue::UNDEFINED)
+ })
+ }) as Box<dyn FnMut() -> js_sys::Promise>)
+ .into_js_value()
+ .unchecked_into::<js_sys::Function>();
+ let opened = if recovery {
+ open_cache_for_recovery_with_outcome(SCOPE.into(), None, Some(grant), None).await
+ } else {
+ open_cache_with_outcome(SCOPE.into(), None, Some(grant), None).await
+ }
+ .expect("a granted open succeeds");
+ assert_eq!(*observed.borrow(), Some(false));
+ assert!(opfs_file_exists(&database).await);
+ close_opened(&opened).await;
+ }
+ destroy_cache(SCOPE.into()).await.expect("destroy cache");
+}
+
+async fn hold_web_lock(lock: &str) {
+ run_js(
+ "return new Promise((granted) => {
+ const request = navigator.locks.request(name, { mode: 'exclusive' }, () =>
+ new Promise((release) => {
+ globalThis.cacheWasmHeldLock = { release, request };
+ granted(true);
+ })
+ );
+ })",
+ lock,
+ )
+ .await;
+}
+
+async fn release_held_web_lock() {
+ run_js(
+ "return (async () => {
+ const held = globalThis.cacheWasmHeldLock;
+ delete globalThis.cacheWasmHeldLock;
+ held.release();
+ await held.request;
+ })()",
+ "",
+ )
+ .await;
+}
+
+fn owner_lock_for(identity: &str) -> String {
+ format!("macro:turso-opfs:v1:{}:{identity}", identity.len())
+}
+
+async fn remove_opfs_pair(identity: &str) {
+ remove_opfs_file(identity).await;
+ remove_opfs_file(&format!("{identity}-wal")).await;
+}
+
+async fn create_opfs_file(name: &str) {
+ run_js(
+ "return (async () => {
+ const root = await navigator.storage.getDirectory();
+ await root.getFileHandle(name, { create: true });
+ })()",
+ name,
+ )
+ .await;
+}
+
+/// Renames a closed main/WAL pair, as if another storage version wrote it.
+async fn move_opfs_pair(from: &str, to: &str) {
+ run_js(
+ "return (async () => {
+ const [from, to] = JSON.parse(name);
+ const root = await navigator.storage.getDirectory();
+ for (const suffix of ['', '-wal']) {
+ await (await root.getFileHandle(from + suffix)).move(to + suffix);
+ }
+ })()",
+ &serde_json::json!([from, to]).to_string(),
+ )
+ .await;
+}
+
+async fn queue_one_mutation(engine: &CacheEngine) {
+ resolved(engine.enqueue_optimistic_mutation(
+ None,
+ "00000000-0000-4000-8000-000000000011".into(),
+ PROPERTY_MUTATION.into(),
+ Some("SetEntityProperty".into()),
+ js(mutation_variables()),
+ js(serde_json::json!({
+ "setEntityProperty": { "id": "prop-1", "displayName": "Queued" }
+ })),
+ JsValue::UNDEFINED,
+ JsValue::UNDEFINED,
+ 1.0,
+ "stale-owner".into(),
+ 0.0,
+ 100.0,
+ ))
+ .await;
+}
+
+async fn stale_cleanup(scope: &str, identity: &str) -> serde_json::Value {
+ from_js(
+ remove_stale_cache_database(scope.into(), identity.into())
+ .await
+ .expect("stale cleanup settles"),
+ )
+}
+
+#[wasm_bindgen_test]
+fn stale_identities_are_only_this_scopes_other_databases() {
+ let scope = "scope-a";
+ let own = cache_database_identity(scope.into());
+ assert_eq!(
+ own,
+ cache_database_name(scope, cache_turso::STORAGE_SCHEMA_VERSION)
+ );
+ assert!(!is_stale_identity(scope, &own));
+ // The unversioned name is stale only once this build's versions move on.
+ assert_eq!(
+ is_stale_identity(scope, "graphql-cache:scope-a"),
+ own != "graphql-cache:scope-a"
+ );
+ assert!(is_stale_identity(scope, "graphql-cache:scope-a:s1.v2.t3"));
+ let versioned = |storage: u32| {
+ format!(
+ "graphql-cache:scope-a:s{}.v{}.t{storage}",
+ cache_core::codec::CACHE_SCHEMA_COMPATIBILITY_EPOCH,
+ cache_core::codec::CACHE_FORMAT_VERSION,
+ )
+ };
+ let current = versioned(cache_turso::STORAGE_SCHEMA_VERSION);
+ // Another spelling of this build's own versions is never opened again.
+ assert_eq!(is_stale_identity(scope, &current), own != current);
+ let newer = versioned(cache_turso::STORAGE_SCHEMA_VERSION + 1);
+ for other in [
+ // A newer build may come back after a rollback, so it keeps its file.
+ newer.as_str(),
+ "graphql-cache:scope-a:s99.v0.t0",
+ "graphql-cache:scope-b:s1.v2.t3",
+ "graphql-cache:scope-ab",
+ "graphql-cache:scope-a-wal",
+ "graphql-cache:scope-a:s1.v2",
+ "graphql-cache:scope-a:s1.v2.t3.x",
+ "graphql-cache:scope-a:sx.v2.t3",
+ "graphql-cache:scope-a:s1.v.t3",
+ ] {
+ assert!(!is_stale_identity(scope, other), "{other}");
+ }
+}
+
+#[wasm_bindgen_test(async)]
+async fn outcome_opens_can_decline_to_wait_for_a_held_owner_lock() {
+ const SCOPE: &str = "cache-wasm-if-available";
+ let database = database_identity(SCOPE);
+ remove_opfs_pair(&database).await;
+ hold_web_lock(&owner_lock_for(&database)).await;
+ for recovery in [false, true] {
+ let error = if recovery {
+ open_cache_for_recovery_with_outcome(SCOPE.into(), None, None, Some(true)).await
+ } else {
+ open_cache_with_outcome(SCOPE.into(), None, None, Some(true)).await
+ }
+ .expect_err("a held owner lock declines the open");
+ assert_eq!(
+ js_sys::Reflect::get(&error, &JsValue::from_str("cacheOwnerLockUnavailable"))
+ .expect("marker lookup"),
+ JsValue::TRUE
+ );
+ assert!(!opfs_file_exists(&database).await);
+ }
+ release_held_web_lock().await;
+ let opened = open_cache_with_outcome(SCOPE.into(), None, None, Some(true))
+ .await
+ .expect("a free owner lock opens");
+ close_opened(&opened).await;
+ destroy_cache(SCOPE.into()).await.expect("destroy cache");
+}
+
+#[wasm_bindgen_test(async)]
+async fn stale_databases_are_removed_only_when_unused_and_empty() {
+ const SCOPE: &str = "cache-wasm-stale-cleanup";
+ let own = database_identity(SCOPE);
+ let unversioned = legacy_cache_database_name(SCOPE);
+ let queued = format!("{unversioned}:s0.v0.t0");
+ let older = format!("{unversioned}:s0.v0.t1");
+ let partial = format!("{unversioned}:s0.v0.t2");
+ for identity in [&own, &unversioned, &queued, &older, &partial] {
+ remove_opfs_pair(identity).await;
+ }
+
+ // An older build's database still queues a mutation, so it is kept.
+ let engine = fresh_engine(SCOPE).await;
+ queue_one_mutation(&engine).await;
+ resolved(engine.close()).await;
+ move_opfs_pair(&own, &queued).await;
+ assert_eq!(
+ stale_cleanup(SCOPE, &queued).await,
+ serde_json::json!({ "outcome": "queued-mutations", "queuedMutations": 1 })
+ );
+ assert!(opfs_file_exists(&queued).await);
+
+ // A live engine of that build holds its lock, so nothing is opened.
+ hold_web_lock(&owner_lock_for(&queued)).await;
+ assert_eq!(
+ stale_cleanup(SCOPE, &queued).await,
+ serde_json::json!({ "outcome": "in-use" })
+ );
+ release_held_web_lock().await;
+
+ // An empty older database and a one-sided pair are deleted outright.
+ let engine = fresh_engine(SCOPE).await;
+ resolved(engine.close()).await;
+ move_opfs_pair(&own, &older).await;
+ assert_eq!(
+ stale_cleanup(SCOPE, &older).await,
+ serde_json::json!({ "outcome": "removed" })
+ );
+ create_opfs_file(&format!("{partial}-wal")).await;
+ assert_eq!(
+ stale_cleanup(SCOPE, &partial).await,
+ serde_json::json!({ "outcome": "removed" })
+ );
+ for identity in [&older, &partial] {
+ assert!(!opfs_file_exists(identity).await);
+ assert!(!opfs_file_exists(&format!("{identity}-wal")).await);
+ }
+
+ // This build's own database and other scopes' databases are refused.
+ for identity in [own.clone(), legacy_cache_database_name("another-scope")] {
+ remove_stale_cache_database(SCOPE.into(), identity)
+ .await
+ .expect_err("not a stale database of this scope");
+ }
+ remove_opfs_pair(&queued).await;
+ remove_opfs_pair(&own).await;
+}
diff --git a/crates/client/turso-opfs/src/browser.rs b/crates/client/turso-opfs/src/browser.rs
index 44e0dbb5bc..0eb1f2ee0b 100644
--- a/crates/client/turso-opfs/src/browser.rs
+++ b/crates/client/turso-opfs/src/browser.rs
@@ -24,7 +24,7 @@ use wasm_bindgen_futures::JsFuture;
use web_sys::{
AbortController, DedicatedWorkerGlobalScope, DomException, FileSystemDirectoryHandle,
FileSystemFileHandle, FileSystemGetFileOptions, FileSystemReadWriteOptions,
- FileSystemRemoveOptions, FileSystemSyncAccessHandle,
+ FileSystemRemoveOptions, FileSystemSyncAccessHandle, WorkerGlobalScope,
};
#[cfg(test)]
@@ -79,6 +79,13 @@ pub enum OpfsErrorKind {
Poisoned,
/// Turso requested a synchronous operation that this adapter forbids.
Unsupported,
+ /// Another context, usually a predecessor worker still being torn down,
+ /// kept a database path open for the whole busy-entry wait. A busy open
+ /// changed neither file and hands its owner back. A busy wipe or reset may
+ /// already have removed the other path and poisons the owner like any
+ /// partial wipe, so the worker must be replaced and the next owner wipes
+ /// again; a busy [`OpfsOwner::remove_and_release`] still releases the lock.
+ Busy,
}
/// A payload-free OPFS adapter error.
@@ -148,7 +155,11 @@ impl OpfsOwner {
pub async fn acquire(database_identity: &str) -> Result<Self, OpfsError> {
let paths = approved_paths(database_identity)?;
let lock_name = owner_lock_name(database_identity);
- let id = acquire_database_lock(database_identity, &lock_name).await?;
+ let id = acquire_database_lock(database_identity, &lock_name, false)
+ .await?
+ .ok_or_else(|| {
+ OpfsError::new(OpfsErrorKind::Lock, "exclusive Web Lock was not granted")
+ })?;
Ok(Self {
id,
paths,
@@ -158,6 +169,24 @@ impl OpfsOwner {
})
}
+ /// Acquires this identity's exclusive Web Lock only if it is free now,
+ /// never queueing behind another holder or waiter.
+ ///
+ /// Returns `None`, having touched no storage, when the lock is taken.
+ pub async fn try_acquire(database_identity: &str) -> Result<Option<Self>, OpfsError> {
+ let paths = approved_paths(database_identity)?;
+ let lock_name = owner_lock_name(database_identity);
+ Ok(acquire_database_lock(database_identity, &lock_name, true)
+ .await?
+ .map(|id| Self {
+ id,
+ paths,
+ lock_name,
+ fresh: false,
+ armed: true,
+ }))
+ }
+
/// Returns the canonical database identity bound into this owner.
pub fn database_path(&self) -> &str {
self.paths.get(FileRole::Main)
@@ -207,6 +236,7 @@ impl OpfsOwner {
/// A one-sided pre-existing pair returns [`OpenResult::ResetRequired`], a
/// type that exposes no Turso I/O and can only be physically reset.
pub async fn open(mut self) -> Result<OpenResult, OpenFailure> {
+ let deadline = BusyDeadline::start();
let paths = self.paths.clone();
let session = match REGISTRY.with(|registry| {
registry
@@ -236,15 +266,16 @@ impl OpfsOwner {
return Err(open_failure_after_cleanup(error, self));
}
- let (handle, was_created) = match open_sync_handle(&root, paths.get(role)).await {
- Ok(value) => value,
- Err(error) => {
- let adapter_error =
- js_error(OpfsErrorKind::Open, "OPFS sync handle open", &error);
- cleanup_failed_open(self.id, session, &adapter_error);
- return Err(open_failure_after_cleanup(adapter_error, self));
- }
- };
+ let (handle, was_created) =
+ match open_sync_handle(&root, paths.get(role), deadline).await {
+ Ok(value) => value,
+ Err(error) => {
+ let adapter_error =
+ entry_error(OpfsErrorKind::Open, "OPFS sync handle open", &error);
+ cleanup_failed_open(self.id, session, &adapter_error);
+ return Err(open_failure_after_cleanup(adapter_error, self));
+ }
+ };
created[role.index()] = was_created;
let handle_id = match REGISTRY.with(|registry| {
@@ -332,6 +363,29 @@ impl OpfsOwner {
guard.disarm();
Ok(())
}
+
+ /// Deletes both approved paths without recreating them, then releases the
+ /// lock. This discards a database that no context uses any more.
+ ///
+ /// Unlike [`Self::recovery_wipe`], a failed removal does not poison the
+ /// registry: the lock is still released, the error is returned, and any
+ /// path left behind can be removed by a later attempt.
+ pub async fn remove_and_release(self) -> Result<(), OpfsError> {
+ REGISTRY
+ .with(|registry| {
+ registry
+ .borrow_mut()
+ .machine
+ .start_wipe(self.id, self.paths.clone())
+ })
+ .map_err(state_error)?;
+ let removed = remove_paths(&self.paths).await;
+ REGISTRY
+ .with(|registry| registry.borrow_mut().machine.finish_wipe(self.id))
+ .map_err(state_error)?;
+ self.release().await?;
+ removed
+ }
}
impl Drop for OpfsOwner {
@@ -1261,10 +1315,13 @@ impl Drop for PendingAcquireGuard {
}
}
+/// Requests the exclusive lock, queueing unless `if_available` is set. With
+/// `if_available`, returns `None` without an owner when the lock is taken.
async fn acquire_database_lock(
database_identity: &str,
lock_name: &str,
-) -> Result<OwnerId, OpfsError> {
+ if_available: bool,
+) -> Result<Option<OwnerId>, OpfsError> {
let worker = js_sys::global()
.dyn_into::<DedicatedWorkerGlobalScope>()
.map_err(|error| js_error(OpfsErrorKind::Lock, "DedicatedWorker lock scope", &error))?;
@@ -1299,12 +1356,23 @@ async fn acquire_database_lock(
&JsValue::from_str("exclusive"),
)
.map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock mode", &error))?;
- Reflect::set(
- options.as_ref(),
- &JsValue::from_str("signal"),
- controller.signal().as_ref(),
- )
- .map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock abort signal", &error))?;
+ // The Web Locks API rejects an abort signal combined with ifAvailable. An
+ // ifAvailable request settles at once, so it needs no cancellation.
+ if if_available {
+ Reflect::set(
+ options.as_ref(),
+ &JsValue::from_str("ifAvailable"),
+ &JsValue::TRUE,
+ )
+ .map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock ifAvailable", &error))?;
+ } else {
+ Reflect::set(
+ options.as_ref(),
+ &JsValue::from_str("signal"),
+ controller.signal().as_ref(),
+ )
+ .map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock abort signal", &error))?;
+ }
let mut acquired_resolve = None;
let mut acquired_reject = None;
@@ -1322,6 +1390,11 @@ async fn acquire_database_lock(
lock_name: lock_name.to_owned(),
};
let grant = Closure::wrap(Box::new(move |lock: JsValue| -> Promise {
+ if (lock.is_null() || lock.is_undefined()) && if_available {
+ // Another context holds or awaits the lock; nothing was claimed.
+ let _ = callback_resolve.call1(&JsValue::UNDEFINED, &JsValue::FALSE);
+ return Promise::resolve(&JsValue::UNDEFINED);
+ }
if lock.is_null() || lock.is_undefined() {
let _ = callback_reject.call1(
&JsValue::UNDEFINED,
@@ -1495,9 +1568,14 @@ async fn acquire_database_lock(
));
}
- JsFuture::from(acquired)
+ let granted = JsFuture::from(acquired)
.await
.map_err(|error| js_error(OpfsErrorKind::Lock, "Web Lock acquisition", &error))?;
+ if granted.as_bool() == Some(false) {
+ REGISTRY.with(|registry| registry.borrow_mut().pending_lock = None);
+ guard.disarm();
+ return Ok(None);
+ }
let owner = REGISTRY.with(|registry| {
let mut registry = registry.borrow_mut();
@@ -1530,7 +1608,7 @@ async fn acquire_database_lock(
Ok(registered.owner)
})?;
guard.disarm();
- Ok(owner)
+ Ok(Some(owner))
}
fn cancel_database_lock_acquire(lock_name: &str) {
@@ -1648,9 +1726,86 @@ async fn worker_root() -> Result<FileSystemDirectoryHandle, JsValue> {
.dyn_into::<FileSystemDirectoryHandle>()
}
+/// How long one open or removal waits for a predecessor to release the files.
+///
+/// Chromium can grant the canonical owner Web Lock to a successor before it
+/// releases the file lock of a predecessor that is still being torn down, such
+/// as a page's cache worker terminated during `pagehide`. Production logs show
+/// that gap outlasting two seconds. Every holder of a sync handle on these
+/// paths first holds that Web Lock, so while this worker holds it,
+/// `NoModificationAllowedError` from an entry operation is a draining
+/// predecessor, not evidence about the files. The wait stays well inside the
+/// coordinator's 20 second database-open watchdog.
+#[cfg(not(test))]
+const BUSY_ENTRY_WAIT_MS: f64 = 10_000.0;
+#[cfg(test)]
+const BUSY_ENTRY_WAIT_MS: f64 = 400.0;
+const BUSY_ENTRY_FIRST_DELAY_MS: i32 = 2;
+const BUSY_ENTRY_MAX_DELAY_MS: i32 = 512;
+
+/// The end of one operation's busy-entry wait, shared by both paths.
+#[derive(Clone, Copy)]
+struct BusyDeadline(f64);
+
+impl BusyDeadline {
+ fn start() -> Self {
+ Self(performance_now() + BUSY_ENTRY_WAIT_MS)
+ }
+}
+
+/// Runs an idempotent OPFS entry operation, retrying only a busy entry until
+/// `deadline`.
+async fn retry_busy_entry(
+ deadline: BusyDeadline,
+ mut operation: impl FnMut() -> Promise,
+) -> Result<JsValue, JsValue> {
+ let mut delay = BUSY_ENTRY_FIRST_DELAY_MS;
+ loop {
+ match JsFuture::from(operation()).await {
+ Err(error) if is_entry_busy(&error) => {
+ let remaining = deadline.0 - performance_now();
+ if remaining <= 0.0 {
+ return Err(error);
+ }
+ sleep_ms(delay.min(remaining.ceil() as i32)).await?;
+ delay = (delay * 2).min(BUSY_ENTRY_MAX_DELAY_MS);
+ }
+ result => return result,
+ }
+ }
+}
+
+fn is_entry_busy(value: &JsValue) -> bool {
+ dom_exception_name(value).as_deref() == Some("NoModificationAllowedError")
+}
+
+/// Classifies an entry that stayed busy until its deadline as
+/// [`OpfsErrorKind::Busy`], keeping the usual message.
+fn entry_error(kind: OpfsErrorKind, operation: &'static str, value: &JsValue) -> OpfsError {
+ if is_entry_busy(value) {
+ OpfsError::new(
+ OpfsErrorKind::Busy,
+ format!("{operation} failed (NoModificationAllowedError)"),
+ )
+ } else {
+ js_error(kind, operation, value)
+ }
+}
+
+async fn sleep_ms(delay: i32) -> Result<(), JsValue> {
+ let worker = js_sys::global().dyn_into::<WorkerGlobalScope>()?;
+ let mut scheduled = Ok(0);
+ let timer = Promise::new(&mut |resolve, _reject| {
+ scheduled = worker.set_timeout_with_callback_and_timeout_and_arguments_0(&resolve, delay);
+ });
+ scheduled?;
+ JsFuture::from(timer).await.map(|_| ())
+}
+
async fn open_sync_handle(
root: &FileSystemDirectoryHandle,
path: &str,
+ deadline: BusyDeadline,
) -> Result<(FileSystemSyncAccessHandle, bool), JsValue> {
let (file, created) = match JsFuture::from(root.get_file_handle(path)).await {
Ok(file) => (file.dyn_into::<FileSystemFileHandle>()?, false),
@@ -1666,7 +1821,7 @@ async fn open_sync_handle(
}
Err(error) => return Err(error),
};
- let handle = JsFuture::from(file.create_sync_access_handle())
+ let handle = retry_busy_entry(deadline, || file.create_sync_access_handle())
.await?
.dyn_into::<FileSystemSyncAccessHandle>()?;
Ok((handle, created))
@@ -1844,11 +1999,12 @@ fn close_sync_handle(handle: &FileSystemSyncAccessHandle) -> Result<(), JsValue>
}
async fn reset_paths(paths: &Paths, recursive: bool) -> Result<(), OpfsError> {
+ let deadline = BusyDeadline::start();
let root = worker_root()
.await
.map_err(|error| js_error(OpfsErrorKind::Remove, "OPFS reset root open", &error))?;
for role in FileRole::ALL {
- remove_if_present(&root, paths.get(role), recursive).await?;
+ remove_if_present(&root, paths.get(role), recursive, deadline).await?;
#[cfg(test)]
if role == FileRole::Main && take_reset_failure(ResetFault::AfterMainRemoval) {
return Err(OpfsError::new(
@@ -1878,22 +2034,37 @@ async fn reset_paths(paths: &Paths, recursive: bool) -> Result<(), OpfsError> {
Ok(())
}
+/// Removes both approved paths, attempting each even if the other fails.
+async fn remove_paths(paths: &Paths) -> Result<(), OpfsError> {
+ let deadline = BusyDeadline::start();
+ let root = worker_root()
+ .await
+ .map_err(|error| js_error(OpfsErrorKind::Remove, "OPFS remove root open", &error))?;
+ let mut first_error = None;
+ for role in FileRole::ALL {
+ if let Err(error) = remove_if_present(&root, paths.get(role), true, deadline).await {
+ first_error.get_or_insert(error);
+ }
+ }
+ first_error.map_or(Ok(()), Err)
+}
+
async fn remove_if_present(
root: &FileSystemDirectoryHandle,
path: &str,
recursive: bool,
+ deadline: BusyDeadline,
) -> Result<(), OpfsError> {
- let remove = if recursive {
- let options = FileSystemRemoveOptions::new();
- options.set_recursive(true);
- root.remove_entry_with_options(path, &options)
- } else {
- root.remove_entry(path)
- };
- match JsFuture::from(remove).await {
+ let options = FileSystemRemoveOptions::new();
+ options.set_recursive(recursive);
+ match retry_busy_entry(deadline, || root.remove_entry_with_options(path, &options)).await {
Ok(_) => Ok(()),
Err(error) if is_not_found(&error) => Ok(()),
- Err(error) => Err(js_error(OpfsErrorKind::Remove, "OPFS path remove", &error)),
+ Err(error) => Err(entry_error(
+ OpfsErrorKind::Remove,
+ "OPFS path remove",
+ &error,
+ )),
}
}
diff --git a/crates/client/turso-opfs/src/browser/test.rs b/crates/client/turso-opfs/src/browser/test.rs
index 4852c5fcdb..ff1f9bcff9 100644
--- a/crates/client/turso-opfs/src/browser/test.rs
+++ b/crates/client/turso-opfs/src/browser/test.rs
@@ -1049,3 +1049,181 @@ async fn real_worker_opfs_contract_and_consuming_lifecycle() {
replace_poisoned_registry_for_test().await;
clean_pair(&uncertain_close_db).await;
}
+
+/// Closes a raw sync handle after `delay_ms`, standing in for a terminated
+/// predecessor whose browser file lock outlives its owner Web Lock.
+fn close_after(handle: FileSystemSyncAccessHandle, delay_ms: i32) {
+ wasm_bindgen_futures::spawn_local(async move {
+ sleep_ms(delay_ms).await.expect("predecessor drain timer");
+ close_sync_handle(&handle).expect("close predecessor handle");
+ });
+}
+
+async fn assert_entry_busy(root: &FileSystemDirectoryHandle, path: &str) {
+ let error = JsFuture::from(root.remove_entry(path))
+ .await
+ .expect_err("predecessor handle locks the entry");
+ assert!(is_entry_busy(&error), "unexpected lock error: {error:?}");
+}
+
+#[wasm_bindgen_test]
+async fn owner_waits_for_a_draining_predecessor_file_lock() {
+ let nonce = js_sys::Date::now() as u64;
+ let database_name = format!("busy-entry-{nonce}.db");
+ let wal_name = format!("{database_name}-wal");
+ clean_pair(&database_name).await;
+ let root = worker_root().await.expect("worker OPFS root");
+
+ // Recovery wipe removes an entry a predecessor still briefly locks.
+ let (predecessor, _) = open_sync_handle(&root, &database_name, BusyDeadline::start())
+ .await
+ .expect("predecessor main handle");
+ assert_entry_busy(&root, &database_name).await;
+ close_after(predecessor, 40);
+ let owner = OpfsOwner::acquire(&database_name)
+ .await
+ .expect("owner lock")
+ .recovery_wipe()
+ .await
+ .expect("wipe waits for the draining predecessor");
+
+ // Opening waits the same way for a sync handle on the WAL.
+ let (predecessor, _) = open_sync_handle(&root, &wal_name, BusyDeadline::start())
+ .await
+ .expect("predecessor WAL handle");
+ assert_entry_busy(&root, &wal_name).await;
+ close_after(predecessor, 40);
+ let owner = connect_ready(open_ready(owner).await)
+ .try_close()
+ .expect("adapter-driven close")
+ .preserve()
+ .expect("preserve idle owner");
+
+ // An open that outwaits a stuck handle reports it as busy, touches
+ // nothing, and keeps the owner usable.
+ let (stuck, _) = open_sync_handle(&root, &wal_name, BusyDeadline::start())
+ .await
+ .expect("stuck WAL handle");
+ let started = performance_now();
+ let failure = owner
+ .open()
+ .await
+ .expect_err("open gives up on a lock that never drains");
+ assert!(performance_now() - started >= BUSY_ENTRY_WAIT_MS);
+ assert_eq!(failure.error().kind(), OpfsErrorKind::Busy);
+ assert_eq!(
+ failure.error().to_string(),
+ "OPFS sync handle open failed (NoModificationAllowedError)"
+ );
+ let owner = failure
+ .into_owner()
+ .expect("a busy open leaves a recoverable owner");
+ assert!(!REGISTRY.with(|registry| registry.borrow().machine.is_poisoned()));
+ close_sync_handle(&stuck).expect("close stuck WAL handle");
+
+ // A wipe that outwaits a stuck handle reports it as busy too.
+ let (stuck, _) = open_sync_handle(&root, &database_name, BusyDeadline::start())
+ .await
+ .expect("stuck main handle");
+ let started = performance_now();
+ let failure = owner
+ .recovery_wipe()
+ .await
+ .expect_err("wipe gives up on a lock that never drains");
+ assert!(performance_now() - started >= BUSY_ENTRY_WAIT_MS);
+ assert_eq!(failure.error().kind(), OpfsErrorKind::Busy);
+ assert_eq!(
+ failure.error().to_string(),
+ "OPFS path remove failed (NoModificationAllowedError)"
+ );
+ assert_poisoned();
+ close_sync_handle(&stuck).expect("close stuck handle");
+ replace_poisoned_registry_for_test().await;
+ clean_pair(&database_name).await;
+}
+
+async fn entry_exists(root: &FileSystemDirectoryHandle, path: &str) -> bool {
+ match JsFuture::from(root.get_file_handle(path)).await {
+ Ok(_) => true,
+ Err(error) if is_not_found(&error) => false,
+ Err(error) => panic!("unexpected OPFS lookup failure: {error:?}"),
+ }
+}
+
+fn registry_has_no_lock() -> bool {
+ REGISTRY.with(|registry| {
+ let registry = registry.borrow();
+ registry.pending_lock.is_none() && registry.owner_lock.is_none()
+ })
+}
+
+#[wasm_bindgen_test]
+async fn try_acquire_never_queues_and_removal_discards_both_paths() {
+ let nonce = js_sys::Date::now() as u64;
+ let database_name = format!("try-acquire-{nonce}.db");
+ let wal_name = format!("{database_name}-wal");
+ let lock_name = owner_lock_name(&database_name);
+ clean_pair(&database_name).await;
+ let root = worker_root().await.expect("worker OPFS root");
+
+ // Another holder makes the lock unavailable; nothing is claimed or queued.
+ let held = acquire_raw_web_lock(&lock_name).await;
+ assert!(
+ OpfsOwner::try_acquire(&database_name)
+ .await
+ .expect("unavailable lock is not an error")
+ .is_none()
+ );
+ assert!(registry_has_no_lock());
+ assert!(!entry_exists(&root, &database_name).await);
+ held.release().await;
+
+ // Once free, the lock is granted and the pair can be created and closed.
+ let owner = OpfsOwner::try_acquire(&database_name)
+ .await
+ .expect("free lock acquisition")
+ .expect("free lock is granted");
+ assert_eq!(owner.lock_name(), lock_name);
+ assert!(!web_lock_available(&lock_name).await);
+ let owner = connect_ready(open_ready(owner).await)
+ .try_close()
+ .expect("adapter-driven close")
+ .preserve()
+ .expect("preserve idle owner");
+ assert!(entry_exists(&root, &database_name).await);
+ assert!(entry_exists(&root, &wal_name).await);
+
+ // Removal deletes both paths without recreating them and frees the lock.
+ owner
+ .remove_and_release()
+ .await
+ .expect("remove an unused pair");
+ assert!(!entry_exists(&root, &database_name).await);
+ assert!(!entry_exists(&root, &wal_name).await);
+ assert!(web_lock_available(&lock_name).await);
+ OpfsOwner::try_acquire(&database_name)
+ .await
+ .expect("reacquire after removal")
+ .expect("lock is free after removal")
+ .remove_and_release()
+ .await
+ .expect("removing absent paths succeeds");
+
+ // A removal that cannot finish still releases without poisoning.
+ let (stuck, _) = open_sync_handle(&root, &database_name, BusyDeadline::start())
+ .await
+ .expect("stuck main handle");
+ let failure = OpfsOwner::try_acquire(&database_name)
+ .await
+ .expect("acquire beside a stuck handle")
+ .expect("lock is free beside a stuck handle")
+ .remove_and_release()
+ .await
+ .expect_err("a stuck handle blocks removal");
+ assert_eq!(failure.kind(), OpfsErrorKind::Busy);
+ assert!(!REGISTRY.with(|registry| registry.borrow().machine.is_poisoned()));
+ assert!(registry_has_no_lock());
+ assert!(web_lock_available(&lock_name).await);
+ close_sync_handle(&stuck).expect("close stuck handle");
+ clean_pair(&database_name).await;
+}