Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(cache): hand the database over while the old build is draining (#9161)

gbirmane2bd10f46f7e
diff --git a/apps/web/docs/graphql-normalized-cache-plan.md b/apps/web/docs/graphql-normalized-cache-plan.md
index 0cb424b60c..159df9aa00 100644
--- a/apps/web/docs/graphql-normalized-cache-plan.md
+++ b/apps/web/docs/graphql-normalized-cache-plan.md
@@ -351,8 +351,12 @@ foregrounding, or explicit draft Retry, with jittered exponential backoff capped
at thirty seconds. A newly registered host restarts an exhausted untouched
coordinator at a new owner epoch with `open-existing`; failed hosts remain
excluded. Draft writes stay blocked until the replacement opens and inspects
-the preserved mutation queue. Unknown storage failures still require explicit
-recovery or an update; retry never rotates the scope or clears persisted data.
+the preserved mutation queue. A read refused because the database was moving
+between owners (`owner-handoff-blocked`) leaves the host running, so recovery
+proves that same host readable on its next attempt instead of registering
+another. Unknown storage failures still require explicit recovery or an update
+(the compose banner offers Reload then); retry never rotates the scope or clears
+persisted data.
A build's coordinator has its build time in the SharedWorker name, so builds
use separate coordinators even when their coordinator script URL is identical.
@@ -365,7 +369,8 @@ registered; a departed tab's lock can outlast it), the coordinator posts a
takeover request on the cross-build channel
`graphql-cache-takeover:{database name}` (`coordinator-takeover.ts`, a contract
every later build must keep). Only builds that open the same database join it,
-and only the coordinator whose engine holds the database answers:
+and only the coordinator whose engine holds the database, or is opening or
+draining it, answers:
- **Yield**, if the requester is strictly newer. Every tab registered there gets
`cache-superseded`. Each page retires its host to the network. The owner's
@@ -374,7 +379,13 @@ and only the coordinator whose engine holds the database answers:
already sent is reported as uncertain rather than sent again, because it may
be in the durable queue that the newer build replays. Nothing is elected
again. The requester's engine, still retrying the lock, opens the same
- database with its data and queued mutations.
+ database with its data and queued mutations. A holder whose engine is still
+ opening or already draining the database (hiding its owner tab starts a
+ drain, and opening the requesting tab usually hides it) answers at once but
+ supersedes its tabs only once that engine is ready or has drained, so no
+ engine is interrupted while its files may be changing. If that engine is
+ lost first, the promised handover completes in place of the reset election,
+ so a build never wipes a database it agreed to hand over.
- **Keep**, if the requester is not newer. The requester fails closed as below.
A superseded page offers a persistent reload prompt. It reloads by itself only
@@ -386,9 +397,12 @@ rollback or from stale HTML, only runs uncached, because a reload could land on
the same build again.
A coordinator that holds nothing but gets a request from a newer build sends
-its tabs there too, without answering. When no yield arrives within a second
-(builds from before handover never answer), when the holder keeps the
-database, or when the engine exhausts its retries, the coordinator fails closed:
+its tabs there too, without answering. An unanswered request is posted again
+every second while the engine keeps retrying the lock: the holder may be
+between owners, or parked with the lock already free, and the requester's own
+retries then take it. When the holder keeps the database, or when the engine
+exhausts its retries (builds from before handover never answer, so their
+holders cost that whole wait), the coordinator fails closed:
it refuses queued requests with `owner-lock-unavailable` and tells every
registered page `cache-unavailable`. Those pages retire their host for the rest
of the page session, without quarantine, toast, or error report; only a tab
diff --git a/apps/web/src/features/email-compose/components/draft-sync-status.tsx b/apps/web/src/features/email-compose/components/draft-sync-status.tsx
index abbc1f93b5..efa75974f2 100644
--- a/apps/web/src/features/email-compose/components/draft-sync-status.tsx
+++ b/apps/web/src/features/email-compose/components/draft-sync-status.tsx
@@ -1,5 +1,6 @@
import { Button } from '@ui';
import { Show } from 'solid-js';
+import { match } from 'ts-pattern';
import type { DraftSyncViewState } from '../core/local-draft';
/** In-composer banner for draft failures; saving itself stays silent. */
@@ -8,6 +9,7 @@ export function DraftSyncStatus(props: {
busy: boolean;
error?: string;
onRetry(): void;
+ onReload(): void;
onKeepEditing(): void;
}) {
return (
@@ -28,9 +30,15 @@ export function DraftSyncStatus(props: {
size="sm"
variant="secondary"
disabled={props.busy}
- onClick={props.onRetry}
+ onClick={
+ state().action === 'reload' ? props.onReload : props.onRetry
+ }
>
- {state().action === 'retry-discard' ? 'Retry discard' : 'Retry'}
+ {match(state().action)
+ .with('retry-discard', () => 'Retry discard')
+ .with('reload', () => 'Reload')
+ .with('retry', undefined, () => 'Retry')
+ .exhaustive()}
</Button>
</Show>
<Show when={state().canKeepEditing}>
diff --git a/apps/web/src/features/email-compose/core/local-draft.ts b/apps/web/src/features/email-compose/core/local-draft.ts
index 048117fea7..4846070a74 100644
--- a/apps/web/src/features/email-compose/core/local-draft.ts
+++ b/apps/web/src/features/email-compose/core/local-draft.ts
@@ -82,7 +82,7 @@ export type DraftSyncViewState = {
message: string;
detail?: string;
failed: boolean;
- action?: 'retry' | 'retry-discard';
+ action?: 'retry' | 'retry-discard' | 'reload';
canKeepEditing: boolean;
};
diff --git a/apps/web/src/features/email-compose/primitives/draft-sync-status.test.ts b/apps/web/src/features/email-compose/primitives/draft-sync-status.test.ts
index b412eeb281..b525bd88ed 100644
--- a/apps/web/src/features/email-compose/primitives/draft-sync-status.test.ts
+++ b/apps/web/src/features/email-compose/primitives/draft-sync-status.test.ts
@@ -139,7 +139,7 @@ it('offers retry for server failures while newer edits remain saved locally', as
it('keeps local edits visible with an actionable synchronization status during recovery', async () => {
const context = createComposeContext();
const [availability, setAvailability] = createSignal<
- 'unavailable' | 'recovering' | 'ready'
+ 'unavailable' | 'recovering' | 'recovery-required' | 'ready'
>('unavailable');
context.drafts.syncAvailability = availability;
context.drafts.saveLocalDraft = vi.fn();
@@ -165,6 +165,8 @@ it('keeps local edits visible with an actionable synchronization status during r
expect(status.state()?.action).toBe('retry');
setAvailability('recovering');
expect(status.state()?.action).toBeUndefined();
+ setAvailability('recovery-required');
+ expect(status.state()?.action).toBe('reload');
setAvailability('ready');
expect(status.state()).toBeUndefined();
});
diff --git a/apps/web/src/features/email-compose/primitives/draft-sync-status.ts b/apps/web/src/features/email-compose/primitives/draft-sync-status.ts
index 1012f8f1a1..1d8a219142 100644
--- a/apps/web/src/features/email-compose/primitives/draft-sync-status.ts
+++ b/apps/web/src/features/email-compose/primitives/draft-sync-status.ts
@@ -7,6 +7,7 @@ import {
createSignal,
onCleanup,
} from 'solid-js';
+import { match } from 'ts-pattern';
import type { EmailDraftStorage } from '../context/compose-capabilities';
import type { DraftSyncViewState } from '../core/local-draft';
@@ -70,7 +71,11 @@ export function createDraftSyncStatus(options: {
? 'Queued changes are preserved. Update or reload Macro to recover local storage.'
: 'Reconnect or retry to resume draft synchronization.',
failed: true,
- action: availability === 'unavailable' ? 'retry' : undefined,
+ action: match(availability)
+ .with('unavailable', () => 'retry' as const)
+ .with('recovery-required', () => 'reload' as const)
+ .with('recovering', () => undefined)
+ .exhaustive(),
canKeepEditing: false,
};
}
diff --git a/apps/web/src/features/email-compose/views/email-compose.tsx b/apps/web/src/features/email-compose/views/email-compose.tsx
index fe3439cf5a..2dc4bc5842 100644
--- a/apps/web/src/features/email-compose/views/email-compose.tsx
+++ b/apps/web/src/features/email-compose/views/email-compose.tsx
@@ -1,3 +1,4 @@
+import { reloadApp } from '@app/observability/reload';
import { FloatRegion } from '@components/app/mobile/float-regions/FloatRegion';
import { SplitHeaderLeft } from '@components/app/split-layout/components/SplitHeader';
import {
@@ -228,6 +229,15 @@ function LoadedEmailComposeView(
setDraftBackMenuOpen(false);
props.host?.goBack?.();
};
+ const reloadForRecovery = async () => {
+ try {
+ await state.flushLocal();
+ } catch (error) {
+ composeContext.notices.reportError(error);
+ return;
+ }
+ reloadApp('user_recovery');
+ };
const SyncStatus = () => (
<DraftSyncStatus
@@ -235,6 +245,7 @@ function LoadedEmailComposeView(
busy={sync.busy()}
error={sync.error()}
onRetry={sync.retry}
+ onReload={() => fireAndForget(reloadForRecovery(), 'email-compose')}
onKeepEditing={sync.keepEditing}
/>
);
diff --git a/apps/web/src/features/email-compose/views/reply-input.tsx b/apps/web/src/features/email-compose/views/reply-input.tsx
index 8e206dea02..3fb0b5e067 100644
--- a/apps/web/src/features/email-compose/views/reply-input.tsx
+++ b/apps/web/src/features/email-compose/views/reply-input.tsx
@@ -1,4 +1,5 @@
import { EmailAttachmentPill } from '@app/features/email-message/components/attachment-pill';
+import { reloadApp } from '@app/observability/reload';
import { useMacroMentionLinkResolver } from '@components/app/split-layout/split-router/mention-links';
import { FileDropOverlay } from '@core/component/FileDropOverlay';
import { buildConfig } from '@core/component/LexicalMarkdown/builder/MarkdownConfigBuilder';
@@ -240,6 +241,15 @@ function LoadedReplyInputView(props: ReplyInputViewProps) {
setDraftBackMenuOpen(false);
props.mobileCompose?.onClose();
};
+ const reloadForRecovery = async () => {
+ try {
+ await state.flushLocal();
+ } catch (error) {
+ composeContext.notices.reportError(error);
+ return;
+ }
+ reloadApp('user_recovery');
+ };
onMount(() => props.mobileCompose?.registerClose(requestClose));
const isMobileCompose = () => props.mobileCompose !== undefined;
const composePortalScope = () =>
@@ -474,6 +484,7 @@ function LoadedReplyInputView(props: ReplyInputViewProps) {
busy={sync.busy()}
error={sync.error()}
onRetry={sync.retry}
+ onReload={() => fireAndForget(reloadForRecovery(), 'email-compose')}
onKeepEditing={sync.keepEditing}
/>
);
diff --git a/apps/web/src/lib/graphql-cache/host/worker-host.ts b/apps/web/src/lib/graphql-cache/host/worker-host.ts
index 25aa5ff26a..8f2041c16b 100644
--- a/apps/web/src/lib/graphql-cache/host/worker-host.ts
+++ b/apps/web/src/lib/graphql-cache/host/worker-host.ts
@@ -33,6 +33,7 @@ import {
type MutationClaim,
type MutationSettlement,
OWNER_EPOCH_LOST_ERROR_CODE,
+ OWNER_HANDOFF_BLOCKED_ERROR_CODE,
OWNER_LOCK_UNAVAILABLE_ERROR_CODE,
parseStorageGeneration,
type QueryUpdate,
@@ -475,8 +476,9 @@ export function createWorkerCacheHost(options: WorkerHostOptions): CacheHost {
function onOwnerHandoffBlocked(): void {
if (state === 'failed' || state === 'suspended' || state === 'disposed')
return;
- const error = new Error(
- 'cache reads unavailable after owner handoff timeout'
+ const error = new CacheResponseError(
+ 'cache reads unavailable after owner handoff timeout',
+ OWNER_HANDOFF_BLOCKED_ERROR_CODE
);
beginRecoveryGeneration();
for (const key of activeOpKeys) lostRegisteredOpKeys.add(key);
diff --git a/apps/web/src/lib/graphql-cache/protocol.ts b/apps/web/src/lib/graphql-cache/protocol.ts
index bd37eef170..28f87d0b4e 100644
--- a/apps/web/src/lib/graphql-cache/protocol.ts
+++ b/apps/web/src/lib/graphql-cache/protocol.ts
@@ -910,6 +910,7 @@ export type CacheRequest = { id: number } & (
export type CacheResponseErrorCode =
| 'owner-epoch-lost'
| 'owner-lock-unavailable'
+ | 'owner-handoff-blocked'
| 'admitted-enqueue-uncertain';
/** Old-owner work was rejected after fenced engine loss and was not replayed. */
@@ -924,6 +925,14 @@ export const OWNER_EPOCH_LOST_ERROR_CODE: CacheResponseErrorCode =
export const OWNER_LOCK_UNAVAILABLE_ERROR_CODE: CacheResponseErrorCode =
'owner-lock-unavailable';
+/**
+ * A read-only request was refused because the database was moving between
+ * owners and the foreground wait ran out. Storage is untouched and the engine
+ * keeps running, so the request can be retried once an owner is ready.
+ */
+export const OWNER_HANDOFF_BLOCKED_ERROR_CODE: CacheResponseErrorCode =
+ 'owner-handoff-blocked';
+
/** An admitted enqueue lost its reply and may already be durably queued. */
export const ADMITTED_ENQUEUE_UNCERTAIN_ERROR_CODE: CacheResponseErrorCode =
'admitted-enqueue-uncertain';
@@ -982,6 +991,7 @@ export const isCacheResponseErrorCode = (
): value is CacheResponseErrorCode =>
value === OWNER_EPOCH_LOST_ERROR_CODE ||
value === OWNER_LOCK_UNAVAILABLE_ERROR_CODE ||
+ value === OWNER_HANDOFF_BLOCKED_ERROR_CODE ||
value === ADMITTED_ENQUEUE_UNCERTAIN_ERROR_CODE;
/** Identifies a coordinator-fenced rejection from a lost owner epoch. */
@@ -1000,6 +1010,14 @@ export const isOwnerLockUnavailableError = (
'errorCode' in value &&
value.errorCode === OWNER_LOCK_UNAVAILABLE_ERROR_CODE;
+/** Identifies a read refused while the database moved between owners. */
+export const isOwnerHandoffBlockedError = (
+ value: unknown
+): value is Error & { errorCode: 'owner-handoff-blocked' } =>
+ value instanceof Error &&
+ 'errorCode' in value &&
+ value.errorCode === OWNER_HANDOFF_BLOCKED_ERROR_CODE;
+
/** Identifies an admitted enqueue whose durable outcome is unknown. */
export const isAdmittedEnqueueUncertainError = (
value: unknown
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts
index e2392e3f72..fa8085d7ae 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-core.test.ts
@@ -232,11 +232,49 @@ describe('CoordinatorCore', () => {
]);
});
- it('hands over only while no engine is changing the files', () => {
+ it('completes the handover once no engine is changing the files', () => {
const activating = new CoordinatorCore('scope');
activating.registerTab('tab-a');
grantStorage(activating, 'tab-a', 1);
expect(activating.supersede()).toEqual([]);
+ expect(activating.handingOver).toBe(true);
+ // The handover completes once the opening engine is ready.
+ expect(
+ activating.engineReady({
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ ownerLockName: OWNER_LOCK,
+ expectedOwnerLockName: OWNER_LOCK,
+ ownerLockHeld: true,
+ databaseActionProof: 'opened-existing',
+ openOutcome: 'opened-existing',
+ })
+ ).toEqual([
+ {
+ kind: 'broadcast-cache-superseded',
+ reason: expect.stringContaining('newer version of the app'),
+ },
+ ]);
+ expect(activating.superseded).toBe(true);
+
+ // A draining engine closes its files first, and nothing is elected after
+ // it even though a visible tab remains.
+ const draining = new CoordinatorCore('scope');
+ draining.registerTab('tab-a');
+ draining.registerTab('tab-b');
+ ready(draining, 'tab-a', 1, 'opened-existing');
+ expect(
+ action(draining.setTabVisibility('tab-a', false), 'drain-owner')
+ ).toMatchObject({ tabId: 'tab-a', handoff: true });
+ expect(draining.supersede()).toEqual([]);
+ expect(draining.handingOver).toBe(true);
+ const drained = draining.engineDrained('tab-a', 1);
+ expect(drained).toContainEqual({
+ kind: 'broadcast-cache-superseded',
+ reason: expect.stringContaining('newer version of the app'),
+ });
+ expect(drained.some((item) => item.kind === 'elect-owner')).toBe(false);
+ expect(draining.state.kind).toBe('waiting-for-tab');
// An idle coordinator's tabs, already uncached, move on too.
const idle = new CoordinatorCore('scope');
@@ -251,6 +289,33 @@ describe('CoordinatorCore', () => {
]);
});
+ it('hands over instead of wiping after losing an engine promised to a newer build', () => {
+ const core = new CoordinatorCore('scope');
+ core.registerTab('tab-a');
+ core.registerTab('tab-b');
+ grantStorage(core, 'tab-a', 1);
+ expect(core.supersede()).toEqual([]);
+
+ core.tabLost('tab-a');
+ expect(core.state.kind).toBe('resetting-after-loss');
+
+ expect(core.resumeAfterLoss()).toEqual([
+ {
+ kind: 'broadcast-cache-superseded',
+ reason: expect.stringContaining('newer version of the app'),
+ },
+ ]);
+ expect(core.superseded).toBe(true);
+ expect(core.state.kind).toBe('waiting-for-tab');
+ expect(core.registerTab('tab-c')).toEqual([
+ {
+ kind: 'notify-cache-unavailable',
+ tabId: 'tab-c',
+ reason: expect.stringContaining('newer version of the app'),
+ },
+ ]);
+ });
+
it('keeps a pending recovery wipe when its files stay busy', () => {
const core = new CoordinatorCore('scope');
core.registerTab('tab-a');
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts
index 07b0d2dd46..fb4269b59d 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-core.ts
@@ -166,6 +166,8 @@ export class CoordinatorCore {
private readonly unavailableTabs = new Map<string, string>();
/** Set once a newer app build took the database over; never cleared. */
private supersededReason: string | undefined;
+ /** A newer build asked while this engine may still be changing the files. */
+ private supersedePending = false;
private readonly queuedRequests: QueuedRequest[] = [];
private readonly inFlight = new Map<RouteId, InFlightRequest>();
private readonly cacheChanges = new CacheChangeBatch();
@@ -206,6 +208,12 @@ export class CoordinatorCore {
return this.supersededReason !== undefined;
}
+ /** Whether a newer app build took the database over, or will once this
+ * engine has settled. */
+ get handingOver(): boolean {
+ return this.supersededReason !== undefined || this.supersedePending;
+ }
+
registerTab(tabId: string, visible = true): CoordinatorAction[] {
if (this.tabs.includes(tabId)) return [];
this.tabs.push(tabId);
@@ -412,6 +420,10 @@ export class CoordinatorCore {
openOutcome: ready.openOutcome,
});
}
+ if (this.supersedePending) {
+ actions.push(...this.completeSupersede());
+ return actions;
+ }
// A hidden engine may have reset storage before it immediately drains.
// Notify hosts of that outcome, but leave queued work for its successor.
const handoff = this.handoffIfNeeded();
@@ -536,7 +548,11 @@ export class CoordinatorCore {
kind: 'waiting-for-tab',
nextDatabaseAction: 'open-existing',
};
- actions.push(...this.activateNext('open-existing'));
+ actions.push(
+ ...(this.supersedePending
+ ? this.completeSupersede()
+ : this.activateNext('open-existing'))
+ );
this.assertInvariants();
return actions;
}
@@ -554,20 +570,26 @@ export class CoordinatorCore {
* using the cache and moves to that build; a tab that registers later only
* runs uncached. The owner's page leaves the way a navigating page does,
* which stops its engine and frees the database, and nothing is elected
- * afterwards. Refused while an engine is starting, recovering, or draining,
- * because its files may be changing.
+ * afterwards. While an engine is opening, recovering, or draining the
+ * database its files may be changing, so the handover waits until that
+ * engine has settled: it completes when the engine becomes ready or drains.
*/
supersede(): CoordinatorAction[] {
- const { kind } = this.stateValue;
+ const state = this.stateValue;
+ if (this.supersededReason !== undefined) return [];
if (
- this.supersededReason !== undefined ||
- (kind !== 'active' &&
- kind !== 'waiting-for-tab' &&
- kind !== 'parked' &&
- kind !== 'failed')
+ state.kind === 'draining' ||
+ state.kind === 'resetting-after-loss' ||
+ (state.kind === 'activating' && startupPhaseMayTouchStorage(state.phase))
) {
+ this.supersedePending = true;
return [];
}
+ return this.completeSupersede();
+ }
+
+ private completeSupersede(): CoordinatorAction[] {
+ this.supersedePending = false;
const reason = SUPERSEDED_REASON;
this.supersededReason = reason;
const actions: CoordinatorAction[] = [
@@ -679,6 +701,16 @@ export class CoordinatorCore {
resumeAfterLoss(): CoordinatorAction[] {
const state = this.stateValue;
if (state.kind !== 'resetting-after-loss') return [];
+ if (this.supersedePending) {
+ // The lost engine was promised to a newer build. Hand over now instead
+ // of electing a reset owner that would wipe the database it agreed to
+ // yield, queued mutations included.
+ this.stateValue = {
+ kind: 'waiting-for-tab',
+ nextDatabaseAction: state.databaseAction,
+ };
+ return this.completeSupersede();
+ }
const candidate = this.chooseCandidate(state.previousTabId);
if (!candidate) {
this.stateValue = {
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts
index 092a17144e..0fb54e97d8 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-router.test.ts
@@ -990,7 +990,7 @@ describe('CoordinatorRouter', () => {
}
};
- it('asks another build for its database and fails closed if nobody answers', async () => {
+ it('asks another build for its database and keeps asking while nobody answers', async () => {
vi.useFakeTimers();
const { router, channel, observations, tabA, tabB, engine } =
await askForAnotherBuildsDatabase();
@@ -1010,9 +1010,24 @@ describe('CoordinatorRouter', () => {
expect(channel.posted).toHaveLength(1);
expect(messagesOfKind(tabB, 'cache-unavailable')).toHaveLength(0);
- // Builds from before handover never answer.
+ // The holder may be between owners and unable to answer yet: the same
+ // request goes out again while the engine keeps retrying the lock.
await vi.advanceTimersByTimeAsync(1);
- expectFailedClosed(tabA, tabB, 'another build keeps the database');
+ expect(channel.posted).toHaveLength(2);
+ expect(channel.posted[1]).toEqual(channel.posted[0]);
+ expect(messagesOfKind(tabA, 'terminate-engine')).toEqual([]);
+ await vi.advanceTimersByTimeAsync(300);
+ expect(channel.posted).toHaveLength(5);
+
+ // Builds from before handover never answer; the engine's own lock budget
+ // ends the attempt.
+ engine.receive({
+ ...version,
+ kind: 'owner-lock-unavailable',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ expectFailedClosed(tabA, tabB, 'the engine stopped retrying');
expect(
observations.filter(
(observation) => observation.ownerEvent === 'owner-lock-unavailable'
@@ -1022,6 +1037,7 @@ describe('CoordinatorRouter', () => {
// Nothing waits in line for the lock; a new tab starts a fresh attempt.
await vi.advanceTimersByTimeAsync(10 * 60_000);
+ expect(channel.posted).toHaveLength(5);
const tabC = new FakePort();
await register(router, tabC, 'tab-c', 2_000);
expect(messagesOfKind(tabC, 'become-owner')).toEqual([
@@ -1347,6 +1363,55 @@ describe('CoordinatorRouter', () => {
expect(channel.posted).toHaveLength(1);
});
+ it('answers a newer build while draining and hands over once drained', async () => {
+ const { router, channel, observations, tabA, tabB, engine, ask } =
+ await holdTheDatabase(1_000);
+ // Hiding the owner tab starts a drain, and the request lands during it.
+ await router.handleTabMessage(tabA as CoordinatorMessagePort, {
+ ...version,
+ kind: 'tab-visibility',
+ tabId: 'tab-a',
+ visible: false,
+ });
+ expect(router.snapshot()?.state.kind).toBe('draining');
+ expect(messagesOfKind(engine, 'drain-engine')).toHaveLength(1);
+
+ ask(2_000);
+ ask(2_000);
+
+ expect(channel.posted).toEqual([
+ {
+ takeover: 1,
+ kind: 'reply',
+ scope: 'scope',
+ requestId: 'request-1',
+ decision: 'yield',
+ },
+ ]);
+ // Tabs keep their cache until the engine has closed the files.
+ for (const tab of [tabA, tabB]) {
+ expect(messagesOfKind(tab, 'cache-superseded')).toEqual([]);
+ }
+
+ engine.receive({
+ ...version,
+ kind: 'engine-drained',
+ tabId: 'tab-a',
+ ownerEpoch: 1,
+ });
+ for (const tab of [tabA, tabB]) {
+ expect(messagesOfKind(tab, 'cache-superseded')).toHaveLength(1);
+ }
+ // Nothing is elected again, although a visible tab remains.
+ expect(messagesOfKind(tabB, 'become-owner')).toEqual([]);
+ expect(router.snapshot()?.state.kind).toBe('waiting-for-tab');
+ expect(
+ observations.filter(
+ (observation) => observation.ownerEvent === 'superseded'
+ )
+ ).toHaveLength(1);
+ });
+
it('sends the tabs of an idle older build to the newer one without answering', async () => {
vi.useFakeTimers();
const channel = fakeTakeoverChannel();
diff --git a/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts b/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts
index 68b9964b77..44fbf9379f 100644
--- a/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts
+++ b/apps/web/src/lib/graphql-cache/worker/coordinator-router.ts
@@ -1788,15 +1788,16 @@ export class CoordinatorRouter {
}
/**
- * Asks the build that holds the database to hand it over. The engine keeps
- * retrying the lock meanwhile. Unless the holder agrees promptly, or the
- * unknown tabs turn out to be this build's, this activation fails closed.
+ * Asks the build that holds the database to hand it over, and keeps asking
+ * while the engine retries the lock: the holder may be between owners
+ * (draining, parked) and unable to answer the first time. The activation
+ * fails closed when the holder keeps the database, or when the engine's own
+ * lock budget runs out without an answer.
*/
private requestTakeover(route: EngineRoute): void {
- const core = this.coreValue;
- if (!core || this.takeover?.ownerEpoch === route.ownerEpoch) return;
- const channel = this.takeoverChannel;
- if (!channel) {
+ if (!this.coreValue || this.takeover?.ownerEpoch === route.ownerEpoch)
+ return;
+ if (!this.takeoverChannel) {
this.closeUnavailable(
route.tabId,
route.ownerEpoch,
@@ -1804,24 +1805,29 @@ export class CoordinatorRouter {
);
return;
}
- const requestId = `${route.ownerEpoch}:${crypto.randomUUID()}`;
const takeover: PendingTakeover = {
tabId: route.tabId,
ownerEpoch: route.ownerEpoch,
- requestId,
+ requestId: `${route.ownerEpoch}:${crypto.randomUUID()}`,
granted: false,
timer: undefined,
};
+ this.takeover = takeover;
+ this.postTakeoverRequest(takeover);
+ }
+
+ private postTakeoverRequest(takeover: PendingTakeover): void {
+ const core = this.coreValue;
+ if (!core) return;
takeover.timer = this.setTimeoutFn(() => {
takeover.timer = undefined;
this.expireTakeover(takeover).catch(reportCoordinatorTaskFailure);
}, this.takeoverReplyTimeoutMs);
- this.takeover = takeover;
- channel.post({
+ this.takeoverChannel?.post({
takeover: CACHE_TAKEOVER_VERSION,
kind: 'request',
scope: core.scope,
- requestId,
+ requestId: takeover.requestId,
buildTime: this.buildTime,
});
}
@@ -1830,6 +1836,7 @@ export class CoordinatorRouter {
* Nobody answered. The unknown liveness lock that prompted the request may
* belong to a tab of this build that had not registered yet, so look again:
* once every live tab is known, keep waiting for the engine's own retries.
+ * Otherwise ask again; the holder may have been unable to answer.
*/
private async expireTakeover(takeover: PendingTakeover): Promise<void> {
if (this.takeover !== takeover || takeover.granted) return;
@@ -1839,11 +1846,7 @@ export class CoordinatorRouter {
this.clearTakeover();
return;
}
- this.closeUnavailable(
- takeover.tabId,
- takeover.ownerEpoch,
- 'another build keeps the database'
- );
+ this.postTakeoverRequest(takeover);
}
private handleTakeoverMessage(message: CacheTakeoverMessage): void {
@@ -1881,14 +1884,20 @@ export class CoordinatorRouter {
/**
* Only the coordinator whose engine holds the database answers, and it
* yields only to a strictly newer build, so builds never trade the database
- * back and forth and a reload never lands on an older build. An idle
+ * back and forth and a reload never lands on an older build. A holder whose
+ * engine is still opening or already draining the database answers at once
+ * and completes the handover when that engine has settled. An idle
* coordinator's tabs already run on the network; they move on silently.
*/
private answerTakeover(request: CacheTakeoverRequest): void {
const core = this.coreValue;
- if (!core || core.superseded) return;
+ if (!core || core.handingOver) return;
const newer = request.buildTime > this.buildTime;
- const holdsDatabase = core.state.kind === 'active';
+ const state = core.state;
+ const holdsDatabase =
+ state.kind === 'active' ||
+ state.kind === 'draining' ||
+ (state.kind === 'activating' && startupPhaseMayTouchStorage(state.phase));
if (holdsDatabase) {
this.takeoverChannel?.post({
takeover: CACHE_TAKEOVER_VERSION,
@@ -1900,7 +1909,6 @@ export class CoordinatorRouter {
}
if (!newer) return;
const actions = core.supersede();
- if (actions.length === 0) return;
if (holdsDatabase) {
this.telemetry.record({
name: 'graphql_cache.owner',
diff --git a/apps/web/src/lib/graphql-cache/worker/startup.ts b/apps/web/src/lib/graphql-cache/worker/startup.ts
index fde293086e..39ae324eca 100644
--- a/apps/web/src/lib/graphql-cache/worker/startup.ts
+++ b/apps/web/src/lib/graphql-cache/worker/startup.ts
@@ -13,8 +13,9 @@ export const OWNER_LOCK_RETRY_DELAYS_MS: readonly number[] = [
25, 50, 100, 200, 400, 800, 1_600, 3_200, 3_200,
];
/** When another build has live tabs, a busy lock is almost certainly theirs.
- * The coordinator asks that build to hand the database over and gives up
- * unless the holder agrees this soon; builds from before handover never do. */
+ * The coordinator asks that build to hand the database over and asks again
+ * this often until it answers or the engine's own lock wait ends; builds from
+ * before handover never answer, so their holders cost that whole wait. */
export const TAKEOVER_REPLY_TIMEOUT_MS = 1_000;
/** Coordinator backstop for an engine that stops reporting while it waits. */
export const OWNER_LOCK_WAIT_TIMEOUT_MS =
diff --git a/apps/web/src/lib/service-clients/service-storage/cache-recovery.test.ts b/apps/web/src/lib/service-clients/service-storage/cache-recovery.test.ts
index cb81a6fff2..1fd2596e23 100644
--- a/apps/web/src/lib/service-clients/service-storage/cache-recovery.test.ts
+++ b/apps/web/src/lib/service-clients/service-storage/cache-recovery.test.ts
@@ -85,3 +85,18 @@ it('pauses offline and fences a completion from the previous session', async ()
expect(recovery.state()).toBe('recovery-required');
recovery.reset();
});
+
+it('retries after a read refused while the database changed owners', async () => {
+ const rebuild = vi.fn(async () => {});
+ const recovery = createCacheRecovery(rebuild);
+ const blocked = Object.assign(
+ new Error('cache reads unavailable after owner handoff timeout'),
+ { errorCode: 'owner-handoff-blocked' }
+ );
+ recovery.failed(blocked);
+ expect(recovery.state()).toBe('unavailable');
+ await vi.advanceTimersByTimeAsync(2_000);
+ expect(rebuild).toHaveBeenCalledOnce();
+ expect(recovery.state()).toBe('ready');
+ recovery.reset();
+});
diff --git a/apps/web/src/lib/service-clients/service-storage/cache-recovery.ts b/apps/web/src/lib/service-clients/service-storage/cache-recovery.ts
index 142847d9c7..10d5b3a138 100644
--- a/apps/web/src/lib/service-clients/service-storage/cache-recovery.ts
+++ b/apps/web/src/lib/service-clients/service-storage/cache-recovery.ts
@@ -1,5 +1,8 @@
import { fireAndForget } from '@core/util/fireAndForget';
-import { isOwnerLockUnavailableError } from '@graphql-cache/protocol';
+import {
+ isOwnerHandoffBlockedError,
+ isOwnerLockUnavailableError,
+} from '@graphql-cache/protocol';
import { CacheBootstrapExhaustedError } from '@graphql-cache/worker/startup';
import { createSignal } from 'solid-js';
@@ -54,10 +57,12 @@ export function createCacheRecovery(rebuild: () => Promise<void>) {
const failed = (error: unknown) => {
failure = error;
// Only these typed errors prove restarting cannot bypass a quarantined
- // journal. Unknown storage/transport failures stay closed for recovery.
+ // journal: nothing opened storage, or a read was refused while the database
+ // moved between owners. Unknown storage/transport failures stay closed.
const recoverable =
error instanceof CacheBootstrapExhaustedError ||
- isOwnerLockUnavailableError(error);
+ isOwnerLockUnavailableError(error) ||
+ isOwnerHandoffBlockedError(error);
setState(recoverable ? 'unavailable' : 'recovery-required');
if (recoverable) schedule();
else {
diff --git a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
index 5fc041fc39..9873881d67 100644
--- a/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
+++ b/apps/web/src/lib/service-clients/service-storage/graphql-soup.ts
@@ -501,8 +501,12 @@ let cacheInitializationFailed = false;
const cacheRecovery = createCacheRecovery(() =>
Telemetry.span('graphql.cache_recovery', async (span) => {
const session = cacheSessionGeneration;
- cachedClient = undefined;
- getGraphqlSoupClient();
+ // A host whose read was refused mid-handoff is still running; prove it
+ // readable now instead of stacking another registration on it.
+ if (!cachedCacheHost || cachedCacheHost.disabled) {
+ cachedClient = undefined;
+ getGraphqlSoupClient();
+ }
const host = cachedCacheHost;
if (!host || host.disabled)
throw new Error('Local cache could not restart.');
diff --git a/docs/agent-guide/email.md b/docs/agent-guide/email.md
index d12bc8dc0e..dbd26df26a 100644
--- a/docs/agent-guide/email.md
+++ b/docs/agent-guide/email.md
@@ -656,8 +656,10 @@ are saved on this device independently of the mutation queue. A failed server
save must leave the draft discoverable in **Drafts** (including grouped views)
and in its reply thread. Saving is silent on desktop and mobile: no saved or
syncing text appears, including offline. A failure shows a banner inside the
-composer, above the recipients, with **Retry** (or **Retry discard**); it never
-appears in the toolbar or the mobile header actions. Verify the banner on mobile
+composer, above the recipients, with **Retry** (or **Retry discard**); while the
+local cache waits for another owner the banner keeps **Retry** and retries by
+itself, and once local storage needs recovery it offers **Reload** instead. It
+never appears in the toolbar or the mobile header actions. Verify the banner on mobile
sits below the header and that editing while failed continues saving locally
without repeatedly submitting the rejected request. If sender metadata is still
unavailable, the local draft is retained; reconnect and select an inbox before