Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(mobile): save the database backup onto the device
Elian Doran0b4fadb45854
diff --git a/.claude/skills/developing-capacitor-mobile/SKILL.md b/.claude/skills/developing-capacitor-mobile/SKILL.mdindex 26419975c2..a33c6a1491 100644--- a/.claude/skills/developing-capacitor-mobile/SKILL.md+++ b/.claude/skills/developing-capacitor-mobile/SKILL.md@@ -83,8 +83,15 @@ sheet.`capacitor.config.json` (iOS only builds what is listed), and `cap update android` to regenerate thetracked `capacitor.settings.gradle` / `app/capacitor.build.gradle`. iOS's `CapApp-SPM/Package.swift`is gitignored and regenerated by CI.-- **Still unfixed: the database backup download** (`downloadDatabase` in `local-bridge.ts`) navigates a- hidden iframe to `/local-backup-download` and hits the same wall.+- **The database backup takes the same last step, from a different source.** `saveDatabase()` in+ `local-bridge.ts` consumes the worker's `BACKUP_STREAM` channel in the page rather than relaying the+ port to the service worker, so no SW is involved (which is also why it works on iOS), and+ `setBackupPinging` stays off — that keepalive exists only for a stream the SW is holding open.+ Two things differ from a download: it goes to `Directory.Documents` under `Trilium/`, not the cache,+ and the folder is **not** swept first, which would delete the previous backup. A dismissed share+ sheet is `done`, not `cancelled`: the file is complete before the sheet opens.+- **`rechunk()` is what makes any source safe to write.** A producer picks its chunk sizes for its own+ reasons — a response body by packet, the backup by database page — and neither is 3-aligned.## Native shellsdiff --git a/apps/client/src/services/backup_download.spec.ts b/apps/client/src/services/backup_download.spec.tsindex 72f97d0978..a35b0810dd 100644--- a/apps/client/src/services/backup_download.spec.ts+++ b/apps/client/src/services/backup_download.spec.ts@@ -1,6 +1,6 @@-import { describe, expect, it } from "vitest";+import { afterEach, describe, expect, it, vi } from "vitest";-import { backupFileName } from "./backup_download";+import { backupFileName, isBackupDownloadSupported, startBackupDownload } from "./backup_download";// What the suggested name itself looks like is settled in commons, which every platform shares;// this is about the file that name becomes.@@ -28,3 +28,46 @@ describe("naming a backup file", () => {expect(backupFileName("com1")).toMatch(dated);});});++describe("taking the backup", () => {+ const realWindow = window as unknown as { standaloneApi?: unknown };++ afterEach(() => {+ delete realWindow.standaloneApi;+ });++ it("hands the backup to the device where one saves it, and to the browser otherwise", async () => {+ const downloadDatabase = vi.fn(async () => ({ status: "done" as const }));+ const saveDatabase = vi.fn(async () => ({+ status: "done" as const,+ location: "/documents/Trilium/b.tnbackup"+ }));+ const onProgress = vi.fn();++ realWindow.standaloneApi = { backup: { downloadDatabase } };+ expect(await startBackupDownload("b.tnbackup", "pw", onProgress)).toEqual({ status: "done" });+ expect(downloadDatabase).toHaveBeenCalledWith("b.tnbackup", "pw", onProgress);++ // The mobile shell has no download manager, so it writes the file itself and says where.+ realWindow.standaloneApi = { backup: { downloadDatabase, saveDatabase } };+ expect(await startBackupDownload("b.tnbackup", "pw", onProgress)).toEqual({+ status: "done",+ location: "/documents/Trilium/b.tnbackup"+ });+ expect(saveDatabase).toHaveBeenCalledWith("b.tnbackup", "pw", onProgress);+ expect(downloadDatabase).toHaveBeenCalledTimes(1);+ });++ it("passes no passphrase at all for an empty one, and fails where nothing backs up this way", async () => {+ const downloadDatabase = vi.fn(async () => ({ status: "done" as const }));+ realWindow.standaloneApi = { backup: { downloadDatabase } };++ await startBackupDownload("b.tnbackup", "");+ // An empty string would be a passphrase, and would lock the container with it.+ expect(downloadDatabase).toHaveBeenCalledWith("b.tnbackup", undefined, undefined);++ delete realWindow.standaloneApi;+ expect(isBackupDownloadSupported()).toBe(false);+ expect(await startBackupDownload("b.tnbackup")).toMatchObject({ status: "failed" });+ });+});diff --git a/apps/client/src/services/backup_download.ts b/apps/client/src/services/backup_download.tsindex 536d255750..f4ed973bd7 100644--- a/apps/client/src/services/backup_download.ts+++ b/apps/client/src/services/backup_download.ts@@ -32,10 +32,13 @@ export function backupFileName(name: string): string {}/**- * Starts the download and resolves with how it ended, which is when the stream behind it has been+ * Starts the backup and resolves with how it ended, which is when the stream behind it has been* fully produced — the closest thing to "finished" the application can see, with the browser's* own download UI carrying the transfer itself.*+ * Inside the mobile shell there is no download manager to carry it, so the backup is written onto+ * the device and offered to the share sheet instead; the result then carries where it went.+ ** @param passphrase encrypts the container. Omitted or empty leaves it unencrypted, which is still* a container: one format, one extension, whatever the user chose.*/@@ -44,10 +47,11 @@ export async function startBackupDownload(passphrase?: string,onProgress?: (sentBytes: number, totalBytes: number) => void): Promise<StandaloneDownloadResult> {- const api = window.standaloneApi;- if (!api) {+ const backup = window.standaloneApi?.backup;+ if (!backup) {return { status: "failed", message: "This platform does not back up by download." };}- return await api.backup.downloadDatabase(fileName, passphrase || undefined, onProgress);+ const take = backup.saveDatabase ?? backup.downloadDatabase;+ return await take.call(backup, fileName, passphrase || undefined, onProgress);}diff --git a/apps/client/src/setup_backup.spec.tsx b/apps/client/src/setup_backup.spec.tsxindex e620431567..9b393d7217 100644--- a/apps/client/src/setup_backup.spec.tsx+++ b/apps/client/src/setup_backup.spec.tsx@@ -11,7 +11,7 @@ const mocks = vi.hoisted(() => ({_fileName: string,_passphrase?: string,_onProgress?: (sentBytes: number, totalBytes: number) => void- ) => ({ status: "done" }) as { status: string; message?: string }+ ) => ({ status: "done" }) as { status: string; message?: string; location?: string })}));@@ -340,6 +340,23 @@ describe("backing up from the setup screen", () => {expect(button("setup.backup-finish")?.disabled).toBe(false);});+ it("names where the file went on a platform that puts it somewhere it can name", async () => {+ // The mobile shell writes the backup itself rather than handing it to a download manager,+ // so it is the only thing that knows where the user should go looking.+ mocks.startBackupDownload.mockResolvedValue({+ status: "done",+ location: "/documents/Trilium/b.tnbackup"+ });+ await reachDownload();++ button("setup.backup-download")?.click();+ await settle();++ const outcome = container.querySelector(".backup-download-outcome")?.textContent;+ expect(outcome).toContain("setup.backup-saved-to");+ expect(outcome).not.toContain("setup.backup-downloaded");+ });+it("counts the download as it goes, since the browser's own progress is out of sight on a phone", async () => {let report: ((sent: number, total: number) => void) | undefined;mocks.startBackupDownload.mockImplementation((_name, _passphrase, onProgress) => {diff --git a/apps/client/src/setup_backup.tsx b/apps/client/src/setup_backup.tsxindex f7eadcb846..d0d49874ca 100644--- a/apps/client/src/setup_backup.tsx+++ b/apps/client/src/setup_backup.tsx@@ -42,6 +42,8 @@ export interface BackupDownload {error?: string;/** How far it has got, once the first bytes have gone. */progress?: { sentBytes: number; totalBytes: number };+ /** Where the file was written, on the platforms that put it somewhere they can name. */+ location?: string;start: () => void;}@@ -58,16 +60,19 @@ export function useBackupDownload(settings: SetupBackupSettings): BackupDownloadconst [ state, setState ] = useState<BackupDownload["state"]>("idle");const [ error, setError ] = useState<string>();const [ progress, setProgress ] = useState<BackupDownload["progress"]>();+ const [ location, setLocation ] = useState<string>();return {fileName,state,error,progress,+ location,start: () => {setState("running");setError(undefined);setProgress(undefined);+ setLocation(undefined);const onProgress = (sentBytes: number, totalBytes: number) =>setProgress({ sentBytes, totalBytes });@@ -75,6 +80,7 @@ export function useBackupDownload(settings: SetupBackupSettings): BackupDownloadvoid startBackupDownload(fileName, settings.passphrase, onProgress).then((result) => {setState(result.status === "done" ? "done" : "failed");setError(result.status === "done" ? undefined : result.message);+ setLocation(result.location);});}};@@ -231,7 +237,9 @@ export function BackupDownloadPanel({ download }: { download: BackupDownload })it, and the button is what the user does next either way. */}{download.state === "done" && (<Admonition type="note" className="backup-download-outcome">- {t("setup.backup-downloaded")}+ {download.location+ ? t("setup.backup-saved-to", { location: download.location })+ : t("setup.backup-downloaded")}</Admonition>)}{download.state === "failed" && (diff --git a/apps/client/src/translations/en/entry.json b/apps/client/src/translations/en/entry.jsonindex 0e5225a80f..c77246d833 100644--- a/apps/client/src/translations/en/entry.json+++ b/apps/client/src/translations/en/entry.json@@ -159,6 +159,7 @@"restore-stage-validating": "Checking the database","restore-stage-swapping": "Putting it in place","restore-stage-migrating": "Opening the database",- "restore-do-not-close": "Do not close this window."+ "restore-do-not-close": "Do not close this window.",+ "backup-saved-to": "The backup has been saved to {{- location}}. Press \"Finish\" to return to the application."}}diff --git a/apps/standalone/src/local-bridge.spec.ts b/apps/standalone/src/local-bridge.spec.tsindex a6e2b99944..a3aeb60353 100644--- a/apps/standalone/src/local-bridge.spec.ts+++ b/apps/standalone/src/local-bridge.spec.ts@@ -2,15 +2,48 @@ import { afterEach, describe, expect, it, vi } from "vitest";// A controllable stand-in for the bundled local-server-worker. vi.hoisted lets the// (hoisted) vi.mock factory share the instance registry with the test body.-const { workerInstances, leadership } = vi.hoisted(() => ({+const { workerInstances, leadership, saved } = vi.hoisted(() => ({workerInstances: [] as MockWorker[],// Most of this suite exercises the leader, which is the tab that owns the// worker. Followers take a different path — see the leadership describe.- leadership: { isLeader: true }+ leadership: { isLeader: true },+ // What saveDatabase() handed the device, and what the device said back.+ saved: {+ fileName: "",+ chunks: [] as Uint8Array[],+ target: null as unknown,+ result: { status: "saved", location: "/documents/Trilium/db.tnbackup" } as {+ status: string;+ location?: string;+ message?: string;+ }+ }}));vi.mock("./leader_election.js", () => ({ isLeader: () => leadership.isLeader }));+// The Capacitor plugins are not there under happy-dom, and what matters on this side is which+// target the backup asked for and what reached it — see capacitor_download.spec.ts for the writing.+vi.mock("./services/capacitor_download.js", () => ({+ BACKUP_TARGET: { directory: "DOCUMENTS", folder: "Trilium", sweep: false },+ saveChunksToDevice: async (+ fileName: string, chunks: AsyncIterable<Uint8Array>, target: unknown+ ) => {+ saved.fileName = fileName;+ saved.target = target;+ saved.chunks = [];+ try {+ for await (const chunk of chunks) {+ saved.chunks.push(chunk);+ }+ } catch (e) {+ // The real one reports a broken source rather than rejecting; see its own spec.+ return { fileName, status: "failed", message: (e as Error).message };+ }+ return { fileName, ...saved.result };+ }+}));+class MockWorker {postMessage = vi.fn();terminate = vi.fn();@@ -33,6 +66,7 @@ async function freshBridge(withServiceWorker = true): Promise<LocalBridge> {vi.resetModules();workerInstances.length = 0;leadership.isLeader = true;+ saved.result = { status: "saved", location: "/documents/Trilium/db.tnbackup" };swHandler = undefined;if (withServiceWorker) {Object.defineProperty(navigator, "serviceWorker", {@@ -430,7 +464,7 @@ describe("backup download keepalive", () => {vi.useFakeTimers();const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response(null));const bridge = await freshBridge();- bridge.startLocalServerWorker();+ void bridge.downloadDatabase("Backup.tnbackup");lastWorker().onmessage?.({ data: { type: "BACKUP_STREAM_ACTIVE", active: true } });await vi.advanceTimersByTimeAsync(25_000);@@ -443,6 +477,21 @@ describe("backup download keepalive", () => {vi.useRealTimers();});++ it("does not ping for a stream the page consumes itself", async () => {+ vi.useFakeTimers();+ const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response(null));+ const bridge = await freshBridge();+ bridge.startLocalServerWorker();++ // saveDatabase() holds the stream open in the page, so there is no service worker waiting+ // on it — and on iOS there is no service worker at all.+ lastWorker().onmessage?.({ data: { type: "BACKUP_STREAM_ACTIVE", active: true } });+ await vi.advanceTimersByTimeAsync(60_000);++ expect(fetchSpy).not.toHaveBeenCalled();+ vi.useRealTimers();+ });});describe("downloadDatabase", () => {@@ -493,6 +542,92 @@ describe("downloadDatabase", () => {});});+describe("saveDatabase", () => {+ /** What the bridge asked the worker to stream, including the port to play it down. */+ function backupMessage(worker: MockWorker): { port?: MessagePort; passphrase?: string } {+ return worker.postMessage.mock.calls+ .map(([ message ]) => message as { type: string; port?: MessagePort; passphrase?: string })+ .find((message) => message.type === "BACKUP_STREAM") ?? {};+ }++ /** The port the bridge handed the worker, played from the worker's side. */+ function backupPort(worker: MockWorker): MessagePort {+ const port = backupMessage(worker).port;+ if (!port) {+ throw new Error("no BACKUP_STREAM sent");+ }+ return port;+ }++ /** Answers the bridge's pulls with `chunks`, then ends — what the local worker does. */+ function serveStream(port: MessagePort, chunks: Uint8Array[]) {+ let next = 0;+ port.onmessage = (event) => {+ const message = event.data as { type?: string };+ if (message?.type !== "pull") {+ return;+ }+ if (next < chunks.length) {+ const chunk = chunks[next++];+ port.postMessage({ type: "chunk", data: chunk.buffer.slice(0) });+ } else {+ port.postMessage({ type: "end" });+ }+ };+ port.postMessage({ type: "begin", byteSize: chunks.reduce((sum, c) => sum + c.length, 0) });+ }++ it("streams the backup to the device and reports where it went", async () => {+ const bridge = await freshBridge();+ const progress: number[] = [];+ const pending = bridge.saveDatabase("db.tnbackup", "hunter2", (sent) => progress.push(sent));++ // The dynamic import of the save module resolves before the stream is asked for.+ await vi.waitFor(() => backupPort(lastWorker()));+ serveStream(backupPort(lastWorker()), [ Uint8Array.from([ 1, 2, 3 ]), Uint8Array.from([ 4, 5 ]) ]);+ lastWorker().onmessage?.({+ data: { type: "BACKUP_STREAM_PROGRESS", sentBytes: 3, totalBytes: 5 }+ });++ expect(await pending).toEqual({ status: "done", location: "/documents/Trilium/db.tnbackup" });+ expect(saved.chunks).toEqual([ Uint8Array.from([ 1, 2, 3 ]), Uint8Array.from([ 4, 5 ]) ]);+ expect(saved.fileName).toBe("db.tnbackup");+ // Documents, not the share sheet's scratch directory: a backup outlives the sheet.+ expect(saved.target).toMatchObject({ directory: "DOCUMENTS", sweep: false });+ // The passphrase rides the worker message rather than any URL.+ expect(backupMessage(lastWorker()).passphrase).toBe("hunter2");+ expect(progress).toEqual([ 3 ]);+ });++ it("is done, not cancelled, when the user dismisses the share sheet", async () => {+ const bridge = await freshBridge();+ saved.result = { status: "cancelled", location: "/documents/Trilium/db.tnbackup" };+ const pending = bridge.saveDatabase("db.tnbackup");++ await vi.waitFor(() => backupPort(lastWorker()));+ serveStream(backupPort(lastWorker()), [ Uint8Array.from([ 1 ]) ]);++ // The file was written before the sheet opened, so there is a backup either way.+ expect(await pending).toEqual({ status: "done", location: "/documents/Trilium/db.tnbackup" });+ });++ it("fails when the stream never begins, and refuses outright in a follower tab", async () => {+ const bridge = await freshBridge();+ const pending = bridge.saveDatabase("db.tnbackup");++ await vi.waitFor(() => backupPort(lastWorker()));+ backupPort(lastWorker()).postMessage({ type: "error", message: "The database is not ready yet." });++ expect(await pending).toMatchObject({+ status: "failed",+ message: "The database is not ready yet."+ });++ leadership.isLeader = false;+ expect(await bridge.saveDatabase("db.tnbackup")).toMatchObject({ status: "failed" });+ });+});+describe("restoreBackup", () => {/** The message the bridge sent to the worker to start a restore. */function restoreMessage(worker: MockWorker) {diff --git a/apps/standalone/src/local-bridge.ts b/apps/standalone/src/local-bridge.tsindex 83a5e52a92..45fa743a1c 100644--- a/apps/standalone/src/local-bridge.ts+++ b/apps/standalone/src/local-bridge.ts@@ -209,6 +209,116 @@ function settlePendingDownload(result: StandaloneDownloadResult): void {}}+/** Where a save in flight wants its progress, which arrives on the worker's own channel. */+let saveProgress: ((sentBytes: number, totalBytes: number) => void) | null = null;++/**+ * Writes the same backup onto the device, for the mobile shell.+ *+ * The service worker drops out of the path entirely: it exists in {@link downloadDatabase} only to+ * turn the stream into a `Response` the browser's download manager will take, and the WebView has+ * no download manager to take it. Here the page consumes the stream itself — the same pull-driven+ * protocol, the same container, the same passphrase — and writes it to the app's documents+ * directory, so it also works on iOS, where a service worker cannot be registered at all.+ *+ * A dismissed share sheet is still a backup: the file is complete before the sheet opens, so that+ * outcome is `done` with the path it went to, not `cancelled`.+ */+export async function saveDatabase(+ fileName: string,+ passphrase?: string,+ onProgress?: (sentBytes: number, totalBytes: number) => void+): Promise<StandaloneDownloadResult> {+ if (!isLeader()) {+ return {+ status: "failed",+ message: "A backup can only be taken from the tab that owns the database. "+ + "Close the other Trilium tabs and try again."+ };+ }++ const { saveChunksToDevice, BACKUP_TARGET } = await import("./services/capacitor_download.js");++ saveProgress = onProgress ?? null;+ try {+ const result = await saveChunksToDevice(fileName, backupChunks(passphrase), BACKUP_TARGET);+ if (result.status === "failed") {+ return { status: "failed", message: result.message };+ }+ return { status: "done", location: result.location };+ } finally {+ saveProgress = null;+ }+}++/**+ * The backup as a stream of bytes, pulled straight off the local worker.+ *+ * The worker answers one `chunk` per `pull` and ends with `end`, which is what carries backpressure+ * back to the database reads: chunks are asked for as fast as they can be written and no faster. A+ * few pulls are kept outstanding so the worker reads the next pages while the current chunk is+ * still crossing the plugin bridge, which is by far the slower of the two.+ */+async function* backupChunks(passphrase: string | undefined): AsyncGenerator<Uint8Array> {+ const worker = startLocalServerWorker();+ const channel = new MessageChannel();+ const port = channel.port1;++ const arrived: unknown[] = [];+ let waiting: ((message: unknown) => void) | null = null;+ port.onmessage = (event) => {+ if (waiting) {+ const resolve = waiting;+ waiting = null;+ resolve(event.data);+ } else {+ arrived.push(event.data);+ }+ };+ const next = (): Promise<unknown> => arrived.length+ ? Promise.resolve(arrived.shift())+ : new Promise((resolve) => { waiting = resolve; });++ worker.postMessage({ type: "BACKUP_STREAM", port: channel.port2, passphrase }, [ channel.port2 ]);++ try {+ const begin = await next() as { type?: string; message?: string } | null;+ if (begin?.type !== "begin") {+ throw new Error(begin?.message ?? "The backup stream did not start.");+ }++ let outstanding = 0;+ for (;;) {+ while (outstanding < MAX_BACKUP_CHUNKS_IN_FLIGHT) {+ outstanding++;+ port.postMessage({ type: "pull" });+ }++ const message = await next() as { type?: string; data?: ArrayBuffer; message?: string } | null;+ outstanding--;++ if (message?.type === "end") {+ return;+ }+ if (message?.type !== "chunk" || !message.data) {+ throw new Error(message?.message ?? "The backup stream failed.");+ }+ yield new Uint8Array(message.data);+ }+ } finally {+ // Ends the stream for a consumer that stopped early — a failed write, say — rather than+ // leaving the worker reading pages nothing will collect.+ port.postMessage({ type: "cancel" });+ port.close();+ }+}++/**+ * How many chunks may be in flight at once. The worker reads ahead this far while the plugin+ * bridge is busy with the previous one; every chunk beyond that is memory held for nothing.+ */+const MAX_BACKUP_CHUNKS_IN_FLIGHT = 4;+export function isLocalApiRequest(url: URL): boolean {return LOCAL_API_PREFIXES.some(p => url.pathname.startsWith(p));}@@ -362,11 +472,14 @@ export function startLocalServerWorker() {// download UI behind the notification shade.if (msg?.type === "BACKUP_STREAM_PROGRESS") {pendingDownload?.onProgress?.(Number(msg.sentBytes), Number(msg.totalBytes));+ saveProgress?.(Number(msg.sentBytes), Number(msg.totalBytes));return;}if (msg?.type === "BACKUP_STREAM_ACTIVE") {- setBackupPinging(msg.active === true);+ // Only a stream the service worker is holding open needs it kept alive; `saveDatabase`+ // consumes its own, and on iOS the ping would be a request to a worker that is not there.+ setBackupPinging(msg.active === true && pendingDownload !== null);if (msg.active === true) {if (pendingDownload) {clearTimeout(pendingDownload.startTimer);diff --git a/apps/standalone/src/main.ts b/apps/standalone/src/main.tsindex dec7feb4f8..fd35a6378d 100644--- a/apps/standalone/src/main.ts+++ b/apps/standalone/src/main.ts@@ -6,7 +6,7 @@ import {import { showErrorOverlay } from "./error-overlay.js";import { installIosInterceptors } from "./ios-interceptors.js";import { claimLeadership } from "./leader_election.js";-import { announceLeadership, attachServiceWorkerBridge, downloadDatabase, registerNativeHttpHandler, restoreBackup, startLocalServerWorker } from "./local-bridge.js";+import { announceLeadership, attachServiceWorkerBridge, downloadDatabase, registerNativeHttpHandler, restoreBackup, saveDatabase, startLocalServerWorker } from "./local-bridge.js";/*** What a cold standalone start passes through, drawn as the splash's progress bar. Weights are@@ -99,9 +99,11 @@ async function bootstrap() {registerNativeHttpHandler(capacitorHttpHandler);// The shell's WebView drops a download the moment the response says `attachment`,- // so the client routes downloads through the share sheet instead.+ // so the client routes downloads through the share sheet instead. The backup takes the+ // same route, off its own stream rather than a response.const { saveUrlToDevice } = await import("./services/capacitor_download.js");standaloneApi.save = { saveUrl: saveUrlToDevice };+ standaloneApi.backup.saveDatabase = saveDatabase;}// 1) Start the local worker ASAP (so /bootstrap is fast) — but only indiff --git a/apps/standalone/src/services/capacitor_download.spec.ts b/apps/standalone/src/services/capacitor_download.spec.tsindex 579ca0abb1..b14a948a11 100644--- a/apps/standalone/src/services/capacitor_download.spec.ts+++ b/apps/standalone/src/services/capacitor_download.spec.ts@@ -1,6 +1,6 @@import { afterEach, describe, expect, it, vi } from "vitest";-import { fileNameOf, saveUrlToDevice } from "./capacitor_download.js";+import { BACKUP_TARGET, fileNameOf, saveChunksToDevice, saveUrlToDevice } from "./capacitor_download.js";interface CapacitorWindow {Capacitor?: unknown;@@ -44,7 +44,11 @@ function installCapacitor({ share }: { share?: ReturnType<typeof vi.fn> } = {})/** The bytes handed to the plugin, decoded back out of the base64 each write carried. */function writtenBytes(written: { data: string }[]): Uint8Array {const binary = written.map((write) => atob(write.data)).join("");- return Uint8Array.from(binary, (char) => char.charCodeAt(0));+ const bytes = new Uint8Array(binary.length);+ for (let i = 0; i < binary.length; i++) {+ bytes[i] = binary.charCodeAt(i);+ }+ return bytes;}/** A response whose body streams in `chunkSize` pieces, as a real network body would. */@@ -141,7 +145,12 @@ describe("capacitor download", () => {const result = await saveUrlToDevice("api/branches/b1/export/subtree/html/t1");- expect(result).toEqual({ status: "saved", fileName: "Export.zip" });+ expect(result).toEqual({+ status: "saved",+ fileName: "Export.zip",+ // The `file://` scheme is stripped: this is shown to a person, not opened.+ location: "/cache/trilium-downloads/Export.zip"+ });expect(removed).toEqual(["trilium-downloads"]);expect(written.map((write) => write.append)).toEqual([false, true]);expect(written.every((write) => write.path === "trilium-downloads/Export.zip")).toBe(true);@@ -162,12 +171,12 @@ describe("capacitor download", () => {headers: { "content-disposition": `attachment; filename="a.bin"` },buffer: bytes})));- expect(await saveUrlToDevice("api/x")).toEqual({ status: "saved", fileName: "a.bin" });+ expect(await saveUrlToDevice("api/x")).toMatchObject({ status: "saved", fileName: "a.bin" });expect(writtenBytes(written)).toEqual(bytes);written.length = 0;vi.stubGlobal("fetch", vi.fn(async () => fakeResponse({})));- expect(await saveUrlToDevice("api/empty.bin")).toEqual({ status: "saved", fileName: "empty.bin" });+ expect(await saveUrlToDevice("api/empty.bin")).toMatchObject({ status: "saved", fileName: "empty.bin" });expect(written).toEqual([expect.objectContaining({ data: "", append: false })]);@@ -176,10 +185,16 @@ describe("capacitor download", () => {it("reports a cancelled share sheet as cancelled, and any other share error as a failure", async () => {installCapacitor({ share: vi.fn(async () => { throw new Error("Share canceled"); }) });vi.stubGlobal("fetch", vi.fn(async () => fakeResponse({ buffer: Uint8Array.from([1]) })));- expect(await saveUrlToDevice("api/a.bin")).toEqual({ status: "cancelled", fileName: "a.bin" });+ // Still a file on disk, which is what `location` says and what makes a dismissed sheet+ // survivable for a backup.+ expect(await saveUrlToDevice("api/a.bin")).toEqual({+ status: "cancelled",+ fileName: "a.bin",+ location: "/cache/trilium-downloads/a.bin"+ });installCapacitor({ share: vi.fn(async () => { throw new Error("No app can open this"); }) });- expect(await saveUrlToDevice("api/a.bin")).toEqual({+ expect(await saveUrlToDevice("api/a.bin")).toMatchObject({status: "failed",fileName: "a.bin",message: "No app can open this"@@ -206,4 +221,64 @@ describe("capacitor download", () => {});});});++ describe("saveChunksToDevice", () => {+ /** Emits `bytes` in pieces of `size`, which is how an arbitrary producer feeds the writer. */+ async function* inPieces(bytes: Uint8Array, size: number) {+ for (let offset = 0; offset < bytes.length; offset += size) {+ yield bytes.slice(offset, offset + size);+ }+ }++ it("realigns a source whose chunks are not a multiple of 3 bytes", async () => {+ const { written } = installCapacitor();+ // 4001 is coprime with 3, so every incoming piece straddles a base64 group. Writing+ // these through unaligned would pad mid-file and corrupt everything after the first+ // piece, which is exactly what the database backup's page-sized chunks would do. The+ // total clears one 768 KiB write, so there is a boundary to get wrong.+ const bytes = Uint8Array.from({ length: 800 * 1024 }, (_, i) => (i * 7) % 251);++ const result = await saveChunksToDevice("db.tnbackup", inPieces(bytes, 4001), BACKUP_TARGET);++ expect(result.status).toBe("saved");+ expect(writtenBytes(written)).toEqual(bytes);+ // Every write but the last is a whole chunk, so every write but the last is 3-aligned.+ expect(written.slice(0, -1).every((write) => atob(write.data).length % 3 === 0)).toBe(true);+ });++ it("writes a backup into documents and leaves the previous ones alone", async () => {+ const { written, removed, sharePlugin } = installCapacitor();++ const result = await saveChunksToDevice(+ "db.tnbackup", inPieces(Uint8Array.from([1, 2, 3]), 3), BACKUP_TARGET+ );++ // Sweeping the folder before each write is right for the share sheet's scratch space+ // and would destroy the previous backup here.+ expect(removed).toEqual([]);+ expect(written).toEqual([+ expect.objectContaining({ path: "Trilium/db.tnbackup", directory: "DOCUMENTS" })+ ]);+ expect(result.location).toBe("/cache/Trilium/db.tnbackup");+ expect(sharePlugin.share).toHaveBeenCalledWith({+ title: "db.tnbackup",+ files: ["file:///cache/Trilium/db.tnbackup"]+ });+ });++ it("fails without sharing when the source throws part-way through", async () => {+ const { sharePlugin } = installCapacitor();+ async function* breaks() {+ yield Uint8Array.from([1, 2, 3]);+ throw new Error("The backup stream failed.");+ }++ expect(await saveChunksToDevice("db.tnbackup", breaks(), BACKUP_TARGET)).toEqual({+ status: "failed",+ fileName: "db.tnbackup",+ message: "The backup stream failed."+ });+ expect(sharePlugin.share).not.toHaveBeenCalled();+ });+ });});diff --git a/apps/standalone/src/services/capacitor_download.ts b/apps/standalone/src/services/capacitor_download.tsindex 34f1be3e08..666e341d9a 100644--- a/apps/standalone/src/services/capacitor_download.ts+++ b/apps/standalone/src/services/capacitor_download.ts@@ -10,13 +10,45 @@ import type { StandaloneSaveResult } from "@triliumnext/commons";* them. The page therefore does the whole job: fetch (still routed to the worker by the service* worker on Android and by the iOS interceptors on `capacitor://`), write to the app's cache* directory, hand the file to the system share sheet.+ *+ * A database backup takes the same last step through {@link saveChunksToDevice}, reading its bytes+ * straight off the local worker rather than out of a response.*/-/** Where a saved file waits between being written and being handed to the share sheet. */-const DOWNLOAD_DIR = "trilium-downloads";+/** Where a kind of saved file is written, and what happens to what is already there. */+export interface SaveTarget {+ /** `Directory`'s value, which the Filesystem plugin takes as a string over the bridge. */+ directory: "CACHE" | "DOCUMENTS";+ /** The folder inside it, so saved files do not litter the directory's root. */+ folder: string;+ /**+ * Whether the folder is emptied before the write. True for scratch space the share sheet reads+ * from once; false where the folder holds the user's own files.+ */+ sweep: boolean;+}++/**+ * A download: transient. The share sheet is what the user does with it, and the copy left behind is+ * rubbish, cleared before the next one. It cannot be cleared *after* the sheet closes, because the+ * receiving app reads the shared URI on its own schedule.+ */+export const DOWNLOAD_TARGET: SaveTarget = {+ directory: "CACHE",+ folder: "trilium-downloads",+ sweep: true+};-/** `Directory.Cache` — the Filesystem plugin takes the enum's string value over the bridge. */-const CACHE = "CACHE";+/**+ * A backup: the opposite. It has to survive the share sheet being dismissed and the phone running+ * low on space, neither of which a cache directory does, and sweeping the folder would destroy the+ * previous backup. Same-day repeats overwrite, which is what the date-based default name means.+ */+export const BACKUP_TARGET: SaveTarget = {+ directory: "DOCUMENTS",+ folder: "Trilium",+ sweep: false+};/*** Bytes per write. A multiple of 3, so each chunk encodes independently: base64 pads any group@@ -37,8 +69,31 @@ export async function saveUrlToDevice(url: string): Promise<StandaloneSaveResult}fileName = fileNameOf(response, url);- const uri = await writeToCache(fileName, response);- return await shareFile(uri, fileName);+ return await saveChunksToDevice(fileName, bodyChunks(response), DOWNLOAD_TARGET);+ } catch (e) {+ return {+ status: "failed",+ fileName,+ message: e instanceof Error ? e.message : String(e)+ };+ }+}++/**+ * Writes a stream of bytes into `target` and offers the file to the share sheet.+ *+ * The file is complete before the sheet opens, and `location` is set as soon as it is on disk, so+ * a `cancelled` result still names a file that exists — which is what makes this usable for a+ * backup, where dismissing the sheet must not mean the backup never happened.+ */+export async function saveChunksToDevice(+ fileName: string,+ chunks: AsyncIterable<Uint8Array>,+ target: SaveTarget+): Promise<StandaloneSaveResult> {+ try {+ const uri = await writeChunks(fileName, chunks, target);+ return { ...await shareFile(uri, fileName), location: displayPath(uri) };} catch (e) {return {status: "failed",@@ -85,37 +140,53 @@ function getPlugin<T>(name: string): T {return plugin as T;}-/** Writes the response into the cache directory a chunk at a time and returns the file's URI. */-async function writeToCache(fileName: string, response: Response): Promise<string> {+/** Writes the chunks into `target` one at a time and returns the finished file's URI. */+async function writeChunks(+ fileName: string,+ chunks: AsyncIterable<Uint8Array>,+ { directory, folder, sweep }: SaveTarget+): Promise<string> {const filesystem = getPlugin<FilesystemPlugin>("Filesystem");- const path = `${DOWNLOAD_DIR}/${fileName}`;+ const path = `${folder}/${fileName}`;- // What the previous save left behind. The share sheet hands the receiving app a URI it reads- // on its own schedule, so a shared file can only be removed before the next save, never after- // `share()` resolves.- await filesystem- .rmdir({ path: DOWNLOAD_DIR, directory: CACHE, recursive: true })- .catch(() => undefined);+ if (sweep) {+ await filesystem+ .rmdir({ path: folder, directory, recursive: true })+ .catch(() => undefined);+ }let started = false;- for await (const chunk of chunksOf(response, CHUNK_BYTES)) {+ for await (const chunk of rechunk(chunks, CHUNK_BYTES)) {const data = toBase64(chunk);if (started) {- await filesystem.appendFile({ path, data, directory: CACHE });+ await filesystem.appendFile({ path, data, directory });} else {- await filesystem.writeFile({ path, data, directory: CACHE, recursive: true });+ await filesystem.writeFile({ path, data, directory, recursive: true });started = true;}}if (!started) {- await filesystem.writeFile({ path, data: "", directory: CACHE, recursive: true });+ await filesystem.writeFile({ path, data: "", directory, recursive: true });}- const { uri } = await filesystem.getUri({ path, directory: CACHE });+ const { uri } = await filesystem.getUri({ path, directory });return uri;}+/**+ * The file's URI as somewhere a person can be told to look. `getUri` answers with a `file://` URL,+ * percent-encoded, which is a path with two layers of machinery on top of it.+ */+function displayPath(uri: string): string {+ const path = uri.replace(/^file:\/\//, "");+ try {+ return decodeURIComponent(path);+ } catch {+ return path;+ }+}+async function shareFile(uri: string, fileName: string): Promise<StandaloneSaveResult> {try {await getPlugin<SharePlugin>("Share").share({ title: fileName, files: [uri] });@@ -130,12 +201,12 @@ async function shareFile(uri: string, fileName: string): Promise<StandaloneSaveR}/**- * Reads the body in pieces of `size` bytes, with whatever is left over emitted last.+ * Reads the body as it arrives.** A response with no `body` — happy-dom under the specs, and a WebView old enough to lack* streaming — is read whole instead.*/-async function* chunksOf(response: Response, size: number): AsyncGenerator<Uint8Array> {+async function* bodyChunks(response: Response): AsyncGenerator<Uint8Array> {const reader = response.body?.getReader();if (!reader) {const whole = new Uint8Array(await response.arrayBuffer());@@ -145,29 +216,73 @@ async function* chunksOf(response: Response, size: number): AsyncGenerator<Uint8return;}- let pending = new Uint8Array(0);for (;;) {const { done, value } = await reader.read();if (value?.length) {- const merged = new Uint8Array(pending.length + value.length);- merged.set(pending, 0);- merged.set(value, pending.length);- pending = merged;-- while (pending.length >= size) {- const chunk = pending.slice(0, size);- pending = pending.slice(size);- yield chunk;- }+ yield value;}if (done) {- break;+ return;+ }+ }+}++/**+ * Regroups a stream into pieces of `size` bytes, with whatever is left over emitted last.+ *+ * This is what keeps every write but the last a multiple of 3 bytes, so each encodes to base64+ * independently. It belongs here rather than in the callers because a source picks its chunk sizes+ * for its own reasons — a response body by packet, the backup stream by database page — and none+ * of them is obliged to know what the writer needs.+ */+async function* rechunk(chunks: AsyncIterable<Uint8Array>, size: number): AsyncGenerator<Uint8Array> {+ // Held as they arrive and joined only when a whole chunk is due. Merging each arrival into one+ // growing buffer instead would copy everything received so far on every arrival, which a+ // database backup pays for by the gigabyte.+ const parts: Uint8Array[] = [];+ let pending = 0;++ for await (const value of chunks) {+ if (!value.length) {+ continue;}+ parts.push(value);+ pending += value.length;++ while (pending >= size) {+ yield take(parts, size);+ pending -= size;+ }+ }++ if (pending) {+ yield take(parts, pending);}+}++/** Removes the first `size` bytes of `parts`, leaving the tail of a part it splits at the front. */+function take(parts: Uint8Array[], size: number): Uint8Array {+ const out = new Uint8Array(size);+ let filled = 0;- if (pending.length) {- yield pending;+ while (filled < size) {+ const part = parts[0];+ if (!part) {+ return out.subarray(0, filled);+ }++ const taken = Math.min(part.length, size - filled);+ out.set(part.subarray(0, taken), filled);+ filled += taken;++ if (taken === part.length) {+ parts.shift();+ } else {+ parts[0] = part.subarray(taken);+ }}++ return out;}/**diff --git a/packages/commons/src/lib/standalone_api_interface.ts b/packages/commons/src/lib/standalone_api_interface.tsindex ffb38b8be3..fb9b459b80 100644--- a/packages/commons/src/lib/standalone_api_interface.ts+++ b/packages/commons/src/lib/standalone_api_interface.ts@@ -47,6 +47,12 @@ export interface StandaloneDownloadResult {status: "done" | "cancelled" | "failed";/** What stopped it, when `status` is `failed`. */message?: string;+ /**+ * Where the file was written, when it went somewhere nameable. A browser download lands+ * wherever the browser puts it and reports nothing back; a save onto a device has a path,+ * which the screen shows because the user otherwise has no way to find it again.+ */+ location?: string;}export interface StandaloneBackupApi {@@ -70,6 +76,22 @@ export interface StandaloneBackupApi {passphrase?: string,onProgress?: (sentBytes: number, totalBytes: number) => void): Promise<StandaloneDownloadResult>;++ /**+ * Writes the same backup onto the device instead, for the mobile shell, whose WebView has no+ * download manager to hand it to.+ *+ * The bytes take the same pull-driven path off the database, so the container, the passphrase+ * and the progress mean exactly what they do above; only the far end differs. The file lands+ * in the app's documents directory and the share sheet then offers to put it somewhere that+ * outlives the device — but the file is already written by then, so a dismissed sheet is still+ * a backup, and the result carries where it went.+ */+ saveDatabase?(+ fileName: string,+ passphrase?: string,+ onProgress?: (sentBytes: number, totalBytes: number) => void+ ): Promise<StandaloneDownloadResult>;}/** How saving a download onto the device ended. */@@ -79,6 +101,8 @@ export interface StandaloneSaveResult {fileName?: string;/** What stopped it, when `status` is `failed`. */message?: string;+ /** Where the file was written, present once it is on disk whatever the share sheet then did. */+ location?: string;}export interface StandaloneSaveApi {