Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
fix(mobile): replace a saved file only once its successor is whole
Elian Doran6bb3d910251a
diff --git a/.claude/skills/developing-capacitor-mobile/SKILL.md b/.claude/skills/developing-capacitor-mobile/SKILL.mdindex 81de59f1d6..1e3a37ed89 100644--- a/.claude/skills/developing-capacitor-mobile/SKILL.md+++ b/.claude/skills/developing-capacitor-mobile/SKILL.md@@ -85,7 +85,12 @@ sheet.holds only what the injected runtime registered (`CapacitorHttp` and the rest of core); the `@capacitor/*`packages exist so `cap sync` wires the **native** code in.- **A shared file cannot be deleted after `share()` resolves** — the receiving app reads the URI on its- own schedule. The cache directory is cleared before the *next* save instead.+ own schedule (a Drive upload can outlive the sheet by minutes). Each download therefore lands in a+ run folder of its own, and a save prunes all previous runs *except the newest*, giving the last+ share one save's grace.+- **Every save writes `<name>.part` and renames into place on success.** A stream can die mid-way —+ `DatabaseChangedError` fires if any write lands during a backup — and truncating the final name+ first would turn the previous good backup into a partial file.- **Android needs no manifest change**: `file_paths.xml` already exposes `<cache-path path="."/>` to the`${applicationId}.fileprovider` the Share plugin looks up, and the cache directory is not externalstorage, so no permission prompt.@@ -103,8 +108,8 @@ sheet.port to the service worker, so no SW is involved (which is also why it works on iOS), and`setBackupPinging` stays off — that keepalive exists only for a stream the SW is holding open.Two things differ from a download: it goes to `Directory.Documents` under `Trilium/`, not the cache,- and the folder is **not** swept first, which would delete the previous backup. A dismissed share- sheet is `done`, not `cancelled`: the file is complete before the sheet opens.+ and nothing there is ever pruned — a repeated name replaces the file only via the `.part` swap. A+ dismissed share sheet is `done`, not `cancelled`: the file is complete before the sheet opens.- **`rechunk()` is what makes any source safe to write.** A producer picks its chunk sizes for its ownreasons — a response body by packet, the backup by database page — and neither is 3-aligned.diff --git a/apps/standalone/src/services/capacitor_download.spec.ts b/apps/standalone/src/services/capacitor_download.spec.tsindex d3cc119188..5f04534449 100644--- a/apps/standalone/src/services/capacitor_download.spec.ts+++ b/apps/standalone/src/services/capacitor_download.spec.ts@@ -15,13 +15,18 @@ interface CapacitorWindow {Capacitor?: unknown;}+/** How a run-folder segment reads inside an asserted path. */+const RUN = "[a-z0-9]+-[a-z0-9]+";+/*** Models the bridge the module reaches through: neither Filesystem nor Share is registered by the* injected runtime, so both arrive via `registerPlugin()` rather than from `Plugins`.*/-function installCapacitor({ share }: { share?: ReturnType<typeof vi.fn> } = {}) {+function installCapacitor({ share, runs }: { share?: ReturnType<typeof vi.fn>; runs?: string[] } = {}) {const written: { path: string; data: string; directory: string; append: boolean }[] = [];const removed: string[] = [];+ const deleted: string[] = [];+ const renamed: { from: string; to: string }[] = [];const filesystem = {writeFile: vi.fn(async (opts: { path: string; data: string; directory: string }) => {@@ -34,6 +39,18 @@ function installCapacitor({ share }: { share?: ReturnType<typeof vi.fn> } = {})getUri: vi.fn(async (opts: { path: string }) => ({ uri: `file:///cache/${opts.path}` })),rmdir: vi.fn(async (opts: { path: string }) => {removed.push(opts.path);+ }),+ deleteFile: vi.fn(async (opts: { path: string }) => {+ deleted.push(opts.path);+ }),+ rename: vi.fn(async (opts: { from: string; to: string }) => {+ renamed.push({ from: opts.from, to: opts.to });+ }),+ readdir: vi.fn(async () => {+ if (!runs) {+ throw new Error("Directory does not exist");+ }+ return { files: runs.map((name) => ({ name })) };})};@@ -47,7 +64,7 @@ function installCapacitor({ share }: { share?: ReturnType<typeof vi.fn> } = {})(window as unknown as CapacitorWindow).Capacitor = { Plugins: {}, registerPlugin };- return { filesystem, sharePlugin, registerPlugin, written, removed };+ return { filesystem, sharePlugin, registerPlugin, written, removed, deleted, renamed };}/**@@ -181,7 +198,7 @@ describe("capacitor download", () => {describe("saveUrlToDevice", () => {it("writes the body in chunks that decode back to the original bytes, then shares the file", async () => {- const { filesystem, sharePlugin, written, removed } = installCapacitor();+ const { filesystem, sharePlugin, written, renamed } = installCapacitor();// Over one chunk, so the write is split and the append path runs. The pattern repeats// at 251 (a prime, coprime with the chunk size) so a chunk boundary landing on the// wrong byte shows up as a mismatch rather than an accidental match.@@ -198,21 +215,40 @@ describe("capacitor download", () => {status: "saved",fileName: "Export.zip",// The `file://` scheme is stripped: this is shown to a person, not opened.- location: "/cache/trilium-downloads/Export.zip"+ location: expect.stringMatching(new RegExp(`^/cache/trilium-downloads/${RUN}/Export\\.zip$`))});- expect(removed).toEqual(["trilium-downloads"]);expect(written.map((write) => write.append)).toEqual([false, true]);- expect(written.every((write) => write.path === "trilium-downloads/Export.zip")).toBe(true);+ // Written beside the final name and renamed over it only once complete.+ const partPattern = new RegExp(`^trilium-downloads/${RUN}/Export\\.zip\\.part$`);+ expect(written.every((write) => partPattern.test(write.path))).toBe(true);+ expect(renamed).toEqual([{+ from: written[0].path,+ to: written[0].path.replace(/\.part$/, "")+ }]);expect(writtenBytes(written)).toEqual(bytes);expect(filesystem.writeFile).toHaveBeenCalledWith(expect.objectContaining({ directory: "CACHE", recursive: true }));expect(sharePlugin.share).toHaveBeenCalledWith({title: "Export.zip",- files: ["file:///cache/trilium-downloads/Export.zip"]+ files: [expect.stringMatching(new RegExp(`^file:///cache/trilium-downloads/${RUN}/Export\\.zip$`))]});});+ it("prunes older download runs but keeps the newest, which can still be being read", async () => {+ const { removed } = installCapacitor({ runs: ["aaaa-x", "cccc-x", "bbbb-x"] });+ vi.stubGlobal("fetch", vi.fn(async () => fakeResponse({+ headers: { "content-disposition": `attachment; filename="a.bin"` },+ buffer: Uint8Array.from([1])+ })));++ await saveUrlToDevice("api/a.bin");++ // The app the previous share sheet handed its file to reads it on its own schedule,+ // so the newest run gets one save's grace.+ expect(removed).toEqual(["trilium-downloads/aaaa-x", "trilium-downloads/bbbb-x"]);+ });+it("reads a response with no streaming body whole, and writes an empty one as an empty file", async () => {const { written } = installCapacitor();const bytes = Uint8Array.from([1, 2, 3, 4, 5]);@@ -239,7 +275,7 @@ describe("capacitor download", () => {expect(await saveUrlToDevice("api/a.bin")).toEqual({status: "cancelled",fileName: "a.bin",- location: "/cache/trilium-downloads/a.bin"+ location: expect.stringMatching(new RegExp(`^/cache/trilium-downloads/${RUN}/a\\.bin$`))});installCapacitor({ share: vi.fn(async () => { throw new Error("No app can open this"); }) });@@ -307,19 +343,22 @@ describe("capacitor download", () => {expect(written.slice(0, -1).every((write) => atob(write.data).length % 3 === 0)).toBe(true);});- it("writes a backup into documents and leaves the previous ones alone", async () => {- const { written, removed, sharePlugin } = installCapacitor();+ it("writes a backup beside the previous one and replaces it only when whole", async () => {+ const { filesystem, written, removed, deleted, renamed, sharePlugin } = installCapacitor();const result = await saveChunksToDevice("db.tnbackup", inPieces(Uint8Array.from([1, 2, 3]), 3), BACKUP_TARGET);- // Sweeping the folder before each write is right for the share sheet's scratch space- // and would destroy the previous backup here.+ // Backups keep their folder: nothing is pruned, and even the same-name predecessor+ // stands until its replacement is complete on disk.expect(removed).toEqual([]);+ expect(filesystem.readdir).not.toHaveBeenCalled();expect(written).toEqual([- expect.objectContaining({ path: "Trilium/db.tnbackup", directory: "DOCUMENTS" })+ expect.objectContaining({ path: "Trilium/db.tnbackup.part", directory: "DOCUMENTS" })]);+ expect(deleted).toEqual(["Trilium/db.tnbackup"]);+ expect(renamed).toEqual([{ from: "Trilium/db.tnbackup.part", to: "Trilium/db.tnbackup" }]);expect(result.location).toBe("/cache/Trilium/db.tnbackup");expect(sharePlugin.share).toHaveBeenCalledWith({title: "db.tnbackup",@@ -340,7 +379,7 @@ describe("capacitor download", () => {expect(written).toEqual([ expect.objectContaining({ data: "", append: false }) ]);expect(filesystem.appendFile).not.toHaveBeenCalled();- expect(posted[0]).toBe(JSON.stringify({ type: "open", path: "/cache/Trilium/db.tnbackup" }));+ expect(posted[0]).toBe(JSON.stringify({ type: "open", path: "/cache/Trilium/db.tnbackup.part" }));expect(posted.at(-1)).toBe(JSON.stringify({ type: "close" }));const streamed = posted.filter((message): message is ArrayBuffer => typeof message !== "string");expect(streamed.length).toBe(2);@@ -352,7 +391,7 @@ describe("capacitor download", () => {});it("fails the save when the sink answers a chunk with an error, without sharing", async () => {- const { sharePlugin } = installCapacitor();+ const { sharePlugin, deleted, renamed } = installCapacitor();const { listeners } = installFileSink({ failOnChunk: true });const result = await saveChunksToDevice(@@ -366,6 +405,9 @@ describe("capacitor download", () => {});expect(sharePlugin.share).not.toHaveBeenCalled();expect(listeners.size).toBe(0);+ // The debris goes; the file under the final name was never touched.+ expect(deleted).toEqual(["Trilium/db.tnbackup.part"]);+ expect(renamed).toEqual([]);});it("encodes through the runtime's own toBase64 when it has one, to the same bytes", async () => {@@ -401,8 +443,8 @@ describe("capacitor download", () => {}});- it("fails without sharing when the source throws part-way through", async () => {- const { sharePlugin } = installCapacitor();+ it("a failed stream never costs the previous backup", async () => {+ const { written, deleted, renamed, sharePlugin } = installCapacitor();async function* breaks() {yield Uint8Array.from([1, 2, 3]);throw new Error("The backup stream failed.");@@ -414,6 +456,13 @@ describe("capacitor download", () => {message: "The backup stream failed."});expect(sharePlugin.share).not.toHaveBeenCalled();++ // The file under the final name was never opened, replaced or deleted: with the+ // date-based default name, a failure here would otherwise turn the previous good+ // backup into a partial file.+ expect(written.every((write) => write.path.endsWith(".part"))).toBe(true);+ expect(renamed).toEqual([]);+ expect(deleted).toEqual(["Trilium/db.tnbackup.part"]);});});});diff --git a/apps/standalone/src/services/capacitor_download.ts b/apps/standalone/src/services/capacitor_download.tsindex b31a6bbafb..dd447f9547 100644--- a/apps/standalone/src/services/capacitor_download.ts+++ b/apps/standalone/src/services/capacitor_download.ts@@ -1,15 +1,11 @@import type { StandaloneSaveResult } from "@triliumnext/commons";/**- * Saving a download inside the Capacitor shell, which has no download manager of its own.- *- * A WebView answers a navigation that resolves to `Content-Disposition: attachment` by calling- * `WebView.setDownloadListener`, and neither Capacitor nor `MainActivity` registers one — the- * response is dropped without a trace. Re-requesting the URL natively would not help either: the- * bytes come from the SQLite worker inside the page, so nothing outside the WebView can fetch- * them. The page therefore does the whole job: fetch (still routed to the worker by the service- * worker on Android and by the iOS interceptors on `capacitor://`), write to the app's cache- * directory, hand the file to the system share sheet.+ * Saving a download inside the Capacitor shell, which has no download manager of its own: the+ * WebView drops a response that says `Content-Disposition: attachment`, silently. The page does+ * the whole job instead — fetch (still routed to the worker by the service worker on Android and+ * by the iOS interceptors on `capacitor://`), write to the app's storage, hand the file to the+ * system share sheet.** A database backup takes the same last step through {@link saveChunksToDevice}, reading its bytes* straight off the local worker rather than out of a response.@@ -22,45 +18,36 @@ export interface SaveTarget {/** The folder inside it, so saved files do not litter the directory's root. */folder: string;/**- * Whether the folder is emptied before the write. True for scratch space the share sheet reads- * from once; false where the folder holds the user's own files.+ * Whether saves are scratch output: each lands in a run folder of its own, and older runs are+ * pruned save by save. False where the folder holds the user's own files, which nothing prunes.*/- sweep: boolean;+ scratch: boolean;}/**- * A download: transient. The share sheet is what the user does with it, and the copy left behind is- * rubbish, cleared before the next one. It cannot be cleared *after* the sheet closes, because the- * receiving app reads the shared URI on its own schedule.+ * A download: scratch. The share sheet is what the user does with it, so only the newest previous+ * run is kept — the app the sheet handed it to can still be reading it.*/export const DOWNLOAD_TARGET: SaveTarget = {directory: "CACHE",folder: "trilium-downloads",- sweep: true+ scratch: true};/**- * A backup: the opposite. It has to survive the share sheet being dismissed and the phone running- * low on space, neither of which a cache directory does, and sweeping the folder would destroy the- * previous backup. Same-day repeats overwrite, which is what the date-based default name means.+ * A backup: the user's file. It has to outlive the share sheet and storage pressure, which a cache+ * directory does not. A repeated name replaces the file, and only once the new one is whole.*/export const BACKUP_TARGET: SaveTarget = {directory: "DOCUMENTS",folder: "Trilium",- sweep: false+ scratch: false};/**- * Bytes per write.- *- * A multiple of 3, so each chunk encodes independently: base64 pads any group narrower than three- * bytes, and a padded group in the middle of a file decodes to the wrong bytes. Every `3n` KiB- * satisfies that.- *- * The size is a trade against the bridge, which is what a write costs: the chunk crosses as base64- * inside a JSON string, so the bytes are walked about five times between here and the file, and- * the chunk, its base64 copy and both sides of the marshalled string are live at once — budget- * roughly four times the chunk. Fewer, larger writes buy fewer round trips at that price.+ * Bytes per write: a multiple of 3, so each chunk encodes to base64 independently — base64 pads+ * any group narrower than three bytes, and a padded group mid-file decodes to the wrong bytes.+ * Larger chunks buy fewer transport round trips and hold more transient copies at once.*/const DEFAULT_CHUNK_BYTES = 3 * 1024 * 1024;@@ -132,6 +119,9 @@ interface FilesystemPlugin {appendFile(opts: WriteOptions): Promise<void>;getUri(opts: { path: string; directory: string }): Promise<{ uri: string }>;rmdir(opts: { path: string; directory: string; recursive: boolean }): Promise<void>;+ deleteFile(opts: { path: string; directory: string }): Promise<void>;+ rename(opts: { from: string; to: string; directory: string }): Promise<void>;+ readdir(opts: { path: string; directory: string }): Promise<{ files: { name: string }[] }>;}interface SharePlugin {@@ -157,36 +147,87 @@ function getPlugin<T>(name: string): T {return plugin as T;}-/** Writes the chunks into `target` one at a time and returns the finished file's URI. */+/** Writes the chunks into `target`, replacing any previous file only on success. */async function writeChunks(fileName: string,chunks: AsyncIterable<Uint8Array>,- { directory, folder, sweep }: SaveTarget+ { directory, folder, scratch }: SaveTarget): Promise<string> {const filesystem = getPlugin<FilesystemPlugin>("Filesystem");- const path = `${folder}/${fileName}`;-- if (sweep) {- await filesystem- .rmdir({ path: folder, directory, recursive: true })- .catch(() => undefined);+ const parent = scratch ? `${folder}/${runName()}` : folder;+ const path = `${parent}/${fileName}`;+ // Every byte lands in a sibling first: a stream that dies mid-way must cost nothing but+ // itself, never the file already under the final name.+ const partPath = `${path}.part`;++ if (scratch) {+ await pruneRuns(filesystem, folder, directory);}- const sink = takeFileSink();- if (sink) {- try {- // The plugin creates the file first, so its own directory mapping — not a guess at- // it — is what turns `path` into the absolute path the sink writes to.- await filesystem.writeFile({ path, data: "", directory, recursive: true });- const { uri } = await filesystem.getUri({ path, directory });-- await writeThroughSink(sink, displayPath(uri), chunks);- return uri;- } finally {- releaseFileSink();+ try {+ const sink = takeFileSink();+ if (sink) {+ try {+ // The plugin creates the file first, so its own directory mapping — not a guess+ // at it — is what turns `partPath` into the absolute path the sink writes to.+ await filesystem.writeFile({ path: partPath, data: "", directory, recursive: true });+ const { uri } = await filesystem.getUri({ path: partPath, directory });+ await writeThroughSink(sink, displayPath(uri), chunks);+ } finally {+ releaseFileSink();+ }+ } else {+ await writeThroughPlugin(filesystem, partPath, directory, chunks);}+ } catch (e) {+ await filesystem.deleteFile({ path: partPath, directory }).catch(() => undefined);+ throw e;+ }++ // Only now, with its replacement complete on disk, does the previous file fall.+ await filesystem.deleteFile({ path, directory }).catch(() => undefined);+ await filesystem.rename({ from: partPath, to: path, directory });++ const { uri } = await filesystem.getUri({ path, directory });+ return uri;+}++/** Names one save's run folder so that later saves sort after earlier ones. */+function runName(): string {+ return `${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 6)}`;+}++/**+ * Deletes previous run folders, except the newest: the app the share sheet handed its file to can+ * still be reading it, and one save's grace is as long as that gets.+ */+async function pruneRuns(+ filesystem: FilesystemPlugin,+ folder: string,+ directory: string+): Promise<void> {+ const listing = await filesystem.readdir({ path: folder, directory }).catch(() => null);+ if (!listing) {+ return;+ }++ const runs = listing.files+ .map((file) => file.name)+ .sort((a, b) => a.length - b.length || (a < b ? -1 : a > b ? 1 : 0));+ for (const run of runs.slice(0, -1)) {+ await filesystem+ .rmdir({ path: `${folder}/${run}`, directory, recursive: true })+ .catch(() => undefined);}+}+/** The plugin-bridge fallback: each chunk crosses as base64 and is appended natively. */+async function writeThroughPlugin(+ filesystem: FilesystemPlugin,+ path: string,+ directory: string,+ chunks: AsyncIterable<Uint8Array>+): Promise<void> {let started = false;// One call is kept in flight while the next chunk encodes, so the encode's cost hides behind// the bridge's. Never two: appends are ordered only because each is issued after the previous@@ -216,17 +257,14 @@ async function writeChunks(if (!started) {await filesystem.writeFile({ path, data: "", directory, recursive: true });}-- const { uri } = await filesystem.getUri({ path, directory });- return uri;}/*** `window.triliumFileSink`: the binary write channel `TriliumFileSink.java` injects on Android.** A `WebMessageListener` object, so it carries raw `ArrayBuffer`s — where the plugin bridge wraps- * every chunk in base64 inside a JSON string that the native side re-parses whole, capping writes- * around 13 MB/s. Absent on iOS, on WebViews too old for ArrayBuffer messages, and on the web.+ * every chunk in base64 inside a JSON string that the native side re-parses whole. Absent on iOS,+ * on WebViews too old for ArrayBuffer messages, and on the web.*/interface NativeFileSink {postMessage(message: string | ArrayBuffer): void;