Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
feat(standalone): import a zip from the picked File without loading it whole
Elian Doranda034da3c26e
diff --git a/.claude/skills/developing-standalone/SKILL.md b/.claude/skills/developing-standalone/SKILL.mdindex afe9b19ab4..49742b3ddb 100644--- a/.claude/skills/developing-standalone/SKILL.md+++ b/.claude/skills/developing-standalone/SKILL.md@@ -32,9 +32,10 @@ apps/standalone/1. **`main.ts`** requires a service worker (secure context: HTTPS or localhost — otherwise it throws a message explaining that), registers `./sw.js` with scope `/`, reloads once if the SW installed but doesn't control the page yet, and exposes `window.standaloneApi` (`restore.importBackup`, `backup.downloadDatabase`) — the client's escape hatch for things carrying a whole file, the twin of desktop's `window.electronApi`. The leader also publishes `localFetch` there (step 3).2. **Leader election** (`leader_election.ts`): the OPFS SAHPool VFS takes *exclusive* sync-access handles, and `createSyncAccessHandle` exists only in a dedicated worker, so exactly one tab per origin can own the database. A **Web Lock** (`trilium-standalone-db`, held for the tab's lifetime) picks it; only the winner calls `startLocalServerWorker()`; when it closes the browser releases the lock and a waiting tab is promoted. Where Web Locks are missing (some WebViews) the tab assumes it is alone.-3. **The leader answers its own API calls.** Winning the lock also publishes `localFetch` (`local-bridge.ts`) on `window.standaloneApi`, and the client prefers it over its XHR transport — in `ajax()` (`apps/client/src/services/server.ts`) and in `setupGlob()`'s `/bootstrap` (`apps/client/src/index.ts`). Without it the leader's own request leaves as an XHR, is intercepted by the service worker, and is posted back into the page it started in, to reach a worker that page owns. `localAjax()` matches the XHR path's contract — headers, JSON body, CSRF refresh-and-retry-once, the `silent*` suppressions, `raw`, and a timeout of `opts.timeoutMs ?? DEFAULT_TIMEOUT`. **Build with `VITE_DISABLE_LOCAL_FETCH=true` to take the service worker's route instead**, which is what compares the two transports. Follower tabs leave `localFetch` unset; so do `upload()`, `chunked_upload.ts` and the LLM stream, which carry `FormData` or want the body as it arrives.-4. **Service worker** (`sw.ts`): cache-first for hashed assets, network-first otherwise; requests under `LOCAL_API_PREFIXES` (`/bootstrap`, `/api/`, `/sync/`, `/search/`, from `local-bridge.ts`) are forwarded with `LOCAL_FETCH` to the **leader** client, found via a cached id, `WHO_IS_LEADER` / `LEADER_REPLY` probing, and one retry on `NOT_LEADER` (leadership can flip while a request is in flight). Since step 3 this carries follower tabs, engine-initiated loads (`<img src="api/images/…">`, fonts, themes) and the call sites that reach past `server.ts`, rather than the whole API surface. Backup downloads stream over a `MessageChannel` from the leader. It must **let `/_trilium_native_http/` fall through** (Android native proxy) and has a defensive guard for `capacitor://`.-5. **The worker** (`local-server-worker.ts`): waits for the previous worker's OPFS access handles to be released (`waitForSahPoolRelease` — a reload races the browser's asynchronous release, and `installOpfsSAHPoolVfs()` reacts to an acquisition failure by *deleting the pool directory*, so the pool must be acquirable before it runs — a wait that reaches its deadline ends the boot instead of installing anyway), then installs the SAHPool VFS **once** — failure is a fatal startup error painted by `error-overlay.ts`, never an in-memory fallback (`TRILIUM_INTEGRATION_TEST=memory` builds seed a fixture into the pool, so writes survive reloads within a test). It then calls `initializeCore({...})` with the `lightweight/*` providers, registers `createConfiguredRouter()` (`BrowserRouter` mimicking Express, path params + query strings), then answers `LOCAL_REQUEST` messages. `DbLock` (`lightweight/db_lock.ts`) serialises `createAsyncRoute` handlers (imports) that hold a transaction across `await`s against the single connection — synchronous routes keep a zero-overhead path. `WS_MESSAGE` replaces the WebSocket in both directions. Anything that escapes after startup is reported as `WORKER_ERROR` for one background task; before startup it means the app never came up and `error-overlay.ts` paints it (the async worker init is invisible to `main.ts`'s try/catch).+3. **The leader answers its own API calls.** Winning the lock also publishes `localFetch` (`local-bridge.ts`) on `window.standaloneApi`, and the client prefers it over its XHR transport — in `ajax()` (`apps/client/src/services/server.ts`) and in `setupGlob()`'s `/bootstrap` (`apps/client/src/index.ts`). Without it the leader's own request leaves as an XHR, is intercepted by the service worker, and is posted back into the page it started in, to reach a worker that page owns. `localAjax()` matches the XHR path's contract — headers, JSON body, CSRF refresh-and-retry-once, the `silent*` suppressions, `raw`, and a timeout of `opts.timeoutMs ?? DEFAULT_TIMEOUT`. **Build with `VITE_DISABLE_LOCAL_FETCH=true` to take the service worker's route instead**, which is what compares the two transports. Follower tabs leave `localFetch` unset. `chunked_upload.ts` and the LLM stream bypass it, since they want the body as it arrives.+4. **Multipart uploads hand over the `File`, not its bytes.** `upload()` and the import's `uploadFiles()` call `uploadForm()` (`server.ts`), which on the leader uses `standaloneApi.localUpload`: it posts the form's fields and the picked `File` as a `{ type: "upload" }` body, and structured cloning passes the `File` by reference. `BrowserRouter` gives the import routes that `File` as `req.file.blob` (`readsUploadsInPlace`, set by the `importUploadMiddleware` marker in `browser_routes.ts`) and reads it into `req.file.buffer` for every other route. A zip importer then reads `{ blob }` through `lightweight/zip_provider.ts` one byte range at a time (central directory from the tail, then each entry by offset, ZIP64 included), so a multi-gigabyte archive is never in memory whole; any other importer calls core's `loadUploadedBytes()` first. On the server the same zip arrives as `{ path }` from multer's temp file.+5. **Service worker** (`sw.ts`): cache-first for hashed assets, network-first otherwise; requests under `LOCAL_API_PREFIXES` (`/bootstrap`, `/api/`, `/sync/`, `/search/`, from `local-bridge.ts`) are forwarded with `LOCAL_FETCH` to the **leader** client, found via a cached id, `WHO_IS_LEADER` / `LEADER_REPLY` probing, and one retry on `NOT_LEADER` (leadership can flip while a request is in flight). Since step 3 this carries follower tabs, engine-initiated loads (`<img src="api/images/…">`, fonts, themes) and the call sites that reach past `server.ts`, rather than the whole API surface. Backup downloads stream over a `MessageChannel` from the leader. It must **let `/_trilium_native_http/` fall through** (Android native proxy) and has a defensive guard for `capacitor://`.+6. **The worker** (`local-server-worker.ts`): waits for the previous worker's OPFS access handles to be released (`waitForSahPoolRelease` — a reload races the browser's asynchronous release, and `installOpfsSAHPoolVfs()` reacts to an acquisition failure by *deleting the pool directory*, so the pool must be acquirable before it runs — a wait that reaches its deadline ends the boot instead of installing anyway), then installs the SAHPool VFS **once** — failure is a fatal startup error painted by `error-overlay.ts`, never an in-memory fallback (`TRILIUM_INTEGRATION_TEST=memory` builds seed a fixture into the pool, so writes survive reloads within a test). It then calls `initializeCore({...})` with the `lightweight/*` providers, registers `createConfiguredRouter()` (`BrowserRouter` mimicking Express, path params + query strings), then answers `LOCAL_REQUEST` messages. `DbLock` (`lightweight/db_lock.ts`) serialises `createAsyncRoute` handlers (imports) that hold a transaction across `await`s against the single connection — synchronous routes keep a zero-overhead path. `WS_MESSAGE` replaces the WebSocket in both directions. Anything that escapes after startup is reported as `WORKER_ERROR` for one background task; before startup it means the app never came up and `error-overlay.ts` paints it (the async worker init is invisible to `main.ts`'s try/catch).## The provider set (`src/lightweight/`)@@ -57,7 +58,7 @@ apps/standalone/| Image | `services/image_provider.ts` | format sniffing only, no resize/compression || LLM | `llm_skills.ts` | this build's half of the LLM stack (`registerStandaloneLlmExtensions`), dynamically imported so skill sheets don't join the startup bundle |-Standalone-only routes (integrity checks, the LLM channel, upload handling that arrives as a buffer instead of a temp file) are registered in `lightweight/browser_routes.ts`; everything else is core's shared route table (`adding-internal-api-route` skill).+Standalone-only routes (integrity checks, the LLM channel, uploads that arrive as a posted `File` instead of a temp file) are registered in `lightweight/browser_routes.ts`; everything else is core's shared route table (`adding-internal-api-route` skill).## Bundle discipline (`vite.config.mts`)diff --git a/apps/client/src/services/import.spec.ts b/apps/client/src/services/import.spec.tsindex 84e35cadf1..c354db0f8d 100644--- a/apps/client/src/services/import.spec.ts+++ b/apps/client/src/services/import.spec.ts@@ -52,78 +52,73 @@ beforeEach(() => {});describe("uploadFiles", () => {+ const fileA = new File([ "a" ], "a.md");+ const fileB = new File([ "b" ], "b.md");+ let uploadForm: ReturnType<typeof vi.fn>;+beforeEach(() => {- ($ as any).ajax = vi.fn(async () => ({}));+ uploadForm = vi.fn(async () => ({}));+ server.uploadForm = uploadForm as typeof server.uploadForm;server.getHeaders = vi.fn(async () => ({ "x-h": "1" })) as typeof server.getHeaders;});it("throws on an unrecognized entity type", async () => {- await expect(uploadFiles("bogus", "p1", ["f"], { shrinkImages: false })).rejects.toThrow(+ await expect(uploadFiles("bogus", "p1", [ fileA ], { shrinkImages: false })).rejects.toThrow("Unrecognized import entity type 'bogus'.");- expect(($ as any).ajax).not.toHaveBeenCalled();+ expect(uploadForm).not.toHaveBeenCalled();});it("returns early without uploading when there are no files", async () => {const result = await uploadFiles("notes", "p1", [], { shrinkImages: false });expect(result).toBeUndefined();- expect(($ as any).ajax).not.toHaveBeenCalled();+ expect(uploadForm).not.toHaveBeenCalled();});- it("posts one ajax request per file, flagging the last, and forwards options + headers", async () => {- await uploadFiles("notes", "parent1", ["a", "b"], {+ it("posts one form per file, flagging the last, and forwards options + headers", async () => {+ await uploadFiles("notes", "parent1", [ fileA, fileB ], {shrinkImages: true,safeImport: "true"});- const ajax = ($ as any).ajax as ReturnType<typeof vi.fn>;- expect(ajax).toHaveBeenCalledTimes(2);-- const first = ajax.mock.calls[0][0];- expect(first.url).toContain("notes/parent1/notes-import");- expect(first.type).toBe("POST");- expect(first.headers).toEqual({ "x-h": "1" });- const firstData = first.data as FormData;- expect(firstData.get("last")).toBe("false");- expect(firstData.get("upload")).toBe("a");- expect(firstData.get("shrinkImages")).toBe("true");- expect(firstData.get("safeImport")).toBe("true");+ expect(uploadForm).toHaveBeenCalledTimes(2);++ const [ firstUrl, first ] = uploadForm.mock.calls[0];+ expect(firstUrl).toBe("notes/parent1/notes-import");+ expect(first).toMatchObject({ method: "POST", headers: { "x-h": "1" }, file: fileA, dataType: "json" });+ expect(first.fields).toMatchObject({ last: "false", shrinkImages: "true", safeImport: "true" });// taskId is shared across files in the same upload batch.- const taskId = firstData.get("taskId");- expect(typeof taskId).toBe("string");+ expect(typeof first.fields.taskId).toBe("string");- const second = ajax.mock.calls[1][0];- const secondData = second.data as FormData;- expect(secondData.get("last")).toBe("true");- expect(secondData.get("upload")).toBe("b");- expect(secondData.get("taskId")).toBe(taskId);+ const [ , second ] = uploadForm.mock.calls[1];+ expect(second.file).toBe(fileB);+ expect(second.fields).toMatchObject({ last: "true", taskId: first.fields.taskId });});it("uses the attachments endpoint for the attachments entity type", async () => {- await uploadFiles("attachments", "host1", ["a"], { shrinkImages: false });- const first = ($ as any).ajax.mock.calls[0][0];- expect(first.url).toContain("notes/host1/attachments-import");+ await uploadFiles("attachments", "host1", [ fileA ], { shrinkImages: false });+ expect(uploadForm.mock.calls[0][0]).toBe("notes/host1/attachments-import");});- it("surfaces an upload error through the toast service via the ajax error callback", async () => {- // Spy on the i18n binding so we can verify xhr.responseText is forwarded+ it("surfaces an upload error through the toast service and rejects", async () => {+ // Spy on the i18n binding so we can verify responseText is forwarded// without depending on i18next interpolation (i18next is uninitialised in// the test env, so t() returns the key and never interpolates). We assert// the call args/structure rather than a human-readable translated string.const tSpy = vi.spyOn(i18n, "t");vi.useFakeTimers();try {- ($ as any).ajax = vi.fn(async (opts: any) => {- opts.error({ responseText: "boom" });- return {};- });- await uploadFiles("notes", "p1", ["a"], { shrinkImages: false });+ const failure = { status: 500, responseText: "boom" };+ uploadForm.mockRejectedValue(failure);+ await expect(uploadFiles("notes", "p1", [ fileA, fileB ], { shrinkImages: false })).rejects.toBe(failure);+ // The first failure ends the batch.+ expect(uploadForm).toHaveBeenCalledTimes(1);// The error toast is deferred (so a WebSocket taskError can claim the taskId first and// win), so the fallback only fires after IMPORT_ERROR_FALLBACK_DELAY — flush it first.vi.runOnlyPendingTimers();expect(toastService.showError).toHaveBeenCalledTimes(1);- // The xhr.responseText ("boom") must be forwarded into the message- // interpolation, not dropped or replaced with the whole xhr object.+ // The responseText ("boom") must be forwarded into the message+ // interpolation, not dropped or replaced with the whole error object.expect(tSpy).toHaveBeenCalledWith("import.failed", { message: "boom" });} finally {tSpy.mockRestore();@@ -132,8 +127,8 @@ describe("uploadFiles", () => {});it("is also reachable through the default export", async () => {- await importService.uploadFiles("notes", "p1", ["a"], { shrinkImages: false });- expect(($ as any).ajax).toHaveBeenCalledTimes(1);+ await importService.uploadFiles("notes", "p1", [ fileA ], { shrinkImages: false });+ expect(uploadForm).toHaveBeenCalledTimes(1);});});diff --git a/apps/client/src/services/import.ts b/apps/client/src/services/import.tsindex 307805b649..00ba25ff38 100644--- a/apps/client/src/services/import.ts+++ b/apps/client/src/services/import.ts@@ -2,7 +2,7 @@ import { ProgressPhase, WebSocketMessage } from "@triliumnext/commons";import appContext from "../components/app_context.js";import { t } from "./i18n.js";-import server from "./server.js";+import server, { type UploadFormError } from "./server.js";import toastService, { type ToastOptionsWithRequiredId } from "./toast.js";import utils from "./utils.js";import ws from "./ws.js";@@ -24,7 +24,7 @@ export interface UploadFilesOptions {replaceUnderscoresWithSpaces?: BooleanLike;}-export async function uploadFiles(entityType: string, parentNoteId: string, files: string[] | File[], options: UploadFilesOptions) {+export async function uploadFiles(entityType: string, parentNoteId: string, files: File[], options: UploadFilesOptions) {if (!["notes", "attachments"].includes(entityType)) {throw new Error(`Unrecognized import entity type '${entityType}'.`);}@@ -39,32 +39,31 @@ export async function uploadFiles(entityType: string, parentNoteId: string, filefor (const file of files) {counter++;- const formData = new FormData();- formData.append("upload", file);- formData.append("taskId", taskId);- formData.append("last", counter === files.length ? "true" : "false");-- for (const key in options) {- formData.append(key, (options as any)[key]);+ const fields: Record<string, string> = {+ taskId,+ last: counter === files.length ? "true" : "false"+ };+ for (const [ key, value ] of Object.entries(options)) {+ fields[key] = String(value);}- await $.ajax({- url: `${window.glob.baseApiUrl}notes/${parentNoteId}/${entityType}-import`,- headers: await server.getHeaders(),- data: formData,- dataType: "json",- type: "POST",- timeout: 60 * 60 * 1000,- error (xhr) {- // Fallback toast for failures that never produced a WebSocket `taskError` (e.g. transport or- // auth errors raised before the importer runs). Deferred so the canonical WebSocket error —- // emitted alongside this same 500 — can claim the taskId first and win.- const message = t("import.failed", { message: xhr.responseText });- setTimeout(() => reportImportError(taskId, message), IMPORT_ERROR_FALLBACK_DELAY);- },- contentType: false, // NEEDED, DON'T REMOVE THIS- processData: false // NEEDED, DON'T REMOVE THIS- });+ try {+ await server.uploadForm(`notes/${parentNoteId}/${entityType}-import`, {+ method: "POST",+ headers: await server.getHeaders(),+ fields,+ file,+ timeoutMs: 60 * 60 * 1000,+ dataType: "json"+ });+ } catch (e) {+ // Fallback toast for failures that never produced a WebSocket `taskError` (e.g. transport or+ // auth errors raised before the importer runs). Deferred so the canonical WebSocket error —+ // emitted alongside this same 500 — can claim the taskId first and win.+ const message = t("import.failed", { message: (e as Partial<UploadFormError>).responseText });+ setTimeout(() => reportImportError(taskId, message), IMPORT_ERROR_FALLBACK_DELAY);+ throw e;+ }}}diff --git a/apps/client/src/services/server.spec.ts b/apps/client/src/services/server.spec.tsindex 1cd3c068c2..8299cd8942 100644--- a/apps/client/src/services/server.spec.ts+++ b/apps/client/src/services/server.spec.ts@@ -602,3 +602,76 @@ describe("localFetch transport", () => {await expect(server.get("url")).rejects.toThrow("Worker error: boom");});});++describe("uploadForm", () => {+ const file = new File([ "zip" ], "notes.zip", { type: "application/zip" });+ const options = {+ method: "POST",+ headers: { "x-csrf-token": "tok", "trilium-hoisted-note-id": null },+ fields: { taskId: "t1" },+ file,+ timeoutMs: 1000+ };++ afterEach(() => {+ delete (window as any).standaloneApi;+ });++ it("hands the File to the worker on a tab that owns it, and parses a JSON reply", async () => {+ const localUpload = vi.fn(async () => new Response(JSON.stringify({ noteId: "n1" }), {+ headers: { "content-type": "application/json" }+ }));+ (window as any).standaloneApi = { localFetch: vi.fn(), localUpload };+ (window as any).$.ajax = () => {+ throw new Error("$.ajax was used even though a tab owns the worker");+ };++ expect(await server.uploadForm("notes/root/notes-import", options)).toEqual({ noteId: "n1" });+ expect(localUpload).toHaveBeenCalledWith("/api/notes/root/notes-import", {+ method: "POST",+ headers: { "x-csrf-token": "tok" },+ fields: { taskId: "t1" },+ file+ });+ });++ it("rejects with the status and text of an error reply, the fields a jqXHR carries", async () => {+ (window as any).standaloneApi = {+ localUpload: vi.fn(async () => new Response("Import failed", { status: 500 }))+ };++ await expect(server.uploadForm("notes/root/notes-import", options))+ .rejects.toEqual({ status: 500, responseText: "Import failed" });+ });++ it("rejects when the worker does not answer in time", async () => {+ vi.useFakeTimers();+ try {+ (window as any).standaloneApi = { localUpload: vi.fn(() => new Promise(() => {})) };+ const pending = expect(server.uploadForm("notes/root/notes-import", options)).rejects.toThrow("Timed out after 1000ms");+ await vi.advanceTimersByTimeAsync(1000);+ await pending;+ } finally {+ vi.useRealTimers();+ }+ });++ it("sends a multipart XHR elsewhere, with the file under `upload`", async () => {+ const ajaxSpy = vi.fn(async () => ({ ok: true }));+ (window as any).$.ajax = ajaxSpy;++ expect(await server.uploadForm("notes/root/notes-import", { ...options, dataType: "json" })).toEqual({ ok: true });+ const opts = (ajaxSpy.mock.calls[0] as unknown[])[0] as any;+ expect(opts).toMatchObject({+ url: "/api/notes/root/notes-import",+ type: "POST",+ dataType: "json",+ timeout: 1000,+ headers: { "x-csrf-token": "tok" },+ contentType: false,+ processData: false+ });+ expect((opts.data as FormData).get("upload")).toBeInstanceOf(File);+ expect((opts.data as FormData).get("taskId")).toBe("t1");+ });+});diff --git a/apps/client/src/services/server.ts b/apps/client/src/services/server.tsindex efc6a31995..b949f40bb1 100644--- a/apps/client/src/services/server.ts+++ b/apps/client/src/services/server.ts@@ -1,3 +1,5 @@+import type { StandaloneApi } from "@triliumnext/commons";+import { t } from "./i18n.js";import { getSetupAuthToken } from "./setup_auth.js";import utils, { isShare } from "./utils.js";@@ -95,19 +97,14 @@ async function remove<T>(url: string, componentId?: string) {}async function upload(url: string, fileToUpload: File, componentId?: string, method = "PUT") {- const formData = new FormData();- formData.append("upload", fileToUpload);-- const doUpload = async () => $.ajax({- url: window.glob.baseApiUrl + url,+ const doUpload = async () => uploadForm(url, {+ method,headers: await getHeaders(componentId ? {"trilium-component-id": componentId} : undefined),- data: formData,- type: method,- timeout: 60 * 60 * 1000,- contentType: false, // NEEDED, DON'T REMOVE THIS- processData: false // NEEDED, DON'T REMOVE THIS+ fields: {},+ file: fileToUpload,+ timeoutMs: UPLOAD_TIMEOUT});try {@@ -123,6 +120,102 @@ async function upload(url: string, fileToUpload: File, componentId?: string, met}}+export interface UploadFormOptions {+ method: string;+ headers: Headers;+ fields: Record<string, string>;+ /** Sent as the form's `upload` field. */+ file: File;+ timeoutMs: number;+ /** Passed to jQuery, which otherwise picks a parser from the response's content type. */+ dataType?: "json";+}++/** What `uploadForm()` rejects with on an error status: the fields of a jqXHR that callers read. */+export interface UploadFormError {+ status: number;+ responseText: string;+}++const UPLOAD_TIMEOUT = 60 * 60 * 1000;++/**+ * Sends a multipart form carrying one file and resolves with the parsed response. The standalone+ * leader tab hands the `File` to its worker through `standaloneApi.localUpload`, so the page never+ * reads it and an import can read a zip one entry at a time; everywhere else it goes out as an XHR.+ * Rejects with an {@link UploadFormError} on an error status.+ */+// eslint-disable-next-line @typescript-eslint/no-explicit-any -- the parsed body of whatever the route answers, as `$.ajax` gives it+async function uploadForm(url: string, options: UploadFormOptions): Promise<any> {+ const headers: Record<string, string> = {};+ for (const [ name, value ] of Object.entries(options.headers)) {+ if (value) {+ headers[name] = value;+ }+ }++ const localUpload = USE_LOCAL_FETCH ? window.standaloneApi?.localUpload : undefined;+ if (localUpload) {+ return await localUploadForm(localUpload, url, headers, options);+ }++ const formData = new FormData();+ formData.append("upload", options.file);+ for (const [ name, value ] of Object.entries(options.fields)) {+ formData.append(name, value);+ }++ return await $.ajax({+ url: window.glob.baseApiUrl + url,+ headers,+ data: formData,+ type: options.method,+ dataType: options.dataType,+ timeout: options.timeoutMs,+ contentType: false, // NEEDED, DON'T REMOVE THIS+ processData: false // NEEDED, DON'T REMOVE THIS+ });+}++async function localUploadForm(+ localUpload: NonNullable<StandaloneApi["localUpload"]>,+ url: string,+ headers: Record<string, string>,+ options: UploadFormOptions+): Promise<unknown> {+ let timeoutHandle: ReturnType<typeof setTimeout> | undefined;+ const expiry = new Promise<never>((_, rej) => {+ timeoutHandle = setTimeout(+ () => rej(new Error(`Timed out after ${options.timeoutMs}ms: ${options.method} ${url}`)),+ options.timeoutMs+ );+ });++ let resp: globalThis.Response;+ try {+ resp = await Promise.race([+ localUpload(window.glob.baseApiUrl + url, {+ method: options.method,+ headers,+ fields: options.fields,+ file: options.file+ }),+ expiry+ ]);+ } finally {+ clearTimeout(timeoutHandle);+ }++ const responseText = await resp.text();+ if (!resp.ok) {+ throw { status: resp.status, responseText } satisfies UploadFormError;+ }+ if ((resp.headers.get("content-type") ?? "").includes("json")) {+ return responseText ? JSON.parse(responseText) : null;+ }+ return responseText;+}+let maxKnownEntityChangeId = 0;let csrfRefreshInProgress: Promise<void> | null = null;@@ -456,6 +549,7 @@ export default {patch,remove,upload,+ uploadForm,// don't remove, used from CKEditor image upload!getHeaders,getMaxKnownEntityChangeId: () => maxKnownEntityChangeIddiff --git a/apps/server/src/zip_provider.ts b/apps/server/src/zip_provider.tsindex 83c7a592dc..92ce05e9c8 100644--- a/apps/server/src/zip_provider.ts+++ b/apps/server/src/zip_provider.ts@@ -108,6 +108,10 @@ async function openZip(source: ZipSource) {return yauzl.fromBufferPromise(buffer.subarray(0, findArchiveEnd(tail, buffer.length)), options);}+ if ("blob" in source) {+ throw new Error("A Blob zip source is only readable in the browser.");+ }+const handle = await fs.promises.open(source.path, "r");let size: number;let end: number;diff --git a/apps/standalone/src/lightweight/browser_router.spec.ts b/apps/standalone/src/lightweight/browser_router.spec.tsindex e8951d7bf4..d01caa46de 100644--- a/apps/standalone/src/lightweight/browser_router.spec.ts+++ b/apps/standalone/src/lightweight/browser_router.spec.ts@@ -1,7 +1,7 @@import { getContext } from "@triliumnext/core";import { afterEach, describe, expect, it, vi } from "vitest";-import { BrowserRouter, createRouter } from "./browser_router.js";+import { BrowserRouter, createRouter, type UploadedFile } from "./browser_router.js";import { dbLock } from "./db_lock.js";const RAW_RESPONSE = Symbol.for("RAW_RESPONSE");@@ -163,6 +163,26 @@ describe("BrowserRouter body parsing", () => {expect(req?.file && Array.from(req.file.buffer)).toEqual([1, 2, 3]);});+ it("takes a posted File's fields, and reads its bytes only for a route that does not read it in place", async () => {+ const file = new File([ new Uint8Array([ 4, 5, 6 ]) ], "notes.zip", { type: "application/zip" });+ const arrayBuffer = vi.spyOn(file, "arrayBuffer");+ const body = { type: "upload", fields: { taskId: "t1" }, file };++ const router = new BrowserRouter();+ let buffered: { body?: unknown; file?: UploadedFile } | undefined;+ let inPlace: { body?: unknown; file?: UploadedFile } | undefined;+ router.post("/upload", (r) => { buffered = r; return {}; });+ router.register("POST", "/import", (r) => { inPlace = r; return {}; }, { readsUploadsInPlace: true });++ await router.dispatch("POST", "http://localhost/import", body);+ expect(inPlace?.body).toEqual({ taskId: "t1" });+ expect(inPlace?.file).toEqual({ originalname: "notes.zip", mimetype: "application/zip", blob: file });+ expect(arrayBuffer).not.toHaveBeenCalled();++ await router.dispatch("POST", "http://localhost/upload", body);+ expect(buffered?.file?.buffer && Array.from(buffered.file.buffer)).toEqual([ 4, 5, 6 ]);+ });+it("warns when multipart parsing fails", async () => {const warn = vi.spyOn(console, "warn").mockImplementation(() => {});const router = new BrowserRouter();diff --git a/apps/standalone/src/lightweight/browser_router.ts b/apps/standalone/src/lightweight/browser_router.tsindex 232fec1314..3636518dbd 100644--- a/apps/standalone/src/lightweight/browser_router.ts+++ b/apps/standalone/src/lightweight/browser_router.ts@@ -8,7 +8,28 @@ import { routes } from "@triliumnext/core";export interface UploadedFile {originalname: string;mimetype: string;- buffer: Uint8Array;+ /** Unset for a {@link LocalUploadBody} file on a route that reads uploads in place. */+ buffer?: Uint8Array;+ /** The file the page posted, for a {@link LocalUploadBody}. */+ blob?: Blob;+}++/**+ * A multipart upload the page posts as its fields and the picked `File` itself. Structured cloning+ * passes the `File` by reference, so no bytes are read until a route asks for them.+ */+export interface LocalUploadBody {+ type: "upload";+ fields: Record<string, string>;+ file?: File;+}++export interface RouteOptions {+ /**+ * Whether the route reads an upload from `UploadedFile.blob` itself, as the import routes do.+ * Every other route gets the upload's bytes in `buffer`.+ */+ readsUploadsInPlace?: boolean;}export interface BrowserRequest {@@ -35,6 +56,7 @@ interface Route {pattern: RegExp;paramNames: string[];handler: RouteHandler;+ options: RouteOptions;}/**@@ -75,6 +97,10 @@ function pathToRegex(path: string): { pattern: RegExp; paramNames: string[] } {};}+function isLocalUploadBody(body: unknown): body is LocalUploadBody {+ return typeof body === "object" && body !== null && (body as LocalUploadBody).type === "upload";+}+/*** Parse query string into an object.*/@@ -138,13 +164,14 @@ export class BrowserRouter {/*** Register a route handler.*/- register(method: string, path: string, handler: RouteHandler): void {+ register(method: string, path: string, handler: RouteHandler, options: RouteOptions = {}): void {const { pattern, paramNames } = pathToRegex(path);this.routes.push({method: method.toUpperCase(),pattern,paramNames,- handler+ handler,+ options});}@@ -183,7 +210,16 @@ export class BrowserRouter {// Parse body based on content-typelet parsedBody = body;let uploadedFile: UploadedFile | undefined;- if (body instanceof ArrayBuffer && headers) {+ if (isLocalUploadBody(body)) {+ parsedBody = body.fields;+ if (body.file) {+ uploadedFile = {+ originalname: body.file.name,+ mimetype: body.file.type || 'application/octet-stream',+ blob: body.file+ };+ }+ } else if (body instanceof ArrayBuffer && headers) {const contentType = headers['content-type'] || headers['Content-Type'] || '';if (contentType.includes('application/json')) {try {@@ -233,6 +269,10 @@ export class BrowserRouter {params[route.paramNames[i]] = decodeURIComponent(match[i + 1]);}+ if (uploadedFile?.blob && !uploadedFile.buffer && !route.options.readsUploadsInPlace) {+ uploadedFile.buffer = new Uint8Array(await uploadedFile.blob.arrayBuffer());+ }+const request: BrowserRequest = {method: upperMethod,url: urlString,diff --git a/apps/standalone/src/lightweight/browser_routes.spec.ts b/apps/standalone/src/lightweight/browser_routes.spec.tsindex a346ec6b1a..26c8a3cfd3 100644--- a/apps/standalone/src/lightweight/browser_routes.spec.ts+++ b/apps/standalone/src/lightweight/browser_routes.spec.ts@@ -157,6 +157,7 @@ describe("registerRoutes (real wiring)", () => {describe("route wrapper branches (via controlled handlers)", () => {type RouteCtx = Parameters<typeof routes.buildSharedApiRoutes>[0];let ctx: RouteCtx;+ const receivedFiles: { import?: unknown; upload?: unknown } = {};function buildRouter(): BrowserRouter {vi.spyOn(routes, "buildSharedApiRoutes").mockImplementation((received: RouteCtx) => {@@ -184,6 +185,13 @@ describe("route wrapper branches (via controlled handlers)", () => {asyncRoute("get", "/t/async-res", [], async (_req: unknown, res: MockRes) => { res.send("async-body"); });asyncRoute("get", "/t/async-noresult", [], async () => ({ done: true }));asyncRouteWithoutTransaction("get", "/t/async-no-tx", [], async () => ({ bare: true }), apiResultHandler);++ asyncRoute("post", "/t/import", [ received.importMiddlewareWithErrorHandling ], async (req: { file?: unknown }) => {+ receivedFiles.import = req.file;+ }, apiResultHandler);+ route("post", "/t/upload", [ received.uploadMiddlewareWithErrorHandling ], (req: { file?: unknown }) => {+ receivedFiles.upload = req.file;+ }, apiResultHandler);});return createConfiguredRouter();}@@ -269,6 +277,18 @@ describe("route wrapper branches (via controlled handlers)", () => {expect(parseJson((await router.dispatch("GET", "http://localhost/api/system-checks")).body)).toEqual({ isCpuArchMismatch: false });});+ it("leaves a posted File unread for the import routes and reads it for the others", async () => {+ const router = buildRouter();+ const file = new File([ "abc" ], "vault.zip", { type: "application/zip" });+ const body = { type: "upload", fields: {}, file };++ await router.dispatch("POST", "http://localhost/t/import", body);+ await router.dispatch("POST", "http://localhost/t/upload", body);++ expect(receivedFiles.import).toEqual({ originalname: "vault.zip", mimetype: "application/zip", blob: file });+ expect(receivedFiles.upload).toMatchObject({ buffer: new TextEncoder().encode("abc") });+ });+it("provides no-op middleware and an init guard", async () => {buildRouter();// No-op middleware stubs do nothing and never throw.diff --git a/apps/standalone/src/lightweight/browser_routes.ts b/apps/standalone/src/lightweight/browser_routes.tsindex 4ef158b6b4..49d1bb6712 100644--- a/apps/standalone/src/lightweight/browser_routes.ts+++ b/apps/standalone/src/lightweight/browser_routes.ts@@ -10,7 +10,7 @@ import type { ShareReply } from '@triliumnext/core/src/share/handlers.js';import { SHARE_ROUTE_PATHS, type ShareRoutePath } from '@triliumnext/core/src/share/route_paths.js';import packageJson from '../../package.json' with { type: 'json' };-import { type BrowserRequest, BrowserRouter } from './browser_router';+import { type BrowserRequest, BrowserRouter, type RouteOptions } from './browser_router';import { dbLock } from './db_lock';/** Minimal response object used by apiResultHandler to capture the processed result. */@@ -41,7 +41,9 @@ function toCoreRequest(req: BrowserRequest): Request {body: req.body,headers,method: req.method,- file: req.file,+ // A file posted for an import route has no `buffer` until an importer that needs one calls+ // `loadUploadedBytes()`, as on the server when multer leaves a zip on disk.+ file: req.file as Request["file"],get originalUrl() { return req.url; },// `fetch` lower-cases the header names it sends, which is the form BrowserRouter stores them in.get: (name: string) => headers[name.toLowerCase()]@@ -99,7 +101,7 @@ function createApiRoute(router: BrowserRouter, transactional: boolean) {* - The resultHandler is applied to post-process the result (entity conversion, status codes).*/function createRoute(router: BrowserRouter) {- return (method: HttpMethod, path: string, _middleware: any[], handler: (req: Request, res: Response) => unknown, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {+ return (method: HttpMethod, path: string, middleware: any[], handler: (req: Request, res: Response) => unknown, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {router.register(method, path, (req: BrowserRequest) => {return dbLock.runShared(() => getContext().init(() => {setContextFromHeaders(req);@@ -122,7 +124,7 @@ function createRoute(router: BrowserRouter) {return result;}));- });+ }, routeOptions(middleware));};}@@ -133,7 +135,7 @@ function createRoute(router: BrowserRouter) {* passed an async callback.*/function createAsyncRoute(router: BrowserRouter, { transactional = true } = {}) {- return (method: HttpMethod, path: string, _middleware: any[], handler: (req: Request, res: Response) => Promise<unknown>, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {+ return (method: HttpMethod, path: string, middleware: any[], handler: (req: Request, res: Response) => Promise<unknown>, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {router.register(method, path, (req: BrowserRequest) => {// Exclusive: this transaction stays open across awaits, so no other// route may touch the connection until it commits. See db_lock.ts.@@ -159,7 +161,7 @@ function createAsyncRoute(router: BrowserRouter, { transactional = true } = {})return result;}) as Promise<unknown>);- });+ }, routeOptions(middleware));};}@@ -282,6 +284,18 @@ function noopMiddleware() {// No-op.}+/**+ * Marks a route whose importer reads a posted `File` in place, which `routeOptions()` turns into+ * `readsUploadsInPlace`. The server's counterpart keeps a zip import on disk for the same reason.+ */+function importUploadMiddleware() {+ // No-op.+}++function routeOptions(middleware: unknown[]): RouteOptions {+ return { readsUploadsInPlace: middleware.includes(importUploadMiddleware) };+}+function checkAppNotInitialized() {if (sql_init.isDbInitialized()) {throw new Error("App already initialized.");@@ -325,9 +339,7 @@ export function registerRoutes(router: BrowserRouter): void {checkCredentials: noopMiddleware,loginRateLimiter: noopMiddleware,uploadMiddlewareWithErrorHandling: noopMiddleware,- // The browser variant has no disk; uploads already arrive as a buffer (see browser_router.ts), so- // imports use the same no-op middleware rather than the server's disk-storage one.- importMiddlewareWithErrorHandling: noopMiddleware,+ importMiddlewareWithErrorHandling: importUploadMiddleware,csrfMiddleware: noopMiddleware});apiRoute('get', '/bootstrap', bootstrapRoute);diff --git a/apps/standalone/src/lightweight/zip_provider.spec.ts b/apps/standalone/src/lightweight/zip_provider.spec.tsindex afcb7f964b..21013ee793 100644--- a/apps/standalone/src/lightweight/zip_provider.spec.ts+++ b/apps/standalone/src/lightweight/zip_provider.spec.ts@@ -1,6 +1,6 @@-import type { ZipEntry } from "@triliumnext/core/src/services/zip_provider.js";+import type { ZipEntry, ZipSource } from "@triliumnext/core/src/services/zip_provider.js";import { strToU8, unzipSync, zipSync } from "fflate";-import { describe, expect, it } from "vitest";+import { describe, expect, it, vi } from "vitest";import BrowserZipProvider from "./zip_provider.js";@@ -14,9 +14,9 @@ function makeZip(files: Record<string, string>): Uint8Array {return zipSync(entries);}-async function readAll(buffer: Uint8Array, encoding?: string): Promise<Record<string, string>> {+async function readAll(source: ZipSource, encoding?: string): Promise<Record<string, string>> {const out: Record<string, string> = {};- await provider.readZipFile(buffer, async (entry: ZipEntry, readContent) => {+ await provider.readZipFile(source, async (entry: ZipEntry, readContent) => {out[entry.fileName] = new TextDecoder().decode(await readContent());}, encoding);return out;@@ -149,10 +149,8 @@ describe("BrowserZipArchive store + backpressure", () => {});describe("BrowserZipProvider path sources are unsupported", () => {- it("readZipFile throws for a { path } source", () => {- // readZipFile isn't async, so the guard throws synchronously before the work promise is created;- // real callers await it inside an async function, where the throw surfaces as a rejection anyway.- expect(() => provider.readZipFile({ path: "/tmp/x.zip" }, async () => {})).toThrow(+ it("readZipFile rejects a { path } source", async () => {+ await expect(provider.readZipFile({ path: "/tmp/x.zip" }, async () => {})).rejects.toThrow("Path-based zip reading is not supported in the browser");});@@ -287,3 +285,163 @@ describe("BrowserZipProvider filename re-decoding", () => {expect(names).toEqual(["þ.txt"]);});});++describe("BrowserZipProvider Blob sources", () => {+ it("reads the same entries and names from a Blob as from the bytes", async () => {+ const zip = makeLegacyZip([+ { placeholder: "AB.txt", nameBytes: GBK_NAME_BYTES },+ { placeholder: "Café.txt" }+ ]);+ const blobSource = { blob: new Blob([ zip ]) };++ expect(await provider.detectFilenameEncoding(blobSource)).toBe("gbk");+ expect(await readAll(blobSource, "gbk")).toEqual(await readAll(zip, "gbk"));+ });++ it("reads one byte range at a time, never the whole archive", async () => {+ const entries: Record<string, string> = {};+ for (let i = 0; i < 8; i++) {+ entries[`note-${i}.txt`] = incompressibleText(40_000, i + 1);+ }+ const zip = makeZip(entries);+ const blob = new Blob([ zip ]);+ const slice = vi.spyOn(blob, "slice");++ expect(await readAll({ blob })).toEqual(entries);+ // Bigger than the tail the reader fetches to find the end-of-central-directory record.+ expect(blob.size).toBeGreaterThan(2 * 0x10000);++ expect(slice).toHaveBeenCalled();+ for (const [ start = 0, end = blob.size ] of slice.mock.calls) {+ expect(end - start).toBeLessThan(blob.size / 2);+ }+ });++ it("reads nothing past the central directory until an entry's content is asked for", async () => {+ const zip = makeZip({ "a.txt": "a".repeat(50_000), "b.txt": "b" });+ const blob = new Blob([ zip ]);+ const slice = vi.spyOn(blob, "slice");++ const names: string[] = [];+ await provider.readZipFile({ blob }, async (entry) => { names.push(entry.fileName); });++ expect(names).toEqual([ "a.txt", "b.txt" ]);+ // The tail holding the end-of-central-directory record, then the directory itself.+ expect(slice).toHaveBeenCalledTimes(2);+ });+});++describe("BrowserZipProvider entry formats", () => {+ it("reads a stored (uncompressed) entry", async () => {+ const zip = zipSync({ "plain.txt": [ strToU8("as is"), { level: 0 } ] });+ expect(await readAll(zip)).toEqual({ "plain.txt": "as is" });+ });++ it("reads a ZIP64 archive, whose sizes and offsets are in 64-bit records", async () => {+ expect(await readAll(makeZip64("big.txt", "sixty-four"))).toEqual({ "big.txt": "sixty-four" });+ expect(await readAll({ blob: new Blob([ makeZip64("big.txt", "sixty-four") ]) })).toEqual({ "big.txt": "sixty-four" });+ });++ it("rejects an entry it cannot read, but only when its content is asked for", async () => {+ const unsupported = patchCentralHeader(makeZip({ "a.txt": "x" }), (view, at) => view.setUint16(at + 10, 12, true));+ const encrypted = patchCentralHeader(makeZip({ "a.txt": "x" }), (view, at) => view.setUint16(at + 8, view.getUint16(at + 8, true) | 1, true));++ expect(await readNames(unsupported)).toEqual([ "a.txt" ]);+ await expect(readAll(unsupported)).rejects.toThrow("Unsupported ZIP compression method 12.");+ await expect(readAll(encrypted)).rejects.toThrow("Encrypted ZIP entries are not supported.");+ });+});++/** Letters from a fixed-seed generator, which deflate cannot shrink much. */+function incompressibleText(length: number, seed: number): string {+ let state = seed;+ let text = "";+ for (let i = 0; i < length; i++) {+ state ^= state << 13;+ state ^= state >>> 17;+ state ^= state << 5;+ state >>>= 0;+ text += String.fromCharCode(97 + state % 26);+ }+ return text;+}++/** Applies `patch` to the first central directory header in `zip`, at the header's offset. */+function patchCentralHeader(zip: Uint8Array, patch: (view: DataView, offset: number) => void): Uint8Array {+ const view = new DataView(zip.buffer, zip.byteOffset, zip.byteLength);+ for (let i = 0; i + 4 <= zip.length; i++) {+ if (view.getUint32(i, true) === 0x02014b50) {+ patch(view, i);+ return zip;+ }+ }+ throw new Error("No central directory header found.");+}++/**+ * Builds a one-entry stored archive in ZIP64 form: every size and offset in the classic records is+ * saturated, and the real values sit in the entry's ZIP64 extra field and the ZIP64 end record.+ */+function makeZip64(name: string, content: string): Uint8Array {+ const nameBytes = strToU8(name);+ const data = strToU8(content);+ const SATURATED = 0xffffffff;++ const local = new DataView(new ArrayBuffer(30));+ local.setUint32(0, 0x04034b50, true);+ local.setUint16(4, 45, true);+ local.setUint32(18, SATURATED, true);+ local.setUint32(22, SATURATED, true);+ local.setUint16(26, nameBytes.length, true);++ const central = new DataView(new ArrayBuffer(46));+ central.setUint32(0, 0x02014b50, true);+ central.setUint16(4, 45, true);+ central.setUint16(6, 45, true);+ central.setUint32(20, SATURATED, true);+ central.setUint32(24, SATURATED, true);+ central.setUint16(28, nameBytes.length, true);+ central.setUint16(30, 28, true);+ central.setUint32(42, SATURATED, true);++ const extra = new DataView(new ArrayBuffer(28));+ extra.setUint16(0, 0x0001, true);+ extra.setUint16(2, 24, true);+ extra.setBigUint64(4, BigInt(data.length), true);+ extra.setBigUint64(12, BigInt(data.length), true);+ extra.setBigUint64(20, 0n, true);++ const directoryOffset = 30 + nameBytes.length + data.length;+ const directorySize = 46 + nameBytes.length + 28;+ const zip64EndOffset = directoryOffset + directorySize;++ const zip64End = new DataView(new ArrayBuffer(56));+ zip64End.setUint32(0, 0x06064b50, true);+ zip64End.setBigUint64(4, 44n, true);+ zip64End.setBigUint64(24, 1n, true);+ zip64End.setBigUint64(32, 1n, true);+ zip64End.setBigUint64(40, BigInt(directorySize), true);+ zip64End.setBigUint64(48, BigInt(directoryOffset), true);++ const locator = new DataView(new ArrayBuffer(20));+ locator.setUint32(0, 0x07064b50, true);+ locator.setBigUint64(8, BigInt(zip64EndOffset), true);+ locator.setUint32(16, 1, true);++ const end = new DataView(new ArrayBuffer(22));+ end.setUint32(0, 0x06054b50, true);+ end.setUint16(8, 0xffff, true);+ end.setUint16(10, 0xffff, true);+ end.setUint32(12, SATURATED, true);+ end.setUint32(16, SATURATED, true);++ const parts = [ local, nameBytes, data, central, nameBytes, extra, zip64End, locator, end ]+ .map((part) => part instanceof DataView ? new Uint8Array(part.buffer) : part);+ const zip = new Uint8Array(parts.reduce((total, part) => total + part.length, 0));+ let offset = 0;+ for (const part of parts) {+ zip.set(part, offset);+ offset += part.length;+ }+ return zip;+}diff --git a/apps/standalone/src/lightweight/zip_provider.ts b/apps/standalone/src/lightweight/zip_provider.tsindex 98738480aa..cc6efbc71f 100644--- a/apps/standalone/src/lightweight/zip_provider.ts+++ b/apps/standalone/src/lightweight/zip_provider.ts@@ -1,13 +1,5 @@import type { FileStream, ZipArchive, ZipEntry, ZipProvider, ZipSource } from "@triliumnext/core/src/services/zip_provider.js";-import { strToU8, unzip, zipSync } from "fflate";--/** The browser/WASM build has no filesystem, so a `path` source is unsupported here — only raw bytes. */-function requireBuffer(source: ZipSource): Uint8Array {- if (!(source instanceof Uint8Array)) {- throw new Error("Path-based zip reading is not supported in the browser; uploads arrive as bytes.");- }- return source;-}+import { inflateSync, strToU8, zipSync } from "fflate";type ZipOutput = {send?: (body: unknown) => unknown;@@ -63,14 +55,32 @@ class BrowserZipArchive implements ZipArchive {export default class BrowserZipProvider implements ZipProvider {async detectFilenameEncoding(source: ZipSource): Promise<string> {- const buffer = requireBuffer(source);- const rawSamples = collectRawFilenameSamples(buffer);+ const entries = await readCentralDirectory(toByteSource(source));+ const rawSamples = collectRawFilenameSamples(entries);if (rawSamples.length === 0) {return "utf-8";}return detectEncodingFromBytes(rawSamples);}+ /**+ * Reads one entry at a time: the central directory first, then each entry's bytes by offset when+ * `readContent()` asks for them. A `Blob` source is therefore never loaded whole.+ */+ async readZipFile(+ source: ZipSource,+ processEntry: (entry: ZipEntry, readContent: () => Promise<Uint8Array>) => Promise<void>,+ filenameEncoding?: string+ ): Promise<void> {+ const byteSource = toByteSource(source);+ for (const record of await readCentralDirectory(byteSource)) {+ await processEntry(+ { fileName: decodeZipFileName(record, filenameEncoding) },+ () => readEntryContent(byteSource, record)+ );+ }+ }+createZipArchive(): ZipArchive {return new BrowserZipArchive();}@@ -78,131 +88,83 @@ export default class BrowserZipProvider implements ZipProvider {createFileStream(_filePath: string): FileStream {throw new Error("File stream creation is not supported in the browser.");}+}- readZipFile(- source: ZipSource,- processEntry: (entry: ZipEntry, readContent: () => Promise<Uint8Array>) => Promise<void>,- filenameEncoding?: string- ): Promise<void> {- const buffer = requireBuffer(source);- return new Promise<void>((res, rej) => {- // Read inside the executor so a malformed archive rejects rather than throwing- // synchronously, matching how an unzip() failure surfaces.- const fileNames = readCentralDirectory(buffer);-- unzip(buffer, async (err, files) => {- if (err) { rej(err); return; }+/** Random access to the archive's bytes, whether they are in memory or behind a `Blob`. */+interface ByteSource {+ size: number;+ read(offset: number, length: number): Promise<Uint8Array>;+}- try {- for (const [fileName, data] of Object.entries(files)) {- await processEntry(- { fileName: decodeZipFileName(fileName, fileNames.get(fileName), filenameEncoding) },- () => Promise.resolve(data)- );- }- res();- } catch (e) {- rej(e);- }- });- });+/** The browser/WASM build has no filesystem, so a `path` source is unsupported here. */+function toByteSource(source: ZipSource): ByteSource {+ if (source instanceof Uint8Array) {+ return {+ size: source.byteLength,+ read: (offset, length) => Promise.resolve(source.subarray(offset, offset + length))+ };+ }+ if ("blob" in source) {+ const { blob } = source;+ return {+ size: blob.size,+ read: async (offset, length) => new Uint8Array(await blob.slice(offset, offset + length).arrayBuffer())+ };}+ throw new Error("Path-based zip reading is not supported in the browser; uploads arrive as bytes or a Blob.");}-/** What an entry's name actually is on disk, as opposed to what fflate made of it. */-interface FileNameInfo {+/** One central directory record: everything needed to find, name and inflate an entry. */+interface CentralDirectoryRecord {/** The name exactly as stored, undecoded. */- rawBytes: Uint8Array;+ rawName: Uint8Array;/** Whether the language encoding flag (general purpose bit 11) is set, i.e. the archive declares UTF-8. */isUtf8Flagged: boolean;+ isEncrypted: boolean;+ compressionMethod: number;+ compressedSize: number;+ uncompressedSize: number;+ localHeaderOffset: number;}-/**- * Collects the raw name bytes of every entry whose encoding is genuinely unknown: the archive did not- * flag it as UTF-8 and it does not decode as UTF-8 either.- *- * A flagged entry is excluded even when its bytes look unusual. Its encoding is already declared, so it- * needs no rescuing — and sampling it would let one such name drag every *other* name in the archive- * through the wrong decoder.- */-function collectRawFilenameSamples(buffer: Uint8Array): Uint8Array[] {- const utf8 = new TextDecoder("utf-8", { fatal: true });- const samples: Uint8Array[] = [];-- for (const { rawBytes, isUtf8Flagged } of readCentralDirectory(buffer).values()) {- if (isUtf8Flagged) {- continue;- }- try {- utf8.decode(rawBytes);- } catch {- samples.push(rawBytes);- }- }-- return samples;-}--/**- * Resolves an entry's name from what fflate decoded plus what the archive actually recorded.- *- * fflate hands names back already decoded, choosing UTF-8 or Latin-1 by the language encoding flag —- * which it never exposes. A flagged name is therefore already correct and is passed through untouched;- * re-deriving bytes from it would truncate every character above U+00FF (a curly apostrophe in a note- * title becoming a control character, and the entry then matching nothing in `!!!meta.json`).- *- * Only an unflagged name is ambiguous — many real-world archives write UTF-8, or a legacy codepage,- * without setting the flag — so those are re-decoded from their raw bytes with the detected encoding.- */-function decodeZipFileName(name: string, info: FileNameInfo | undefined, encoding?: string): string {- // A name missing from the central directory (a duplicate entry, say) keeps fflate's reading.- if (!info || info.isUtf8Flagged) {- return name;- }-- try {- return new TextDecoder(encoding || "utf-8", { fatal: true }).decode(info.rawBytes);- } catch {- if (encoding && encoding !== "utf-8") {- // Encoding detection was wrong for this entry, try UTF-8- try {- return new TextDecoder("utf-8", { fatal: true }).decode(info.rawBytes);- } catch {- return name;- }- }- return name;- }-}-+const LOCAL_HEADER_SIGNATURE = 0x04034b50;const CENTRAL_HEADER_SIGNATURE = 0x02014b50;const END_OF_CENTRAL_DIRECTORY_SIGNATURE = 0x06054b50;+const ZIP64_END_OF_CENTRAL_DIRECTORY_SIGNATURE = 0x06064b50;+const ZIP64_LOCATOR_SIGNATURE = 0x07064b50;+const LOCAL_HEADER_LENGTH = 30;const CENTRAL_HEADER_LENGTH = 46;const END_OF_CENTRAL_DIRECTORY_LENGTH = 22;+const ZIP64_LOCATOR_LENGTH = 20;+const ZIP64_END_OF_CENTRAL_DIRECTORY_LENGTH = 56;+/** The longest comment the end-of-central-directory record can carry after itself. */+const MAX_COMMENT_LENGTH = 0xffff;+const ZIP64_EXTRA_FIELD_ID = 0x0001;+/** A 32-bit size or offset that holds this value is in the entry's ZIP64 extra field instead. */+const ZIP64_SATURATED = 0xffffffff;+/** General purpose bit 0: the entry is encrypted. */+const ENCRYPTED_FLAG = 0x1;/** General purpose bit 11: the file name and comment are UTF-8. */const UTF8_FLAG = 0x800;+const METHOD_STORED = 0;+const METHOD_DEFLATED = 8;/**- * Reads the archive's central directory for the two facts fflate discards — each entry's raw name bytes- * and its language encoding flag — keyed by the string fflate produces for that entry, which is the only- * handle the callers have on it.- *- * This also replaces what used to be a second full `unzip()` pass done purely to sample file names: the- * central directory carries them verbatim, so no entry has to be inflated to read one.+ * Reads the end-of-central-directory record (and its ZIP64 counterpart, for an archive past 4 GiB or+ * 65535 entries), then the central directory it points to.*/-function readCentralDirectory(buffer: Uint8Array): Map<string, FileNameInfo> {- const entries = new Map<string, FileNameInfo>();- if (buffer.byteLength < END_OF_CENTRAL_DIRECTORY_LENGTH) {+async function readCentralDirectory(source: ByteSource): Promise<CentralDirectoryRecord[]> {+ if (source.size < END_OF_CENTRAL_DIRECTORY_LENGTH) {throw new Error("Truncated ZIP: no end-of-central-directory record.");}- const view = new DataView(buffer.buffer, buffer.byteOffset, buffer.byteLength);+ const tailStart = Math.max(0, source.size - END_OF_CENTRAL_DIRECTORY_LENGTH - MAX_COMMENT_LENGTH);+ const tail = await source.read(tailStart, source.size - tailStart);+ const tailView = viewOf(tail);- // The end-of-central-directory record is last, but a trailing comment of up to 64 KiB may follow it.let eocd = -1;- const earliest = Math.max(0, buffer.byteLength - END_OF_CENTRAL_DIRECTORY_LENGTH - 0xffff);- for (let i = buffer.byteLength - END_OF_CENTRAL_DIRECTORY_LENGTH; i >= earliest; i--) {- if (view.getUint32(i, true) === END_OF_CENTRAL_DIRECTORY_SIGNATURE) {+ for (let i = tail.byteLength - END_OF_CENTRAL_DIRECTORY_LENGTH; i >= 0; i--) {+ if (tailView.getUint32(i, true) === END_OF_CENTRAL_DIRECTORY_SIGNATURE) {eocd = i;break;}@@ -211,47 +173,166 @@ function readCentralDirectory(buffer: Uint8Array): Map<string, FileNameInfo> {throw new Error("Truncated ZIP: no end-of-central-directory record.");}- // Walk the directory by header rather than by the recorded entry count, which saturates at 65535- // (and is then only recoverable from a ZIP64 record). A ZIP64 offset saturates to 0xFFFFFFFF, which- // lands past `end` and simply yields no entries — every name then keeps fflate's own decoding.- let offset = view.getUint32(eocd + 16, true);- const end = Math.min(buffer.byteLength, offset + view.getUint32(eocd + 12, true));+ let directorySize = tailView.getUint32(eocd + 12, true);+ let directoryOffset = tailView.getUint32(eocd + 16, true);++ const locator = eocd - ZIP64_LOCATOR_LENGTH;+ if (locator >= 0 && tailView.getUint32(locator, true) === ZIP64_LOCATOR_SIGNATURE) {+ const zip64Offset = readUint64(tailView, locator + 8);+ const zip64 = viewOf(await source.read(zip64Offset, ZIP64_END_OF_CENTRAL_DIRECTORY_LENGTH));+ if (zip64.byteLength < ZIP64_END_OF_CENTRAL_DIRECTORY_LENGTH+ || zip64.getUint32(0, true) !== ZIP64_END_OF_CENTRAL_DIRECTORY_SIGNATURE) {+ throw new Error("Corrupt ZIP: the ZIP64 end-of-central-directory record is missing.");+ }+ directorySize = readUint64(zip64, 40);+ directoryOffset = readUint64(zip64, 48);+ }++ if (directoryOffset + directorySize > source.size) {+ throw new Error("Corrupt ZIP: the central directory lies past the end of the file.");+ }++ const directory = await source.read(directoryOffset, directorySize);+ const view = viewOf(directory);+ const records: CentralDirectoryRecord[] = [];- while (offset + CENTRAL_HEADER_LENGTH <= end && view.getUint32(offset, true) === CENTRAL_HEADER_SIGNATURE) {- const isUtf8Flagged = !!(view.getUint16(offset + 8, true) & UTF8_FLAG);+ // Walked by header rather than by the recorded entry count, which saturates at 65535.+ let offset = 0;+ while (offset + CENTRAL_HEADER_LENGTH <= directory.byteLength+ && view.getUint32(offset, true) === CENTRAL_HEADER_SIGNATURE) {+ const flags = view.getUint16(offset + 8, true);const nameLength = view.getUint16(offset + 28, true);const extraLength = view.getUint16(offset + 30, true);const commentLength = view.getUint16(offset + 32, true);const nameStart = offset + CENTRAL_HEADER_LENGTH;+ const extraStart = nameStart + nameLength;+ if (extraStart + extraLength > directory.byteLength) {+ throw new Error("Corrupt ZIP: a central directory record runs past the directory.");+ }- if (nameStart + nameLength > end) {- break;+ const record: CentralDirectoryRecord = {+ rawName: directory.subarray(nameStart, extraStart),+ isUtf8Flagged: !!(flags & UTF8_FLAG),+ isEncrypted: !!(flags & ENCRYPTED_FLAG),+ compressionMethod: view.getUint16(offset + 10, true),+ compressedSize: view.getUint32(offset + 20, true),+ uncompressedSize: view.getUint32(offset + 24, true),+ localHeaderOffset: view.getUint32(offset + 42, true)+ };+ applyZip64Extra(record, view, extraStart, extraLength);+ records.push(record);++ offset = extraStart + extraLength + commentLength;+ }++ return records;+}++/**+ * Replaces the saturated 32-bit fields with the 64-bit values from the ZIP64 extra field, which+ * carries only the saturated ones, in the order uncompressed size, compressed size, local header offset.+ */+function applyZip64Extra(record: CentralDirectoryRecord, view: DataView, start: number, length: number) {+ const end = start + length;+ for (let offset = start; offset + 4 <= end;) {+ const id = view.getUint16(offset, true);+ const size = view.getUint16(offset + 2, true);+ let field = offset + 4;+ if (id === ZIP64_EXTRA_FIELD_ID) {+ for (const key of ["uncompressedSize", "compressedSize", "localHeaderOffset"] as const) {+ if (record[key] === ZIP64_SATURATED && field + 8 <= offset + 4 + size) {+ record[key] = readUint64(view, field);+ field += 8;+ }+ }+ return;}+ offset += 4 + size;+ }+}- const rawBytes = buffer.subarray(nameStart, nameStart + nameLength);- entries.set(decodeAsFflateDoes(rawBytes, isUtf8Flagged), { rawBytes, isUtf8Flagged });+/** Reads an entry's bytes through its local header, which can carry a different extra field length. */+async function readEntryContent(source: ByteSource, record: CentralDirectoryRecord): Promise<Uint8Array> {+ if (record.isEncrypted) {+ throw new Error("Encrypted ZIP entries are not supported.");+ }++ const header = viewOf(await source.read(record.localHeaderOffset, LOCAL_HEADER_LENGTH));+ if (header.byteLength < LOCAL_HEADER_LENGTH || header.getUint32(0, true) !== LOCAL_HEADER_SIGNATURE) {+ throw new Error("Corrupt ZIP: an entry's local header is missing.");+ }+ const dataStart = record.localHeaderOffset + LOCAL_HEADER_LENGTH+ + header.getUint16(26, true) + header.getUint16(28, true);+ const data = await source.read(dataStart, record.compressedSize);+ if (data.byteLength < record.compressedSize) {+ throw new Error("Truncated ZIP: an entry's data runs past the end of the file.");+ }- offset = nameStart + nameLength + extraLength + commentLength;+ switch (record.compressionMethod) {+ case METHOD_STORED:+ return data;+ case METHOD_DEFLATED:+ return inflateSync(data, { out: new Uint8Array(record.uncompressedSize) });+ default:+ throw new Error(`Unsupported ZIP compression method ${record.compressionMethod}.`);}+}++function viewOf(bytes: Uint8Array): DataView {+ return new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);+}- return entries;+/** ZIP64 values fit in a double for any archive a browser can hold. */+function readUint64(view: DataView, offset: number): number {+ return Number(view.getBigUint64(offset, true));}/**- * Reproduces the string fflate hands back for an entry, so a central directory record can be matched to- * it: UTF-8 when the archive flags the name as such, otherwise one character per byte.+ * Collects the raw name bytes of every entry whose encoding is genuinely unknown: the archive did not+ * flag it as UTF-8 and it does not decode as UTF-8 either.*- * The Latin-1 branch cannot use `TextDecoder`, whose "latin1"/"iso-8859-1" labels are aliases of- * windows-1252 and so map 0x80-0x9F to different code points than the plain byte-to-char widening- * fflate performs.+ * A flagged entry is excluded even when its bytes look unusual. Its encoding is already declared, so it+ * needs no rescuing, and sampling it would let one such name drag every other name in the archive+ * through the wrong decoder.+ */+function collectRawFilenameSamples(records: CentralDirectoryRecord[]): Uint8Array[] {+ const utf8 = new TextDecoder("utf-8", { fatal: true });+ const samples: Uint8Array[] = [];++ for (const { rawName, isUtf8Flagged } of records) {+ if (isUtf8Flagged) {+ continue;+ }+ try {+ utf8.decode(rawName);+ } catch {+ samples.push(rawName);+ }+ }++ return samples;+}++/**+ * Decodes an entry's name. A UTF-8-flagged name is decoded as UTF-8. An unflagged name is ambiguous,+ * since many archivers write UTF-8 or a legacy codepage without setting the flag, so it is decoded with+ * the detected encoding, then UTF-8, and failing both one character per byte.*/-function decodeAsFflateDoes(rawBytes: Uint8Array, isUtf8Flagged: boolean): string {- if (isUtf8Flagged) {- return new TextDecoder("utf-8").decode(rawBytes);+function decodeZipFileName(record: CentralDirectoryRecord, encoding?: string): string {+ if (record.isUtf8Flagged) {+ return new TextDecoder("utf-8").decode(record.rawName);+ }++ for (const label of encoding && encoding !== "utf-8" ? [encoding, "utf-8"] : ["utf-8"]) {+ try {+ return new TextDecoder(label, { fatal: true }).decode(record.rawName);+ } catch {+ // An unknown label or bytes invalid in it: try the next one.+ }}let name = "";- for (const byte of rawBytes) {+ for (const byte of record.rawName) {name += String.fromCharCode(byte);}return name;diff --git a/apps/standalone/src/local-bridge.spec.ts b/apps/standalone/src/local-bridge.spec.tsindex 103e12d7b3..fa41fe9c91 100644--- a/apps/standalone/src/local-bridge.spec.ts+++ b/apps/standalone/src/local-bridge.spec.ts@@ -397,6 +397,39 @@ describe("pending request rejection", () => {});});+describe("localUpload", () => {+ it("posts the fields and the File itself, transferring nothing, and resolves with the reply", async () => {+ const bridge = await freshBridge();+ const file = new File([ "zip" ], "notes.zip");+ const promise = bridge.localUpload("api/notes/root/notes-import", {+ method: "POST",+ headers: { "x-csrf-token": "t" },+ fields: { taskId: "t1" },+ file+ });++ const worker = lastWorker();+ await vi.waitFor(() => expect(worker.postMessage).toHaveBeenCalledWith(+ expect.objectContaining({+ type: "LOCAL_REQUEST",+ request: {+ url: new URL("api/notes/root/notes-import", location.href).href,+ method: "POST",+ headers: { "x-csrf-token": "t" },+ body: { type: "upload", fields: { taskId: "t1" }, file }+ }+ }),+ []+ ));+ // The File goes by reference: the posted body holds the very same object.+ const posted = worker.postMessage.mock.calls.at(-1)?.[0] as { id: string; request: { body: { file: File } } };+ expect(posted.request.body.file).toBe(file);++ worker.onmessage?.({ data: { type: "LOCAL_RESPONSE", id: posted.id, response: { status: 200, headers: {} } } });+ expect((await promise).status).toBe(200);+ });+});+describe("localFetch", () => {it("posts a GET LOCAL_REQUEST with no body transfer and builds a Response from the reply", async () => {const bridge = await freshBridge();diff --git a/apps/standalone/src/local-bridge.ts b/apps/standalone/src/local-bridge.tsindex 4397be714d..97dfc2acc9 100644--- a/apps/standalone/src/local-bridge.ts+++ b/apps/standalone/src/local-bridge.ts@@ -1,7 +1,8 @@import type {StandaloneDownloadResult,StandaloneRestoreProgress,- StandaloneRestoreResult+ StandaloneRestoreResult,+ StandaloneUploadInit} from "@triliumnext/commons";import { reportSplashPhase } from "../../client/src/services/splash.js";@@ -388,7 +389,6 @@ function isShareRequest(pathname: string): boolean {export async function localFetch(request: Request): Promise<Response> {startLocalServerWorker();- const id = Math.random().toString(36).slice(2);const headersObj: Record<string, string> = {};for (const [k, v] of request.headers.entries()) headersObj[k] = v;@@ -396,13 +396,33 @@ export async function localFetch(request: Request): Promise<Response> {? null: await request.arrayBuffer();+ return requestFromWorker({ url: request.url, method: request.method, headers: headersObj, body }, body ? [body] : []);+}++/**+ * Sends a multipart upload to the worker as its fields and the picked `File` itself, which+ * structured cloning passes by reference. Unlike `localFetch()`, nothing reads the file on the page,+ * and an importer can read a zip from it one entry at a time.+ */+export function localUpload(url: string, init: StandaloneUploadInit): Promise<Response> {+ return requestFromWorker({+ url: new URL(url, location.href).href,+ method: init.method,+ headers: init.headers ?? {},+ body: { type: "upload", fields: init.fields, file: init.file }+ }, []);+}++async function requestFromWorker(+ request: { url: string; method: string; headers: Record<string, string>; body: unknown },+ transfer: Transferable[]+): Promise<Response> {+ const worker = startLocalServerWorker();++ const id = Math.random().toString(36).slice(2);const response = await new Promise<{ status: number; headers: Record<string, string>; body?: ArrayBuffer }>((resolve, reject) => {pending.set(id, { resolve, reject });- localWorker!.postMessage({- type: "LOCAL_REQUEST",- id,- request: { url: request.url, method: request.method, headers: headersObj, body }- }, body ? [body] : []);+ worker.postMessage({ type: "LOCAL_REQUEST", id, request }, transfer);});const respHeaders = new Headers();diff --git a/apps/standalone/src/main.spec.ts b/apps/standalone/src/main.spec.tsindex 21b141b662..1aaa8348d6 100644--- a/apps/standalone/src/main.spec.ts+++ b/apps/standalone/src/main.spec.ts@@ -8,6 +8,7 @@ const mocks = vi.hoisted(() => ({downloadDatabase: vi.fn(),announceLeadership: vi.fn(),localFetch: vi.fn(),+ localUpload: vi.fn(),capacitorHttpHandler: vi.fn(),saveUrlToDevice: vi.fn(),requestSecurityChange: vi.fn(),@@ -26,6 +27,7 @@ vi.mock("./local-bridge.js", () => ({downloadDatabase: mocks.downloadDatabase,announceLeadership: mocks.announceLeadership,localFetch: mocks.localFetch,+ localUpload: mocks.localUpload,requestSecurityChange: mocks.requestSecurityChange}));vi.mock("./leader_election.js", () => ({@@ -104,6 +106,7 @@ describe("bootstrap", () => {await vi.waitFor(() => expect(mocks.announceLeadership).toHaveBeenCalled());// The leader also answers the client's API calls directly, skipping that route.expect(window.standaloneApi?.localFetch).toBe(mocks.localFetch);+ expect(window.standaloneApi?.localUpload).toBe(mocks.localUpload);// And its worker is the one holding the settings file, so it is the tab that can change// what this instance is allowed to run.expect(window.standaloneApi?.security).toBeDefined();@@ -123,6 +126,7 @@ describe("bootstrap", () => {// A follower must keep calling through the service worker: with no worker of its own,// a localFetch here would answer from nothing.expect(window.standaloneApi?.localFetch).toBeUndefined();+ expect(window.standaloneApi?.localUpload).toBeUndefined();// Nor can it change a security setting: the file that holds them is locked by the// leader's worker, which this tab has no way to reach.diff --git a/apps/standalone/src/main.ts b/apps/standalone/src/main.tsindex 8c2637f89a..b464dba062 100644--- a/apps/standalone/src/main.ts+++ b/apps/standalone/src/main.ts@@ -6,7 +6,7 @@ import {import { showErrorOverlay } from "./error-overlay.js";import { installIosInterceptors } from "./ios-interceptors.js";import { claimLeadership } from "./leader_election.js";-import { announceLeadership, attachServiceWorkerBridge, downloadDatabase, localFetch, registerNativeHttpHandler, restoreBackup, saveDatabase, startLocalServerWorker } from "./local-bridge.js";+import { announceLeadership, attachServiceWorkerBridge, downloadDatabase, localFetch, localUpload, registerNativeHttpHandler, restoreBackup, saveDatabase, startLocalServerWorker } from "./local-bridge.js";import { createSecurityApi } from "./security_gate.js";/**@@ -121,6 +121,7 @@ async function bootstrap() {// The leader answers API requests from its own worker, so the client's server.ts// can skip the service-worker round trip.standaloneApi.localFetch = localFetch;+ standaloneApi.localUpload = localUpload;// Its worker is also the one holding the lock on the security settings file, so it is// the only tab that can change what this instance is allowed to run. See security_gate.ts.standaloneApi.security = createSecurityApi();diff --git a/packages/commons/src/lib/standalone_api_interface.ts b/packages/commons/src/lib/standalone_api_interface.tsindex abdae4e854..76b622282b 100644--- a/packages/commons/src/lib/standalone_api_interface.ts+++ b/packages/commons/src/lib/standalone_api_interface.ts@@ -150,4 +150,18 @@ export interface StandaloneApi {* would pay the XHR → service worker → page round trip to end up in the same worker.*/localFetch?(request: Request): Promise<Response>;+ /**+ * Sends a multipart upload to the same worker as `localFetch`, passing the picked `File` by+ * reference instead of reading it into the request body. Set alongside `localFetch`.+ */+ localUpload?(url: string, init: StandaloneUploadInit): Promise<Response>;+}++/** A multipart upload for {@link StandaloneApi.localUpload}: the form's text fields and its one file. */+export interface StandaloneUploadInit {+ method: string;+ headers?: Record<string, string>;+ fields: Record<string, string>;+ /** Sent as the form's `upload` field, as multer expects on the server. */+ file?: File;}diff --git a/packages/trilium-core/src/routes/api/import.ts b/packages/trilium-core/src/routes/api/import.tsindex cf84bb635b..d0c072980a 100644--- a/packages/trilium-core/src/routes/api/import.ts+++ b/packages/trilium-core/src/routes/api/import.ts@@ -1,6 +1,6 @@import type { Request } from "../../http_interface";-import type { File } from "../../services/import/common.js";+import { type File, loadUploadedBytes } from "../../services/import/common.js";type ImportRequest<P> = Omit<Request<P>, "file"> & { file?: File };@@ -109,6 +109,7 @@ async function importAttachmentsToNote(req: ImportRequest<{ parentNoteId: string// unlike in note import, we let the events run, because a huge number of attachments is not likelytry {+ await loadUploadedBytes(file);singleImportService.importAttachment(taskContext, file, parentNote);// An imported image is stored once its compression answers, which is after the call above// has returned. Waited out for the same reason the note import waits: what is reported asdiff --git a/packages/trilium-core/src/services/import/common.ts b/packages/trilium-core/src/services/import/common.tsindex 2d37ef6ae0..f147c5f5e9 100644--- a/packages/trilium-core/src/services/import/common.ts+++ b/packages/trilium-core/src/services/import/common.ts@@ -1,11 +1,23 @@export interface File {originalname: string;mimetype: string;+ /**+ * The file's bytes. Unset for a zip import that arrives as `path` or `blob`; call+ * {@link loadUploadedBytes} before reading it on any path that does not read the zip in place.+ */buffer: string | Buffer | Uint8Array;/*** Filesystem path of the uploaded file when it was streamed to disk (server multer disk storage).- * Lets a zip be read in place instead of buffering it; absent for the browser/WASM upload, which- * only has `buffer`.+ * Lets a zip be read in place instead of buffering it.*/path?: string;+ /** The file as the page picked it (browser/WASM), which lets a zip be read in place. */+ blob?: Blob;+}++/** Reads a `blob`-backed upload into `buffer`, for the importers that need the whole file. */+export async function loadUploadedBytes(file: File): Promise<void> {+ if (file.buffer === undefined && file.blob) {+ file.buffer = new Uint8Array(await file.blob.arrayBuffer());+ }}diff --git a/packages/trilium-core/src/services/import/dispatch.spec.ts b/packages/trilium-core/src/services/import/dispatch.spec.tsindex 9f20b901f8..33ccbff161 100644--- a/packages/trilium-core/src/services/import/dispatch.spec.ts+++ b/packages/trilium-core/src/services/import/dispatch.spec.ts@@ -168,4 +168,38 @@ describe("importFile (dispatch)", () => {expect(stubbed.importZip).toHaveBeenCalledWith(taskContext, { path: "/tmp/v.zip" }, parentNote);});});++ describe("a Blob upload (standalone, where the page posts the picked File)", () => {+ it("hands a zip importer the Blob without reading it", async () => {+ const blob = new Blob([ "zip bytes" ]);+ const arrayBuffer = vi.spyOn(blob, "arrayBuffer");+ const file = { originalname: "notes.zip", mimetype: "application/zip", blob } as File;++ await importFile(taskContext, file, parentNote, makeOptions(), "notion");++ expect(stubbed.importNotion).toHaveBeenCalledWith(taskContext, { blob }, parentNote);+ expect(arrayBuffer).not.toHaveBeenCalled();+ expect(file.buffer).toBeUndefined();+ });++ it("reads the Blob into the buffer for an importer that needs the whole file", async () => {+ const file = { originalname: "page.md", mimetype: "text/markdown", blob: new Blob([ "# Hi" ]) } as File;++ await importFile(taskContext, file, parentNote, makeOptions());++ expect(stubbed.importSingleFile).toHaveBeenCalledWith(taskContext, file, parentNote);+ expect(new TextDecoder().decode(file.buffer as Uint8Array)).toBe("# Hi");+ });++ it("reads the Blob for a zip the user chose not to explode", async () => {+ const file = { originalname: "keep.zip", mimetype: "application/zip", blob: new Blob([ "PK" ]) } as File;++ await importFile(taskContext, file, parentNote, makeOptions({ explodeArchives: false }));++ expect(stubbed.importZip).not.toHaveBeenCalled();+ expect(stubbed.importSingleFile).toHaveBeenCalledWith(taskContext, file, parentNote);+ expect(file.buffer).toBeInstanceOf(Uint8Array);+ });+ });});+diff --git a/packages/trilium-core/src/services/import/dispatch.ts b/packages/trilium-core/src/services/import/dispatch.tsindex dc2fafbd69..2a1a2200e2 100644--- a/packages/trilium-core/src/services/import/dispatch.ts+++ b/packages/trilium-core/src/services/import/dispatch.ts@@ -4,7 +4,7 @@ import type TaskContext from "../task_context.js";import { extname } from "../utils/path.js";import type { ZipSource } from "../zip_provider.js";import anytypeImportService from "./anytype/importer.js";-import type { File } from "./common.js";+import { type File, loadUploadedBytes } from "./common.js";import enexImportService from "./enex.js";import keepImportService from "./keep/importer.js";import notionImportService from "./notion/importer.js";@@ -32,8 +32,8 @@ export interface ImportOptions {* importer identically. The caller owns the surrounding CLS setup (`disableEntityEvents`,* `ignoreEntityChangeIds`) and the `becca_loader.load()` afterwards.*- * A `.zip` is read from `file.path` in place (streamed per entry) when present, falling back to- * `file.buffer` for the browser/WASM upload that has no temp file.+ * A `.zip` is read in place from `file.path` or `file.blob` when present, falling back to+ * `file.buffer`.*/export default async function importFile(taskContext: TaskContext<"importNotes">, file: File, parentNote: BNote, options: ImportOptions, format?: string): Promise<BNote | (string | number)[] | null> {const imported = await routeToImporter(taskContext, file, parentNote, options, format);@@ -55,12 +55,13 @@ export default async function importFile(taskContext: TaskContext<"importNotes">async function routeToImporter(taskContext: TaskContext<"importNotes">, file: File, parentNote: BNote, options: ImportOptions, format?: string): Promise<BNote | (string | number)[] | null> {const extension = extname(file.originalname).toLowerCase();- // Every zip-reading importer (generic + the tagged providers) reads the archive straight from the temp- // file (server disk storage / native pick) so a multi-GB zip is streamed per entry — never held in one- // buffer, and free of fs.readFile's ~2 GiB ceiling. Falls back to the buffer when there's no path (the- // browser/WASM upload). The `typeof file.buffer !== "string"` guards below ensure the no-path fallback- // is real bytes, not a string body.- const zipSource: ZipSource = file.path ? { path: file.path } : file.buffer as Uint8Array;+ // Every zip-reading importer reads the archive in place, from the temp file on the server or the+ // picked `Blob` in the browser, so a multi-GB zip is read one entry at a time. The buffer is the+ // fallback for an upload that has neither. The `typeof file.buffer !== "string"` guards below ensure+ // that fallback is real bytes, not a string body.+ const zipSource: ZipSource = file.path ? { path: file.path }+ : file.blob ? { blob: file.blob }+ : file.buffer as Uint8Array;if (format === "notion" && (file.path || typeof file.buffer !== "string")) {// An explicit format always wins over extension sniffing: a Notion export is just a `.zip`,@@ -88,7 +89,10 @@ async function routeToImporter(taskContext: TaskContext<"importNotes">, file: Fi});}return await zipImportService.importZip(taskContext, zipSource, parentNote);- } else if (extension === ".opml" && options.explodeArchives) {+ }++ await loadUploadedBytes(file);+ if (extension === ".opml" && options.explodeArchives) {return await opmlImportService.importOpml(taskContext, file.buffer, parentNote);} else if (extension === ".enex" && options.explodeArchives) {return await enexImportService.importEnex(taskContext, file, parentNote);diff --git a/packages/trilium-core/src/services/zip_provider.ts b/packages/trilium-core/src/services/zip_provider.tsindex b1ecb16bdc..46210dd669 100644--- a/packages/trilium-core/src/services/zip_provider.ts+++ b/packages/trilium-core/src/services/zip_provider.ts@@ -1,9 +1,9 @@/**- * A zip to read: either its raw bytes, or — server/desktop only — a filesystem path the provider opens- * and reads *in place* (no full-buffer copy, no ~2 GiB `fs.readFile` ceiling). The browser/WASM provider- * only supports the byte form.+ * A zip to read: its raw bytes, a filesystem path (server/desktop only) or a `Blob` (browser/WASM+ * only). A path or a `Blob` is read in place, one byte range at a time, so the archive is never held+ * in memory whole.*/-export type ZipSource = Uint8Array | { path: string };+export type ZipSource = Uint8Array | { path: string } | { blob: Blob };export interface ZipEntry {fileName: string;