Skip to content

Build your personal knowledge base with Trilium Notes

feat(standalone): import a zip from the picked File without loading it whole

Elian Doranda034da3c26e
diff --git a/.claude/skills/developing-standalone/SKILL.md b/.claude/skills/developing-standalone/SKILL.md
index afe9b19ab4..49742b3ddb 100644
--- a/.claude/skills/developing-standalone/SKILL.md
+++ b/.claude/skills/developing-standalone/SKILL.md
@@ -32,9 +32,10 @@ apps/standalone/
1. **`main.ts`** requires a service worker (secure context: HTTPS or localhost — otherwise it throws a message explaining that), registers `./sw.js` with scope `/`, reloads once if the SW installed but doesn't control the page yet, and exposes `window.standaloneApi` (`restore.importBackup`, `backup.downloadDatabase`) — the client's escape hatch for things carrying a whole file, the twin of desktop's `window.electronApi`. The leader also publishes `localFetch` there (step 3).
2. **Leader election** (`leader_election.ts`): the OPFS SAHPool VFS takes *exclusive* sync-access handles, and `createSyncAccessHandle` exists only in a dedicated worker, so exactly one tab per origin can own the database. A **Web Lock** (`trilium-standalone-db`, held for the tab's lifetime) picks it; only the winner calls `startLocalServerWorker()`; when it closes the browser releases the lock and a waiting tab is promoted. Where Web Locks are missing (some WebViews) the tab assumes it is alone.
-3. **The leader answers its own API calls.** Winning the lock also publishes `localFetch` (`local-bridge.ts`) on `window.standaloneApi`, and the client prefers it over its XHR transport — in `ajax()` (`apps/client/src/services/server.ts`) and in `setupGlob()`'s `/bootstrap` (`apps/client/src/index.ts`). Without it the leader's own request leaves as an XHR, is intercepted by the service worker, and is posted back into the page it started in, to reach a worker that page owns. `localAjax()` matches the XHR path's contract — headers, JSON body, CSRF refresh-and-retry-once, the `silent*` suppressions, `raw`, and a timeout of `opts.timeoutMs ?? DEFAULT_TIMEOUT`. **Build with `VITE_DISABLE_LOCAL_FETCH=true` to take the service worker's route instead**, which is what compares the two transports. Follower tabs leave `localFetch` unset; so do `upload()`, `chunked_upload.ts` and the LLM stream, which carry `FormData` or want the body as it arrives.
-4. **Service worker** (`sw.ts`): cache-first for hashed assets, network-first otherwise; requests under `LOCAL_API_PREFIXES` (`/bootstrap`, `/api/`, `/sync/`, `/search/`, from `local-bridge.ts`) are forwarded with `LOCAL_FETCH` to the **leader** client, found via a cached id, `WHO_IS_LEADER` / `LEADER_REPLY` probing, and one retry on `NOT_LEADER` (leadership can flip while a request is in flight). Since step 3 this carries follower tabs, engine-initiated loads (`<img src="api/images/…">`, fonts, themes) and the call sites that reach past `server.ts`, rather than the whole API surface. Backup downloads stream over a `MessageChannel` from the leader. It must **let `/_trilium_native_http/` fall through** (Android native proxy) and has a defensive guard for `capacitor://`.
-5. **The worker** (`local-server-worker.ts`): waits for the previous worker's OPFS access handles to be released (`waitForSahPoolRelease` — a reload races the browser's asynchronous release, and `installOpfsSAHPoolVfs()` reacts to an acquisition failure by *deleting the pool directory*, so the pool must be acquirable before it runs — a wait that reaches its deadline ends the boot instead of installing anyway), then installs the SAHPool VFS **once** — failure is a fatal startup error painted by `error-overlay.ts`, never an in-memory fallback (`TRILIUM_INTEGRATION_TEST=memory` builds seed a fixture into the pool, so writes survive reloads within a test). It then calls `initializeCore({...})` with the `lightweight/*` providers, registers `createConfiguredRouter()` (`BrowserRouter` mimicking Express, path params + query strings), then answers `LOCAL_REQUEST` messages. `DbLock` (`lightweight/db_lock.ts`) serialises `createAsyncRoute` handlers (imports) that hold a transaction across `await`s against the single connection — synchronous routes keep a zero-overhead path. `WS_MESSAGE` replaces the WebSocket in both directions. Anything that escapes after startup is reported as `WORKER_ERROR` for one background task; before startup it means the app never came up and `error-overlay.ts` paints it (the async worker init is invisible to `main.ts`'s try/catch).
+3. **The leader answers its own API calls.** Winning the lock also publishes `localFetch` (`local-bridge.ts`) on `window.standaloneApi`, and the client prefers it over its XHR transport — in `ajax()` (`apps/client/src/services/server.ts`) and in `setupGlob()`'s `/bootstrap` (`apps/client/src/index.ts`). Without it the leader's own request leaves as an XHR, is intercepted by the service worker, and is posted back into the page it started in, to reach a worker that page owns. `localAjax()` matches the XHR path's contract — headers, JSON body, CSRF refresh-and-retry-once, the `silent*` suppressions, `raw`, and a timeout of `opts.timeoutMs ?? DEFAULT_TIMEOUT`. **Build with `VITE_DISABLE_LOCAL_FETCH=true` to take the service worker's route instead**, which is what compares the two transports. Follower tabs leave `localFetch` unset. `chunked_upload.ts` and the LLM stream bypass it, since they want the body as it arrives.
+4. **Multipart uploads hand over the `File`, not its bytes.** `upload()` and the import's `uploadFiles()` call `uploadForm()` (`server.ts`), which on the leader uses `standaloneApi.localUpload`: it posts the form's fields and the picked `File` as a `{ type: "upload" }` body, and structured cloning passes the `File` by reference. `BrowserRouter` gives the import routes that `File` as `req.file.blob` (`readsUploadsInPlace`, set by the `importUploadMiddleware` marker in `browser_routes.ts`) and reads it into `req.file.buffer` for every other route. A zip importer then reads `{ blob }` through `lightweight/zip_provider.ts` one byte range at a time (central directory from the tail, then each entry by offset, ZIP64 included), so a multi-gigabyte archive is never in memory whole; any other importer calls core's `loadUploadedBytes()` first. On the server the same zip arrives as `{ path }` from multer's temp file.
+5. **Service worker** (`sw.ts`): cache-first for hashed assets, network-first otherwise; requests under `LOCAL_API_PREFIXES` (`/bootstrap`, `/api/`, `/sync/`, `/search/`, from `local-bridge.ts`) are forwarded with `LOCAL_FETCH` to the **leader** client, found via a cached id, `WHO_IS_LEADER` / `LEADER_REPLY` probing, and one retry on `NOT_LEADER` (leadership can flip while a request is in flight). Since step 3 this carries follower tabs, engine-initiated loads (`<img src="api/images/…">`, fonts, themes) and the call sites that reach past `server.ts`, rather than the whole API surface. Backup downloads stream over a `MessageChannel` from the leader. It must **let `/_trilium_native_http/` fall through** (Android native proxy) and has a defensive guard for `capacitor://`.
+6. **The worker** (`local-server-worker.ts`): waits for the previous worker's OPFS access handles to be released (`waitForSahPoolRelease` — a reload races the browser's asynchronous release, and `installOpfsSAHPoolVfs()` reacts to an acquisition failure by *deleting the pool directory*, so the pool must be acquirable before it runs — a wait that reaches its deadline ends the boot instead of installing anyway), then installs the SAHPool VFS **once** — failure is a fatal startup error painted by `error-overlay.ts`, never an in-memory fallback (`TRILIUM_INTEGRATION_TEST=memory` builds seed a fixture into the pool, so writes survive reloads within a test). It then calls `initializeCore({...})` with the `lightweight/*` providers, registers `createConfiguredRouter()` (`BrowserRouter` mimicking Express, path params + query strings), then answers `LOCAL_REQUEST` messages. `DbLock` (`lightweight/db_lock.ts`) serialises `createAsyncRoute` handlers (imports) that hold a transaction across `await`s against the single connection — synchronous routes keep a zero-overhead path. `WS_MESSAGE` replaces the WebSocket in both directions. Anything that escapes after startup is reported as `WORKER_ERROR` for one background task; before startup it means the app never came up and `error-overlay.ts` paints it (the async worker init is invisible to `main.ts`'s try/catch).
## The provider set (`src/lightweight/`)
@@ -57,7 +58,7 @@ apps/standalone/
| Image | `services/image_provider.ts` | format sniffing only, no resize/compression |
| LLM | `llm_skills.ts` | this build's half of the LLM stack (`registerStandaloneLlmExtensions`), dynamically imported so skill sheets don't join the startup bundle |
-Standalone-only routes (integrity checks, the LLM channel, upload handling that arrives as a buffer instead of a temp file) are registered in `lightweight/browser_routes.ts`; everything else is core's shared route table (`adding-internal-api-route` skill).
+Standalone-only routes (integrity checks, the LLM channel, uploads that arrive as a posted `File` instead of a temp file) are registered in `lightweight/browser_routes.ts`; everything else is core's shared route table (`adding-internal-api-route` skill).
## Bundle discipline (`vite.config.mts`)
diff --git a/apps/client/src/services/import.spec.ts b/apps/client/src/services/import.spec.ts
index 84e35cadf1..c354db0f8d 100644
--- a/apps/client/src/services/import.spec.ts
+++ b/apps/client/src/services/import.spec.ts
@@ -52,78 +52,73 @@ beforeEach(() => {
});
describe("uploadFiles", () => {
+ const fileA = new File([ "a" ], "a.md");
+ const fileB = new File([ "b" ], "b.md");
+ let uploadForm: ReturnType<typeof vi.fn>;
+
beforeEach(() => {
- ($ as any).ajax = vi.fn(async () => ({}));
+ uploadForm = vi.fn(async () => ({}));
+ server.uploadForm = uploadForm as typeof server.uploadForm;
server.getHeaders = vi.fn(async () => ({ "x-h": "1" })) as typeof server.getHeaders;
});
it("throws on an unrecognized entity type", async () => {
- await expect(uploadFiles("bogus", "p1", ["f"], { shrinkImages: false })).rejects.toThrow(
+ await expect(uploadFiles("bogus", "p1", [ fileA ], { shrinkImages: false })).rejects.toThrow(
"Unrecognized import entity type 'bogus'."
);
- expect(($ as any).ajax).not.toHaveBeenCalled();
+ expect(uploadForm).not.toHaveBeenCalled();
});
it("returns early without uploading when there are no files", async () => {
const result = await uploadFiles("notes", "p1", [], { shrinkImages: false });
expect(result).toBeUndefined();
- expect(($ as any).ajax).not.toHaveBeenCalled();
+ expect(uploadForm).not.toHaveBeenCalled();
});
- it("posts one ajax request per file, flagging the last, and forwards options + headers", async () => {
- await uploadFiles("notes", "parent1", ["a", "b"], {
+ it("posts one form per file, flagging the last, and forwards options + headers", async () => {
+ await uploadFiles("notes", "parent1", [ fileA, fileB ], {
shrinkImages: true,
safeImport: "true"
});
- const ajax = ($ as any).ajax as ReturnType<typeof vi.fn>;
- expect(ajax).toHaveBeenCalledTimes(2);
-
- const first = ajax.mock.calls[0][0];
- expect(first.url).toContain("notes/parent1/notes-import");
- expect(first.type).toBe("POST");
- expect(first.headers).toEqual({ "x-h": "1" });
- const firstData = first.data as FormData;
- expect(firstData.get("last")).toBe("false");
- expect(firstData.get("upload")).toBe("a");
- expect(firstData.get("shrinkImages")).toBe("true");
- expect(firstData.get("safeImport")).toBe("true");
+ expect(uploadForm).toHaveBeenCalledTimes(2);
+
+ const [ firstUrl, first ] = uploadForm.mock.calls[0];
+ expect(firstUrl).toBe("notes/parent1/notes-import");
+ expect(first).toMatchObject({ method: "POST", headers: { "x-h": "1" }, file: fileA, dataType: "json" });
+ expect(first.fields).toMatchObject({ last: "false", shrinkImages: "true", safeImport: "true" });
// taskId is shared across files in the same upload batch.
- const taskId = firstData.get("taskId");
- expect(typeof taskId).toBe("string");
+ expect(typeof first.fields.taskId).toBe("string");
- const second = ajax.mock.calls[1][0];
- const secondData = second.data as FormData;
- expect(secondData.get("last")).toBe("true");
- expect(secondData.get("upload")).toBe("b");
- expect(secondData.get("taskId")).toBe(taskId);
+ const [ , second ] = uploadForm.mock.calls[1];
+ expect(second.file).toBe(fileB);
+ expect(second.fields).toMatchObject({ last: "true", taskId: first.fields.taskId });
});
it("uses the attachments endpoint for the attachments entity type", async () => {
- await uploadFiles("attachments", "host1", ["a"], { shrinkImages: false });
- const first = ($ as any).ajax.mock.calls[0][0];
- expect(first.url).toContain("notes/host1/attachments-import");
+ await uploadFiles("attachments", "host1", [ fileA ], { shrinkImages: false });
+ expect(uploadForm.mock.calls[0][0]).toBe("notes/host1/attachments-import");
});
- it("surfaces an upload error through the toast service via the ajax error callback", async () => {
- // Spy on the i18n binding so we can verify xhr.responseText is forwarded
+ it("surfaces an upload error through the toast service and rejects", async () => {
+ // Spy on the i18n binding so we can verify responseText is forwarded
// without depending on i18next interpolation (i18next is uninitialised in
// the test env, so t() returns the key and never interpolates). We assert
// the call args/structure rather than a human-readable translated string.
const tSpy = vi.spyOn(i18n, "t");
vi.useFakeTimers();
try {
- ($ as any).ajax = vi.fn(async (opts: any) => {
- opts.error({ responseText: "boom" });
- return {};
- });
- await uploadFiles("notes", "p1", ["a"], { shrinkImages: false });
+ const failure = { status: 500, responseText: "boom" };
+ uploadForm.mockRejectedValue(failure);
+ await expect(uploadFiles("notes", "p1", [ fileA, fileB ], { shrinkImages: false })).rejects.toBe(failure);
+ // The first failure ends the batch.
+ expect(uploadForm).toHaveBeenCalledTimes(1);
// The error toast is deferred (so a WebSocket taskError can claim the taskId first and
// win), so the fallback only fires after IMPORT_ERROR_FALLBACK_DELAY — flush it first.
vi.runOnlyPendingTimers();
expect(toastService.showError).toHaveBeenCalledTimes(1);
- // The xhr.responseText ("boom") must be forwarded into the message
- // interpolation, not dropped or replaced with the whole xhr object.
+ // The responseText ("boom") must be forwarded into the message
+ // interpolation, not dropped or replaced with the whole error object.
expect(tSpy).toHaveBeenCalledWith("import.failed", { message: "boom" });
} finally {
tSpy.mockRestore();
@@ -132,8 +127,8 @@ describe("uploadFiles", () => {
});
it("is also reachable through the default export", async () => {
- await importService.uploadFiles("notes", "p1", ["a"], { shrinkImages: false });
- expect(($ as any).ajax).toHaveBeenCalledTimes(1);
+ await importService.uploadFiles("notes", "p1", [ fileA ], { shrinkImages: false });
+ expect(uploadForm).toHaveBeenCalledTimes(1);
});
});
diff --git a/apps/client/src/services/import.ts b/apps/client/src/services/import.ts
index 307805b649..00ba25ff38 100644
--- a/apps/client/src/services/import.ts
+++ b/apps/client/src/services/import.ts
@@ -2,7 +2,7 @@ import { ProgressPhase, WebSocketMessage } from "@triliumnext/commons";
import appContext from "../components/app_context.js";
import { t } from "./i18n.js";
-import server from "./server.js";
+import server, { type UploadFormError } from "./server.js";
import toastService, { type ToastOptionsWithRequiredId } from "./toast.js";
import utils from "./utils.js";
import ws from "./ws.js";
@@ -24,7 +24,7 @@ export interface UploadFilesOptions {
replaceUnderscoresWithSpaces?: BooleanLike;
}
-export async function uploadFiles(entityType: string, parentNoteId: string, files: string[] | File[], options: UploadFilesOptions) {
+export async function uploadFiles(entityType: string, parentNoteId: string, files: File[], options: UploadFilesOptions) {
if (!["notes", "attachments"].includes(entityType)) {
throw new Error(`Unrecognized import entity type '${entityType}'.`);
}
@@ -39,32 +39,31 @@ export async function uploadFiles(entityType: string, parentNoteId: string, file
for (const file of files) {
counter++;
- const formData = new FormData();
- formData.append("upload", file);
- formData.append("taskId", taskId);
- formData.append("last", counter === files.length ? "true" : "false");
-
- for (const key in options) {
- formData.append(key, (options as any)[key]);
+ const fields: Record<string, string> = {
+ taskId,
+ last: counter === files.length ? "true" : "false"
+ };
+ for (const [ key, value ] of Object.entries(options)) {
+ fields[key] = String(value);
}
- await $.ajax({
- url: `${window.glob.baseApiUrl}notes/${parentNoteId}/${entityType}-import`,
- headers: await server.getHeaders(),
- data: formData,
- dataType: "json",
- type: "POST",
- timeout: 60 * 60 * 1000,
- error (xhr) {
- // Fallback toast for failures that never produced a WebSocket `taskError` (e.g. transport or
- // auth errors raised before the importer runs). Deferred so the canonical WebSocket error —
- // emitted alongside this same 500 — can claim the taskId first and win.
- const message = t("import.failed", { message: xhr.responseText });
- setTimeout(() => reportImportError(taskId, message), IMPORT_ERROR_FALLBACK_DELAY);
- },
- contentType: false, // NEEDED, DON'T REMOVE THIS
- processData: false // NEEDED, DON'T REMOVE THIS
- });
+ try {
+ await server.uploadForm(`notes/${parentNoteId}/${entityType}-import`, {
+ method: "POST",
+ headers: await server.getHeaders(),
+ fields,
+ file,
+ timeoutMs: 60 * 60 * 1000,
+ dataType: "json"
+ });
+ } catch (e) {
+ // Fallback toast for failures that never produced a WebSocket `taskError` (e.g. transport or
+ // auth errors raised before the importer runs). Deferred so the canonical WebSocket error —
+ // emitted alongside this same 500 — can claim the taskId first and win.
+ const message = t("import.failed", { message: (e as Partial<UploadFormError>).responseText });
+ setTimeout(() => reportImportError(taskId, message), IMPORT_ERROR_FALLBACK_DELAY);
+ throw e;
+ }
}
}
diff --git a/apps/client/src/services/server.spec.ts b/apps/client/src/services/server.spec.ts
index 1cd3c068c2..8299cd8942 100644
--- a/apps/client/src/services/server.spec.ts
+++ b/apps/client/src/services/server.spec.ts
@@ -602,3 +602,76 @@ describe("localFetch transport", () => {
await expect(server.get("url")).rejects.toThrow("Worker error: boom");
});
});
+
+describe("uploadForm", () => {
+ const file = new File([ "zip" ], "notes.zip", { type: "application/zip" });
+ const options = {
+ method: "POST",
+ headers: { "x-csrf-token": "tok", "trilium-hoisted-note-id": null },
+ fields: { taskId: "t1" },
+ file,
+ timeoutMs: 1000
+ };
+
+ afterEach(() => {
+ delete (window as any).standaloneApi;
+ });
+
+ it("hands the File to the worker on a tab that owns it, and parses a JSON reply", async () => {
+ const localUpload = vi.fn(async () => new Response(JSON.stringify({ noteId: "n1" }), {
+ headers: { "content-type": "application/json" }
+ }));
+ (window as any).standaloneApi = { localFetch: vi.fn(), localUpload };
+ (window as any).$.ajax = () => {
+ throw new Error("$.ajax was used even though a tab owns the worker");
+ };
+
+ expect(await server.uploadForm("notes/root/notes-import", options)).toEqual({ noteId: "n1" });
+ expect(localUpload).toHaveBeenCalledWith("/api/notes/root/notes-import", {
+ method: "POST",
+ headers: { "x-csrf-token": "tok" },
+ fields: { taskId: "t1" },
+ file
+ });
+ });
+
+ it("rejects with the status and text of an error reply, the fields a jqXHR carries", async () => {
+ (window as any).standaloneApi = {
+ localUpload: vi.fn(async () => new Response("Import failed", { status: 500 }))
+ };
+
+ await expect(server.uploadForm("notes/root/notes-import", options))
+ .rejects.toEqual({ status: 500, responseText: "Import failed" });
+ });
+
+ it("rejects when the worker does not answer in time", async () => {
+ vi.useFakeTimers();
+ try {
+ (window as any).standaloneApi = { localUpload: vi.fn(() => new Promise(() => {})) };
+ const pending = expect(server.uploadForm("notes/root/notes-import", options)).rejects.toThrow("Timed out after 1000ms");
+ await vi.advanceTimersByTimeAsync(1000);
+ await pending;
+ } finally {
+ vi.useRealTimers();
+ }
+ });
+
+ it("sends a multipart XHR elsewhere, with the file under `upload`", async () => {
+ const ajaxSpy = vi.fn(async () => ({ ok: true }));
+ (window as any).$.ajax = ajaxSpy;
+
+ expect(await server.uploadForm("notes/root/notes-import", { ...options, dataType: "json" })).toEqual({ ok: true });
+ const opts = (ajaxSpy.mock.calls[0] as unknown[])[0] as any;
+ expect(opts).toMatchObject({
+ url: "/api/notes/root/notes-import",
+ type: "POST",
+ dataType: "json",
+ timeout: 1000,
+ headers: { "x-csrf-token": "tok" },
+ contentType: false,
+ processData: false
+ });
+ expect((opts.data as FormData).get("upload")).toBeInstanceOf(File);
+ expect((opts.data as FormData).get("taskId")).toBe("t1");
+ });
+});
diff --git a/apps/client/src/services/server.ts b/apps/client/src/services/server.ts
index efc6a31995..b949f40bb1 100644
--- a/apps/client/src/services/server.ts
+++ b/apps/client/src/services/server.ts
@@ -1,3 +1,5 @@
+import type { StandaloneApi } from "@triliumnext/commons";
+
import { t } from "./i18n.js";
import { getSetupAuthToken } from "./setup_auth.js";
import utils, { isShare } from "./utils.js";
@@ -95,19 +97,14 @@ async function remove<T>(url: string, componentId?: string) {
}
async function upload(url: string, fileToUpload: File, componentId?: string, method = "PUT") {
- const formData = new FormData();
- formData.append("upload", fileToUpload);
-
- const doUpload = async () => $.ajax({
- url: window.glob.baseApiUrl + url,
+ const doUpload = async () => uploadForm(url, {
+ method,
headers: await getHeaders(componentId ? {
"trilium-component-id": componentId
} : undefined),
- data: formData,
- type: method,
- timeout: 60 * 60 * 1000,
- contentType: false, // NEEDED, DON'T REMOVE THIS
- processData: false // NEEDED, DON'T REMOVE THIS
+ fields: {},
+ file: fileToUpload,
+ timeoutMs: UPLOAD_TIMEOUT
});
try {
@@ -123,6 +120,102 @@ async function upload(url: string, fileToUpload: File, componentId?: string, met
}
}
+export interface UploadFormOptions {
+ method: string;
+ headers: Headers;
+ fields: Record<string, string>;
+ /** Sent as the form's `upload` field. */
+ file: File;
+ timeoutMs: number;
+ /** Passed to jQuery, which otherwise picks a parser from the response's content type. */
+ dataType?: "json";
+}
+
+/** What `uploadForm()` rejects with on an error status: the fields of a jqXHR that callers read. */
+export interface UploadFormError {
+ status: number;
+ responseText: string;
+}
+
+const UPLOAD_TIMEOUT = 60 * 60 * 1000;
+
+/**
+ * Sends a multipart form carrying one file and resolves with the parsed response. The standalone
+ * leader tab hands the `File` to its worker through `standaloneApi.localUpload`, so the page never
+ * reads it and an import can read a zip one entry at a time; everywhere else it goes out as an XHR.
+ * Rejects with an {@link UploadFormError} on an error status.
+ */
+// eslint-disable-next-line @typescript-eslint/no-explicit-any -- the parsed body of whatever the route answers, as `$.ajax` gives it
+async function uploadForm(url: string, options: UploadFormOptions): Promise<any> {
+ const headers: Record<string, string> = {};
+ for (const [ name, value ] of Object.entries(options.headers)) {
+ if (value) {
+ headers[name] = value;
+ }
+ }
+
+ const localUpload = USE_LOCAL_FETCH ? window.standaloneApi?.localUpload : undefined;
+ if (localUpload) {
+ return await localUploadForm(localUpload, url, headers, options);
+ }
+
+ const formData = new FormData();
+ formData.append("upload", options.file);
+ for (const [ name, value ] of Object.entries(options.fields)) {
+ formData.append(name, value);
+ }
+
+ return await $.ajax({
+ url: window.glob.baseApiUrl + url,
+ headers,
+ data: formData,
+ type: options.method,
+ dataType: options.dataType,
+ timeout: options.timeoutMs,
+ contentType: false, // NEEDED, DON'T REMOVE THIS
+ processData: false // NEEDED, DON'T REMOVE THIS
+ });
+}
+
+async function localUploadForm(
+ localUpload: NonNullable<StandaloneApi["localUpload"]>,
+ url: string,
+ headers: Record<string, string>,
+ options: UploadFormOptions
+): Promise<unknown> {
+ let timeoutHandle: ReturnType<typeof setTimeout> | undefined;
+ const expiry = new Promise<never>((_, rej) => {
+ timeoutHandle = setTimeout(
+ () => rej(new Error(`Timed out after ${options.timeoutMs}ms: ${options.method} ${url}`)),
+ options.timeoutMs
+ );
+ });
+
+ let resp: globalThis.Response;
+ try {
+ resp = await Promise.race([
+ localUpload(window.glob.baseApiUrl + url, {
+ method: options.method,
+ headers,
+ fields: options.fields,
+ file: options.file
+ }),
+ expiry
+ ]);
+ } finally {
+ clearTimeout(timeoutHandle);
+ }
+
+ const responseText = await resp.text();
+ if (!resp.ok) {
+ throw { status: resp.status, responseText } satisfies UploadFormError;
+ }
+ if ((resp.headers.get("content-type") ?? "").includes("json")) {
+ return responseText ? JSON.parse(responseText) : null;
+ }
+ return responseText;
+}
+
let maxKnownEntityChangeId = 0;
let csrfRefreshInProgress: Promise<void> | null = null;
@@ -456,6 +549,7 @@ export default {
patch,
remove,
upload,
+ uploadForm,
// don't remove, used from CKEditor image upload!
getHeaders,
getMaxKnownEntityChangeId: () => maxKnownEntityChangeId
diff --git a/apps/server/src/zip_provider.ts b/apps/server/src/zip_provider.ts
index 83c7a592dc..92ce05e9c8 100644
--- a/apps/server/src/zip_provider.ts
+++ b/apps/server/src/zip_provider.ts
@@ -108,6 +108,10 @@ async function openZip(source: ZipSource) {
return yauzl.fromBufferPromise(buffer.subarray(0, findArchiveEnd(tail, buffer.length)), options);
}
+ if ("blob" in source) {
+ throw new Error("A Blob zip source is only readable in the browser.");
+ }
+
const handle = await fs.promises.open(source.path, "r");
let size: number;
let end: number;
diff --git a/apps/standalone/src/lightweight/browser_router.spec.ts b/apps/standalone/src/lightweight/browser_router.spec.ts
index e8951d7bf4..d01caa46de 100644
--- a/apps/standalone/src/lightweight/browser_router.spec.ts
+++ b/apps/standalone/src/lightweight/browser_router.spec.ts
@@ -1,7 +1,7 @@
import { getContext } from "@triliumnext/core";
import { afterEach, describe, expect, it, vi } from "vitest";
-import { BrowserRouter, createRouter } from "./browser_router.js";
+import { BrowserRouter, createRouter, type UploadedFile } from "./browser_router.js";
import { dbLock } from "./db_lock.js";
const RAW_RESPONSE = Symbol.for("RAW_RESPONSE");
@@ -163,6 +163,26 @@ describe("BrowserRouter body parsing", () => {
expect(req?.file && Array.from(req.file.buffer)).toEqual([1, 2, 3]);
});
+ it("takes a posted File's fields, and reads its bytes only for a route that does not read it in place", async () => {
+ const file = new File([ new Uint8Array([ 4, 5, 6 ]) ], "notes.zip", { type: "application/zip" });
+ const arrayBuffer = vi.spyOn(file, "arrayBuffer");
+ const body = { type: "upload", fields: { taskId: "t1" }, file };
+
+ const router = new BrowserRouter();
+ let buffered: { body?: unknown; file?: UploadedFile } | undefined;
+ let inPlace: { body?: unknown; file?: UploadedFile } | undefined;
+ router.post("/upload", (r) => { buffered = r; return {}; });
+ router.register("POST", "/import", (r) => { inPlace = r; return {}; }, { readsUploadsInPlace: true });
+
+ await router.dispatch("POST", "http://localhost/import", body);
+ expect(inPlace?.body).toEqual({ taskId: "t1" });
+ expect(inPlace?.file).toEqual({ originalname: "notes.zip", mimetype: "application/zip", blob: file });
+ expect(arrayBuffer).not.toHaveBeenCalled();
+
+ await router.dispatch("POST", "http://localhost/upload", body);
+ expect(buffered?.file?.buffer && Array.from(buffered.file.buffer)).toEqual([ 4, 5, 6 ]);
+ });
+
it("warns when multipart parsing fails", async () => {
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
const router = new BrowserRouter();
diff --git a/apps/standalone/src/lightweight/browser_router.ts b/apps/standalone/src/lightweight/browser_router.ts
index 232fec1314..3636518dbd 100644
--- a/apps/standalone/src/lightweight/browser_router.ts
+++ b/apps/standalone/src/lightweight/browser_router.ts
@@ -8,7 +8,28 @@ import { routes } from "@triliumnext/core";
export interface UploadedFile {
originalname: string;
mimetype: string;
- buffer: Uint8Array;
+ /** Unset for a {@link LocalUploadBody} file on a route that reads uploads in place. */
+ buffer?: Uint8Array;
+ /** The file the page posted, for a {@link LocalUploadBody}. */
+ blob?: Blob;
+}
+
+/**
+ * A multipart upload the page posts as its fields and the picked `File` itself. Structured cloning
+ * passes the `File` by reference, so no bytes are read until a route asks for them.
+ */
+export interface LocalUploadBody {
+ type: "upload";
+ fields: Record<string, string>;
+ file?: File;
+}
+
+export interface RouteOptions {
+ /**
+ * Whether the route reads an upload from `UploadedFile.blob` itself, as the import routes do.
+ * Every other route gets the upload's bytes in `buffer`.
+ */
+ readsUploadsInPlace?: boolean;
}
export interface BrowserRequest {
@@ -35,6 +56,7 @@ interface Route {
pattern: RegExp;
paramNames: string[];
handler: RouteHandler;
+ options: RouteOptions;
}
/**
@@ -75,6 +97,10 @@ function pathToRegex(path: string): { pattern: RegExp; paramNames: string[] } {
};
}
+function isLocalUploadBody(body: unknown): body is LocalUploadBody {
+ return typeof body === "object" && body !== null && (body as LocalUploadBody).type === "upload";
+}
+
/**
* Parse query string into an object.
*/
@@ -138,13 +164,14 @@ export class BrowserRouter {
/**
* Register a route handler.
*/
- register(method: string, path: string, handler: RouteHandler): void {
+ register(method: string, path: string, handler: RouteHandler, options: RouteOptions = {}): void {
const { pattern, paramNames } = pathToRegex(path);
this.routes.push({
method: method.toUpperCase(),
pattern,
paramNames,
- handler
+ handler,
+ options
});
}
@@ -183,7 +210,16 @@ export class BrowserRouter {
// Parse body based on content-type
let parsedBody = body;
let uploadedFile: UploadedFile | undefined;
- if (body instanceof ArrayBuffer && headers) {
+ if (isLocalUploadBody(body)) {
+ parsedBody = body.fields;
+ if (body.file) {
+ uploadedFile = {
+ originalname: body.file.name,
+ mimetype: body.file.type || 'application/octet-stream',
+ blob: body.file
+ };
+ }
+ } else if (body instanceof ArrayBuffer && headers) {
const contentType = headers['content-type'] || headers['Content-Type'] || '';
if (contentType.includes('application/json')) {
try {
@@ -233,6 +269,10 @@ export class BrowserRouter {
params[route.paramNames[i]] = decodeURIComponent(match[i + 1]);
}
+ if (uploadedFile?.blob && !uploadedFile.buffer && !route.options.readsUploadsInPlace) {
+ uploadedFile.buffer = new Uint8Array(await uploadedFile.blob.arrayBuffer());
+ }
+
const request: BrowserRequest = {
method: upperMethod,
url: urlString,
diff --git a/apps/standalone/src/lightweight/browser_routes.spec.ts b/apps/standalone/src/lightweight/browser_routes.spec.ts
index a346ec6b1a..26c8a3cfd3 100644
--- a/apps/standalone/src/lightweight/browser_routes.spec.ts
+++ b/apps/standalone/src/lightweight/browser_routes.spec.ts
@@ -157,6 +157,7 @@ describe("registerRoutes (real wiring)", () => {
describe("route wrapper branches (via controlled handlers)", () => {
type RouteCtx = Parameters<typeof routes.buildSharedApiRoutes>[0];
let ctx: RouteCtx;
+ const receivedFiles: { import?: unknown; upload?: unknown } = {};
function buildRouter(): BrowserRouter {
vi.spyOn(routes, "buildSharedApiRoutes").mockImplementation((received: RouteCtx) => {
@@ -184,6 +185,13 @@ describe("route wrapper branches (via controlled handlers)", () => {
asyncRoute("get", "/t/async-res", [], async (_req: unknown, res: MockRes) => { res.send("async-body"); });
asyncRoute("get", "/t/async-noresult", [], async () => ({ done: true }));
asyncRouteWithoutTransaction("get", "/t/async-no-tx", [], async () => ({ bare: true }), apiResultHandler);
+
+ asyncRoute("post", "/t/import", [ received.importMiddlewareWithErrorHandling ], async (req: { file?: unknown }) => {
+ receivedFiles.import = req.file;
+ }, apiResultHandler);
+ route("post", "/t/upload", [ received.uploadMiddlewareWithErrorHandling ], (req: { file?: unknown }) => {
+ receivedFiles.upload = req.file;
+ }, apiResultHandler);
});
return createConfiguredRouter();
}
@@ -269,6 +277,18 @@ describe("route wrapper branches (via controlled handlers)", () => {
expect(parseJson((await router.dispatch("GET", "http://localhost/api/system-checks")).body)).toEqual({ isCpuArchMismatch: false });
});
+ it("leaves a posted File unread for the import routes and reads it for the others", async () => {
+ const router = buildRouter();
+ const file = new File([ "abc" ], "vault.zip", { type: "application/zip" });
+ const body = { type: "upload", fields: {}, file };
+
+ await router.dispatch("POST", "http://localhost/t/import", body);
+ await router.dispatch("POST", "http://localhost/t/upload", body);
+
+ expect(receivedFiles.import).toEqual({ originalname: "vault.zip", mimetype: "application/zip", blob: file });
+ expect(receivedFiles.upload).toMatchObject({ buffer: new TextEncoder().encode("abc") });
+ });
+
it("provides no-op middleware and an init guard", async () => {
buildRouter();
// No-op middleware stubs do nothing and never throw.
diff --git a/apps/standalone/src/lightweight/browser_routes.ts b/apps/standalone/src/lightweight/browser_routes.ts
index 4ef158b6b4..49d1bb6712 100644
--- a/apps/standalone/src/lightweight/browser_routes.ts
+++ b/apps/standalone/src/lightweight/browser_routes.ts
@@ -10,7 +10,7 @@ import type { ShareReply } from '@triliumnext/core/src/share/handlers.js';
import { SHARE_ROUTE_PATHS, type ShareRoutePath } from '@triliumnext/core/src/share/route_paths.js';
import packageJson from '../../package.json' with { type: 'json' };
-import { type BrowserRequest, BrowserRouter } from './browser_router';
+import { type BrowserRequest, BrowserRouter, type RouteOptions } from './browser_router';
import { dbLock } from './db_lock';
/** Minimal response object used by apiResultHandler to capture the processed result. */
@@ -41,7 +41,9 @@ function toCoreRequest(req: BrowserRequest): Request {
body: req.body,
headers,
method: req.method,
- file: req.file,
+ // A file posted for an import route has no `buffer` until an importer that needs one calls
+ // `loadUploadedBytes()`, as on the server when multer leaves a zip on disk.
+ file: req.file as Request["file"],
get originalUrl() { return req.url; },
// `fetch` lower-cases the header names it sends, which is the form BrowserRouter stores them in.
get: (name: string) => headers[name.toLowerCase()]
@@ -99,7 +101,7 @@ function createApiRoute(router: BrowserRouter, transactional: boolean) {
* - The resultHandler is applied to post-process the result (entity conversion, status codes).
*/
function createRoute(router: BrowserRouter) {
- return (method: HttpMethod, path: string, _middleware: any[], handler: (req: Request, res: Response) => unknown, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {
+ return (method: HttpMethod, path: string, middleware: any[], handler: (req: Request, res: Response) => unknown, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {
router.register(method, path, (req: BrowserRequest) => {
return dbLock.runShared(() => getContext().init(() => {
setContextFromHeaders(req);
@@ -122,7 +124,7 @@ function createRoute(router: BrowserRouter) {
return result;
}));
- });
+ }, routeOptions(middleware));
};
}
@@ -133,7 +135,7 @@ function createRoute(router: BrowserRouter) {
* passed an async callback.
*/
function createAsyncRoute(router: BrowserRouter, { transactional = true } = {}) {
- return (method: HttpMethod, path: string, _middleware: any[], handler: (req: Request, res: Response) => Promise<unknown>, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {
+ return (method: HttpMethod, path: string, middleware: any[], handler: (req: Request, res: Response) => Promise<unknown>, resultHandler?: ((req: Request, res: ResultHandlerResponse, result: unknown) => unknown) | null) => {
router.register(method, path, (req: BrowserRequest) => {
// Exclusive: this transaction stays open across awaits, so no other
// route may touch the connection until it commits. See db_lock.ts.
@@ -159,7 +161,7 @@ function createAsyncRoute(router: BrowserRouter, { transactional = true } = {})
return result;
}) as Promise<unknown>);
- });
+ }, routeOptions(middleware));
};
}
@@ -282,6 +284,18 @@ function noopMiddleware() {
// No-op.
}
+/**
+ * Marks a route whose importer reads a posted `File` in place, which `routeOptions()` turns into
+ * `readsUploadsInPlace`. The server's counterpart keeps a zip import on disk for the same reason.
+ */
+function importUploadMiddleware() {
+ // No-op.
+}
+
+function routeOptions(middleware: unknown[]): RouteOptions {
+ return { readsUploadsInPlace: middleware.includes(importUploadMiddleware) };
+}
+
function checkAppNotInitialized() {
if (sql_init.isDbInitialized()) {
throw new Error("App already initialized.");
@@ -325,9 +339,7 @@ export function registerRoutes(router: BrowserRouter): void {
checkCredentials: noopMiddleware,
loginRateLimiter: noopMiddleware,
uploadMiddlewareWithErrorHandling: noopMiddleware,
- // The browser variant has no disk; uploads already arrive as a buffer (see browser_router.ts), so
- // imports use the same no-op middleware rather than the server's disk-storage one.
- importMiddlewareWithErrorHandling: noopMiddleware,
+ importMiddlewareWithErrorHandling: importUploadMiddleware,
csrfMiddleware: noopMiddleware
});
apiRoute('get', '/bootstrap', bootstrapRoute);
diff --git a/apps/standalone/src/lightweight/zip_provider.spec.ts b/apps/standalone/src/lightweight/zip_provider.spec.ts
index afcb7f964b..21013ee793 100644
--- a/apps/standalone/src/lightweight/zip_provider.spec.ts
+++ b/apps/standalone/src/lightweight/zip_provider.spec.ts
@@ -1,6 +1,6 @@
-import type { ZipEntry } from "@triliumnext/core/src/services/zip_provider.js";
+import type { ZipEntry, ZipSource } from "@triliumnext/core/src/services/zip_provider.js";
import { strToU8, unzipSync, zipSync } from "fflate";
-import { describe, expect, it } from "vitest";
+import { describe, expect, it, vi } from "vitest";
import BrowserZipProvider from "./zip_provider.js";
@@ -14,9 +14,9 @@ function makeZip(files: Record<string, string>): Uint8Array {
return zipSync(entries);
}
-async function readAll(buffer: Uint8Array, encoding?: string): Promise<Record<string, string>> {
+async function readAll(source: ZipSource, encoding?: string): Promise<Record<string, string>> {
const out: Record<string, string> = {};
- await provider.readZipFile(buffer, async (entry: ZipEntry, readContent) => {
+ await provider.readZipFile(source, async (entry: ZipEntry, readContent) => {
out[entry.fileName] = new TextDecoder().decode(await readContent());
}, encoding);
return out;
@@ -149,10 +149,8 @@ describe("BrowserZipArchive store + backpressure", () => {
});
describe("BrowserZipProvider path sources are unsupported", () => {
- it("readZipFile throws for a { path } source", () => {
- // readZipFile isn't async, so the guard throws synchronously before the work promise is created;
- // real callers await it inside an async function, where the throw surfaces as a rejection anyway.
- expect(() => provider.readZipFile({ path: "/tmp/x.zip" }, async () => {})).toThrow(
+ it("readZipFile rejects a { path } source", async () => {
+ await expect(provider.readZipFile({ path: "/tmp/x.zip" }, async () => {})).rejects.toThrow(
"Path-based zip reading is not supported in the browser"
);
});
@@ -287,3 +285,163 @@ describe("BrowserZipProvider filename re-decoding", () => {
expect(names).toEqual(["þ.txt"]);
});
});
+
+describe("BrowserZipProvider Blob sources", () => {
+ it("reads the same entries and names from a Blob as from the bytes", async () => {
+ const zip = makeLegacyZip([
+ { placeholder: "AB.txt", nameBytes: GBK_NAME_BYTES },
+ { placeholder: "Café.txt" }
+ ]);
+ const blobSource = { blob: new Blob([ zip ]) };
+
+ expect(await provider.detectFilenameEncoding(blobSource)).toBe("gbk");
+ expect(await readAll(blobSource, "gbk")).toEqual(await readAll(zip, "gbk"));
+ });
+
+ it("reads one byte range at a time, never the whole archive", async () => {
+ const entries: Record<string, string> = {};
+ for (let i = 0; i < 8; i++) {
+ entries[`note-${i}.txt`] = incompressibleText(40_000, i + 1);
+ }
+ const zip = makeZip(entries);
+ const blob = new Blob([ zip ]);
+ const slice = vi.spyOn(blob, "slice");
+
+ expect(await readAll({ blob })).toEqual(entries);
+ // Bigger than the tail the reader fetches to find the end-of-central-directory record.
+ expect(blob.size).toBeGreaterThan(2 * 0x10000);
+
+ expect(slice).toHaveBeenCalled();
+ for (const [ start = 0, end = blob.size ] of slice.mock.calls) {
+ expect(end - start).toBeLessThan(blob.size / 2);
+ }
+ });
+
+ it("reads nothing past the central directory until an entry's content is asked for", async () => {
+ const zip = makeZip({ "a.txt": "a".repeat(50_000), "b.txt": "b" });
+ const blob = new Blob([ zip ]);
+ const slice = vi.spyOn(blob, "slice");
+
+ const names: string[] = [];
+ await provider.readZipFile({ blob }, async (entry) => { names.push(entry.fileName); });
+
+ expect(names).toEqual([ "a.txt", "b.txt" ]);
+ // The tail holding the end-of-central-directory record, then the directory itself.
+ expect(slice).toHaveBeenCalledTimes(2);
+ });
+});
+
+describe("BrowserZipProvider entry formats", () => {
+ it("reads a stored (uncompressed) entry", async () => {
+ const zip = zipSync({ "plain.txt": [ strToU8("as is"), { level: 0 } ] });
+ expect(await readAll(zip)).toEqual({ "plain.txt": "as is" });
+ });
+
+ it("reads a ZIP64 archive, whose sizes and offsets are in 64-bit records", async () => {
+ expect(await readAll(makeZip64("big.txt", "sixty-four"))).toEqual({ "big.txt": "sixty-four" });
+ expect(await readAll({ blob: new Blob([ makeZip64("big.txt", "sixty-four") ]) })).toEqual({ "big.txt": "sixty-four" });
+ });
+
+ it("rejects an entry it cannot read, but only when its content is asked for", async () => {
+ const unsupported = patchCentralHeader(makeZip({ "a.txt": "x" }), (view, at) => view.setUint16(at + 10, 12, true));
+ const encrypted = patchCentralHeader(makeZip({ "a.txt": "x" }), (view, at) => view.setUint16(at + 8, view.getUint16(at + 8, true) | 1, true));
+
+ expect(await readNames(unsupported)).toEqual([ "a.txt" ]);
+ await expect(readAll(unsupported)).rejects.toThrow("Unsupported ZIP compression method 12.");
+ await expect(readAll(encrypted)).rejects.toThrow("Encrypted ZIP entries are not supported.");
+ });
+});
+
+/** Letters from a fixed-seed generator, which deflate cannot shrink much. */
+function incompressibleText(length: number, seed: number): string {
+ let state = seed;
+ let text = "";
+ for (let i = 0; i < length; i++) {
+ state ^= state << 13;
+ state ^= state >>> 17;
+ state ^= state << 5;
+ state >>>= 0;
+ text += String.fromCharCode(97 + state % 26);
+ }
+ return text;
+}
+
+/** Applies `patch` to the first central directory header in `zip`, at the header's offset. */
+function patchCentralHeader(zip: Uint8Array, patch: (view: DataView, offset: number) => void): Uint8Array {
+ const view = new DataView(zip.buffer, zip.byteOffset, zip.byteLength);
+ for (let i = 0; i + 4 <= zip.length; i++) {
+ if (view.getUint32(i, true) === 0x02014b50) {
+ patch(view, i);
+ return zip;
+ }
+ }
+ throw new Error("No central directory header found.");
+}
+
+/**
+ * Builds a one-entry stored archive in ZIP64 form: every size and offset in the classic records is
+ * saturated, and the real values sit in the entry's ZIP64 extra field and the ZIP64 end record.
+ */
+function makeZip64(name: string, content: string): Uint8Array {
+ const nameBytes = strToU8(name);
+ const data = strToU8(content);
+ const SATURATED = 0xffffffff;
+
+ const local = new DataView(new ArrayBuffer(30));
+ local.setUint32(0, 0x04034b50, true);
+ local.setUint16(4, 45, true);
+ local.setUint32(18, SATURATED, true);
+ local.setUint32(22, SATURATED, true);
+ local.setUint16(26, nameBytes.length, true);
+
+ const central = new DataView(new ArrayBuffer(46));
+ central.setUint32(0, 0x02014b50, true);
+ central.setUint16(4, 45, true);
+ central.setUint16(6, 45, true);
+ central.setUint32(20, SATURATED, true);
+ central.setUint32(24, SATURATED, true);
+ central.setUint16(28, nameBytes.length, true);
+ central.setUint16(30, 28, true);
+ central.setUint32(42, SATURATED, true);
+
+ const extra = new DataView(new ArrayBuffer(28));
+ extra.setUint16(0, 0x0001, true);
+ extra.setUint16(2, 24, true);
+ extra.setBigUint64(4, BigInt(data.length), true);
+ extra.setBigUint64(12, BigInt(data.length), true);
+ extra.setBigUint64(20, 0n, true);
+
+ const directoryOffset = 30 + nameBytes.length + data.length;
+ const directorySize = 46 + nameBytes.length + 28;
+ const zip64EndOffset = directoryOffset + directorySize;
+
+ const zip64End = new DataView(new ArrayBuffer(56));
+ zip64End.setUint32(0, 0x06064b50, true);
+ zip64End.setBigUint64(4, 44n, true);
+ zip64End.setBigUint64(24, 1n, true);
+ zip64End.setBigUint64(32, 1n, true);
+ zip64End.setBigUint64(40, BigInt(directorySize), true);
+ zip64End.setBigUint64(48, BigInt(directoryOffset), true);
+
+ const locator = new DataView(new ArrayBuffer(20));
+ locator.setUint32(0, 0x07064b50, true);
+ locator.setBigUint64(8, BigInt(zip64EndOffset), true);
+ locator.setUint32(16, 1, true);
+
+ const end = new DataView(new ArrayBuffer(22));
+ end.setUint32(0, 0x06054b50, true);
+ end.setUint16(8, 0xffff, true);
+ end.setUint16(10, 0xffff, true);
+ end.setUint32(12, SATURATED, true);
+ end.setUint32(16, SATURATED, true);
+
+ const parts = [ local, nameBytes, data, central, nameBytes, extra, zip64End, locator, end ]
+ .map((part) => part instanceof DataView ? new Uint8Array(part.buffer) : part);
+ const zip = new Uint8Array(parts.reduce((total, part) => total + part.length, 0));
+ let offset = 0;
+ for (const part of parts) {
+ zip.set(part, offset);
+ offset += part.length;
+ }
+ return zip;
+}
diff --git a/apps/standalone/src/lightweight/zip_provider.ts b/apps/standalone/src/lightweight/zip_provider.ts
index 98738480aa..cc6efbc71f 100644
--- a/apps/standalone/src/lightweight/zip_provider.ts
+++ b/apps/standalone/src/lightweight/zip_provider.ts
@@ -1,13 +1,5 @@
import type { FileStream, ZipArchive, ZipEntry, ZipProvider, ZipSource } from "@triliumnext/core/src/services/zip_provider.js";
-import { strToU8, unzip, zipSync } from "fflate";
-
-/** The browser/WASM build has no filesystem, so a `path` source is unsupported here — only raw bytes. */
-function requireBuffer(source: ZipSource): Uint8Array {
- if (!(source instanceof Uint8Array)) {
- throw new Error("Path-based zip reading is not supported in the browser; uploads arrive as bytes.");
- }
- return source;
-}
+import { inflateSync, strToU8, zipSync } from "fflate";
type ZipOutput = {
send?: (body: unknown) => unknown;
@@ -63,14 +55,32 @@ class BrowserZipArchive implements ZipArchive {
export default class BrowserZipProvider implements ZipProvider {
async detectFilenameEncoding(source: ZipSource): Promise<string> {
- const buffer = requireBuffer(source);
- const rawSamples = collectRawFilenameSamples(buffer);
+ const entries = await readCentralDirectory(toByteSource(source));
+ const rawSamples = collectRawFilenameSamples(entries);
if (rawSamples.length === 0) {
return "utf-8";
}
return detectEncodingFromBytes(rawSamples);
}
+ /**
+ * Reads one entry at a time: the central directory first, then each entry's bytes by offset when
+ * `readContent()` asks for them. A `Blob` source is therefore never loaded whole.
+ */
+ async readZipFile(
+ source: ZipSource,
+ processEntry: (entry: ZipEntry, readContent: () => Promise<Uint8Array>) => Promise<void>,
+ filenameEncoding?: string
+ ): Promise<void> {
+ const byteSource = toByteSource(source);
+ for (const record of await readCentralDirectory(byteSource)) {
+ await processEntry(
+ { fileName: decodeZipFileName(record, filenameEncoding) },
+ () => readEntryContent(byteSource, record)
+ );
+ }
+ }
+
createZipArchive(): ZipArchive {
return new BrowserZipArchive();
}
@@ -78,131 +88,83 @@ export default class BrowserZipProvider implements ZipProvider {
createFileStream(_filePath: string): FileStream {
throw new Error("File stream creation is not supported in the browser.");
}
+}
- readZipFile(
- source: ZipSource,
- processEntry: (entry: ZipEntry, readContent: () => Promise<Uint8Array>) => Promise<void>,
- filenameEncoding?: string
- ): Promise<void> {
- const buffer = requireBuffer(source);
- return new Promise<void>((res, rej) => {
- // Read inside the executor so a malformed archive rejects rather than throwing
- // synchronously, matching how an unzip() failure surfaces.
- const fileNames = readCentralDirectory(buffer);
-
- unzip(buffer, async (err, files) => {
- if (err) { rej(err); return; }
+/** Random access to the archive's bytes, whether they are in memory or behind a `Blob`. */
+interface ByteSource {
+ size: number;
+ read(offset: number, length: number): Promise<Uint8Array>;
+}
- try {
- for (const [fileName, data] of Object.entries(files)) {
- await processEntry(
- { fileName: decodeZipFileName(fileName, fileNames.get(fileName), filenameEncoding) },
- () => Promise.resolve(data)
- );
- }
- res();
- } catch (e) {
- rej(e);
- }
- });
- });
+/** The browser/WASM build has no filesystem, so a `path` source is unsupported here. */
+function toByteSource(source: ZipSource): ByteSource {
+ if (source instanceof Uint8Array) {
+ return {
+ size: source.byteLength,
+ read: (offset, length) => Promise.resolve(source.subarray(offset, offset + length))
+ };
+ }
+ if ("blob" in source) {
+ const { blob } = source;
+ return {
+ size: blob.size,
+ read: async (offset, length) => new Uint8Array(await blob.slice(offset, offset + length).arrayBuffer())
+ };
}
+ throw new Error("Path-based zip reading is not supported in the browser; uploads arrive as bytes or a Blob.");
}
-/** What an entry's name actually is on disk, as opposed to what fflate made of it. */
-interface FileNameInfo {
+/** One central directory record: everything needed to find, name and inflate an entry. */
+interface CentralDirectoryRecord {
/** The name exactly as stored, undecoded. */
- rawBytes: Uint8Array;
+ rawName: Uint8Array;
/** Whether the language encoding flag (general purpose bit 11) is set, i.e. the archive declares UTF-8. */
isUtf8Flagged: boolean;
+ isEncrypted: boolean;
+ compressionMethod: number;
+ compressedSize: number;
+ uncompressedSize: number;
+ localHeaderOffset: number;
}
-/**
- * Collects the raw name bytes of every entry whose encoding is genuinely unknown: the archive did not
- * flag it as UTF-8 and it does not decode as UTF-8 either.
- *
- * A flagged entry is excluded even when its bytes look unusual. Its encoding is already declared, so it
- * needs no rescuing — and sampling it would let one such name drag every *other* name in the archive
- * through the wrong decoder.
- */
-function collectRawFilenameSamples(buffer: Uint8Array): Uint8Array[] {
- const utf8 = new TextDecoder("utf-8", { fatal: true });
- const samples: Uint8Array[] = [];
-
- for (const { rawBytes, isUtf8Flagged } of readCentralDirectory(buffer).values()) {
- if (isUtf8Flagged) {
- continue;
- }
- try {
- utf8.decode(rawBytes);
- } catch {
- samples.push(rawBytes);
- }
- }
-
- return samples;
-}
-
-/**
- * Resolves an entry's name from what fflate decoded plus what the archive actually recorded.
- *
- * fflate hands names back already decoded, choosing UTF-8 or Latin-1 by the language encoding flag —
- * which it never exposes. A flagged name is therefore already correct and is passed through untouched;
- * re-deriving bytes from it would truncate every character above U+00FF (a curly apostrophe in a note
- * title becoming a control character, and the entry then matching nothing in `!!!meta.json`).
- *
- * Only an unflagged name is ambiguous — many real-world archives write UTF-8, or a legacy codepage,
- * without setting the flag — so those are re-decoded from their raw bytes with the detected encoding.
- */
-function decodeZipFileName(name: string, info: FileNameInfo | undefined, encoding?: string): string {
- // A name missing from the central directory (a duplicate entry, say) keeps fflate's reading.
- if (!info || info.isUtf8Flagged) {
- return name;
- }
-
- try {
- return new TextDecoder(encoding || "utf-8", { fatal: true }).decode(info.rawBytes);
- } catch {
- if (encoding && encoding !== "utf-8") {
- // Encoding detection was wrong for this entry, try UTF-8
- try {
- return new TextDecoder("utf-8", { fatal: true }).decode(info.rawBytes);
- } catch {
- return name;
- }
- }
- return name;
- }
-}
-
+const LOCAL_HEADER_SIGNATURE = 0x04034b50;
const CENTRAL_HEADER_SIGNATURE = 0x02014b50;
const END_OF_CENTRAL_DIRECTORY_SIGNATURE = 0x06054b50;
+const ZIP64_END_OF_CENTRAL_DIRECTORY_SIGNATURE = 0x06064b50;
+const ZIP64_LOCATOR_SIGNATURE = 0x07064b50;
+const LOCAL_HEADER_LENGTH = 30;
const CENTRAL_HEADER_LENGTH = 46;
const END_OF_CENTRAL_DIRECTORY_LENGTH = 22;
+const ZIP64_LOCATOR_LENGTH = 20;
+const ZIP64_END_OF_CENTRAL_DIRECTORY_LENGTH = 56;
+/** The longest comment the end-of-central-directory record can carry after itself. */
+const MAX_COMMENT_LENGTH = 0xffff;
+const ZIP64_EXTRA_FIELD_ID = 0x0001;
+/** A 32-bit size or offset that holds this value is in the entry's ZIP64 extra field instead. */
+const ZIP64_SATURATED = 0xffffffff;
+/** General purpose bit 0: the entry is encrypted. */
+const ENCRYPTED_FLAG = 0x1;
/** General purpose bit 11: the file name and comment are UTF-8. */
const UTF8_FLAG = 0x800;
+const METHOD_STORED = 0;
+const METHOD_DEFLATED = 8;
/**
- * Reads the archive's central directory for the two facts fflate discards — each entry's raw name bytes
- * and its language encoding flag — keyed by the string fflate produces for that entry, which is the only
- * handle the callers have on it.
- *
- * This also replaces what used to be a second full `unzip()` pass done purely to sample file names: the
- * central directory carries them verbatim, so no entry has to be inflated to read one.
+ * Reads the end-of-central-directory record (and its ZIP64 counterpart, for an archive past 4 GiB or
+ * 65535 entries), then the central directory it points to.
*/
-function readCentralDirectory(buffer: Uint8Array): Map<string, FileNameInfo> {
- const entries = new Map<string, FileNameInfo>();
- if (buffer.byteLength < END_OF_CENTRAL_DIRECTORY_LENGTH) {
+async function readCentralDirectory(source: ByteSource): Promise<CentralDirectoryRecord[]> {
+ if (source.size < END_OF_CENTRAL_DIRECTORY_LENGTH) {
throw new Error("Truncated ZIP: no end-of-central-directory record.");
}
- const view = new DataView(buffer.buffer, buffer.byteOffset, buffer.byteLength);
+ const tailStart = Math.max(0, source.size - END_OF_CENTRAL_DIRECTORY_LENGTH - MAX_COMMENT_LENGTH);
+ const tail = await source.read(tailStart, source.size - tailStart);
+ const tailView = viewOf(tail);
- // The end-of-central-directory record is last, but a trailing comment of up to 64 KiB may follow it.
let eocd = -1;
- const earliest = Math.max(0, buffer.byteLength - END_OF_CENTRAL_DIRECTORY_LENGTH - 0xffff);
- for (let i = buffer.byteLength - END_OF_CENTRAL_DIRECTORY_LENGTH; i >= earliest; i--) {
- if (view.getUint32(i, true) === END_OF_CENTRAL_DIRECTORY_SIGNATURE) {
+ for (let i = tail.byteLength - END_OF_CENTRAL_DIRECTORY_LENGTH; i >= 0; i--) {
+ if (tailView.getUint32(i, true) === END_OF_CENTRAL_DIRECTORY_SIGNATURE) {
eocd = i;
break;
}
@@ -211,47 +173,166 @@ function readCentralDirectory(buffer: Uint8Array): Map<string, FileNameInfo> {
throw new Error("Truncated ZIP: no end-of-central-directory record.");
}
- // Walk the directory by header rather than by the recorded entry count, which saturates at 65535
- // (and is then only recoverable from a ZIP64 record). A ZIP64 offset saturates to 0xFFFFFFFF, which
- // lands past `end` and simply yields no entries — every name then keeps fflate's own decoding.
- let offset = view.getUint32(eocd + 16, true);
- const end = Math.min(buffer.byteLength, offset + view.getUint32(eocd + 12, true));
+ let directorySize = tailView.getUint32(eocd + 12, true);
+ let directoryOffset = tailView.getUint32(eocd + 16, true);
+
+ const locator = eocd - ZIP64_LOCATOR_LENGTH;
+ if (locator >= 0 && tailView.getUint32(locator, true) === ZIP64_LOCATOR_SIGNATURE) {
+ const zip64Offset = readUint64(tailView, locator + 8);
+ const zip64 = viewOf(await source.read(zip64Offset, ZIP64_END_OF_CENTRAL_DIRECTORY_LENGTH));
+ if (zip64.byteLength < ZIP64_END_OF_CENTRAL_DIRECTORY_LENGTH
+ || zip64.getUint32(0, true) !== ZIP64_END_OF_CENTRAL_DIRECTORY_SIGNATURE) {
+ throw new Error("Corrupt ZIP: the ZIP64 end-of-central-directory record is missing.");
+ }
+ directorySize = readUint64(zip64, 40);
+ directoryOffset = readUint64(zip64, 48);
+ }
+
+ if (directoryOffset + directorySize > source.size) {
+ throw new Error("Corrupt ZIP: the central directory lies past the end of the file.");
+ }
+
+ const directory = await source.read(directoryOffset, directorySize);
+ const view = viewOf(directory);
+ const records: CentralDirectoryRecord[] = [];
- while (offset + CENTRAL_HEADER_LENGTH <= end && view.getUint32(offset, true) === CENTRAL_HEADER_SIGNATURE) {
- const isUtf8Flagged = !!(view.getUint16(offset + 8, true) & UTF8_FLAG);
+ // Walked by header rather than by the recorded entry count, which saturates at 65535.
+ let offset = 0;
+ while (offset + CENTRAL_HEADER_LENGTH <= directory.byteLength
+ && view.getUint32(offset, true) === CENTRAL_HEADER_SIGNATURE) {
+ const flags = view.getUint16(offset + 8, true);
const nameLength = view.getUint16(offset + 28, true);
const extraLength = view.getUint16(offset + 30, true);
const commentLength = view.getUint16(offset + 32, true);
const nameStart = offset + CENTRAL_HEADER_LENGTH;
+ const extraStart = nameStart + nameLength;
+ if (extraStart + extraLength > directory.byteLength) {
+ throw new Error("Corrupt ZIP: a central directory record runs past the directory.");
+ }
- if (nameStart + nameLength > end) {
- break;
+ const record: CentralDirectoryRecord = {
+ rawName: directory.subarray(nameStart, extraStart),
+ isUtf8Flagged: !!(flags & UTF8_FLAG),
+ isEncrypted: !!(flags & ENCRYPTED_FLAG),
+ compressionMethod: view.getUint16(offset + 10, true),
+ compressedSize: view.getUint32(offset + 20, true),
+ uncompressedSize: view.getUint32(offset + 24, true),
+ localHeaderOffset: view.getUint32(offset + 42, true)
+ };
+ applyZip64Extra(record, view, extraStart, extraLength);
+ records.push(record);
+
+ offset = extraStart + extraLength + commentLength;
+ }
+
+ return records;
+}
+
+/**
+ * Replaces the saturated 32-bit fields with the 64-bit values from the ZIP64 extra field, which
+ * carries only the saturated ones, in the order uncompressed size, compressed size, local header offset.
+ */
+function applyZip64Extra(record: CentralDirectoryRecord, view: DataView, start: number, length: number) {
+ const end = start + length;
+ for (let offset = start; offset + 4 <= end;) {
+ const id = view.getUint16(offset, true);
+ const size = view.getUint16(offset + 2, true);
+ let field = offset + 4;
+ if (id === ZIP64_EXTRA_FIELD_ID) {
+ for (const key of ["uncompressedSize", "compressedSize", "localHeaderOffset"] as const) {
+ if (record[key] === ZIP64_SATURATED && field + 8 <= offset + 4 + size) {
+ record[key] = readUint64(view, field);
+ field += 8;
+ }
+ }
+ return;
}
+ offset += 4 + size;
+ }
+}
- const rawBytes = buffer.subarray(nameStart, nameStart + nameLength);
- entries.set(decodeAsFflateDoes(rawBytes, isUtf8Flagged), { rawBytes, isUtf8Flagged });
+/** Reads an entry's bytes through its local header, which can carry a different extra field length. */
+async function readEntryContent(source: ByteSource, record: CentralDirectoryRecord): Promise<Uint8Array> {
+ if (record.isEncrypted) {
+ throw new Error("Encrypted ZIP entries are not supported.");
+ }
+
+ const header = viewOf(await source.read(record.localHeaderOffset, LOCAL_HEADER_LENGTH));
+ if (header.byteLength < LOCAL_HEADER_LENGTH || header.getUint32(0, true) !== LOCAL_HEADER_SIGNATURE) {
+ throw new Error("Corrupt ZIP: an entry's local header is missing.");
+ }
+ const dataStart = record.localHeaderOffset + LOCAL_HEADER_LENGTH
+ + header.getUint16(26, true) + header.getUint16(28, true);
+ const data = await source.read(dataStart, record.compressedSize);
+ if (data.byteLength < record.compressedSize) {
+ throw new Error("Truncated ZIP: an entry's data runs past the end of the file.");
+ }
- offset = nameStart + nameLength + extraLength + commentLength;
+ switch (record.compressionMethod) {
+ case METHOD_STORED:
+ return data;
+ case METHOD_DEFLATED:
+ return inflateSync(data, { out: new Uint8Array(record.uncompressedSize) });
+ default:
+ throw new Error(`Unsupported ZIP compression method ${record.compressionMethod}.`);
}
+}
+
+function viewOf(bytes: Uint8Array): DataView {
+ return new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+}
- return entries;
+/** ZIP64 values fit in a double for any archive a browser can hold. */
+function readUint64(view: DataView, offset: number): number {
+ return Number(view.getBigUint64(offset, true));
}
/**
- * Reproduces the string fflate hands back for an entry, so a central directory record can be matched to
- * it: UTF-8 when the archive flags the name as such, otherwise one character per byte.
+ * Collects the raw name bytes of every entry whose encoding is genuinely unknown: the archive did not
+ * flag it as UTF-8 and it does not decode as UTF-8 either.
*
- * The Latin-1 branch cannot use `TextDecoder`, whose "latin1"/"iso-8859-1" labels are aliases of
- * windows-1252 and so map 0x80-0x9F to different code points than the plain byte-to-char widening
- * fflate performs.
+ * A flagged entry is excluded even when its bytes look unusual. Its encoding is already declared, so it
+ * needs no rescuing, and sampling it would let one such name drag every other name in the archive
+ * through the wrong decoder.
+ */
+function collectRawFilenameSamples(records: CentralDirectoryRecord[]): Uint8Array[] {
+ const utf8 = new TextDecoder("utf-8", { fatal: true });
+ const samples: Uint8Array[] = [];
+
+ for (const { rawName, isUtf8Flagged } of records) {
+ if (isUtf8Flagged) {
+ continue;
+ }
+ try {
+ utf8.decode(rawName);
+ } catch {
+ samples.push(rawName);
+ }
+ }
+
+ return samples;
+}
+
+/**
+ * Decodes an entry's name. A UTF-8-flagged name is decoded as UTF-8. An unflagged name is ambiguous,
+ * since many archivers write UTF-8 or a legacy codepage without setting the flag, so it is decoded with
+ * the detected encoding, then UTF-8, and failing both one character per byte.
*/
-function decodeAsFflateDoes(rawBytes: Uint8Array, isUtf8Flagged: boolean): string {
- if (isUtf8Flagged) {
- return new TextDecoder("utf-8").decode(rawBytes);
+function decodeZipFileName(record: CentralDirectoryRecord, encoding?: string): string {
+ if (record.isUtf8Flagged) {
+ return new TextDecoder("utf-8").decode(record.rawName);
+ }
+
+ for (const label of encoding && encoding !== "utf-8" ? [encoding, "utf-8"] : ["utf-8"]) {
+ try {
+ return new TextDecoder(label, { fatal: true }).decode(record.rawName);
+ } catch {
+ // An unknown label or bytes invalid in it: try the next one.
+ }
}
let name = "";
- for (const byte of rawBytes) {
+ for (const byte of record.rawName) {
name += String.fromCharCode(byte);
}
return name;
diff --git a/apps/standalone/src/local-bridge.spec.ts b/apps/standalone/src/local-bridge.spec.ts
index 103e12d7b3..fa41fe9c91 100644
--- a/apps/standalone/src/local-bridge.spec.ts
+++ b/apps/standalone/src/local-bridge.spec.ts
@@ -397,6 +397,39 @@ describe("pending request rejection", () => {
});
});
+describe("localUpload", () => {
+ it("posts the fields and the File itself, transferring nothing, and resolves with the reply", async () => {
+ const bridge = await freshBridge();
+ const file = new File([ "zip" ], "notes.zip");
+ const promise = bridge.localUpload("api/notes/root/notes-import", {
+ method: "POST",
+ headers: { "x-csrf-token": "t" },
+ fields: { taskId: "t1" },
+ file
+ });
+
+ const worker = lastWorker();
+ await vi.waitFor(() => expect(worker.postMessage).toHaveBeenCalledWith(
+ expect.objectContaining({
+ type: "LOCAL_REQUEST",
+ request: {
+ url: new URL("api/notes/root/notes-import", location.href).href,
+ method: "POST",
+ headers: { "x-csrf-token": "t" },
+ body: { type: "upload", fields: { taskId: "t1" }, file }
+ }
+ }),
+ []
+ ));
+ // The File goes by reference: the posted body holds the very same object.
+ const posted = worker.postMessage.mock.calls.at(-1)?.[0] as { id: string; request: { body: { file: File } } };
+ expect(posted.request.body.file).toBe(file);
+
+ worker.onmessage?.({ data: { type: "LOCAL_RESPONSE", id: posted.id, response: { status: 200, headers: {} } } });
+ expect((await promise).status).toBe(200);
+ });
+});
+
describe("localFetch", () => {
it("posts a GET LOCAL_REQUEST with no body transfer and builds a Response from the reply", async () => {
const bridge = await freshBridge();
diff --git a/apps/standalone/src/local-bridge.ts b/apps/standalone/src/local-bridge.ts
index 4397be714d..97dfc2acc9 100644
--- a/apps/standalone/src/local-bridge.ts
+++ b/apps/standalone/src/local-bridge.ts
@@ -1,7 +1,8 @@
import type {
StandaloneDownloadResult,
StandaloneRestoreProgress,
- StandaloneRestoreResult
+ StandaloneRestoreResult,
+ StandaloneUploadInit
} from "@triliumnext/commons";
import { reportSplashPhase } from "../../client/src/services/splash.js";
@@ -388,7 +389,6 @@ function isShareRequest(pathname: string): boolean {
export async function localFetch(request: Request): Promise<Response> {
startLocalServerWorker();
- const id = Math.random().toString(36).slice(2);
const headersObj: Record<string, string> = {};
for (const [k, v] of request.headers.entries()) headersObj[k] = v;
@@ -396,13 +396,33 @@ export async function localFetch(request: Request): Promise<Response> {
? null
: await request.arrayBuffer();
+ return requestFromWorker({ url: request.url, method: request.method, headers: headersObj, body }, body ? [body] : []);
+}
+
+/**
+ * Sends a multipart upload to the worker as its fields and the picked `File` itself, which
+ * structured cloning passes by reference. Unlike `localFetch()`, nothing reads the file on the page,
+ * and an importer can read a zip from it one entry at a time.
+ */
+export function localUpload(url: string, init: StandaloneUploadInit): Promise<Response> {
+ return requestFromWorker({
+ url: new URL(url, location.href).href,
+ method: init.method,
+ headers: init.headers ?? {},
+ body: { type: "upload", fields: init.fields, file: init.file }
+ }, []);
+}
+
+async function requestFromWorker(
+ request: { url: string; method: string; headers: Record<string, string>; body: unknown },
+ transfer: Transferable[]
+): Promise<Response> {
+ const worker = startLocalServerWorker();
+
+ const id = Math.random().toString(36).slice(2);
const response = await new Promise<{ status: number; headers: Record<string, string>; body?: ArrayBuffer }>((resolve, reject) => {
pending.set(id, { resolve, reject });
- localWorker!.postMessage({
- type: "LOCAL_REQUEST",
- id,
- request: { url: request.url, method: request.method, headers: headersObj, body }
- }, body ? [body] : []);
+ worker.postMessage({ type: "LOCAL_REQUEST", id, request }, transfer);
});
const respHeaders = new Headers();
diff --git a/apps/standalone/src/main.spec.ts b/apps/standalone/src/main.spec.ts
index 21b141b662..1aaa8348d6 100644
--- a/apps/standalone/src/main.spec.ts
+++ b/apps/standalone/src/main.spec.ts
@@ -8,6 +8,7 @@ const mocks = vi.hoisted(() => ({
downloadDatabase: vi.fn(),
announceLeadership: vi.fn(),
localFetch: vi.fn(),
+ localUpload: vi.fn(),
capacitorHttpHandler: vi.fn(),
saveUrlToDevice: vi.fn(),
requestSecurityChange: vi.fn(),
@@ -26,6 +27,7 @@ vi.mock("./local-bridge.js", () => ({
downloadDatabase: mocks.downloadDatabase,
announceLeadership: mocks.announceLeadership,
localFetch: mocks.localFetch,
+ localUpload: mocks.localUpload,
requestSecurityChange: mocks.requestSecurityChange
}));
vi.mock("./leader_election.js", () => ({
@@ -104,6 +106,7 @@ describe("bootstrap", () => {
await vi.waitFor(() => expect(mocks.announceLeadership).toHaveBeenCalled());
// The leader also answers the client's API calls directly, skipping that route.
expect(window.standaloneApi?.localFetch).toBe(mocks.localFetch);
+ expect(window.standaloneApi?.localUpload).toBe(mocks.localUpload);
// And its worker is the one holding the settings file, so it is the tab that can change
// what this instance is allowed to run.
expect(window.standaloneApi?.security).toBeDefined();
@@ -123,6 +126,7 @@ describe("bootstrap", () => {
// A follower must keep calling through the service worker: with no worker of its own,
// a localFetch here would answer from nothing.
expect(window.standaloneApi?.localFetch).toBeUndefined();
+ expect(window.standaloneApi?.localUpload).toBeUndefined();
// Nor can it change a security setting: the file that holds them is locked by the
// leader's worker, which this tab has no way to reach.
diff --git a/apps/standalone/src/main.ts b/apps/standalone/src/main.ts
index 8c2637f89a..b464dba062 100644
--- a/apps/standalone/src/main.ts
+++ b/apps/standalone/src/main.ts
@@ -6,7 +6,7 @@ import {
import { showErrorOverlay } from "./error-overlay.js";
import { installIosInterceptors } from "./ios-interceptors.js";
import { claimLeadership } from "./leader_election.js";
-import { announceLeadership, attachServiceWorkerBridge, downloadDatabase, localFetch, registerNativeHttpHandler, restoreBackup, saveDatabase, startLocalServerWorker } from "./local-bridge.js";
+import { announceLeadership, attachServiceWorkerBridge, downloadDatabase, localFetch, localUpload, registerNativeHttpHandler, restoreBackup, saveDatabase, startLocalServerWorker } from "./local-bridge.js";
import { createSecurityApi } from "./security_gate.js";
/**
@@ -121,6 +121,7 @@ async function bootstrap() {
// The leader answers API requests from its own worker, so the client's server.ts
// can skip the service-worker round trip.
standaloneApi.localFetch = localFetch;
+ standaloneApi.localUpload = localUpload;
// Its worker is also the one holding the lock on the security settings file, so it is
// the only tab that can change what this instance is allowed to run. See security_gate.ts.
standaloneApi.security = createSecurityApi();
diff --git a/packages/commons/src/lib/standalone_api_interface.ts b/packages/commons/src/lib/standalone_api_interface.ts
index abdae4e854..76b622282b 100644
--- a/packages/commons/src/lib/standalone_api_interface.ts
+++ b/packages/commons/src/lib/standalone_api_interface.ts
@@ -150,4 +150,18 @@ export interface StandaloneApi {
* would pay the XHR → service worker → page round trip to end up in the same worker.
*/
localFetch?(request: Request): Promise<Response>;
+ /**
+ * Sends a multipart upload to the same worker as `localFetch`, passing the picked `File` by
+ * reference instead of reading it into the request body. Set alongside `localFetch`.
+ */
+ localUpload?(url: string, init: StandaloneUploadInit): Promise<Response>;
+}
+
+/** A multipart upload for {@link StandaloneApi.localUpload}: the form's text fields and its one file. */
+export interface StandaloneUploadInit {
+ method: string;
+ headers?: Record<string, string>;
+ fields: Record<string, string>;
+ /** Sent as the form's `upload` field, as multer expects on the server. */
+ file?: File;
}
diff --git a/packages/trilium-core/src/routes/api/import.ts b/packages/trilium-core/src/routes/api/import.ts
index cf84bb635b..d0c072980a 100644
--- a/packages/trilium-core/src/routes/api/import.ts
+++ b/packages/trilium-core/src/routes/api/import.ts
@@ -1,6 +1,6 @@
import type { Request } from "../../http_interface";
-import type { File } from "../../services/import/common.js";
+import { type File, loadUploadedBytes } from "../../services/import/common.js";
type ImportRequest<P> = Omit<Request<P>, "file"> & { file?: File };
@@ -109,6 +109,7 @@ async function importAttachmentsToNote(req: ImportRequest<{ parentNoteId: string
// unlike in note import, we let the events run, because a huge number of attachments is not likely
try {
+ await loadUploadedBytes(file);
singleImportService.importAttachment(taskContext, file, parentNote);
// An imported image is stored once its compression answers, which is after the call above
// has returned. Waited out for the same reason the note import waits: what is reported as
diff --git a/packages/trilium-core/src/services/import/common.ts b/packages/trilium-core/src/services/import/common.ts
index 2d37ef6ae0..f147c5f5e9 100644
--- a/packages/trilium-core/src/services/import/common.ts
+++ b/packages/trilium-core/src/services/import/common.ts
@@ -1,11 +1,23 @@
export interface File {
originalname: string;
mimetype: string;
+ /**
+ * The file's bytes. Unset for a zip import that arrives as `path` or `blob`; call
+ * {@link loadUploadedBytes} before reading it on any path that does not read the zip in place.
+ */
buffer: string | Buffer | Uint8Array;
/**
* Filesystem path of the uploaded file when it was streamed to disk (server multer disk storage).
- * Lets a zip be read in place instead of buffering it; absent for the browser/WASM upload, which
- * only has `buffer`.
+ * Lets a zip be read in place instead of buffering it.
*/
path?: string;
+ /** The file as the page picked it (browser/WASM), which lets a zip be read in place. */
+ blob?: Blob;
+}
+
+/** Reads a `blob`-backed upload into `buffer`, for the importers that need the whole file. */
+export async function loadUploadedBytes(file: File): Promise<void> {
+ if (file.buffer === undefined && file.blob) {
+ file.buffer = new Uint8Array(await file.blob.arrayBuffer());
+ }
}
diff --git a/packages/trilium-core/src/services/import/dispatch.spec.ts b/packages/trilium-core/src/services/import/dispatch.spec.ts
index 9f20b901f8..33ccbff161 100644
--- a/packages/trilium-core/src/services/import/dispatch.spec.ts
+++ b/packages/trilium-core/src/services/import/dispatch.spec.ts
@@ -168,4 +168,38 @@ describe("importFile (dispatch)", () => {
expect(stubbed.importZip).toHaveBeenCalledWith(taskContext, { path: "/tmp/v.zip" }, parentNote);
});
});
+
+ describe("a Blob upload (standalone, where the page posts the picked File)", () => {
+ it("hands a zip importer the Blob without reading it", async () => {
+ const blob = new Blob([ "zip bytes" ]);
+ const arrayBuffer = vi.spyOn(blob, "arrayBuffer");
+ const file = { originalname: "notes.zip", mimetype: "application/zip", blob } as File;
+
+ await importFile(taskContext, file, parentNote, makeOptions(), "notion");
+
+ expect(stubbed.importNotion).toHaveBeenCalledWith(taskContext, { blob }, parentNote);
+ expect(arrayBuffer).not.toHaveBeenCalled();
+ expect(file.buffer).toBeUndefined();
+ });
+
+ it("reads the Blob into the buffer for an importer that needs the whole file", async () => {
+ const file = { originalname: "page.md", mimetype: "text/markdown", blob: new Blob([ "# Hi" ]) } as File;
+
+ await importFile(taskContext, file, parentNote, makeOptions());
+
+ expect(stubbed.importSingleFile).toHaveBeenCalledWith(taskContext, file, parentNote);
+ expect(new TextDecoder().decode(file.buffer as Uint8Array)).toBe("# Hi");
+ });
+
+ it("reads the Blob for a zip the user chose not to explode", async () => {
+ const file = { originalname: "keep.zip", mimetype: "application/zip", blob: new Blob([ "PK" ]) } as File;
+
+ await importFile(taskContext, file, parentNote, makeOptions({ explodeArchives: false }));
+
+ expect(stubbed.importZip).not.toHaveBeenCalled();
+ expect(stubbed.importSingleFile).toHaveBeenCalledWith(taskContext, file, parentNote);
+ expect(file.buffer).toBeInstanceOf(Uint8Array);
+ });
+ });
});
+
diff --git a/packages/trilium-core/src/services/import/dispatch.ts b/packages/trilium-core/src/services/import/dispatch.ts
index dc2fafbd69..2a1a2200e2 100644
--- a/packages/trilium-core/src/services/import/dispatch.ts
+++ b/packages/trilium-core/src/services/import/dispatch.ts
@@ -4,7 +4,7 @@ import type TaskContext from "../task_context.js";
import { extname } from "../utils/path.js";
import type { ZipSource } from "../zip_provider.js";
import anytypeImportService from "./anytype/importer.js";
-import type { File } from "./common.js";
+import { type File, loadUploadedBytes } from "./common.js";
import enexImportService from "./enex.js";
import keepImportService from "./keep/importer.js";
import notionImportService from "./notion/importer.js";
@@ -32,8 +32,8 @@ export interface ImportOptions {
* importer identically. The caller owns the surrounding CLS setup (`disableEntityEvents`,
* `ignoreEntityChangeIds`) and the `becca_loader.load()` afterwards.
*
- * A `.zip` is read from `file.path` in place (streamed per entry) when present, falling back to
- * `file.buffer` for the browser/WASM upload that has no temp file.
+ * A `.zip` is read in place from `file.path` or `file.blob` when present, falling back to
+ * `file.buffer`.
*/
export default async function importFile(taskContext: TaskContext<"importNotes">, file: File, parentNote: BNote, options: ImportOptions, format?: string): Promise<BNote | (string | number)[] | null> {
const imported = await routeToImporter(taskContext, file, parentNote, options, format);
@@ -55,12 +55,13 @@ export default async function importFile(taskContext: TaskContext<"importNotes">
async function routeToImporter(taskContext: TaskContext<"importNotes">, file: File, parentNote: BNote, options: ImportOptions, format?: string): Promise<BNote | (string | number)[] | null> {
const extension = extname(file.originalname).toLowerCase();
- // Every zip-reading importer (generic + the tagged providers) reads the archive straight from the temp
- // file (server disk storage / native pick) so a multi-GB zip is streamed per entry — never held in one
- // buffer, and free of fs.readFile's ~2 GiB ceiling. Falls back to the buffer when there's no path (the
- // browser/WASM upload). The `typeof file.buffer !== "string"` guards below ensure the no-path fallback
- // is real bytes, not a string body.
- const zipSource: ZipSource = file.path ? { path: file.path } : file.buffer as Uint8Array;
+ // Every zip-reading importer reads the archive in place, from the temp file on the server or the
+ // picked `Blob` in the browser, so a multi-GB zip is read one entry at a time. The buffer is the
+ // fallback for an upload that has neither. The `typeof file.buffer !== "string"` guards below ensure
+ // that fallback is real bytes, not a string body.
+ const zipSource: ZipSource = file.path ? { path: file.path }
+ : file.blob ? { blob: file.blob }
+ : file.buffer as Uint8Array;
if (format === "notion" && (file.path || typeof file.buffer !== "string")) {
// An explicit format always wins over extension sniffing: a Notion export is just a `.zip`,
@@ -88,7 +89,10 @@ async function routeToImporter(taskContext: TaskContext<"importNotes">, file: Fi
});
}
return await zipImportService.importZip(taskContext, zipSource, parentNote);
- } else if (extension === ".opml" && options.explodeArchives) {
+ }
+
+ await loadUploadedBytes(file);
+ if (extension === ".opml" && options.explodeArchives) {
return await opmlImportService.importOpml(taskContext, file.buffer, parentNote);
} else if (extension === ".enex" && options.explodeArchives) {
return await enexImportService.importEnex(taskContext, file, parentNote);
diff --git a/packages/trilium-core/src/services/zip_provider.ts b/packages/trilium-core/src/services/zip_provider.ts
index b1ecb16bdc..46210dd669 100644
--- a/packages/trilium-core/src/services/zip_provider.ts
+++ b/packages/trilium-core/src/services/zip_provider.ts
@@ -1,9 +1,9 @@
/**
- * A zip to read: either its raw bytes, or — server/desktop only — a filesystem path the provider opens
- * and reads *in place* (no full-buffer copy, no ~2 GiB `fs.readFile` ceiling). The browser/WASM provider
- * only supports the byte form.
+ * A zip to read: its raw bytes, a filesystem path (server/desktop only) or a `Blob` (browser/WASM
+ * only). A path or a `Blob` is read in place, one byte range at a time, so the archive is never held
+ * in memory whole.
*/
-export type ZipSource = Uint8Array | { path: string };
+export type ZipSource = Uint8Array | { path: string } | { blob: Blob };
export interface ZipEntry {
fileName: string;