Build your personal knowledge base with Trilium Notes
TypeScript electronelectron-appknowledge-baseknowledge-graphknowledge-managementknowledge-management-graphlocal-firstnote-managmentnote-takernote-takingnotebooknotesnotes-apppersonal-knowledge-basepersonal-wikiscriptableself-hostedself-hostingwiki
/**
* A small pool of worker processes that compress images, so the decoding stops holding the thread
* that serves the application.
*
* Decoding is synchronous from beginning to end — jpeg-js, pngjs and UPNG all run to completion
* without yielding — so an image being compressed on the main thread is an application that has
* stopped answering for as long as it takes. Moving that off-thread is worth more than the extra
* cores it also buys.
*
* Processes rather than worker threads, which were tried first and cannot be used: in the Electron
* main process a worker thread's isolate wedges on this workload. Its garbage collector hands
* sweeping to V8 background tasks that never complete there, so the hundreds of megabytes a decode
* allocates are never truly reclaimed — a forced full collection returns without them, and a second
* one never returns at all. The first image per thread works, the next waits on that sweep forever;
* in between sit runs that crawl, seconds of collector stall per image. A forked child is a main
* isolate on Node's own platform, where collection simply works — under the plain server, which
* never had the problem, and under Electron, which fork serves by re-running its binary as Node.
*
* Nothing here is load-bearing. Every way this can fail — a worker file that is not where it was
* expected, a process that will not start, one that dies mid-image — ends in the same place: the
* caller is told so once and compresses in this process instead, exactly as it always did.
*
* This is the Node half of the platform, reached only through the server's own image provider —
* the server itself and the desktop app, whose main process is Node too. The standalone build has
* no decoder to run in a thread and never arrives here; it answers for itself.
*/
import { type ChildProcess, fork } from "node:child_process";
import { existsSync } from "node:fs";
import { availableParallelism, totalmem } from "node:os";
import { join } from "node:path";
import { getLog } from "@triliumnext/core";
import type { ImageCompressionOutcome, ImageCompressionRequest } from "@triliumnext/core/src/services/image_provider.js";
import type { ImageWorkerRequest, ImageWorkerResponse, ImageWorkerTrace } from "./image_worker.js";
/**
* How many images may be compressed at once, or 1 where they cannot be compressed off-thread at
* all — which is the same answer as "do it here", and is what the caller schedules against.
*
* How much of the machine to take depends on whose machine it is.
*
* The desktop app runs on one person's computer, doing a job that person just asked for and is
* waiting on, so it takes a core per worker and leaves one for the thread handing out the images —
* which is also the thread the window is answering on. A server is not anyone's computer in that
* sense: it may be serving other people while this runs, and a maintenance job nobody is watching
* has no business taking the whole of it, so it stays with a handful.
*
* A ceiling rather than a promise either way: what actually runs at once is whatever the memory
* budget admits, which on large images is far fewer. Worth knowing that decoding is bound by memory
* bandwidth as much as by processor, so the last threads on a wide machine are worth much less than
* the first — this is what the machine allows, not what it will reward.
*
* `availableParallelism` rather than the raw core count: it follows the affinity the process was
* given, so a container pinned to two cores is answered as two.
*
* Bounded by memory besides. Each worker may grow a {@link WORKER_HEAP_MB} heap, and their sum is
* kept inside half of what the machine has — a four-gigabyte NAS is answered 1, not 4, because the
* failure past that bound is not the compression slowing down but the machine paging, which stops
* everything on it. Coarse on purpose: in a container `totalmem` reads the host rather than the
* cgroup, so this guards the obvious overcommit rather than accounting for what is genuinely free.
*/
export function compressionConcurrency(): number {
if (!workerEntry()) {
return 1;
}
const byMemory = Math.floor(totalmem() / 2 / (WORKER_HEAP_MB * 1024 * 1024));
const byCores = isDesktopApp()
? availableParallelism() - 1
: Math.min(availableParallelism(), SERVER_MAX_WORKERS);
return Math.max(1, Math.min(byCores, byMemory));
}
/** What a shared machine is allowed to spend on a job nobody is waiting for. */
const SERVER_MAX_WORKERS = 4;
/**
* Whether this is the desktop app rather than a server.
*
* Read from the runtime rather than from core's platform flags: those are settled during startup,
* and this is asked for whenever a run begins.
*/
function isDesktopApp(): boolean {
return process.versions.electron !== undefined;
}
/**
* Compresses one image in a worker, or answers `null` if there is no worker to do it in — which the
* caller reads as "then do it yourself" rather than as a failure of the image.
*
* @param budgetMb the most this decode may allocate, which the caller has already set aside for it.
*/
export async function compressInWorker(
buffer: Uint8Array,
request: ImageCompressionRequest,
budgetMb: number
): Promise<ImageCompressionOutcome | null> {
if (!workerEntry()) {
return disable("no worker entry point was found");
}
const worker = await acquire();
if (!worker) {
return null;
}
try {
const response = await send(worker, buffer, request, budgetMb);
// The worker's own logging, written here because here is where a logger exists.
response.logs.forEach((line) => getLog().info(line));
if (response.error) {
throw new Error(response.error);
}
release(worker);
return response.outcome ?? null;
} catch (e: unknown) {
// Retired rather than released: a process that failed us may well be dead, and handing it to
// the next image would post a message nothing is listening for — which looks like the run
// hanging rather than like one image having failed.
retire(worker);
// No worker has ever so much as announced itself, so this is not one process having died
// but this installation being unable to run them — an environment whose loader cannot read
// the entry point, say. Answered as "there are no workers", which is the truth of it: this
// image and every one after it is compressed by the caller, exactly as before there were
// workers.
if (!everSpoke) {
return disable(`they do not work here (${firstLine(e)})`);
}
// Workers run here — one announced itself, which took spawning, the loader and the channel
// all working — so this failure belongs to the image: a decode that blew through the heap,
// say, taking its worker with it. Raised rather than answered with a fallback, and
// emphatically not treated as the installation being broken: decoding this image on the
// calling thread would stop the application for the length of a decode — the very freeze
// the pool exists to prevent, handed to exactly the image that just proved too much for a
// worker. The image is reported untouched and the run goes on.
throw e;
}
}
/**
* Stops the workers that are not doing anything, so a Trilium that has finished compressing is not
* still holding them.
*
* An idle child is still a process, holding its runtime and whatever its last decode grew it to.
* Several of them sitting on that is memory the rest of the machine would rather have, and
* terminating them gives it back outright.
*
* Only the free ones: a busy worker is not in the pool to be found here, and is left to finish.
*/
export function shutdownImageWorkers() {
started -= idle.length;
idle.forEach((pooled) => {
pooled.stopped = true;
pooled.worker.kill();
});
idle.length = 0;
}
/**
* How long the pool waits, having been given nothing to do, before letting its workers go.
*
* Long enough that the images of one run do not pay to start a process each, short enough that a
* run which has finished stops costing anything soon after. Nothing schedules a run, so any wait at
* all is only ever bridging the gap between two images.
*/
const IDLE_SHUTDOWN_MS = 30 * 1000;
let idleTimer: NodeJS.Timeout | undefined;
/** Restarted whenever a worker comes free, so the countdown only ever runs on a quiet pool. */
function scheduleShutdown() {
clearTimeout(idleTimer);
idleTimer = setTimeout(shutdownImageWorkers, IDLE_SHUTDOWN_MS);
// Never a reason for this to be what holds the process open.
idleTimer.unref();
}
interface PooledWorker {
worker: ChildProcess;
/** Missing exactly when the spawn failed, in which case its error event is on the way. */
pid: number | undefined;
/**
* Settled when the child first speaks, or when it dies trying. Waited out before sending:
* unlike a thread's queue, a process's channel does not hold messages for a listener that has
* not attached yet, and the child only listens once its module has loaded.
*/
ready: Promise<void>;
/** Set when we stopped it ourselves, so its exit is not reported as something going wrong. */
stopped?: boolean;
pending?: (response: ImageWorkerResponse) => void;
failed?: (error: unknown) => void;
}
const idle: PooledWorker[] = [];
const waiting: ((worker: PooledWorker | null) => void)[] = [];
let started = 0;
let disabled = false;
/**
* Whether any worker has ever sent anything back — its startup announce being the first thing one
* says. That is proof the installation can run workers at all, which is the line between "give up
* on workers for good" and "this one image failed": everything a worker needs to exist — spawning,
* the loader, the channel home — has demonstrably worked once.
*/
let everSpoke = false;
async function acquire(): Promise<PooledWorker | null> {
if (disabled) {
return null;
}
const free = idle.pop();
if (free) {
return free;
}
if (started < compressionConcurrency()) {
const spawned = spawn();
if (spawned) {
started++;
return spawned;
}
return null;
}
// Every worker is busy, so this one queues for the next to come free. That queue is ordinary
// and can be deep: images arriving into the database are handed over one call at a time by
// callers that know nothing of each other, and an import produces hundreds at once.
//
// Which makes it the pool's business to answer every caller in it. Whoever waits here is past
// the point where they could have started a worker for themselves, so from here they are
// answered by `release`, by `retire`, or by `disable` — and if any of those three forgets
// them, they wait for the life of the process. An image whose compression never comes back is
// an image never written: the note was created empty and is filled in when the answer arrives.
return new Promise((resolve) => waiting.push(resolve));
}
function release(worker: PooledWorker) {
const next = waiting.shift();
if (next) {
next(worker);
return;
}
idle.push(worker);
// Nothing was waiting for it, so this may well have been the last image of a run.
scheduleShutdown();
}
/**
* Takes a worker out of the pool for good, and never puts it back.
*
* Whoever was waiting for a free worker is answered with none rather than with this one: no process
* is coming to replace it, and a caller told there is none compresses the image itself. Waiting on
* a worker that no longer exists is the one outcome worse than not having workers at all.
*/
function retire(worker: PooledWorker) {
started--;
worker.stopped = true;
worker.worker.kill();
const next = waiting.shift();
if (!next) {
return;
}
// The capacity this worker held is free again, and whoever is queued for it is already past
// the point where they could have started one for themselves — they queued because the pool
// was full, and it no longer is. Starting the replacement here is what keeps the queue moving;
// without it a pool whose workers all died would leave everyone behind them waiting on a
// release that can no longer come.
//
// Only where workers are known to run, though. If none has ever spoken, another is not going
// to either, and starting one would make this caller wait out a second process failing to come
// up before hearing the same answer it can be given now.
const replacement = everSpoke ? spawn() : null;
if (replacement) {
started++;
next(replacement);
} else {
// Nothing can be started, which `spawn` has already answered for by disabling workers
// outright — and that answers everyone else. This one was taken off the queue first.
next(null);
}
}
/**
* The heap a worker is given: sized to the work, and neither inherited nor unbounded.
*
* Both directions matter, which is why this is a fixed figure rather than a floor.
*
* Too small and a decode does not fail, it collects — over and over, taking several times longer
* while looking perfectly healthy. Whatever started this process sized its heap for running an
* application, not for decoding photographs.
*
* Too large is the subtler half. V8 collects lazily against its ceiling, so a child allowed four
* gigabytes will happily accumulate several before it bothers, and four such children can put a
* machine into paging — where the symptom is not the compression slowing down but everything else
* stopping, the process that hosts them included.
*
* This is what one decode actually needs: the memory budget a single image may claim, and room to
* work around it. A worker only ever holds one image at a time, so it never needs more.
*/
const WORKER_HEAP_MB = 2048;
function spawn(): PooledWorker | null {
const entry = workerEntry();
/* v8 ignore next 3 -- guarded by compressionConcurrency() answering 1 when there is no entry,
which stops the caller ever asking. Kept because "no entry" and "cannot spawn" are one
fallback, and this is the one that must not throw. */
if (!entry) {
return disable("no worker entry point was found");
}
try {
const startedAt = Date.now();
const worker = fork(entry.file, {
execArgv: [ ...(entry.execArgv ?? []), `--max-old-space-size=${WORKER_HEAP_MB}` ],
// The default serialization is JSON, which has no idea what to do with an image's
// bytes; this is the structured clone a worker thread's messages always had.
serialization: "advanced"
});
const readiness: { arrived?: () => void; died?: (error: unknown) => void } = {};
const ready = new Promise<void>((resolve, reject) => {
readiness.arrived = resolve;
readiness.died = reject;
});
// A worker that dies while idle has nobody awaiting its readiness; that is not a leak
// worth a warning about an unhandled rejection.
ready.catch(() => {});
const pooled: PooledWorker = { worker, pid: worker.pid, ready };
// A process that comes up says so within a second, so a long silence here is not a slow
// boot but a hung one — and without a deadline of its own it would sit on the run for the
// whole of the reply timeout, which is sized for decodes, not for saying hello.
const bootTimer = setTimeout(() => {
getLog().info(`Image Compression Tool: worker ${pooled.pid} did not come up within `
+ `${WORKER_BOOT_TIMEOUT_MS}ms; giving up on it`);
readiness.died?.(new Error("Image worker did not come up in time."));
}, WORKER_BOOT_TIMEOUT_MS);
bootTimer.unref();
// Rare enough to say out loud without asking: a handful of lines per process, and the ones
// that answer "did a worker ever start, and did it stay up" — which is the whole question
// when a run stops making progress.
getLog().info(`Image Compression Tool: starting a worker with a `
+ `${WORKER_HEAP_MB} MB heap from ${entry.file}`);
pooled.worker.on("spawn",
() => getLog().info(`Image Compression Tool: worker ${pooled.pid} online in ${Date.now() - startedAt}ms`));
pooled.worker.on("message", (response: ImageWorkerResponse | ImageWorkerTrace) => {
clearTimeout(bootTimer);
everSpoke = true;
readiness.arrived?.();
// Traces arrive while the image is still being worked on, so they are written straight
// out rather than waited for; only the answer settles the request.
if ("trace" in response) {
getLog().info(`Image Compression Tool: ${response.trace}`);
return;
}
pooled.pending?.(response);
});
// A worker that dies takes its in-flight image with it; the image is failed, not the run.
pooled.worker.on("error", (error) => {
clearTimeout(bootTimer);
getLog().info(`Image Compression Tool: worker ${pooled.pid} errored: ${firstLine(error)}`);
readiness.died?.(error);
pooled.failed?.(error);
});
pooled.worker.on("exit", (code, signal) => {
clearTimeout(bootTimer);
// A worker we stopped ourselves exits with a failure code too, so saying so plainly is
// the difference between "this is the pool tidying up" and "something died mid-image".
getLog().info(pooled.stopped
? `Image Compression Tool: worker ${pooled.pid} stopped`
: `Image Compression Tool: worker ${pooled.pid} exited unexpectedly with code ${code ?? signal}`
+ `${pooled.pending ? " while working on an image" : " while idle"}`);
readiness.died?.(new Error(`Image worker exited with code ${code ?? signal}.`));
pooled.failed?.(new Error(`Image worker exited with code ${code ?? signal}.`));
});
// Never a reason for the pool to be what holds this process open; both handles count.
pooled.worker.unref();
pooled.worker.channel?.unref();
return pooled;
} catch (e: unknown) {
return disable(`they could not be started (${(e as Error).message})`);
}
}
/** The gist of a failure. A worker's error arrives with its stack attached, which a log line is not. */
function firstLine(error: unknown): string {
return String(error instanceof Error ? error.message : error).split(/\r?\n/)[0];
}
/**
* Gives up on workers for the rest of the process, saying so once.
*
* Once rather than per image: a run over a subtree would otherwise write the same line a thousand
* times, and the condition is a property of this installation rather than of any one image.
*/
function disable(reason: string): null {
if (!disabled) {
disabled = true;
getLog().info(
`Image Compression Tool: compressing in this process because ${reason}.`);
}
// Everyone still queued is answered, not left holding a promise for a worker that is now never
// coming. Nothing else would ever answer them — the queue is drained by workers coming free,
// and there are to be no more workers — so without this they wait for the life of the process,
// and whoever is awaiting them waits with them. An image whose compression never returns is an
// image that is never written: the note is created empty and filled in when the compression
// answers, so a promise that never settles is a note left at nothing.
answerEveryoneWaiting();
return null;
}
/** Tells every queued caller there is no worker for it, so each one compresses its own image. */
function answerEveryoneWaiting() {
while (waiting.length > 0) {
waiting.shift()?.(null);
}
}
function send(
worker: PooledWorker,
buffer: Uint8Array,
request: ImageCompressionRequest,
budgetMb: number
): Promise<ImageWorkerResponse> {
const message: ImageWorkerRequest = { id: ++lastRequestId, buffer, request, budgetMb };
const sentAt = Date.now();
trace(`sending image #${message.id} (${buffer.byteLength} bytes) to worker ${worker.pid}`
+ ` — ${started} started, ${idle.length} idle, ${waiting.length} waiting`);
return new Promise<ImageWorkerResponse>((resolve, reject) => {
// A worker that took an image and went quiet would otherwise hold the run open with nothing
// to show for it. Generous, because a large decode legitimately takes minutes; expiring is
// treated as the worker having failed, and the image is compressed here instead.
const expiry = setTimeout(() => {
getLog().info(
`Image Compression Tool: worker ${worker.pid} did not answer for image `
+ `#${message.id} within ${WORKER_REPLY_TIMEOUT_MS}ms; giving up on it`);
reject(new Error("Image worker did not answer in time."));
}, WORKER_REPLY_TIMEOUT_MS);
worker.pending = (response) => {
clearTimeout(expiry);
trace(`worker ${worker.pid} answered image #${message.id} in ${Date.now() - sentAt}ms`);
resolve(response);
};
worker.failed = (error) => {
clearTimeout(expiry);
reject(error);
};
// Only once the child is listening; a message sent into its boot is a message lost. The
// bytes are copied rather than shared either way — the caller still holds them to report
// the original size, and to fall back on if this goes wrong.
worker.ready.then(
() => worker.worker.send(message, (error) => {
if (error) {
worker.failed?.(error);
}
}),
(error: unknown) => worker.failed?.(error)
);
}).finally(() => {
worker.pending = undefined;
worker.failed = undefined;
});
}
/**
* How long a worker is given to answer before it is written off.
*
* Deliberately far longer than any decode should take, because the cost of being wrong in this
* direction is small and the cost of being wrong in the other is not. Two minutes looked generous
* against images that decode in seconds — until an image near the size ceiling, sharing the memory
* bus with three others, legitimately took longer than that. Killing it did not save any time: the
* work fell back onto the thread serving the application, blocked it for twenty seconds, and made
* the next worker time out in turn.
*
* So this is for a worker that has genuinely stopped, not for one that is taking a while. A run
* meets that at most once per worker, and the images are not waiting on anything else meanwhile.
*/
const WORKER_REPLY_TIMEOUT_MS = 15 * 60 * 1000;
/**
* How long a fresh process is given to announce itself before it is written off.
*
* Kept apart from the reply timeout above, which is sized for decodes that legitimately take
* minutes: coming up is under a second of work, and a process silent for this long past it is not
* going to speak. Generous all the same — a first spawn on a loaded machine, behind a virus
* scanner reading the binary, has taken whole seconds without anything being wrong.
*/
const WORKER_BOOT_TIMEOUT_MS = 30 * 1000;
/**
* Per-image tracing, off unless asked for: a cleanup over a large tree would otherwise write two
* lines per image into the log a user reads. Set `TRILIUM_IMAGE_WORKER_DEBUG` to follow a run image
* by image; the worker reads the same variable and traces its own side to stdout.
*/
function trace(message: string) {
if (process.env.TRILIUM_IMAGE_WORKER_DEBUG) {
getLog().info(`Image Compression Tool: ${message}`);
}
}
let lastRequestId = 0;
interface WorkerEntry {
file: string;
execArgv?: string[];
}
/**
* Where the worker's code is, which differs by how the server was started.
*
* Bundled, the worker sits beside the bundle as an entry point of its own. Running from sources it
* is the TypeScript, started with this process's own loader flags — which works wherever those
* flags are what makes TypeScript readable, and fails harmlessly wherever they are not. A loader
* that arrived through `NODE_OPTIONS` instead — the desktop's dev launcher does this — reaches the
* child through its environment, which fork passes on by itself.
*
* Answered once and remembered, including the answer that there is none: this is a question about
* an installation, and asking the filesystem it per image would be its own small waste.
*/
function workerEntry(): WorkerEntry | null {
if (resolvedEntry === undefined) {
resolvedEntry = [
{ file: join(__dirname, "services", "image_worker.cjs") },
{ file: join(__dirname, "image_worker.cjs") },
{ file: join(__dirname, "image_worker.ts"), execArgv: process.execArgv }
].find((candidate) => existsSync(candidate.file)) ?? null;
}
return resolvedEntry;
}
let resolvedEntry: WorkerEntry | null | undefined;