Skip to content

Build your personal knowledge base with Trilium Notes

docs(skill): record how to benchmark backend search

Elian Doran6273c6f93490
diff --git a/.claude/skills/benchmarking-search/SKILL.md b/.claude/skills/benchmarking-search/SKILL.md
new file mode 100644
index 0000000000..e51aabe347
--- /dev/null
+++ b/.claude/skills/benchmarking-search/SKILL.md
@@ -0,0 +1,137 @@
+---
+name: benchmarking-search
+description: Use when measuring or optimising Trilium's backend search — "why is autocomplete slow on a big database?", "where does a search spend its time?", "did this change actually make search faster?", or any before/after on `packages/trilium-core/src/services/search`. Boots core against a read-only snapshot of a real database, times a query, and attributes a CPU profile by caller or by callee. Includes the measurement discipline that separates a real win from machine noise — several plausible "wins" in this area have turned out to be drift. Don't write a new timing harness or profile parser; both live here.
+---
+
+# Benchmarking Trilium's backend search
+
+Search performance only reproduces on a real database. A synthetic fixture has uniform titles, no clones, no inherited attributes and a shallow tree, and none of the costs that dominate in practice show up. Always measure against a snapshot of a real document.
+
+| | reads | answers |
+|---|---|---|
+| `bench-search.mts` | a database snapshot | *how long does this query take, and how does that change* |
+| `bench-profile.mts` | a `.cpuprofile` it writes | *which function, called by whom, and what does it contain* |
+
+## 1. Snapshot a real database, read-only
+
+Never benchmark against a live document — and never let the harness write to one.
+
+```bash
+# macOS: ~/Library/Application Support/trilium-data ; Linux: ~/.local/share/trilium-data
+cd ~/Library/Application\ Support/trilium-data
+sqlite3 "file:document.db?mode=ro" ".backup '/path/to/bench/document.db'"
+```
+
+`.backup` uses SQLite's backup API, so it is safe while the app is running and handles the WAL. The source is opened `mode=ro` and never written. A 1.9 GB document copies in about seven seconds.
+
+The snapshot must be named `document.db` and sit in a directory of its own — that directory becomes `TRILIUM_DATA_DIR`.
+
+## 2. Run the benchmark
+
+`bench-search.mts` has to live under `apps/server/` to run: `@triliumnext/core` is a workspace package that pnpm links only into `apps/server/node_modules`, and ESM resolves bare specifiers from the importing file's own location.
+
+```bash
+cp .claude/skills/benchmarking-search/bench-search.mts apps/server/
+
+TRILIUM_RESOURCE_DIR=$PWD/apps/server/src \
+TRILIUM_DATA_DIR=/path/to/bench \
+TRILIUM_GENERAL_READONLY=true \
+TRILIUM_GENERAL_NOBACKUP=true \
+BENCH_ITERATIONS=40 \
+ node --import tsx apps/server/bench-search.mts a t 'the quick brown fox'
+
+rm apps/server/bench-search.mts # when finished
+```
+
+| variable | why |
+|---|---|
+| `TRILIUM_RESOURCE_DIR` | assets live under `src/assets` in dev; without it `resource_dir.ts` calls `process.exit(1)` at import time |
+| `TRILIUM_GENERAL_READONLY` | the harness **refuses to start** without it, so a snapshot is never written to |
+| `TRILIUM_GENERAL_NOBACKUP` | stops the backup service touching the snapshot directory |
+| `BENCH_ITERATIONS` | default 5. Use 40 for anything you intend to act on |
+| `BENCH_PROFILE` | write a `.cpuprofile` covering only the search calls |
+| `BENCH_AUTOCOMPLETE` | `=1` sets `autocomplete: true`, taking `NoteFlatTextExp`'s single-token path |
+| `BENCH_ONE_PASS` | `=1` disables two-pass ranking, i.e. the pre-`ab6dc34c15` behaviour |
+
+It boots core exactly as `apps/server/src/main.ts` does, minus the HTTP server, so the code under test is the real thing.
+
+## 3. Profile and attribute
+
+```bash
+BENCH_PROFILE=/tmp/a.cpuprofile ... node --import tsx apps/server/bench-search.mts a
+
+# top functions by self time
+node --import tsx .claude/skills/benchmarking-search/bench-profile.mts /tmp/a.cpuprofile 15
+
+# who calls removeDiacritic, walking past thin wrappers
+node --import tsx .claude/skills/benchmarking-search/bench-profile.mts /tmp/a.cpuprofile 8 \
+ removeDiacritic "normalize,normalizeSearchText,tokenizeIntoWords"
+
+# what NoteFlatTextExp.execute spends its time on, by callee
+node --import tsx .claude/skills/benchmarking-search/bench-profile.mts /tmp/a.cpuprofile 10 \
+ "children:execute@note_flat_text"
+```
+
+The skip list matters. `removeDiacritic` is called ~100% from `normalize`, which is called ~100% from `normalizeSearchText` — attributing one level up tells you nothing. Skipping those names lands the blame on code that *chose* to normalize.
+
+The `@file` suffix on `children:` narrows to one implementation; `execute` is a method on every expression type.
+
+## 4. Measurement discipline
+
+**This is the most important section.** Search work in this repo has repeatedly produced plausible numbers that were wrong.
+
+**Calibrate before trusting anything.** Run the *same code* three times. Observed on a developer machine running Trilium alongside: **±10% at 12 iterations, ±4-5% at 40**. Anything smaller than the floor is unresolvable, and reporting it as a win is a mistake.
+
+**Never compare measurements taken minutes apart.** The machine drifts. One change here was recorded at 189 ms, and the identical code measured 210 ms later in the same session. Always measure both arms **back to back**, ideally in one command.
+
+**Prefer phase timers to wall clock.** They are in-process, unsampled, and immune to profiler overhead. They are not in the codebase — add them temporarily around the stages of `performSearch`:
+
+```ts
+const t0 = performance.now();
+const noteSet = expression.execute(allNoteSet, executionContext, searchContext);
+const executeMs = performance.now() - t0;
+// ... same around result construction, the computeScore loop, and the sort
+getLog().info(`phases: execute=${executeMs.toFixed(1)} build=${buildMs.toFixed(1)} ...`);
+```
+
+Then take the **minimum of each phase independently** across iterations — the least-contended estimate of each — rather than the phases of the best single iteration.
+
+Phase timers catch what wall clock hides. A per-search cache keyed on `` `${noteId}-${parentNoteId}` `` showed a *consistent 4-8% wall-clock improvement across every query*, and the phase it targeted had gone **58.4 ms → 62.1 ms, i.e. slower**. The tell: `executeMs` had also "improved", and the change could not possibly touch `executeMs`. **If a phase your change cannot reach appears to move, you are reading noise.**
+
+**Self time undercounts.** Ranking leads by self time led to sizing one at ~23 ms when its subtree was 42.9 ms. Use `children:` for anything whose cost is in its callees.
+
+**Strings are the usual culprit.** Almost every real win here was an allocation removed, not an algorithm improved: a template literal built per call, a `Map` key concatenated per lookup, `String.normalize("NFD")` re-run over a constant. Prefer nested maps over combined string keys — `Map<parent, Map<child, V>>` allocates nothing per lookup.
+
+## 5. Where the time goes
+
+For a query matching most of the database (a single letter matches ~84% of a 22k-note document, since flat text includes `noteId`, `type` and `mime`):
+
+| phase | what happens | typical share |
+|---|---|---|
+| `execute` | `getCandidateNotes` scans every flat text; `searchPathTowardsRoot` resolves a note path per candidate | ~50% |
+| `build` | a `SearchResult` per match, each resolving its path segment titles | ~19% |
+| `score` | `computeScore` per match: title, path and content contributions | ~27% |
+| `sort` | full sort; the comparator's `notePathTitle` tiebreak fires constantly on equal scores | ~6% |
+
+Every phase is **O(matches)**. That is the ceiling: no micro-optimisation halves a query that matches 18,000 notes. Reaching a large win means processing fewer results — which is what two-pass ranking does — or matching fewer notes, which is a product decision.
+
+The slowest shape is not the biggest result set but **zero matches with several tokens**: a 5-token query with no matches costs ~650 ms, because fewer than five good results triggers the progressive search's second pass and the whole scan runs twice, with cost linear in token count. That is the shape behind issue #10712.
+
+## 6. What has already been done
+
+Eight commits on `perf/autocomplete-debounce` (PR #11542), each with its measurements in the commit body. Read those before re-treading:
+
+- **Two-pass ranking** — score without the path, shortlist, then fully score the shortlist. −43% on large result sets; the only structural win.
+- **Result cap** at 25 — detailing went 155-309 ms → 21-50 ms, response 212 KB → 16-30 KB.
+- **Client debounce** rewritten — was `notesCount / 20` ms capped at a second, measured from the last *search* rather than the last keystroke.
+- Four commits removing per-result re-derivation of query-side and note-side strings.
+
+Tried and rejected, with numbers, so they are not retried blindly:
+
+| | why |
+|---|---|
+| Reordering the archived filter after the fulltext match | broke 3 `parse.spec` assertions; `execute` +6 ms |
+| Per-search `getAllNotePaths` cache | copy-on-extend allocation exactly cancelled the sharing |
+| Lazy `notePathTitle` alone | `build` −16%, `sort` **+68%** — the sort tiebreak reads it |
+| Lazy `notePathArray` | broke 5 tests for ~2% |
+| `autocomplete: true` (the single-token fast path) | faster on 1 char, **slower** on 2+, and it changes results — it keeps candidates matching only via inherited attributes |
diff --git a/.claude/skills/benchmarking-search/bench-profile.mts b/.claude/skills/benchmarking-search/bench-profile.mts
new file mode 100644
index 0000000000..2e8a734955
--- /dev/null
+++ b/.claude/skills/benchmarking-search/bench-profile.mts
@@ -0,0 +1,185 @@
+/**
+ * Aggregates a V8 `.cpuprofile` by self time, so the output names the functions actually burning
+ * the CPU rather than the call tree that reached them.
+ *
+ * node --import tsx apps/server/bench-profile.mts <profile.cpuprofile> [topN]
+ */
+
+import fs from "fs";
+import path from "path";
+
+interface CallFrame {
+ functionName: string;
+ url: string;
+ lineNumber: number;
+}
+
+interface ProfileNode {
+ id: number;
+ callFrame: CallFrame;
+ children?: number[];
+}
+
+interface Profile {
+ nodes: ProfileNode[];
+ samples: number[];
+ timeDeltas: number[];
+}
+
+const [ profilePath, topArg ] = process.argv.slice(2);
+const TOP = Number(topArg ?? 30);
+
+if (!profilePath) {
+ console.error("usage: bench-profile.mts <profile.cpuprofile> [topN]");
+ process.exit(1);
+}
+
+const profile: Profile = JSON.parse(fs.readFileSync(profilePath, "utf-8"));
+const nodesById = new Map(profile.nodes.map((node) => [ node.id, node ]));
+
+// Self time per node: a sample is attributed to the frame that was executing when it was taken.
+const selfMicrosByNode = new Map<number, number>();
+let totalMicros = 0;
+
+for (const [ index, nodeId ] of profile.samples.entries()) {
+ const delta = profile.timeDeltas[index] ?? 0;
+ if (delta <= 0) {
+ continue;
+ }
+
+ selfMicrosByNode.set(nodeId, (selfMicrosByNode.get(nodeId) ?? 0) + delta);
+ totalMicros += delta;
+}
+
+function label(frame: CallFrame) {
+ const name = frame.functionName || "(anonymous)";
+ if (!frame.url) {
+ return name;
+ }
+
+ const file = frame.url.startsWith("file://") ? path.relative(process.cwd(), frame.url.slice(7)) : frame.url;
+ return `${name} ${file}:${frame.lineNumber + 1}`;
+}
+
+// Several nodes can share a call frame (the same function reached by different paths), so fold them.
+const selfMicrosByFrame = new Map<string, number>();
+
+for (const [ nodeId, micros ] of selfMicrosByNode) {
+ const node = nodesById.get(nodeId);
+ if (!node) {
+ continue;
+ }
+
+ const key = label(node.callFrame);
+ selfMicrosByFrame.set(key, (selfMicrosByFrame.get(key) ?? 0) + micros);
+}
+
+const attributeTo = process.argv[4] ?? "";
+const skip = new Set((process.argv[5] ?? "").split(",").filter(Boolean));
+
+// "children:<fn>" breaks that function's subtree down by direct callee, which says where a phase
+// spends its time rather than which leaf is hottest.
+if (attributeTo.startsWith("children:")) {
+ // "children:execute@note_flat_text" narrows to one file, since a name like `execute` is
+ // implemented by every expression type.
+ const [ target, inFile ] = attributeTo.slice("children:".length).split("@");
+ const matches = (frame: CallFrame) =>
+ frame.functionName === target && (!inFile || frame.url.includes(inFile));
+
+ const totalOf = new Map<number, number>();
+ const totalFor = (id: number): number => {
+ const cached = totalOf.get(id);
+ if (cached !== undefined) {
+ return cached;
+ }
+
+ const node = nodesById.get(id);
+ let total = selfMicrosByNode.get(id) ?? 0;
+ for (const child of node?.children ?? []) {
+ total += totalFor(child);
+ }
+
+ totalOf.set(id, total);
+ return total;
+ };
+
+ const byChild = new Map<string, number>();
+ let ownSelf = 0;
+ let subtree = 0;
+
+ for (const node of profile.nodes) {
+ if (!matches(node.callFrame)) {
+ continue;
+ }
+
+ ownSelf += selfMicrosByNode.get(node.id) ?? 0;
+ subtree += totalFor(node.id);
+
+ for (const child of node.children ?? []) {
+ const childNode = nodesById.get(child);
+ if (!childNode) {
+ continue;
+ }
+
+ const key = label(childNode.callFrame);
+ byChild.set(key, (byChild.get(key) ?? 0) + totalFor(child));
+ }
+ }
+
+ console.log(`${target}: ${(subtree / 1000).toFixed(1)}ms subtree, ${(ownSelf / 1000).toFixed(1)}ms self\n`);
+ for (const [ name, micros ] of [ ...byChild.entries() ].sort((a, b) => b[1] - a[1]).slice(0, TOP)) {
+ console.log(`${(micros / 1000).toFixed(1).padStart(9)} ${((micros / subtree) * 100).toFixed(1).padStart(5)} ${name}`);
+ }
+ process.exit(0);
+}
+
+// With a function name as the third argument, attribute that function's self time to whoever
+// called it, which says why a hot leaf is hot rather than merely that it is.
+
+if (attributeTo) {
+ const parentOf = new Map<number, number>();
+ for (const node of profile.nodes) {
+ for (const child of node.children ?? []) {
+ parentOf.set(child, node.id);
+ }
+ }
+
+ const byCaller = new Map<string, number>();
+ let attributed = 0;
+
+ for (const [ nodeId, micros ] of selfMicrosByNode) {
+ const node = nodesById.get(nodeId);
+ if (node?.callFrame.functionName !== attributeTo) {
+ continue;
+ }
+
+ // Walk past thin wrappers so the blame lands on code that chose to normalize, not on the
+ // one-line helper it went through.
+ let ancestor = parentOf.get(nodeId);
+ while (ancestor !== undefined && skip.has(nodesById.get(ancestor)?.callFrame.functionName ?? "")) {
+ ancestor = parentOf.get(ancestor);
+ }
+
+ const parent = ancestor === undefined ? undefined : nodesById.get(ancestor);
+ const key = parent ? label(parent.callFrame) : "(root)";
+ byCaller.set(key, (byCaller.get(key) ?? 0) + micros);
+ attributed += micros;
+ }
+
+ console.log(`${attributeTo}: ${(attributed / 1000).toFixed(1)}ms self time, by caller\n`);
+ for (const [ name, micros ] of [ ...byCaller.entries() ].sort((a, b) => b[1] - a[1]).slice(0, TOP)) {
+ console.log(`${(micros / 1000).toFixed(1).padStart(9)} ${((micros / attributed) * 100).toFixed(1).padStart(5)} ${name}`);
+ }
+ process.exit(0);
+}
+
+const ranked = [ ...selfMicrosByFrame.entries() ].sort((a, b) => b[1] - a[1]).slice(0, TOP);
+
+console.log(`total sampled: ${(totalMicros / 1000).toFixed(1)}ms across ${profile.samples.length} samples\n`);
+console.log(`${"self ms".padStart(9)} ${"%".padStart(5)} function`);
+
+for (const [ name, micros ] of ranked) {
+ const ms = (micros / 1000).toFixed(1).padStart(9);
+ const share = ((micros / totalMicros) * 100).toFixed(1).padStart(5);
+ console.log(`${ms} ${share} ${name}`);
+}
diff --git a/.claude/skills/benchmarking-search/bench-search.mts b/.claude/skills/benchmarking-search/bench-search.mts
new file mode 100644
index 0000000000..fe76344a65
--- /dev/null
+++ b/.claude/skills/benchmarking-search/bench-search.mts
@@ -0,0 +1,148 @@
+/**
+ * Search benchmark against a real database.
+ *
+ * Boots core exactly as `main.ts` does, minus the HTTP server, and opens whatever database
+ * `TRILIUM_DATA_DIR` points at. Run it read-only against a snapshot, never a live document:
+ *
+ * TRILIUM_RESOURCE_DIR=$PWD/apps/server/src TRILIUM_DATA_DIR=<dir> \
+ * TRILIUM_GENERAL_READONLY=true TRILIUM_GENERAL_NOBACKUP=true \
+ * node --import tsx apps/server/bench-search.mts <query>...
+ */
+
+import fs from "fs";
+import { Session } from "inspector";
+import path from "path";
+import { promisify } from "util";
+
+const QUERIES = process.argv.slice(2).length > 0 ? process.argv.slice(2) : [ "a", "s", "t", "te", "tec", "tech" ];
+const ITERATIONS = Number(process.env.BENCH_ITERATIONS ?? 5);
+
+/** `apps/server` builds to CommonJS, so a default export arrives wrapped when tsx loads it here. */
+async function loadDefault<T>(specifier: string): Promise<T> {
+ const mod: any = await import(specifier);
+ const outer = mod.default ?? mod;
+ return (outer?.default ?? outer) as T;
+}
+
+async function boot() {
+ const { initializeCore, options } = await import("@triliumnext/core");
+ const config = await loadDefault<any>("./src/services/config.js");
+ const dataDirs = await loadDefault<any>("./src/services/data_dir.js");
+
+ if (!config.General.readOnly) {
+ throw new Error("Refusing to run: set TRILIUM_GENERAL_READONLY=true so the snapshot is never written to.");
+ }
+
+ const BetterSqlite3Provider = await loadDefault<any>("./src/sql_provider.js");
+ const ServerLogService = await loadDefault<any>("./src/log_provider.js");
+ const { RESOURCE_DIR } = await import("./src/services/resource_dir.js");
+ const { consumeSetupMarker, setupPlatform } = await import("./src/services/setup_marker.js");
+
+ const dbProvider = new BetterSqlite3Provider();
+ dbProvider.loadFromFile(dataDirs.DOCUMENT_PATH, true);
+
+ await initializeCore({
+ dbConfig: { provider: dbProvider, isReadOnly: true, onTransactionCommit() {}, onTransactionRollback() {} },
+ crypto: new (await loadDefault<any>("./src/crypto_provider.js"))(),
+ zip: new (await loadDefault<any>("./src/zip_provider.js"))(),
+ zipExportProviderFactory: (await import("./src/services/export/zip/factory.js")).serverZipExportProviderFactory,
+ request: new (await loadDefault<any>("./src/services/request.js"))(),
+ executionContext: new (await loadDefault<any>("./src/cls_provider.js"))(),
+ messaging: new (await loadDefault<any>("./src/services/ws_messaging_provider.js"))(),
+ schema: (await import("./src/core_assets.js")).loadCoreSchema(),
+ platform: new (await loadDefault<any>("./src/platform_provider.js"))(),
+ log: new ServerLogService(),
+ translations: (await import("./src/services/i18n.js")).initializeTranslationsWithParams,
+ getDemoArchive: async () => fs.readFileSync(path.join(RESOURCE_DIR, "db", "demo.zip")),
+ inAppHelp: new (await loadDefault<any>("./src/in_app_help_provider.js"))(),
+ backup: new (await loadDefault<any>("./src/backup_provider.js"))(options),
+ image: (await import("./src/services/image_provider.js")).serverImageProvider,
+ config,
+ setupMarker: consumeSetupMarker(),
+ setupPlatform,
+ extraAppInfo: { nodeVersion: process.version, dataDirectory: path.resolve(dataDirs.TRILIUM_DATA_DIR) }
+ } as any);
+}
+
+function summarise(label: string, samples: number[]) {
+ const sorted = [ ...samples ].sort((a, b) => a - b);
+ const sum = sorted.reduce((total, value) => total + value, 0);
+ return `${label.padEnd(9)} min=${sorted[0].toFixed(1).padStart(7)} med=${sorted[Math.floor(sorted.length / 2)].toFixed(1).padStart(7)}`
+ + ` max=${sorted[sorted.length - 1].toFixed(1).padStart(7)} mean=${(sum / sorted.length).toFixed(1).padStart(7)}`;
+}
+
+/** Samples only the search calls, so booting and becca loading stay out of the profile. */
+async function withProfile<T>(outPath: string | undefined, run: () => Promise<T>): Promise<T> {
+ if (!outPath) {
+ return run();
+ }
+
+ const session = new Session();
+ session.connect();
+ const post = promisify(session.post.bind(session)) as (method: string, params?: unknown) => Promise<any>;
+
+ await post("Profiler.enable");
+ await post("Profiler.setSamplingInterval", { interval: 100 });
+ await post("Profiler.start");
+
+ try {
+ return await run();
+ } finally {
+ const { profile } = await post("Profiler.stop");
+ fs.writeFileSync(outPath, JSON.stringify(profile));
+ session.disconnect();
+ console.log(`\nprofile written to ${outPath}`);
+ }
+}
+
+async function main() {
+ const bootAt = performance.now();
+ await boot();
+
+ const { becca, becca_loader, cls, search: searchService, SearchContext } = await import("@triliumnext/core");
+
+ if (Object.keys(becca.notes).length === 0) {
+ cls.init(() => becca_loader.load());
+ }
+ console.log(`booted in ${(performance.now() - bootAt).toFixed(0)}ms, ${Object.keys(becca.notes).length} notes in becca\n`);
+
+ await withProfile(process.env.BENCH_PROFILE, async () => {
+ for (const query of QUERIES) {
+ const samples: number[] = [];
+ let matched = 0;
+
+ for (let i = 0; i < ITERATIONS; i++) {
+ cls.init(() => {
+ const context = new SearchContext({
+ fastSearch: true,
+ includeArchivedNotes: false,
+ includeHiddenNotes: true,
+ fuzzyAttributeSearch: true,
+ ignoreInternalAttributes: true,
+ ancestorNoteId: "root",
+ // BENCH_AUTOCOMPLETE=1 takes NoteFlatTextExp's single-token path, which skips
+ // the recursive parent walk and keeps every flat-text candidate.
+ autocomplete: process.env.BENCH_AUTOCOMPLETE === "1",
+ // Mirrors `searchNotesForAutocomplete`.
+ rankInTwoPasses: process.env.BENCH_ONE_PASS !== "1"
+ });
+
+ const startedAt = performance.now();
+ const results = searchService.findResultsWithQuery(query, context);
+ samples.push(performance.now() - startedAt);
+ matched = results.length;
+ });
+ }
+
+ console.log(`${summarise(JSON.stringify(query), samples)} matched=${matched}`);
+ }
+ });
+}
+
+main().then(
+ () => process.exit(0),
+ (error) => {
+ console.error(error);
+ process.exit(1);
+ }
+);