Skip to content

Build your personal knowledge base with Trilium Notes

docs(dev): document sync tombstones, zip libraries and cross-process cache divergence

Elian Doran3e478cd70803
diff --git a/docs/Developer Guide/!!!meta.json b/docs/Developer Guide/!!!meta.json
index 6f4b2cbbad..3ad6328f32 100644
--- a/docs/Developer Guide/!!!meta.json
+++ b/docs/Developer Guide/!!!meta.json
@@ -1543,6 +1543,33 @@
]
}
]
+ },
+ {
+ "isClone": false,
+ "noteId": "dEpZ1pLibs4Z",
+ "notePath": [
+ "jdjRLhLV3TtI",
+ "qalhAaJoQ7AN",
+ "dEpZ1pLibs4Z"
+ ],
+ "title": "Compression libraries",
+ "notePosition": 30,
+ "prefix": null,
+ "isExpanded": false,
+ "type": "text",
+ "mime": "text/html",
+ "attributes": [
+ {
+ "type": "label",
+ "name": "shareAlias",
+ "value": "compression-libraries",
+ "isInheritable": false,
+ "position": 20
+ }
+ ],
+ "format": "markdown",
+ "dataFileName": "Compression libraries.md",
+ "attachments": []
}
]
},
@@ -2769,6 +2796,34 @@
"format": "markdown",
"dataFileName": "Content hashing.md",
"attachments": []
+ },
+ {
+ "isClone": false,
+ "noteId": "sYnCT0mbst1E",
+ "notePath": [
+ "jdjRLhLV3TtI",
+ "yeqU0zo0ZQ83",
+ "n9wYW9nUTynV",
+ "sYnCT0mbst1E"
+ ],
+ "title": "Entity change tombstones",
+ "notePosition": 20,
+ "prefix": null,
+ "isExpanded": false,
+ "type": "text",
+ "mime": "text/html",
+ "attributes": [
+ {
+ "type": "label",
+ "name": "shareAlias",
+ "value": "entity-change-tombstones",
+ "isInheritable": false,
+ "position": 20
+ }
+ ],
+ "format": "markdown",
+ "dataFileName": "Entity change tombstones.md",
+ "attachments": []
}
]
},
@@ -2962,6 +3017,33 @@
"format": "markdown",
"dataFileName": "Error [TransformError] The package @esbuildlinux-x64 could not be found, and is needed by esbuild.md",
"attachments": []
+ },
+ {
+ "isClone": false,
+ "noteId": "trBshtRcNt5X",
+ "notePath": [
+ "jdjRLhLV3TtI",
+ "YjerxU7Aii8X",
+ "trBshtRcNt5X"
+ ],
+ "title": "A note is listed in Recent Changes but missing from the tree",
+ "notePosition": 30,
+ "prefix": null,
+ "isExpanded": false,
+ "type": "text",
+ "mime": "text/html",
+ "attributes": [
+ {
+ "type": "label",
+ "name": "shareAlias",
+ "value": "recent-changes-missing-note",
+ "isInheritable": false,
+ "position": 20
+ }
+ ],
+ "format": "markdown",
+ "dataFileName": "A note is listed in Recent Changes but missing from the tree.md",
+ "attachments": []
}
]
},
diff --git a/docs/Developer Guide/Developer Guide/Concepts/Synchronisation/Entity change tombstones.md b/docs/Developer Guide/Developer Guide/Concepts/Synchronisation/Entity change tombstones.md
new file mode 100644
index 0000000000..cc35092a67
--- /dev/null
+++ b/docs/Developer Guide/Developer Guide/Concepts/Synchronisation/Entity change tombstones.md
@@ -0,0 +1,48 @@
+# Entity change tombstones
+When an entity is erased, its row in `entity_changes` is **not** deleted — it is flagged. `erase#setEntityChangesAsErased` sets `isErased = true`, refreshes `utcDateChanged` and re-pushes the row through `putEntityChangeWithForcedChange`. The tombstone is what tells every peer "this entity is gone"; without it, a peer that still holds the entity would simply push it back.
+
+## Tombstones are never collected
+
+There is no garbage collection for them. There is no cluster-wide acknowledged-id watermark, no TTL, and no compaction pass. The `entity_changes` table therefore grows monotonically with **lifetime deletion activity**, not with the amount of data a user currently has.
+
+Two things amplify this:
+
+* **Deletions cascade.** Erasing a note tombstones its branches, attributes and revisions too (`erase#eraseNotes`), so the count tracks edit-and-delete volume rather than note count.
+* **Full initial sync replays the entire history.** A fresh client pulls every row, tombstones included. There is no snapshot or bootstrap path that would let it start from the current state.
+
+A real database observed in 2026 held ~165 000 entity changes against only ~12 000 live entities (1 954 notes, 7 200 attributes, 644 attachments) — roughly **85 % tombstones**.
+
+## Why they cannot simply be deleted
+
+`content_hash#getEntityHashes` folds the flag into the per-sector hash:
+
+```js
+entityHashMap[sector] = (entityHashMap[sector] || "") + hash + isErased;
+```
+
+Erased rows are therefore part of the hash a peer compares against. Deleting tombstones on one instance diverges its sector hashes from any peer that still has them, `checkContentHashes` then fails on nearly every sector, and `addEntityChangesForSector` re-pushes **all** rows for those sectors — including the erased ones, because `putEntityChange` preserves `isErased`. The tombstones come back.
+
+A manual `DELETE FROM entity_changes WHERE isErased = 1` only sticks if it is done on every instance at once, or if the server is the sole surviving source and every client is wiped or brand new (a fresh device has no tombstones to push back). Back up first, and only do it with all clients caught up. It is id-safe: `entity_changes` uses an `AUTOINCREMENT` high-water mark, so ids are never reused.
+
+## The built-in options do not help
+
+* **`fillEntityChanges`** runs `DELETE FROM entity_changes WHERE isErased = 0` and rebuilds the live rows. Tombstones are untouched, and the rebuilt rows get new ids, which makes peers re-pull them.
+* **`forceFullSync`** resets `lastSyncedPull`/`lastSyncedPush` to 0 and re-pulls everything, tombstones included. It makes the problem worse.
+
+## Blobs are the one exception
+
+The same wall was already hit for blobs and special-cased in `erase#eraseUnusedBlobs`, which purges their change rows outright instead of tombstoning them:
+
+```js
+// blobs are not marked as erased in entity_changes, they are just purged completely
+// this is because technically every keystroke can create a new blob and there would be just too many
+sql.executeMany(`DELETE FROM entity_changes WHERE entityName = 'blobs' AND entityId IN (???)`, unusedBlobIds);
+```
+
+That exception was never generalised, so every other entity type still accumulates.
+
+## Where it actually bites
+
+The cost is invisible on native clients — better-sqlite3 handles a few hundred thousand rows without complaint. It is only fatal on the WASM targets (standalone in the browser, and the Capacitor mobile app built on it), where SQLite runs synchronously and a long transaction blocks the thread. This is latent debt that the newer targets exposed rather than created.
+
+Any real fix has to address the hash coupling above. The plausible directions are a per-instance acknowledged-id watermark that permits coordinated collection, a snapshot/bootstrap path so fresh clients do not replay history, or generalising the blob purge to other high-churn entity types.
diff --git a/docs/Developer Guide/Developer Guide/Dependencies/Compression libraries.md b/docs/Developer Guide/Developer Guide/Dependencies/Compression libraries.md
new file mode 100644
index 0000000000..698c07a5c2
--- /dev/null
+++ b/docs/Developer Guide/Developer Guide/Dependencies/Compression libraries.md
@@ -0,0 +1,24 @@
+# Compression libraries
+The repository ships **three** independent ZIP implementations. This looks like an obvious consolidation target and is not one — each is anchored by a constraint the others cannot satisfy. This page exists so the analysis is not repeated.
+
+| Library | Declared by | Used for |
+|---|---|---|
+| `jszip` | `packages/commons` | The XLSX container, via `exceljs`; also `renderSpreadsheetToCsvZip` |
+| `fflate` | `apps/standalone` | The browser note `ZipProvider` (`lightweight/zip_provider.ts`) |
+| `archiver` + `yauzl` | `apps/server` | The Node `ZipProvider` used by server and desktop |
+
+## jszip cannot be removed
+
+`exceljs` — commons' XLSX read/write engine and a genuine runtime dependency — hard-depends on jszip for the XLSX zip container. `pnpm why jszip` resolves it as `jszip → exceljs → @triliumnext/commons`. Any bundle that reads or writes spreadsheets pulls jszip in regardless of what else changes.
+
+`commons/src/lib/spreadsheet/render_to_csv.ts` also uses jszip directly. Switching that one call site to fflate does **not** remove jszip (exceljs still drags it in) and is mildly counterproductive: it would add fflate to the server bundle, where jszip is already present, in a feature that has therefore already paid for jszip.
+
+## fflate is not interchangeable with it either
+
+fflate is the deliberate lighter choice for the browser: roughly 30 KB against jszip's ~95 KB, in a bundle where that matters, with custom CP437 filename handling for the standalone note ZipProvider. Replacing it with jszip would grow the standalone bundle to remove a dependency that cannot be removed anyway.
+
+## A trap if jszip is ever dropped
+
+`packages/commons` is intentionally **browser-pure** — its library `tsconfig` does not include `@types/node`; only the spec `tsconfig` does. jszip's `index.d.ts` carries a `/// <reference types="node" />`, which has been silently supplying the `Buffer` global type to `parse_from_xlsx.ts`.
+
+If jszip ever does leave commons, `bytesToBase64`'s reference to `Buffer` breaks. The fix is to reach it through `globalThis` rather than re-adding `@types/node`, which would undo the browser-purity constraint.
diff --git a/docs/Developer Guide/Developer Guide/Troubleshooting/A note is listed in Recent Changes but missing from the tree.md b/docs/Developer Guide/Developer Guide/Troubleshooting/A note is listed in Recent Changes but missing from the tree.md
new file mode 100644
index 0000000000..bfda750963
--- /dev/null
+++ b/docs/Developer Guide/Developer Guide/Troubleshooting/A note is listed in Recent Changes but missing from the tree.md
@@ -0,0 +1,40 @@
+# A note is listed in Recent Changes but missing from the tree
+## Symptom
+
+A note that was definitely created does not appear in the tree. Recent Changes lists it, but shows it as **Deleted**, and hovering or trying to restore it fails with:
+
+```
+NotFoundError: Deleted note 'X' was not found.
+```
+
+The row is live in the database. Restarting the application makes the note appear normally.
+
+This looks like Becca or Froca corruption. It usually is not.
+
+## Cause: two processes on one data directory
+
+Entity-change propagation is **in-process only**. A note created by process A goes into A's Becca and into the shared SQLite file — but process B's Becca never hears about it, because the notification never leaves A.
+
+The two views then disagree, in a way that produces exactly the symptom above:
+
+* **Recent Changes reads the database directly.** `routes/api/recent_changes.ts` queries `notes` by raw SQL, so it sees the new row.
+* **The tree and `tree/load` read Becca**, which in process B has no such note. The client renders any entry it cannot resolve through Froca as a deleted-note link.
+* **Hovering that link calls `/api/deleted-notes/:noteId/metadata`**, which selects `... WHERE noteId = ? AND isDeleted = 1` and throws `NotFoundError` when it finds nothing — and it finds nothing precisely because the note is *alive*.
+
+Restarting the second instance reloads Becca from disk, which is why the note then appears.
+
+## Confirming it
+
+Check for more than one Trilium process bound to the same data directory. On Windows:
+
+```powershell
+Get-NetTCPConnection -LocalPort 37840,37841 -State Listen | Select-Object LocalPort, OwningProcess
+```
+
+then match the owning PIDs against the running executables. An installed build and a locally built desktop app both default to `%APPDATA%\trilium-data`, so the two share `document.db` without either warning about it. They also share the daily log file, and log lines carry no PID — so one process's output is indistinguishable from the other's, which is worth knowing before reading the log for clues.
+
+The same applies to any automation pointed at a running instance: a REST or MCP client talks to whichever process owns that port, and its writes land in that process's Becca — not in the window being watched.
+
+## Avoiding it
+
+Give each instance its own data directory (`TRILIUM_DATA_DIR`) whenever two are running at once. The development server already does this by default (`apps/server/data`); the installed application and a locally built desktop app do not.