Skip to content

Soul-driven AI agent with permission-hardened tools, token budgets, and multi-channel access. Runs 24/7 from CLI, Telegram or More.

docs: ADR-018 SQLite backend selection; 1.3.1 changelog entry

Salman Qureshi0428945f4bcf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 65bf927b..3000c41a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,10 @@
- **SSRF guard** — unchanged; see the 1.2.3 entry below (hardened in 1.3.0 by #121 and #124; response record in `docs/security/ssrf-108-response.md`).
- **Disclosure process** — `SECURITY.md` added (private reporting, response targets, scope); issue-closure and advisory drafts in `docs/security/2026-10-backlog-closure.md`.
+### Fixed
+
+- **SQLite in the standalone binaries and on toolchain-less installs** (ROADMAP P1.9, #96, ADR-018) — the binaries bundled better-sqlite3's JavaScript but never its native addon, so every SQLite probe failed inside them: Second Brain was silently disabled and bots/boards fell back to JSON. A shared driver (`src/utils/sqlite-driver.ts`) now selects **better-sqlite3 → bun:sqlite → node:sqlite → JSON**: binaries use Bun's embedded engine, Node ≥ 22.13 installs work without a compiler via `node:sqlite`, and the better-sqlite3 path is untouched. Second Brain, the bots queue, boards and the cloud pool-search cache all go through it; the pool cache now degrades to uncached instead of throwing when no engine loads. New `mercury doctor --storage` prints the active backend (with why the others were rejected), every database file, its size and row counts, and warns loudly when storage is JSON-only. `MERCURY_SQLITE_BACKEND=better-sqlite3|bun:sqlite|node:sqlite|json` forces a backend. Contract tests run the same assertions against every engine the process can load (`src/utils/sqlite-driver.test.ts`).
+
## 1.3.0 — Mercury Bots
Persistent, persona-scoped agents that run **outside your conversation** — each with its own persona, workspace, skills, permissions, and durable queue. Onboard specialists, chain them into multi-level fleets, and let them work 24/7 while you keep talking to Mercury.
diff --git a/DECISIONS.md b/DECISIONS.md
index 91446b27..a06ee201 100644
--- a/DECISIONS.md
+++ b/DECISIONS.md
@@ -135,3 +135,13 @@
- **Context**: PR #126 (SeashoreShi) fixes issue #122: macOS release builds produced `web.tar.gz` containing an AppleDouble entry (`._web`) because `web/` carries xattrs. bsdtar hides the entry from `tar -tzf` so build-time verification never saw it, but GNU tar lists it and `install.sh`'s `^web/` path guard rejected the asset — breaking `mercury update` on Linux. The PR also upgrades sub-agent `allowedTools` from advisory (prompt text only — the model still received every tool) to a hard runtime filter via a new `filterToolsByAllowlist` util.
- **Decision**: Accept and maintain. Verified empirically on macOS before merge: an xattr-bearing `web/` directory produced `._web` in the archive without `COPYFILE_DISABLE=1` and nothing with it. Full suite green (487 tests), `tsc --noEmit` clean. The allowlist change is a security positive (defense in depth — children can no longer invoke tools outside their allowlist, e.g. `list_agents`/`stop_agent`); the runtime filter subsumes the prompt-text filtering in `sub-agent.ts` line 516, which remains as harmless documentation. Merge commit authored by Mercury, feature commits retain the original author.
- **Consequence**: Release archives stay safe for GNU tar consumers; sub-agent tool scoping is now enforced at the SDK boundary rather than trusted to the model's cooperation. Maintenance owned by Mercury: any new tool surface exposed to sub-agents must respect `resolveTools()` (not `capabilities.getTools()` directly), and release builds on macOS must keep `COPYFILE_DISABLE=1` — re-verify archive contents with GNU tar or the `verify-standalone-release.cjs` path check before publishing.
+
+## ADR-018: SQLite backend selection: better-sqlite3 → bun:sqlite → node:sqlite → JSON
+
+- **Context**: Four stores depend on SQLite — Second Brain (`src/memory/second-brain-db.ts`), the bots job queue (`src/bots/queue.ts`), boards (`src/core/board-db.ts`) and the cloud pool-search cache (`src/cloud/pool-search.ts`). Each carried its own `require('better-sqlite3')` probe. The standalone binaries (`bun build --compile`) bundle better-sqlite3's JavaScript but not its native addon, so inside a binary every probe failed on every target: Second Brain was silently disabled and bots/boards fell back to JSON (ROADMAP P1.9, §F, #96). Toolchain-less npm installs (Termux, no compiler) hit the same wall. Meanwhile both runtimes ship an embedded engine — `bun:sqlite` in Bun and `node:sqlite` (`DatabaseSync`) in Node ≥ 22.5 (unflagged in 22.13 / 23.4) — with the same synchronous shape better-sqlite3 has.
+- **Decision**: One shared driver, `src/utils/sqlite-driver.ts`, exposes the minimal synchronous API the call sites already use (`openSqlite` → `prepare().run/get/all`, `exec`, `pragma`, `transaction`, `close`) and selects a backend once per process, in this order: **better-sqlite3** (when it loads and opens a file — the npm default, zero behaviour change: statements are handed through untouched) → **bun:sqlite** (when `process.versions.bun`) → **node:sqlite** (Node ≥ 22.5) → **none**, in which case `openSqlite()` returns `null` and callers fall back to JSON exactly as before (Second Brain and the pool cache stay disabled; the queue and boards use their JSON stores). Engines are loaded with a `createRequire` string require so tsup/tsc never resolve `bun:sqlite`/`node:sqlite` (both are also tsup externals). The driver normalises the engine differences that would otherwise be silent data bugs: bun:sqlite binds NULL for bare named keys unless they carry the `@`/`:`/`$` prefix and returns `null` from `get()`; node:sqlite rejects `undefined`/boolean bindings, throws on named keys the SQL does not mention, returns null-prototype rows, has no `transaction()` and throws on double `close()`. `MERCURY_SQLITE_BACKEND=better-sqlite3|bun:sqlite|node:sqlite|json` forces a backend; `describeSqliteBackend()` reports the probe outcome and `mercury doctor --storage` prints backend, DB paths, sizes, row counts and a loud warning when storage is JSON-only. `isBetterSqlite3Available()` remains as a deprecated alias of `isSqliteAvailable()` so web/API callers keep working.
+- **Alternatives considered**:
+ - Shipping better-sqlite3's prebuilt `.node` per target next to the binary — doubles the release matrix, needs a loader path per OS, and still leaves toolchain-less npm installs broken.
+ - sql.js (WASM) as the universal engine — asynchronous init, whole-file-in-memory, no WAL; already used only as the dashboard's read-only fallback and unsuitable as a durable write path for the queue.
+ - Making JSON the only store — loses FTS5 search for Second Brain and the lease semantics the bots queue relies on.
+- **Consequence**: Standalone binaries get Second Brain, bots and boards on bun:sqlite with no native addon; Node 22.13+ installs work without a compiler; Node 20 keeps better-sqlite3 (a single process never mixes engines, but the files are plain SQLite so switching runtimes keeps the data). Node 22.5–22.12 only expose node:sqlite behind `--experimental-sqlite`, and Node 22.x prints one `ExperimentalWarning` when it loads. The driver's contract test (`src/utils/sqlite-driver.test.ts`) runs the same assertions against every engine the test process can load — CI should run it under Node 20, Node 22+ and `bun test` (P2.6). FTS5 is required from every engine (Node's and Bun's builds ship it). New SQLite call sites must go through the driver, bind named parameters with bare keys, and never assume `get()` returns `null`.