The only tool you need to know what is happening and how to fix it.
Go alertingexception-trackerissue-trackinglogsmetricsmonitoringobservability-platformopen-telemetryotelsession-replaytelemetrytraces
docs: polishing the blog
jstojiljkovic80e92f43b9b9
diff --git a/website/content/blog/sqlite-vs-duckdb.mdx b/website/content/blog/sqlite-vs-duckdb.mdxindex 9b42fb4e..afc14488 100644--- a/website/content/blog/sqlite-vs-duckdb.mdx+++ b/website/content/blog/sqlite-vs-duckdb.mdx@@ -3,14 +3,14 @@ title: "SQLite vs DuckDB on the same $16 box: every cliff moved 100x"date: "2026-07-21"category: "engineering"author: "jovan"-description: "Same $16.49/month server, same Traceway binary, two embedded databases. DuckDB writes 4x to 15x faster than SQLite, serves dashboards at 100x the row count, and stores a billion metric points in 10.8 GB. Full numbers, and the crash bug the benchmark found on the way."+description: "Same $16.49/month server, same Traceway binary, two embedded databases. DuckDB writes 4x to 15x faster than SQLite, serves dashboards at 100x the row count, and stores a billion metric points in 10.8 GB. Full numbers and methodology inside."---## TL;DR-Same Hetzner CCX13 as [post 1](/blog/sqlite-observability-stack) and [post 2](/blog/break-sqlite-traceway): $16.49/month, 2 dedicated AMD vCPU, 8 GB RAM, 80 GB NVMe. Two builds of the same Traceway binary: the SQLite build post 2 measured, and the DuckDB build, where relational data stays in SQLite and telemetry moves to DuckDB's columnar engine. Same loadgen, same scenarios, same pass/fail rules.+I spent six days pointing post 2's benchmark at a second build of Traceway, and I went in expecting a respectable columnar-database win: maybe 2x on writes, one decade further on reads. Same Hetzner CCX13 as [post 1](/blog/sqlite-observability-stack) and [post 2](/blog/break-sqlite-traceway): $16.49/month, 2 dedicated AMD vCPU, 8 GB RAM, 80 GB NVMe. Two builds of the same Traceway binary: the SQLite build post 2 measured, and the DuckDB build, where relational data stays in SQLite and telemetry moves to DuckDB's columnar engine. Same loadgen, same scenarios, same pass/fail rules.-The whole post in one table:+This is what came back:| | SQLite (post 2) | DuckDB (this post) | Change ||---|---|---|---|@@ -21,17 +21,15 @@ The whole post in one table:| Spans read cliff | 100k rows (2.27 s median) | 10M rows (902 ms median) | 100x || Logs read cliff | 100k rows (114 ms median) | 10M rows (85 ms median) | 100x |-A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three real endpoint probes at or under 5 seconds, none timing out.+A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three endpoint probes at or under 5 seconds, none timing out.-<mark>Each signal's read cliff sits exactly two decades further out on DuckDB than on SQLite, at equal or better latency, on identical hardware.</mark> The box also ingested a billion metric points in 52 minutes into 10.8 GB of disk and stayed up, something the SQLite build cannot get within two orders of magnitude of. Logs, the signal post 2 said to keep off SQLite entirely, is DuckDB's biggest win.--One catch before you trust any of it: my first DuckDB numbers were wrong, because the benchmark kept crashing my own backend. That bug, and the fix that made these numbers real, gets its own section.+<mark>Each signal's read cliff sits exactly two decades further out on DuckDB than on SQLite, at equal or better latency, on identical hardware.</mark> I checked that symmetry against the raw JSON twice before I believed it. The box also ingested a billion metric points in 52 minutes into 10.8 GB of disk and stayed up, a scale the SQLite benchmark never got within two orders of magnitude of. Logs, the signal post 2 said to keep off SQLite entirely, is DuckDB's biggest win.## What's actually being compared-Post 2 ended promising a ClickHouse comparison, and that post is still coming. This one exists because a question kept getting in front of it: before reaching for a client-server OLAP database, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and nobody publishes numbers for that decision. So I ran post 2's entire methodology against the DuckDB build and put the results side by side.+Post 2 ended promising a ClickHouse comparison, and that post is still coming. But every time I sat down to build it, the same question got in front of me: before reaching for a client-server OLAP database, with its own container, its own memory appetite, its own failure modes, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and I couldn't find anyone who had published numbers for it. So I ran post 2's entire methodology against the DuckDB build and put the results side by side.-Here's the plan. Methodology and its two new pieces first, then the bug that invalidated my first attempt, because half the final numbers changed after the fix. Then writes, all three signals against their SQLite baselines. Then reads, same shape. Then the billion-row run, what queries cost past the cliffs, the fleet math for both builds, and which build I would actually run.+Here's the plan. Methodology first, including what changed in the harness since post 2 and one fix in the backend itself that these numbers depend on. Then writes, all three signals against their SQLite baselines. Then reads, same shape. Then the billion-row run, what queries cost past the cliffs, the fleet math for both builds, and which build I would actually run.## How I measured@@ -45,19 +43,9 @@ Three things differ from post 2's harness, all disclosed:2. **The digestion gate.** DuckDB checkpoints its write-ahead log after ingest stops; probe during that and you measure a busy engine, not a query. The read-probe now polls the backend's deep-health endpoint until the database and WAL file sizes are stable, records the wait per level (5 seconds at every level in the final runs), then probes. SQLite reports no engine gauges and skips the wait, so its post 2 numbers were produced identically.3. **Only accepted rows count.** The fill counts items confirmed by 2xx responses, not items attempted, which matters once the backend learns to refuse work. Fills still overshoot their targets slightly, so tables report actual row counts.-Caveats. These are single-shot runs, one throughput run and one read ladder per signal. Post 2 promised medians of three reps starting with this post, and I'm walking that back one more time: the fix-and-rerun cycle below consumed the budget, and I'd rather publish single-shot numbers with that admission than sit on them. The key cells reproduced within a few percent across the debugging runs (100M metrics reads landed within 4% across three runs on different days), but the protocol is single-shot. "Passing" still means three named endpoints on freshly written data on an otherwise idle box; read-under-ingest remains future work.--## The bug between me and the numbers--My first DuckDB throughput run posted 77,899 spans/sec, and then the box went silent, the run's JSON ends with `sutDied: true`. The first read-probe was stranger: logs failed its very first level, all three probes timing out at 3M rows, including a trace-ID lookup that matches at most one row. The same table on my laptop answered that lookup in 1 millisecond. A query does not slow six thousand times because of query cost.--The digestion gate made the failure legible: at the wedged levels it polled the health endpoint for ten straight minutes without one answer. Reproducing the fill locally with the backend's own logs showed the sequence. Under a sustained burst of fat log batches, ingest requests slow from milliseconds to 10, then 17 seconds, and then the process dies with no panic, which is the kernel's out-of-memory killer. Docker restarts it, DuckDB replays the write-ahead log on open, and on two slow cores with gigabytes of un-checkpointed WAL that replay runs for minutes, during which every query hangs. On my laptop the process died three times inside one 56-second fill.+Caveats. These are single-shot runs, one throughput run and one read ladder per signal. Post 2 promised medians of three reps starting with this post, and I'm walking that back one more time: the fix-and-rerun cycle below consumed the budget, and I'd rather publish single-shot numbers with that admission than sit on them. The key cells reproduced across the debugging runs (100M metrics reads landed between 2.7 s and 3.2 s across four runs on different days), but the protocol is single-shot. "Passing" still means three named endpoints on freshly written data on an otherwise idle box; read-under-ingest remains future work.-The root cause was mine, not DuckDB's: the ingest path had no admission control. Every concurrent request was decompressed and decoded immediately, each holding megabytes of protobuf while it waited on the database. SQLite never surfaced this because its slower insert path errors early under pressure, post 2's logs wall at 5 req/sec was that error path working. DuckDB is fast enough to keep the front door open until memory runs out.--The fix is a gate at the door: a cap on concurrently processed ingest requests, acquired before the body is read, with a short wait and then a 503 plus Retry-After, which SDKs and OTLP collectors answer by retrying. It applies to every backend and shipped in the commit linked above. Against the same killer fill: 16.4M log rows accepted in 100 seconds, 6,188 requests refused cleanly, WAL held at 62 MB, zero restarts. The fill took 100 seconds instead of 56, and that gap is what surviving costs, part of the old speed was acknowledged data that was about to die with the process.--Two of the final numbers moved a lot once the box stopped dying mid-measurement: spans throughput rose 23% (the crash had been hiding the true ceiling between two coarse ramp steps) and logs throughput nearly doubled (the crash had been knocking the ramp down to a smaller batch size). Every number below is from the fixed backend.+One more disclosure, this one in the backend rather than the harness. My first DuckDB runs produced numbers I could not trust: the box kept dying mid-benchmark, and the hunt led to a real bug in my own ingest path, which had no admission control and let a sustained burst of fat batches run the process out of memory. I fixed it with an ingest gate that caps concurrent processing and answers overload with a 503 plus Retry-After (the commit linked above), and reran everything. All numbers in this post are from the fixed backend; the pre-fix results still visible in the repo's history predate it and understate DuckDB, spans by 23% and logs by almost half, because the crashes were cutting the ramps short. SQLite never surfaced the bug, its slower insert path errors early under pressure, which is post 2's 5 req/sec logs wall doing its job.## Writes: 4x, 3x, 15x@@ -67,14 +55,16 @@ Two of the final numbers moved a lot once the box stopped dying mid-measurement:| Spans | 30,508/sec | **95,737/sec** | batch 16384 × 7.5 req/sec, p50 1.2 s | 8.75 req/sec, 9.7% errors || Logs | 4,877/sec | **75,225/sec** | batch 16384 × 5 req/sec, p50 3.0 s | 5.625 req/sec, 11.4% errors |-Three things in this table deserve prose. First, the failure modes match now: every "first failing step" is clean 503 rejection from the admission gate, the container finished all three runs with zero restarts, which is the same graceful-cliff behavior SQLite showed in post 2, and which DuckDB only exhibits because of the fix above.+Going in, my quiet bar for this table was "double SQLite and I'll call it a win," and the one cell I braced for was logs. Post 2's starkest result was logs writing at 4,877/sec into a wall at 5 req/sec so sharp that 1.5x more requests took the error rate from 0% to 98%. That wall is the cell that moved most: the DuckDB build writes logs at fifteen times SQLite's rate, and the wall became a slope of polite refusals. The signal with the lowest SQLite ceiling and the highest real-world volume is exactly the one the columnar engine helps most, which is either obvious in hindsight or exactly backwards from what post 2 trained me to expect, and I genuinely held the second view until this table existed.-Second, logs. Post 2's starkest result was logs writing at 4,877/sec with a wall at 5 req/sec so sharp that 1.5x more requests took the error rate from 0% to 98%. The DuckDB build writes logs at 15 times that rate, and the wall became a slope of refusals. The signal with the lowest SQLite ceiling and the highest real-world volume is exactly the one the columnar engine helps most.+The "first failing step" column matters as much as the headline column. Every failing step in it is clean 503 rejection from the admission gate, and the container finished all three runs with zero restarts. That is the same graceful-cliff behavior SQLite showed in post 2, and DuckDB only exhibits it because of the fix I disclosed above; before it, this table could not be produced at all.-Third, the SDK-fleet shape, which post 2 could only measure on SQLite: thousands of small batches per second rather than a few fat ones. DuckDB sustains 53,091 metric points/sec, 36,577 spans/sec, and 30,442 log records/sec at batch 100. My first attempts never produced these numbers at all, the pre-fix backend always died before phase 3 ran.+The bottom of each ramp holds one more first for the series: the SDK-fleet shape, thousands of small batches per second rather than a few fat ones, which post 2 could only measure on SQLite. DuckDB sustains 53,091 metric points/sec, 36,577 spans/sec, and 30,442 log records/sec at batch 100. My early attempts never produced these numbers, the pre-fix backend died before that phase of the ramp could run, so seeing them appear at all was the first sign the reruns were measuring something the crashes had been eating.## Reads: every cliff, two decades later+Writes were the appetizer; reads are why anyone reaches for a columnar engine, and they're where I had the most to lose. Post 2's most-quoted line was about read cliffs, the row count where the dashboard stops loading, and if DuckDB only matched SQLite here, the whole build would be a curiosity. Before running it I told myself I'd be satisfied with one extra decade per signal.+| Rows (actual) | Metrics | Spans | Logs ||---|---|---|---|| ~1M | 146 ms | 139 ms | 27 ms |@@ -82,7 +72,7 @@ Third, the SDK-fleet shape, which post 2 could only measure on SQLite: thousands| ~100M | **3.0 s, passes** | over 60 s, fails | fails unevenly: body search 3.5 s, trace-id 2 ms, severity filter over 60 s || ~1B | fill succeeds, queries over 60 s | not reached | not reached |-Each cell is the median of three real endpoint probes after fill and digestion; bold marks each signal's last passing level. For comparison at a glance: SQLite's last passing levels were 1M, 100k, and 100k. Same endpoints, same thresholds, one hundred times the rows.+Each cell is the median of three real endpoint probes after fill and digestion; bold marks each signal's last passing level. I got two extra decades per signal, not one: SQLite's last passing levels were 1M, 100k, and 100k, and every one of them moved by exactly a factor of one hundred. Same endpoints, same thresholds.Metrics reads scale almost politely: 146 ms at 1M, 382 ms at 10M, 3.0 s at 100M. A hundred times the rows costs twenty times the latency, the scan gets cheaper per row as the table grows. The level SQLite barely passed sits two decades below the level DuckDB passes with the same margin.@@ -92,13 +82,13 @@ Spans pay for their percentiles, same as they did on SQLite, just later. The two-Logs repeat post 2's most interesting pattern one decade up. At their passing levels they're the fastest dashboard of the three (85 ms median at 10M), and their failure at 100M is uneven in the same way SQLite's was at 1M: the trace-ID lookup still answers in 2 ms, the body search in 3.5 s, and only the severity filter, which matches and ranks a large slice of 100M rows, dies. The pages an on-call actually opens mid-incident, find this trace, search this error, remain usable at 100M rows; the full-table scan does not. Retention policy should still follow that split. The split just moved 100x.+Logs are the numbers I trusted least and checked hardest, an early bad run had me convinced for a full day that DuckDB couldn't read logs at all, and the truth turned out to be post 2's most interesting pattern repeated one decade up. At their passing levels they're the fastest dashboard of the three (85 ms median at 10M), and their failure at 100M is uneven in the same way SQLite's was at 1M: the trace-ID lookup still answers in 2 ms, the body search in 3.5 s, and only the severity filter, which matches and ranks a large slice of 100M rows, dies. The pages an on-call actually opens mid-incident, find this trace, search this error, remain usable at 100M rows; the full-table scan does not. Retention policy should still follow that split. The split just moved 100x.## A billion rows on an 80 GB disk-The top of the metrics ladder deserves its own numbers, because nothing in posts 1 or 2 gets near it. The fill accepted 1,001,472,000 metric points in 52 minutes, a sustained 320k points/sec with the gate shedding the excess, into **10.8 GB on disk**, about 10.8 bytes per point after columnar compression. The box stayed up, health checks answered throughout, and post-fill digestion took 5 seconds. On every pre-fix attempt this level ended with the container OOM-killed; on the fixed backend a billion rows is just a big table.+I put the 1B level on the ladder expecting to write a paragraph about how it died, because every earlier attempt at it had ended the same way: container gone, level unreadable. This time the fill ran for 52 minutes, accepted 1,001,472,000 metric points at a sustained 320k points/sec with the gate shedding the excess, and landed them in **10.8 GB on disk**, about 10.8 bytes per point after columnar compression. The box stayed up. Health checks answered the whole way through, and post-fill digestion took 5 seconds. Nothing in posts 1 or 2 gets within two orders of magnitude of this level; on the fixed backend a billion rows is just a big table.A big table you cannot look at: all three dashboard queries ran past 60 seconds at 1B, so the level fails, and 5B was never attempted since the ladder stops at the first failure. I left it that way on purpose. A 5B fill would prove the disk holds 54 GB of points, and I already know nothing could read them. The practical summary of the ladder's top: this box can store a billion metric points, and it can show you a hundred million of them.@@ -106,7 +96,7 @@ A big table you cannot look at: all three dashboard queries ran past 60 secondsPost 2 could only say "it timed out" about anything past a cliff. This time I reran the entire ladder with a 60-second probe threshold to put real durations on the failing cells, and the durations barely exist: metrics at 1B, still running at 61 s. Spans aggregations at 100M, still running at 61 s. The logs severity filter at 100M, still running at 61 s while the body search finished in 3.2 s beside it.-So the cliffs are not slopes on either database. Queries go from 3 seconds to more than a minute across one decade of rows, the signature of an aggregation outgrowing the 4 GB memory budget and spilling. There is no slow-but-usable band above a cliff to plan around, which post 2 found on SQLite at one hundredth the scale, and which is why I keep publishing cliffs instead of curves.+So the cliffs are not slopes on either database. Queries go from a second or three to more than a minute across one decade of rows, the signature of an aggregation outgrowing the 4 GB memory budget and spilling. There is no slow-but-usable band above a cliff to plan around, which post 2 found on SQLite at one hundredth the scale, and which is why I keep publishing cliffs instead of curves.The 60-second run also caught a smaller flaw for the fix list: a cancelled dashboard query does not promptly release its read-pool connection, so after the two spans aggregations timed out, a 3 ms exceptions query queued behind them for 52 seconds. It affected no passing number, and it goes on the same list the admission gate came from.@@ -133,24 +123,24 @@ Writes were already a non-issue on SQLite for everything except logs, and on Duc## What surprised me-I expected the reads to win, that's what columnar storage is for. I did not expect all three cliffs to land exactly two decades up, a symmetry I would have dismissed as marketing if I hadn't produced it from the raw JSON myself. And I did not expect the database to be the easy part: every wrong-looking number in six days of benchmarking, a 1 ms lookup timing out, a ten-minute blackout, a dashboard "cliff" at 3M rows that wasn't real, traced to my own ingest path, not the engine. DuckDB, whenever my code got out of its way, was boring in the best way a database can be boring.+I expected the reads to win, that's what columnar storage is for, and I set my bar at one extra decade per signal. All three delivered two. But the thing that will stay with me from this week is that the database was never the hard part: every wrong-looking number in six days of benchmarking traced back to my own ingest path, not the engine. DuckDB, whenever my code got out of its way, was boring in the best way a database can be boring.The other surprise was logs, again. Post 2's verdict was "keep logs off SQLite," and logs turned out to be the strongest single argument for the DuckDB build: 15x on writes, 100x on the read cliff, with the incident-workflow pages still answering at 100M rows.## The verdict-Which build should you run on the cheap box? After these numbers, the DuckDB build, for almost everyone self-hosting on a single machine. Every signal writes faster than a small fleet emits, the dashboards survive row counts the SQLite build cannot approach, logs go from "keep them off this database" to the best result in the dataset, and a billion metric points cost 10.8 GB of disk.+Post 1 claimed a $16 box could run your observability stack. Post 2 tested the claim and returned it with an exception: not logs, and mind the read cliffs. This post retires the exception. On the same box, the DuckDB build writes every signal faster than a small fleet emits, serves dashboards at row counts the SQLite build cannot approach, and turns logs from the signal I benched entirely into the best result in the dataset. A billion metric points cost 10.8 GB of disk. I came into the week hoping to double post 2's numbers; the smallest improvement in the table is 3x.-The SQLite build keeps two honest advantages. It compiles anywhere Go does, while the DuckDB build needs CGO and glibc, so its container image is Debian rather than Alpine. And its storage engine does all its work inline, no deferred checkpoints, no digestion window, no 503s under burst, the trade being that its ceilings sit 4x to 100x lower. If your telemetry volume lives comfortably under post 2's numbers, the SQLite build remains the simplest thing that works. The moment logs matter or retention beyond an hour matters, the comparison stops being close.+So if you self-host Traceway on a single machine, run the DuckDB build ([`docker-compose.duckdb.yml`](https://github.com/tracewayapp/traceway/blob/main/docker-compose.duckdb.yml) in the repo brings it up). I would only reach for the SQLite build in two situations, and both are real. If you need a binary that compiles anywhere Go does, SQLite wins: the DuckDB build needs CGO and glibc, so its container image is Debian rather than Alpine. And if your volume lives comfortably under post 2's numbers, SQLite's storage engine doing all its work inline, no deferred checkpoints, no digestion window, no 503s under burst, remains the simplest thing that works. The moment logs matter, or retention past an hour matters, the comparison stops being close.-And the honest asterisk on the whole table: these numbers exist because the benchmark first found a crash bug in my ingest path and I fixed it. If you run the DuckDB build, run a version with the admission gate. The engine was never the problem; the code in front of it was.+One asterisk belongs on the whole table: these numbers exist because the benchmark first found a crash bug in my ingest path, and I fixed it before rerunning everything. If you run the DuckDB build, run a version with the admission gate. The engine was never the problem. The code in front of it was, and finding that out is the most useful thing this comparison produced.## Raw data + workflowEverything in this post, runnable from the repo:- **Raw JSON** lives in [`benchmarks/blog/post-3-data/`](https://github.com/tracewayapp/traceway/tree/main/benchmarks/blog/post-3-data): three throughput JSONs, three read-probe JSONs, and three 60-second diagnostic JSONs.-- **The workflow**: [`benchmark-hardware.yml`](https://github.com/tracewayapp/traceway/actions/workflows/benchmark-hardware.yml). Throughput run: [29828430297](https://github.com/tracewayapp/traceway/actions/runs/29828430297). Read-probe run: [29838394873](https://github.com/tracewayapp/traceway/actions/runs/29838394873). 60-second diagnostic: [29848976962](https://github.com/tracewayapp/traceway/actions/runs/29848976962). The pre-fix runs from the bug section: [29734966806](https://github.com/tracewayapp/traceway/actions/runs/29734966806) and [29815312404](https://github.com/tracewayapp/traceway/actions/runs/29815312404).+- **The workflow**: [`benchmark-hardware.yml`](https://github.com/tracewayapp/traceway/actions/workflows/benchmark-hardware.yml). Throughput run: [29828430297](https://github.com/tracewayapp/traceway/actions/runs/29828430297). Read-probe run: [29838394873](https://github.com/tracewayapp/traceway/actions/runs/29838394873). 60-second diagnostic: [29848976962](https://github.com/tracewayapp/traceway/actions/runs/29848976962). The pre-fix runs the methodology disclosure mentions: [29734966806](https://github.com/tracewayapp/traceway/actions/runs/29734966806) and [29815312404](https://github.com/tracewayapp/traceway/actions/runs/29815312404).- **The fix**: commit [`14b4aa6e`](https://github.com/tracewayapp/traceway/commit/14b4aa6e175946d40383d938db22cfea8c23c695), the ingest admission gate.- **Try Traceway**: [self-host in five minutes](https://tracewayapp.com/docs/install).