Skip to content

OpenPanel is an open-source web and product analytics platform, an open-source alternative to Mixpanel with optional self-hosting.

15.9 KB
Raw Blame

M18-003 — the cache client's fail-fast, measured

What /track costs when the Redis cache is unreachable, before and after the change in packages/redis/redis.ts. Every number below is a paste from a run on this box; nothing here is estimated.

Box — 4 cores, 15 GB, Linux 6.8.0-137; Redis 7.2.5 on 127.0.0.1:6379; Postgres, ClickHouse and Redpanda local. Date 2026-09-13. Base revision 714dae9d. "Before" is that revision's packages/redis/redis.ts verbatim; "after" is the same tree with this task's change applied and nothing else.

Reproducing

cd apps/api
# the fault harness — spawns its own API on :3399 behind a TCP proxy for Redis
LOG_LEVEL=warn dotenv -e ../../.env -- bun e2e/redis-fail-fast.ts
# phases A-C only (the recovery-window comparison)
LOG_LEVEL=warn M18_QUICK=1 M18_OUTAGE_MS=60000 dotenv -e ../../.env -- bun e2e/redis-fail-fast.ts
# what a legitimate command costs, which is what bounds commandTimeout
dotenv -e ../../.env -- bun e2e/redis-command-cost.ts

The comparison runs ("before", commandTimeout only, ioredis's default backoff) are the same harness against an edited createFailFastCacheClient — the option under test removed, nothing else — and the tree was restored to the committed version afterwards, verified by hash.

The harness never touches the shared server's data: the API under test is pointed at a local TCP proxy it can stop (refuse, a docker stop) or freeze (blackhole, a docker pause), on Redis database 3 and its own Kafka topic. The operator's dev stack keeps its own connection to :6379 throughout.

The headline

Fault Shape Before After
Redis stopped warm-auth /track 2 of 3 no answer in 30 s, the third 500 at 30,001 ms 200 × 10, p50 10.0 ms
Redis stopped cold-auth /track 2 of 3 no answer in 30 s, one 500 at 30,001 ms 500 × 10, p50 1.3 ms
Redis stopped browser-origin /track 500 at 12,041 ms, one no answer in 30 s 500 × 10, p50 1.0 ms
Redis stopped /healthcheck no answer in 30 s 503 in 17 ms
Redis frozen warm-auth /track 5 of 5 no answer in 30 s 200 × 5 at 1,010 ms
Redis frozen browser-origin /track 5 of 5 no answer in 30 s 500 × 5 at 502 ms
booted with Redis down first /track no answer in 30 s 500 in 516 ms

The before column reproduces drill 02 (verification/drills/02-redis-unavailable.md): 12,041 ms here against its 12,039 ms, and the same "half the requests never answered" shape. The 41,978 ms outlier drill 02 saw is above this harness's 30 s client timeout, so it shows up as no-answer(AbortError) instead.

The behavioural acceptance criterion is the first two rows: warm auth keeps its normal latency and still answers 200 (10.0 ms p50 against a 11.7 ms p50 healthy baseline — the auth resolves from the process-local LRU and the session lookup degrades to the deterministic id), and cold auth fails promptly (1.3 ms) instead of hanging.

Raw output

Before — 714dae9d unmodified (M18_OUTAGE_PROBES=3)

── A. healthy baseline ──
warm-secret /track                 n=20 200×20                 p50=10.8ms p95=12.1ms max=12.1ms
browser-origin /track              n=20 200×20                 p50=11.0ms p95=14.5ms max=14.5ms

── B. sustained outage (20s, refuse) ──
warm-secret /track                 n=3 no-answer(AbortError)×2 500×1 p50=30001.2ms p95=30001.6ms max=30001.6ms
cold-unknown /track                n=3 no-answer(AbortError)×2 500×1 p50=30001.2ms p95=30001.7ms max=30001.7ms
browser-origin /track              n=3 500×2 no-answer(AbortError)×1 p50=12041.4ms p95=30001.5ms max=30001.5ms
/healthz/live                      status=200 1.2ms
/healthz/ready                     status=200 4.5ms
/healthcheck                       status=no-answer(AbortError) 30001.3ms

── C. recovery ──
first browser-origin 200 after restore: 64ms (status 200)
browser-origin /track              n=20 200×20                 p50=10.7ms p95=18.8ms max=18.8ms

── D. 1000ms blip under load ──
warm-secret, control               n=2745 200×2745               p50=10.8ms p95=12.8ms max=18.1ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=3
warm-secret, blip                  n=2313 200×2313               p50=10.7ms p95=12.2ms max=1074.2ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=3
browser-origin, control            n=2490 200×2490               p50=12.0ms p95=13.9ms max=23.2ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=2
browser-origin, blip               n=2030 200×2030               p50=12.2ms p95=13.9ms max=1069.8ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=2

── E. blackhole (frozen dependency) ──
warm-secret /track                 n=5 no-answer(AbortError)×5 p50=30000.9ms p95=30001.0ms max=30001.0ms
browser-origin /track              n=5 no-answer(AbortError)×5 p50=30000.8ms p95=30001.0ms max=30001.0ms

── F. boot with Redis down ──
booted with redis unreachable: true (2235ms)
  /healthz/live                    status=200 4.7ms
  /healthz/ready                   status=200 2.0ms
  /healthcheck                     status=503 10552.1ms
  cold /track (never had redis)      n=1 no-answer(AbortError)×1 p50=30001.0ms p95=30001.0ms max=30001.0ms
  /track once redis returns: status=200 15.2ms
  /healthz/ready                   status=200 1.1ms
  /healthcheck                     status=200 12.5ms

After — with the change

── A. healthy baseline ──
warm-secret /track                 n=20 200×20                 p50=11.7ms p95=13.0ms max=13.0ms
browser-origin /track              n=20 200×20                 p50=11.8ms p95=14.6ms max=14.6ms

── B. sustained outage (20s, refuse) ──
warm-secret /track                 n=10 200×10                 p50=10.0ms p95=11.8ms max=11.8ms
cold-unknown /track                n=10 500×10                 p50=1.3ms p95=3.0ms max=3.0ms
browser-origin /track              n=10 500×10                 p50=1.0ms p95=1.4ms max=1.4ms
/healthz/live                      status=200 0.9ms
/healthz/ready                     status=200 2.4ms
/healthcheck                       status=503 17.1ms

── C. recovery ──
first browser-origin 200 after restore: 336ms (status 200)
browser-origin /track              n=20 200×20                 p50=11.0ms p95=12.1ms max=12.1ms

── D. 1000ms blip under load ──
warm-secret, control               n=2620 200×2620               p50=11.3ms p95=13.5ms max=19.0ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=2
warm-secret, blip                  n=2830 200×2830               p50=10.8ms p95=12.7ms max=18.4ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=2
browser-origin, control            n=2320 200×2320               p50=12.5ms p95=18.2ms max=26.5ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=3
browser-origin, blip               n=6840 200×1980 500×4860      p50=1.2ms p95=13.0ms max=17.8ms
                                   failures=4860 over 1220ms of wall clock, distinct sessionIds=3

── E. blackhole (frozen dependency) ──
warm-secret /track                 n=5 200×5                  p50=1009.7ms p95=1010.1ms max=1010.1ms
browser-origin /track              n=5 500×5                  p50=501.8ms p95=501.9ms max=501.9ms

── F. boot with Redis down ──
booted with redis unreachable: true (2224ms)
  /healthz/live                    status=200 2.5ms
  /healthz/ready                   status=200 3.7ms
  /healthcheck                     status=503 503.5ms
  cold /track (never had redis)      n=1 500×1                  p50=515.8ms p95=515.8ms max=515.8ms
  /track once redis returns: status=200 74.5ms
  /healthz/ready                   status=200 0.7ms
  /healthcheck                     status=200 6.7ms

The mechanism, and what it costs

Two settings, because there are two faults, and commandTimeout alone does not fix the one that matters.

1. The offline queue goes off once the client has connected — a command issued while the socket is known-down is rejected in microseconds instead of queued behind a 20-attempt reconnect cycle. This is what produces the 10.0 ms warm /track and the 1.3 ms cold refusal in phase B. commandTimeout on its own would have made those 500 ms each, not 1 ms.

2. commandTimeout: 500 — the offline queue cannot see the fault where the socket is UP and the server answers nothing. Phase E is that fault: 30 s of nothing before, 1,010 ms after for the warm shape (two cache calls in series, each cut off at the 500 ms deadline) and 502 ms for the browser shape.

The cost is the blip (phase D — a 1 s outage under 5 concurrent clients, each shape run twice, once without the fault as a control):

Shape Before After
warm-auth 0 failures, one request stretched to 1,074 ms 0 failures, max 18.4 ms
browser-origin 0 failures, one request stretched to 1,070 ms 4,860 × 500 over 1,220 ms

That is the whole trade, stated plainly: a 1 s Redis blip used to cost the browser-shaped /track a ~1 s stall and no errors; it now costs it ~1.2 s of 500s. The warm-auth shape pays nothing either way — it degrades to the deterministic session id. distinct sessionIds is unchanged between control and blip in every run, so no session was split by the faster failure.

The third configuration, measured too

commandTimeout alone — the offline queue left ON — is a real middle ground and was measured as its own run, same harness, same box:

before (neither) commandTimeout only fail-fast + commandTimeout (chosen)
outage, warm-auth /track 2 of 3 unanswered in 30 s 200 at 1,011 ms 200 at 10.0 ms
outage, cold-auth /track 2 of 3 unanswered in 30 s 500 at 502 ms 500 at 1.3 ms
outage, /healthcheck unanswered in 30 s 503 in 504 ms 503 in 17 ms
1 s blip, browser-origin 0 failures, one 1,070 ms stall 5 failures over 504 ms 4,860 failures over 1,220 ms
frozen server, warm-auth unanswered in 30 s 200 at 1,010 ms 200 at 1,010 ms

commandTimeout alone is much the gentler option on the blip — ioredis arms the deadline in sendCommand, before the writable check, so a queued command is under it too, and only the five requests that waited past 500 ms failed. But it does not meet this task's behavioural criterion: a warm-auth /track during the outage takes 1,011 ms, not "tens of ms", because every request still pays the full deadline twice over for a dependency whose answer it is going to discard. Fail-fast is what turns that back into 10.0 ms. The chosen configuration is the combination, and the blip is what it costs.

Its phases B and D, verbatim:

── B. sustained outage (20s, refuse) ──
warm-secret /track                 n=10 200×10                 p50=1011.3ms p95=1011.9ms max=1011.9ms
cold-unknown /track                n=10 500×10                 p50=502.1ms p95=504.3ms max=504.3ms
browser-origin /track              n=10 500×10                 p50=502.2ms p95=502.5ms max=502.5ms
/healthz/live                      status=200 0.9ms
/healthz/ready                     status=200 3.8ms
/healthcheck                       status=503 503.9ms

── D. 1000ms blip under load ──
warm-secret, control               n=2520 200×2520               p50=11.8ms p95=13.8ms max=21.9ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=3
warm-secret, blip                  n=2118 200×2118               p50=11.6ms p95=13.8ms max=1010.9ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=2
browser-origin, control            n=2355 200×2355               p50=12.8ms p95=14.9ms max=20.7ms
                                   failures=0 over 0ms of wall clock, distinct sessionIds=2
browser-origin, blip               n=1950 200×1945 500×5         p50=12.8ms p95=14.7ms max=514.5ms
                                   failures=5 over 504ms of wall clock, distinct sessionIds=3

Why the reconnect backoff is capped at 500 ms

With the offline queue off, the reconnect backoff becomes user-visible: while ioredis waits out its next attempt, every command is rejected, so the failures outlive the outage. ioredis's default is min(attempt * 50, 2000) ms, and the attempt counter keeps climbing for as long as the server is away (measured directly: attempt 28, next delay 1,400 ms, 19 s into an outage). Time from "Redis is back" to "/track answers 200 again", phase C, three runs each at 20 s and two each at 60 s:

Outage ioredis default cap (2,000 ms) this change (500 ms)
20 s 390 / 395 / 396 ms 349 / 340 / 342 ms
60 s 1,137 / 1,135 ms 416 / 413 ms

The cap costs nothing (a reconnect to a live local server is sub-millisecond) and bounds that window at roughly half a second however long the outage was.

Why 500 ms and not 100 ms

commandTimeout applies to every command the cache client issues, including the event buffer's. If it is anywhere near a busy flush, a healthy server starts failing as if it were down — and under M18-001 a failed flush refuses to commit Kafka offsets. bun e2e/redis-command-cost.ts, same box, same day:

redis 127.0.0.1:6379 db 4, mean row 998B, batch 4000
MULTI 4000x rpush (shutdown flush)     n=20 payload=3992000B p50=21.04ms p95=33.95ms p99=33.95ms max=33.95ms
MULTI 100x rpush (steady state)        n=200 payload=99800B p50=0.70ms p95=0.81ms p99=1.77ms max=1.86ms
LRANGE 0..3999 (cron flush)            n=100 payload=3992000B p50=5.68ms p95=12.86ms p99=21.26ms max=21.26ms
GET (auth / cache lookup)              n=2000 p50=0.11ms p95=0.14ms p99=0.18ms max=4.45ms

500 ms is ~15x the heaviest legitimate command (a 4 MB MULTI of 4,000 rpush, 34 ms) and ~2,800x a GET. 100 ms would be only 3x the flush, which is too close for a box under load.

Boot ordering — the one real risk

With the offline queue off from construction, ioredis rejects every command issued before the first connect completes, so boot order would decide whether a command works. The queue therefore stays on until the client's first ready; a process that has never reached Redis falls back to commandTimeout, which is bounded. Phase F is the proof: an API booted with Redis already unreachable comes up in 2,224 ms (2,235 ms before, so booting is not slower), /healthz/live and /healthz/ready answer 200 in single-digit ms, /healthcheck reports 503 in 504 ms instead of hanging for 10.5 s, the first /track fails in 516 ms instead of never answering, and once Redis returns the same process serves 200s again (74.5 ms) with /healthcheck back to 200. packages/redis/redis.test.ts pins the same property as a unit test: a command issued in the same tick as the client's construction resolves rather than throwing "Stream isn't writeable".

Interaction with M18-001

M18-001's shutdown flushes the event buffer's micro-batch to Redis and refuses to commit offsets if that flush fails. Fail-fast makes that flush fail faster, which is correct only if a fast rejection still reads as a failure. ioredis resolves a MULTI whose individual commands failed, so the property that matters is that a fail-fast client rejects the exec() — pinned by packages/redis/redis.test.ts ("MULTI under fail-fast", rejects in under 100 ms once the server is gone). The rest of the chain is already covered: event-buffer.test.ts:370 (the buffer rethrows and keeps the events) and shutdown.test.ts:113 (a rejected flushEventBuffer returns SHUTDOWN_EXIT_FAILED, so nothing is committed).

Not covered here

The duplicate from drill 02's finding (1) — an event landing in ClickHouse more than once after a handler failure plus a rebalance — is out of scope for M18-003 and untouched by this change.