:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Global Rate Limiter — Design and Implementation
Overview
The global rate limiter enforces per-entity event throughput limits across a fleet of capture nodes. It uses a 2-epoch sliding window counter in Redis for global state, a leaky bucket local decay model for zero-Redis-I/O hot-path decisions, a background tick that sends each node's counts to Redis and reads back fleet counts in bounded, chunked commands, and pressure-tiered adaptive sync to minimize read volume for low-utilization entities.
Architecture
┌──────────────────────────────────────────────────────────────────────┐
│ Capture Node │
│ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Hot Path (check_limit_internal) │ │
│ │ │ │
│ │ 1. Enqueue UpdateRequest to mpsc channel (count > 0) │ │
│ │ 2. Lookup entity in the local moka cache │ │
│ │ 3. Decide whether to push the key to pending_sync (DashSet) │ │
│ │ 4. Add this request to local_pending on the CacheEntry │ │
│ │ 5. Compare effective_level() against the threshold │ │
│ │ → Allowed / Limited, with the read-outage guard │ │
│ │ │ │
│ │ ⚡ Zero Redis I/O — all decisions are local │ │
│ └───────────┬──────────────────────────────┬────────────────────┘ │
│ │ mpsc channel │ pending_sync set │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Background task (spawned tokio task) │ │
│ │ │ │
│ │ Between ticks: merge each UpdateRequest into write_batch │ │
│ │ HashMap<(entity_key, epoch), count> │ │
│ │ │ │
│ │ Every tick_interval (default 1s): │ │
│ │ 1. Take up to max_sync_keys_per_tick keys from pending_sync │ │
│ │ 2. Purge write entries whose epoch can no longer be read │ │
│ │ 3. Take up to max_sync_keys_per_tick write entries │ │
│ │ 4. Per Redis instance, writes first, then reads: │ │
│ │ WRITES: INCRBY + EXPIREAT per (key, epoch), chunked │ │
│ │ READS: MGET [curr_epoch, prev_epoch] per entity, chunked │ │
│ │ 5. Apply each read chunk as it lands: │ │
│ │ - Compute weighted_count from the 2-epoch response │ │
│ │ - Measure drift vs local estimate │ │
│ │ - Update CacheEntry (estimated_count, pressure, synced_at) │ │
│ │ - Reset local_pending to 0 │ │
│ │ - Record tier transitions for observability │ │
│ │ 6. Record read health per instance │ │
│ └───────────────────────────┬────────────────────────────────────┘ │
│ │ │
└──────────────────────────────┼───────────────────────────────────────┘
│
▼
┌─────────────────────┐
│ Redis │
│ │
│ Key format: │
│ {prefix}:{key}:{e} │
│ │
│ e = epoch number │
│ dies 1-2 epochs on │
└─────────────────────┘
Key Concepts
1. Two-Epoch Sliding Window Counter
Instead of N discrete time-bucket keys, each entity uses exactly 2 Redis keys: a current epoch counter and a previous epoch counter.
epoch = floor(unix_timestamp / window_interval)
Redis keys for entity "team_42":
{prefix}:team_42:{epoch} ← current
{prefix}:team_42:{epoch - 1} ← previous
The estimated count uses weighted interpolation based on how far into the current epoch we are:
progress = (now % window_interval) / window_interval // 0.0 .. 1.0
estimated = prev_count × (1.0 - progress) + current_count
This produces a smooth, continuously-updating estimate that's more accurate than fixed buckets.
Count
│
│ prev epoch current epoch
│ ┌──────────┐ ┌──────────┐
│ │▓▓▓▓▓▓▓▓▓▓│ │░░░░░░ │
│ │▓▓▓▓▓▓▓▓▓▓│ │░░░░░░ │
│ └──────────┘ └──────────┘
│ weight: weight:
│ (1 - progress) 1.0
│
└──────────────────────────────────── Time
epoch N-1 ↑ epoch N
boundary
2. Leaky Bucket Local Decay
Each CacheEntry stores the last-known weighted count from Redis (estimated_count)
and the time it was synced (synced_at).
synced_at is None until the first Redis read lands. A never-synced entry is
due for a sync as soon as it clears min_sync_floor; a synced one waits out its
pressure tier. Collapsing the two delays a new key's first read by a full tier
interval, during which the node cannot see the fleet.
A known, tolerated race.
The request path's get-modify-insert can overwrite the entry a Redis read just refreshed.
The overwriting entry keeps the older estimate and the older synced_at, so it stays due for a sync, and the key's next request queues the read that corrects it.
Until then its estimate can be lower or higher than the fresh read.
It is higher when the key's previous epoch was over its threshold, because the window then drains faster than the leak rate.
The collision window is microseconds per event against one read per key every 7.5 to 60 seconds.
A fix via moka's per-key entry API cost ~0.3 µs per evaluation (+31% on this crate's bench) for no measurable change in decisions, so it is deliberately not applied.
Between syncs, the estimate decays at the configured leak rate:
leak_rate = threshold / window_interval
effective_level(entry, now) =
max(0, estimated_count - leak_rate × elapsed) + local_pending
This approximates the drain of the sliding window and keeps the local estimate useful for much longer than a simple stale/fresh binary. It matches the real drain only when the previous epoch held exactly the threshold. Below that the local estimate drains too fast, and above it too slowly.
Once reads have failed for max_read_outage, local_pending cannot limit a key.
local_pending is a correction to the last fleet count this node read, and only a successful read clears it.
Nothing else ages it, so while reads to Redis fail it grows without bound.
Given a long enough read outage, one node's count reaches the threshold on its own and limits a key that is under its limit.
Everywhere else the limiter fails open when Redis cannot answer, so this path would fail closed.
So this node records, for each Redis instance, when its current run of failed reads began.
A tick with any successful read clears the record, a tick whose every read failed starts it if it is not running, and a tick with no reads leaves it alone.
A quiet period before a failure therefore never counts. A quiet period after a failure does, because no read has confirmed the count since.
A read fails only when the client cannot complete it.
With the read/write split client, a replica read that the primary then answers counts as a success, so the guard trips only when both fail.
The record is wall-clock time, not a tick count, because during an outage one tick can wait seconds on timeouts and reconnects.
Once reads have failed for max_read_outage, a key that instance owns is judged by its decayed fleet count alone:
limit_level = reads failing ? max(0, estimated_count - leak_rate × elapsed)
: effective_level
The fleet count still drains as the window moves, so a key known to be over its limit keeps being limited until the evidence runs out. The guard gates only the limit decision. It does not reset the entry, change its counts, or change when it syncs, and it runs only for a key already over its threshold.
The signal is per instance, not global.
With more than one instance, a global signal would let a healthy instance hide a dead one, and the dead one's keys would fail closed again.
max_read_outage defaults to one window_interval.
A key under its limit sees fewer than threshold events in any one window, so this node's count for it can reach the threshold only once that count spans more than one window.
The count spans the outage plus the time since the key's last read before it, which is about one sync cadence of the key's tier.
So a false limit can happen only in that last cadence before the guard engages, and only for a key whose traffic lands mostly on this node.
For example, with steady traffic and a window eight times the sync interval, only a key above about 94% of its limit is exposed, for at most half a sync interval.
To close even that gap, set max_read_outage to window_interval - sync_interval or less. That bound holds while sync_interval is at most a third of window_interval.
Level
│
│ ╲ ← estimated_count decays
│ ╲ ····· ← local_pending accumulates
│ ╲ ····
│ ╲ ····
│ ╲ ····
│ ╳ ← effective_level = decayed global + local pending
│ ···· ╲
│ ···· ╲
│ ╲
│──────────────────╲──────────────────── threshold
│ ╲
└──────────────────────────────────── Time
synced_at next sync
3. Background Pipeline
All counter I/O runs in one background tokio task per limiter, so the hot path (request evaluation) never blocks on Redis. A second task refreshes custom-key thresholds when a dynamic source is configured.
The background task uses tokio::select! over two sources:
┌───────────────────────────┐ ┌──────────────────────────────┐
│ mpsc channel (writes) │ │ tokio interval (tick) │
│ │ │ │
│ UpdateRequest { │ │ Every tick_interval: │
│ key, count, timestamp │ │ → take bounded slices of │
│ } │ │ pending_sync and writes │
│ │ │ → send writes, then reads │
│ Merged between ticks │ │ → apply read results │
│ into: │ │ │
│ HashMap<(key, epoch), │ │ │
│ count> │ │ │
└───────────────────────────┘ └──────────────────────────────┘
The channel is not read while a tick runs, so updates wait in it until the tick ends.
The loop takes updates in batches of up to UPDATE_DRAIN_BATCH (256) per receive. Tokio's cooperative budget allows a task 128 budgeted operations per poll, and each receive is one, so one update per receive lets a CPU-starved pod drain slower than events arrive.
Each tick first takes every update already queued. A tick that runs longer than tick_interval leaves the next tick ready at once, and without this drain the random select! could pick the tick again before the channel.
A full channel drops the update and increments global_rate_limiter_error_total with cause=channel_full. It writes no log line.
Each tick, per Redis instance:
- Writes go first:
INCRBY key delta+EXPIREAT key deadlinefor each(entity, epoch)with pending counts. - Reads follow:
MGET [current_epoch_key, prev_epoch_key]for each entity taken frompending_sync. - Writes go first because a read result resets the key's
local_pending, so the read must already include this tick's writes. - Each side is split into commands of at most
max_keys_per_commandRedis keys, so a read command carries half that many entities. - At most
max_concurrent_commandscommands run at once per instance. A tick with both reads and writes costs at least two round trips, and more when the batch is large.
4. Adaptive Sync (Pressure Tiers)
Not all entities need to be synced at the same frequency. The system assigns each entity a pressure tier based on how close it is to the threshold:
Pressure tiers and sync frequency:
┌─────────────────────────────────────────────────────────────┐
│ │
│ ┌───────────┐ ┌──────────────┐ ┌─────────┐ ┌───────────┐ │
│ │ Idle │ │ Low │ │ Normal │ │ Hot │ │
│ │ < 10% │ │ 10% - 50% │ │50% - 80%│ │ ≥ 80% │ │
│ │ │ │ │ │ │ │ │ │
│ │ 4× base, │ │ 4× interval │ │ 1× base │ │ ½× base │ │
│ │above floor│ │ (60s) │ │ (15s) │ │ (7.5s) │ │
│ └───────────┘ └──────────────┘ └─────────┘ └───────────┘ │
│ │
│ 0% 10% 50% 80% 100% │
│ ───────────────────────────────────────────────────────── │
│ pressure (level / threshold) │
└─────────────────────────────────────────────────────────────┘
A key below min_sync_floor does not sync at all, whatever its tier.
An idle key above the floor syncs on the Low cadence, so a key that is hot across the fleet but cold on every node is still discovered.
The tier is recomputed on every request.
It uses the larger of the current level / threshold and the pressure at the last sync, so local traffic moves a key to a faster cadence at once.
tier_transitions_total counts changes only when a read lands.
Tier boundaries are pressure-based (level / threshold), so they apply correctly to custom keys regardless of threshold magnitude. A custom key with a 100× higher limit than the default will sync at the same relative cadence when at equivalent pressure.
Tier-distribution gauges come from a throttled full scan. cache_size is cheap (entry_count) and emits every tick. The per-tier sync_tier_gauge needs a full cache.iter() scan, so it runs only every TIER_SCAN_INTERVAL_TICKS (a constant, 30): the distribution moves slowly and prod metrics dedup to 60s, so scanning every tick would be wasted work under load. The scan runs in the background tick task, off the per-event hot path.
5. Timeouts
Each Redis command runs under two timeouts, and the shorter one fires first:
- The limiter's own timer,
global_read_timeoutorglobal_write_timeout(250ms by default). - The Redis client's response timeout. In capture it is
GLOBAL_RATE_LIMIT_REDIS_RESPONSE_TIMEOUT_MS, elseREDIS_RESPONSE_TIMEOUT_MS(100ms by default).
A client timeout is counted under cause="redis_write" for writes and cause="redis_error" for reads.
The write_timeout and read_timeout causes mean the limiter's own timer fired.
With the read/write split client, a replica read that fails with a recoverable error, such as a timeout, is retried on the primary. So one read can use two client timeouts, and the limiter's read timer can still fire.
A write that times out is not retried. If the command already reached Redis it still applies. If it was still queued in the client, the client drops it, so a timed-out write may or may not have counted.
Data Model
CacheEntry (local moka cache)
struct CacheEntry {
estimated_count: f64, // weighted count from last Redis sync
synced_at: Option<Instant>, // when we last read from Redis; None until the first read
local_pending: u64, // events counted locally since last sync, reset to 0 on sync
pressure: f64, // effective_level / threshold at last sync
}
local_pending is reset to 0 when fresh data arrives from Redis.
Since estimated_count already includes the events this node wrote via INCRBY, preserving local_pending would double-count them.
Events whose write has not landed when the read does drop out of the local estimate.
That covers events counted after the tick took its write batch, and writes deferred past the per-tick cap.
They appear in a later read once their write lands, unless their epoch ages out first and the tick purges them.
Redis Key Model
Key: {prefix}:{entity_key}:{epoch_number}
Value: integer counter (INCRBY)
Dies: (epoch + 2) × window_interval + grace + a per-key offset inside one window
grace is the part of global_cache_ttl above 2 × window_interval, and zero by default.
With zero grace, up to 3 keys per entity exist at any time. Reads consult 2 (current + previous
epoch); the offset that spreads expiry can hold the generation before those
alive for up to one more window.
Two deployments that share a key prefix must agree on window_interval.
The epoch number is floor(unix / window_interval), so a mismatch splits the
shared counter into separate key namespaces. Capture's AI byte budget is one
such namespace (its prefix carries no capture_mode). The
global_rate_limiter_window_seconds{scope} gauge publishes each process's
running value so an alert can compare them.
Configuration
Capture runs two limiters from these settings: the per-(token, distinct_id) limiter and the AI byte budget. Where the two read different env vars, both are listed.
Rate limiting behavior
| Parameter | Default | Env var (capture) | Description |
|---|---|---|---|
global_threshold |
1,000,000 | GLOBAL_RATE_LIMIT_TOKEN_DISTINCTID_THRESHOLD (capture default 300,000); AI: AI_BYTE_LIMIT_PER_SECOND × the AI window |
Default limit per window per key |
window_interval |
60s | GLOBAL_RATE_LIMIT_WINDOW_INTERVAL_SECS; AI: AI_BYTE_LIMIT_WINDOW_INTERVAL_SECS, else the former |
Sliding window size for the 2-epoch counter |
sync_interval |
15s | GLOBAL_RATE_LIMIT_SYNC_INTERVAL_SECS |
Base re-sync cadence. The pressure tier scales it |
tick_interval |
1s | GLOBAL_RATE_LIMIT_TICK_INTERVAL_MS |
Background pipeline cadence |
min_sync_floor |
10 | GLOBAL_RATE_LIMIT_MIN_SYNC_FLOOR (the AI limiter uses 0) |
Minimum local level before a key is read. Capped at 1% of the key's threshold |
max_read_outage |
window_interval |
GLOBAL_RATE_LIMIT_MAX_READ_OUTAGE_SECS |
How long, in wall-clock time, reads to an instance must keep failing before its keys stop limiting on this node's unconfirmed counts. A replica read that the primary answers counts as a success. Derived from the configured window, not the default. 0 disables the guard |
custom_keys |
empty | GLOBAL_RATE_LIMIT_TOKEN_DISTINCTID_OVERRIDES_CSV; AI: AI_BYTE_LIMIT_OVERRIDES_CSV |
Per-key threshold overrides (key=limit,...) |
custom_key_source |
none | GLOBAL_RATE_LIMIT_CUSTOM_THRESHOLD_KEY |
Redis key of a JSON threshold map that replaces custom_keys once fetched. Per-(token, distinct_id) limiter only |
custom_key_refresh_interval |
60s | GLOBAL_RATE_LIMIT_CUSTOM_THRESHOLD_REFRESH_SECS |
Refresh cadence for custom_key_source |
Throughput bounds
| Parameter | Default | Env var (capture) | Description |
|---|---|---|---|
max_sync_keys_per_tick |
20,000 | GLOBAL_RATE_LIMIT_MAX_SYNC_KEYS_PER_TICK |
Keys read per tick, and write entries sent per tick. The rest wait for later ticks |
max_keys_per_command |
2,000 | GLOBAL_RATE_LIMIT_MAX_KEYS_PER_COMMAND |
Redis keys per command. A read costs two keys per entity |
max_concurrent_commands |
4 | GLOBAL_RATE_LIMIT_MAX_CONCURRENT_COMMANDS |
Commands in flight at once per instance |
max_write_batch_entries |
200,000 | GLOBAL_RATE_LIMIT_MAX_WRITE_BATCH_ENTRIES |
Buffered (key, epoch) entries. At the cap, updates for new keys drop and are counted |
max_pending_sync_entries |
200,000 | GLOBAL_RATE_LIMIT_MAX_PENDING_SYNC_ENTRIES |
Queued syncs. At the cap, new sync requests drop and re-queue on the key's next request |
channel_capacity |
1,000,000 | — | mpsc channel buffer for async update requests. A full channel drops the update |
Local cache (Moka)
| Parameter | Default | Env var (capture) | Description |
|---|---|---|---|
local_cache_max_entries |
300,000 | GLOBAL_RATE_LIMIT_TOKEN_DISTINCTID_LOCAL_CACHE_MAX_ENTRIES (capture default 5,000,000); AI: AI_BYTE_LIMIT_LOCAL_CACHE_MAX_ENTRIES |
Entry cap. Moka's default TinyLFU policy can refuse a new key at the cap, and that key then takes the always-allowed miss path |
local_cache_ttl |
600s | GLOBAL_RATE_LIMIT_LOCAL_CACHE_TTL_SECS |
Time since the entry was last written. Every request and every read rewrites the entry, so this restarts each time. At or above local_cache_idle_timeout it never fires first |
local_cache_idle_timeout |
300s | GLOBAL_RATE_LIMIT_LOCAL_CACHE_IDLE_TIMEOUT_SECS |
Evicts entries no request touched within this time. Hot keys never idle-expire; cold keys reclaim slots |
Redis
| Parameter | Default | Env var (capture) | Description |
|---|---|---|---|
global_cache_ttl |
120s | — | Only the part above 2 × window_interval is used, as clock-skew grace. Keys expire at the absolute deadline from epoch_expire_at, so a smaller value cannot shorten their life |
global_read_timeout |
250ms | GLOBAL_RATE_LIMIT_READ_TIMEOUT_MS |
The limiter's cap on one read command. See Timeouts |
global_write_timeout |
250ms | GLOBAL_RATE_LIMIT_WRITE_TIMEOUT_MS |
The limiter's cap on one write command. See Timeouts |
redis_key_prefix |
@posthog/global_rate_limiter |
— | Prefix for all Redis keys. Capture derives the per-(token, distinct_id) prefix from capture_mode; the AI prefix carries none |
Capture connects each limiter to GLOBAL_RATE_LIMIT_REDIS_URL, with reads going to GLOBAL_RATE_LIMIT_REDIS_READER_URL when that is set.
Without GLOBAL_RATE_LIMIT_REDIS_URL, both limiters share capture's main Redis client, and the GLOBAL_RATE_LIMIT_REDIS_*_TIMEOUT_MS settings do nothing.
The per-(token, distinct_id) limiter can shard its keys across two clusters. GLOBAL_RATE_LIMIT_REDIS_SHARDS=2 adds GLOBAL_RATE_LIMIT_REDIS_SHARD_1_URL and its optional GLOBAL_RATE_LIMIT_REDIS_SHARD_1_READER_URL as the second instance.
Capture connects to the second cluster only when the shard count is 2, so the URLs can ship before the switch. A shard count of 2 without both writer URLs fails startup.
The AI byte limiter always uses the first cluster.
Clamps and tuning
new() clamps three settings and logs a warning when it does:
| Parameter | Clamped to at least | What an undersized value breaks |
|---|---|---|
local_cache_idle_timeout |
window_interval |
Entries expire mid-window and the next request takes the always-allowed miss path |
local_cache_ttl |
window_interval |
Same, for an entry that no request or read rewrites for that long |
global_cache_ttl |
2 × window_interval |
Nothing, because expiry is absolute and a smaller value only removes grace. Capture never sets it, so the warning fires at startup whenever the window is above 60s |
- An entry idles out only when no request touches it for
local_cache_idle_timeout, so a key's sync cadence never evicts an active key. - Under high key cardinality with cold-skewed traffic, a shorter idle timeout reclaims slots faster.
min_sync_floorgates reads only. It never drops a count; it only suppresses theMGETfor keys too far under their threshold to be limited. Counts can still be lost to a full channel, the write-batch cap, a failed write, or the stale-epoch purge.- Writes are bounded separately.
absorb_updatemerges updates by(key, epoch)between ticks, so write volume scales with distinct active keys, not event rate.
Request Flow
Request arrives
│
▼
┌──────────────────────────┐
│ Custom mode, no override │──── Yes ──→ NOT APPLICABLE
└──────┬───────────────────┘
│ No
▼
┌──────────────────────────┐
│ Enqueue the count to the │ (any count > 0, limited or not)
│ mpsc channel │
└──────┬───────────────────┘
▼
┌──────────────────────────┐
│ Cache hit? │──── No ──→ Insert entry with this count,
└──────┬───────────────────┘ queue a sync if at or above
│ Yes the floor, ALLOW
▼
┌──────────────────────────┐
│ Below the floor? no sync │
│ Else due by tier cadence │──── Due ──→ Push to pending_sync
└──────┬───────────────────┘
▼
┌──────────────────────────┐
│ Add this count to │
│ local_pending │
└──────┬───────────────────┘
▼
┌──────────────────────────┐
│ level >= threshold? │──── No ──→ ALLOW
└──────┬───────────────────┘
│ Yes
▼
┌──────────────────────────┐
│ Reads to the key's │
│ instance failing for │──── No ──→ LIMIT
│ max_read_outage? │
└──────┬───────────────────┘
│ Yes
▼
┌──────────────────────────┐
│ Decayed fleet count │──── Yes ──→ LIMIT
│ >= threshold? │
└──────┬───────────────────┘
│ No
▼
ALLOW (counted in outage_limit_skipped_total)
Multi-Redis Partitioning
When multiple Redis instances are configured, work is split by a stable hash of the key, modulo the instance count:
Entity keys Redis instances
┌──────────┐
│ team_1 │ ──hash──→ Redis[0]
│ team_2 │ ──hash──→ Redis[2]
│ team_3 │ ──hash──→ Redis[0]
│ team_4 │ ──hash──→ Redis[1]
│ ... │
└──────────┘
Each partition executes its pipeline independently and in parallel.
Single-instance mode (common case) skips partitioning entirely.
Before raising the instance count
Partitioning is dormant at one instance: instance_index returns 0 and never
hashes. Two things to know before that changes.
- Salt any new per-key derivation.
select_redis_clientandepoch_expire_atboth hash the entity key. If they share a hash, a shard count that shares a factor with the window leaves each shard only a fraction of the expiry offsets and the expiry burst returns per shard. SeeEXPIRY_HASH_DOMAIN. Jitter itself does not need a stable hash, because a disagreement only moves a deadline inside its bounded window. Shard selection does, and uses SipHash-1-3 for that reason. - Changing the instance count splits counters while the rollout is in flight. Pods on the old count and the new one route the same key to different instances, so each sees part of its traffic. This under-counts and therefore fails open, never over-enforces, and it clears once every pod agrees and the current epoch keys expire. Modulo sharding moves N/(N+1) of the keys when the count goes from N to N+1; a consistent-hashing scheme would move about 1/(N+1) of them and shrink this window, but it cannot remove it.
- In capture, change only
GLOBAL_RATE_LIMIT_REDIS_SHARDS, and in one rollout. Every pod must list the clusters in the same order, which the numbered URL variables fix. Ship the shard 1 URLs first with the count at 1, then change the count. Changing it back moves the keys back with the same transient.
Redis load
- The hot path issues no Redis I/O.
- Writes: each tick sends one INCRBY and one EXPIREAT per distinct
(key, epoch)this node saw since the last drain, up tomax_sync_keys_per_tick. - Every node that saw a key writes it, so write volume grows with fleet size as well as with the number of active keys.
- Reads: each tick reads two epoch keys per key that is at or above the floor and due by its tier, up to
max_sync_keys_per_tick.
Metrics
| Metric | Type | Purpose |
|---|---|---|
global_rate_limiter_eval_counts_total |
Counter | Core allow/limit decisions |
global_rate_limiter_cache_counts_total |
Counter | Cache hit/miss/sync_queued, exactly one per evaluation |
global_rate_limiter_outage_limit_skipped_total |
Counter | Limits withheld because the key's instance was in a read outage. It stays at zero while reads work |
global_rate_limiter_read_failing_seconds |
Gauge | Seconds since reads to one instance began failing, per redis_idx, set on each tick that reads. Zero while reads succeed. The guard engages once it reaches max_read_outage |
global_rate_limiter_pipeline_ms |
Histogram | Latency of successful commands only, with no read/write label, so failed and timed-out commands are absent |
global_rate_limiter_tick_ms |
Histogram | Duration of ticks that had work |
global_rate_limiter_pipeline_size |
Histogram | Keys read and (key, epoch) entries written per tick, by op |
global_rate_limiter_commands_per_tick |
Histogram | Commands issued per tick per instance after chunking, by op |
global_rate_limiter_pending_sync_size |
Gauge | Keys read this tick, capped at max_sync_keys_per_tick. For the backlog, use sync_deferred_size |
global_rate_limiter_sync_deferred_size |
Gauge | Keys still queued for sync after a tick took its slice |
global_rate_limiter_write_deferred_size |
Gauge | (key, epoch) entries still batched after a tick took its slice |
global_rate_limiter_update_channel_depth |
Gauge | Updates waiting in the channel when a tick starts. Near zero while the loop keeps up |
global_rate_limiter_sync_skipped_total |
Counter | Syncs not queued because the key is below the floor |
global_rate_limiter_sync_tier_gauge |
Gauge | Entity distribution across tiers (scanned every TIER_SCAN_INTERVAL_TICKS) |
global_rate_limiter_tier_transitions_total |
Counter | Tier changes, counted when a read lands |
global_rate_limiter_cache_size |
Gauge | Live local cache entry count vs cap |
global_rate_limiter_window_seconds |
Gauge | Configured window_interval per scope. Deployments that share a Redis key prefix must report the same value, or their epoch keys diverge and the shared counter splits |
global_rate_limiter_eviction_total |
Counter | Cache evictions by cause (size/expired/explicit) |
global_rate_limiter_estimate_drift |
Histogram | Local vs Redis accuracy |
global_rate_limiter_sync_staleness_ms |
Histogram | Real staleness at access time. A large share sits past one minute in normal operation, from idle keys that stopped syncing below the floor, so read its tail as expected rather than as an outage |
global_rate_limiter_error_total |
Counter | By step and cause: failed commands (redis_write, redis_error, write_timeout, read_timeout, see Timeouts), drops at the caps (channel_full, write_batch_full, pending_sync_full), and stale_epoch_purged |
global_rate_limiter_records_total |
Counter | Keys moved by successful commands, by op: write entries for writes, two result slots per entity for reads |
global_rate_limiter_custom_thresholds_loaded |
Gauge | Custom thresholds applied at the last successful refresh |
global_rate_limiter_custom_thresholds_last_refresh_timestamp |
Gauge | Unix time of the last successful refresh |
global_rate_limiter_custom_thresholds_fetch_total |
Counter | Refresh fetches by result. Not scoped to a limiter |
File Layout
rust/common/limiters/
├── src/
│ ├── global_rate_limiter.rs ← Core implementation and unit tests
│ ├── custom_key_source.rs ← Dynamic custom-key thresholds from Redis
│ └── ...
├── tests/
│ └── global_rate_limiter_integration_tests.rs ← Real-Redis tests
├── benches/
│ └── global_rate_limiter.rs ← Benchmarks
└── Cargo.toml
rust/capture/src/
├── global_rate_limiter.rs ← Capture service wrapper
├── config.rs ← Env config for both capture limiters
└── prometheus.rs ← Histogram bucket registration