Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

49.0 KB
Raw Blame

HyperCache System

PostHog's HyperCache provides multi-tier caching with Redis → S3 → Database fallback. It's designed for high-traffic, read-heavy endpoints where pre-caching every possible value is worth the storage cost.

Architecture overview

┌─────────────────────────────────────────────────────────────────┐
│                         Client Request                          │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Redis (Layer 1)                          │
│                        TTL: 30 days                             │
│                        Latency: ~1-2ms                          │
└─────────────────────────────────────────────────────────────────┘
                                │ miss
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                         S3 (Layer 2)                            │
│                        Latency: ~50-100ms                       │
│                        Warms Redis on hit                       │
└─────────────────────────────────────────────────────────────────┘
                                │ miss
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Database (Layer 3)                         │
│                        Latency: ~200-500ms                      │
│                        Warms Redis + S3                         │
└─────────────────────────────────────────────────────────────────┘

Core components

Component File Purpose
HyperCache class posthog/storage/hypercache.py Multi-tier cache with fallback
Local evaluation posthog/models/feature_flag/local_evaluation.py Feature flag caching for SDKs
Remote config posthog/models/remote_config.py Client configuration caching
Team caching posthog/models/team/team_caching.py Authentication team lookup cache

HyperCache class

The HyperCache class in posthog/storage/hypercache.py provides the core caching logic.

Creating an instance

from posthog.storage.hypercache import HyperCache

cache = HyperCache(
    namespace="feature_flags",           # Category name for metrics
    value="flags_with_cohorts.json",     # Value identifier for metrics
    load_fn=lambda key: load_data(key),  # Fallback loader when cache misses
    token_based=False,                   # Use team ID (False) or API token (True)
    cache_ttl=60 * 60 * 24 * 30,        # 30 days for cache hits
    cache_miss_ttl=60 * 60 * 24,        # 1 day for cache misses
    enable_etag=True,                    # Enable HTTP 304 support
)

Cache key formats

The token_based parameter determines the cache key structure:

  • Team ID-based (token_based=False): cache/teams/{team_id}/{namespace}/{value}
  • Token-based (token_based=True): cache/team_tokens/{api_token}/{namespace}/{value}

Key methods

Method Purpose
get_from_cache(key) Get cached data, returns None on miss
get_from_cache_with_source(key) Returns (data, source) where source is redis, s3, or db
get_if_none_match(key, etag) ETag support for HTTP 304 responses
update_cache(key) Force refresh from database
set_cache_value(key, data) Write to both Redis and S3
clear_cache(key) Delete from Redis and S3

ETag support

HyperCache supports HTTP 304 "Not Modified" responses to reduce bandwidth:

data, etag, modified = cache.get_if_none_match(team, client_etag)

if not modified:
    return HttpResponse(status=304, headers={"ETag": etag})
else:
    return JsonResponse(data, headers={"ETag": etag})

ETags are computed as SHA256 hashes of the JSON content.

Service cache (Rust)

The feature-flags Rust evaluation service uses a separate HyperCache instance defined in products/feature_flags/backend/flags_cache.py. Unlike the local evaluation cache (which serves SDKs with cohort definitions and group type mappings), the service cache provides raw flag data plus pre-computed dependency metadata so the Rust service can evaluate flags in the correct order without recomputing the dependency graph on every request.

Cache instance

# products/feature_flags/backend/flags_cache.py
flags_hypercache = HyperCache(
    namespace="feature_flags",
    value="flags.json",
    load_fn=lambda key: _get_feature_flags_for_service(HyperCache.team_from_key(key)),
    cache_ttl=settings.FLAGS_CACHE_TTL,
    cache_miss_ttl=settings.FLAGS_CACHE_MISS_TTL,
    cache_alias=FLAGS_DEDICATED_CACHE_ALIAS if FLAGS_DEDICATED_CACHE_ALIAS in settings.CACHES else None,
    batch_load_fn=_get_feature_flags_for_teams_batch,
    expiry_sorted_set_key=FLAGS_CACHE_EXPIRY_SORTED_SET,
)

The _get_feature_flags_for_service function fetches all flags for a team (including inactive, but excluding deleted and encrypted remote config flags), then returns a cache payload trimmed to the flags worth caching. The Rust service filters out inactive flags at request time via filtered_out_flag_ids.

Before anything serializes or reads a flag, _omit_unsupported_flags classifies each stored filters document with detect_config_format (products/feature_flags/backend/facade/config.py). A config version 1 document (no version, or a numeric 1) is published, and so is an active config version 2 document that the shared validator (validate_config, under the deployed MAX_FEATURE_FLAG_FILTER_SIZE_BYTES) admits; that document is published verbatim and carries no cohort or flag references. Any other v2 document, an unsupported discriminator, a document that is not a JSON object, or an evaluable v1 document whose release conditions cannot be read is omitted, together with every flag whose dependency conditions reference it, transitively. That applies to inactive and archived rows too, so an inactive v2 or non-object row is never blanked into a v1-shaped {"groups": []} entry, and a dependent with flag_evaluates_to: false on it never matches against a target the matcher never evaluated. The Rust reader parses the published v2 document again; a document Python admits but Rust rejects is served as that one flag with failed: true and the flag_data_parsing_error reason while the team's other flags evaluate. The rebuild still succeeds with the remaining flags, the stored rows are not modified, and the omitted ids are logged. Unevaluable v1 rows are not read (_is_unevaluable), so a disabled row with an unreadable document keeps its established behavior: kept and blanked when referenced, dropped otherwise. Cohort references and flag dependencies are then read from the surviving flags' filters through products/feature_flags/backend/facade/references.py.

Because inactive flags are filtered before the matcher reads filters, an inactive flag can never affect a response, so the payload keeps only evaluable flags plus the inactive flags that another flag's dependency conditions reference. A referenced entry is load-bearing: the matcher pre-seeds its id as false, so a dependent with flag_evaluates_to: false on a disabled flag still matches instead of missing a dependency. _drop_unreferenced_unevaluable_flags removes the rest, evaluation_metadata is computed on the surviving set, and _blank_inactive_filters replaces the kept unevaluable flags' filters with an empty {"groups": []} before the payload is written. build_flags_cache in rust/feature-flags/src/flags/cache_builder.rs writes the same entry and applies the same omission, drop, and blanking rule: omit_unsupported_flags classifies the raw documents that from_pg_keeping_undecodable loads, keeping an active v2 row whose parse succeeded. The service's PostgreSQL fallback runs the same classification, so a Redis miss serves the flag set the cache would have carried. Both builders run the same fixture, rust/feature-flags/tests/fixtures/flags_cache_config_formats.json. One known divergence remains: Rust decodes v1 documents into typed structs, so an evaluable row it cannot decode (a property filter without a type key, for example) is omitted there with its dependents, while Python, which only reads the cohort and flag references, still publishes it. When the Rust service reads a Python-written entry that holds such a row, it skips that one flag, counts it in flags_flag_malformed_filter_total, and marks every flag that depends on it as missing a dependency, so the team's other flags still evaluate. That response sets errorsWhileComputingFlags, and so does a PostgreSQL fallback response that omits such a row, so an SDK keeps its stored value for the missing flag. An entry the Rust builder wrote already lacks the row, so a response served from it does not report the missing flag. verify_team_flags reports and repairs such a team, attributed to the writer that produced it. Python rejects a v1 row as well when its groups, or a group's properties, is neither falsy nor a list of objects, so neither builder publishes that row and no response reports it. An entry written before the builders omitted unsupported non-v1 rows can hold an inactive non-v1 row already blanked to {"groups": []}; its cached bytes cannot recover the original format, so the entry is rebuilt from the database rows by the next invalidation, refresh sweep, or verifier repair rather than reclassified in place. Until then an unreferenced blanked row is inert for the matcher, and an active dependent of one reports as STALE_IN_CACHE and triggers the repair. Parity is per team, not per byte: each team has one primary writer (teams whose invalidation routes to Kafka via KAFKA_ROUTING_FLAG get the Rust builder, every other team Python), the Python verifier remains a repair writer for every team, and the two serializers order keys differently — so what must match is the flag set, fields, and metadata, not the bytes or etag. Verifier fixes on the flags cache carry a writer label (posthog_hypercache_verify_fixes_total{cache_type="flags", writer="rust"|"python"|"unknown"}), attributed by evaluating the same routing flag (get_team_primary_flags_writer in flags_cache.py): a fix on a rust-routed team is the parity signal that the Rust builder diverged, which the unattributed counter blends into Python's baseline repair noise. unknown means the routing flag couldn't be evaluated at fix time, so an attribution outage can't masquerade as a clean Rust ramp. Old-shape entries that still carry unreferenced inactive rows stay valid: the matcher never reads those rows, and verify_team_flags suppresses them instead of reporting STALE_IN_CACHE, so they converge through flag edits and TTL rather than a fleet-wide repair. Deploy Django ahead of the Rust images: an older Python verifier reports a Rust-written entry's dropped rows as missing, repairs them back, and the next Rust build removes them again. The reverse skew is safe — the newer verifier tolerates the extra rows old writers leave.

Cache payload structure

{
  "flags": [
    /* serialized flag dicts */
  ],
  "evaluation_metadata": {
    "dependency_stages": [[1, 5], [3], [7]],
    "flags_with_missing_deps": [9, 12],
    "transitive_deps": { "3": [1, 5], "7": [1, 3, 5] }
  }
}

The evaluation_metadata fields:

Field Type Description
dependency_stages list[list[int]] Flag IDs grouped by evaluation order. Stage 0 contains flags with no dependencies; stage N depends only on flags in stages 0…N-1. Flags within the same stage can be evaluated in parallel.
flags_with_missing_deps list[int] Flag IDs whose dependencies are missing, cyclic, or transitively broken. The Rust service treats these as evaluation errors.
transitive_deps dict[str, list[int]] Map of stringified flag ID to the sorted list of all its transitive dependency flag IDs.

Dependency computation

The _compute_flag_dependencies function in flags_cache.py builds the evaluation metadata using Kahn's algorithm (layered topological sort). The algorithm:

  1. Extracts direct dependencies from each flag's filters.groups[*].properties where type == "flag".
  2. Builds an in-degree map and reverse-dependency edges across all flags.
  3. Peels layers of zero-in-degree nodes, computing transitive dependency closures as it goes. Each layer becomes one evaluation stage.
  4. Detects cycles — any flag still with in-degree > 0 after all layers are peeled is a cycle participant. Cycled flags and any flag that transitively depends on them are added to flags_with_missing_deps.

This matches the Rust fallback path's petgraph-based cycle handling, where all cycle participants are excluded from stages (not just back-edge targets).

Local evaluation caching

Feature flag local evaluation uses a HyperCache instance in products/feature_flags/backend/local_evaluation.py:

flag_definitions_hypercache = HyperCache(
    namespace="feature_flags",
    value="flags_with_cohorts.json",
    load_fn=lambda key: _get_flags_response_for_local_evaluation(HyperCache.team_from_key(key)),
    enable_etag=True,
    # Set only when FLAGS_REDIS_URL is configured. See "Dedicated flags Redis" below.
    cache_alias=FLAGS_DEDICATED_CACHE_ALIAS,
    secondary_cache_alias="default",
)

It includes full cohort definitions and group type mappings, since all current SDKs support cohort evaluation locally. A legacy flag_definitions_without_cohorts_hypercache variant — pre-flattened cohort filters for SDKs too old to evaluate cohorts locally — was removed once nothing served it to real clients anymore.

The builder classifies stored filters before reading cohort references, transforming dependencies, or serializing flags. Only absent or numeric version 1 configurations enter the legacy feed. Classification includes inactive and deleted targets before existing lifecycle filtering omits them. Unsupported formats are expected exclusions; malformed flags increment posthog_flag_definitions_processing_error. A malformed flag or reachable cohort removes the affected flag and its transitive dependents while independent flags remain available. Direct and nested cohort references must be integers or strings that parse as integers; booleans, floats, and unparseable values are rejected before serialization. Legacy cohort property dictionaries can use type or values as property names; a grouped expression requires both keys. Dependencies on excluded targets are omitted even when the condition expects false. An inconclusive dependency does not always force an SDK to use server evaluation: a later condition can return a different variant. For example, if the first condition selects blue when the target is true and the next always selects green, omitting only the target can make the SDK return green instead of blue. Removing the dependent prevents that local answer, at the cost of sending the whole flag to server evaluation or the caller's local-only default. Supported v1 missing targets, inactive targets, cycles, cohort scoping, mappings, and metadata retain their existing behavior. The internal flags.json producer keeps its separate rejection and inactive-filter behavior.

Readers serve cached bodies and answer 304s without re-checking them, so the exclusion relies on deployment order. For well-formed v1 data with string dependency references, this builder preserves the older builder's definitions. Integer dependency references now resolve to flag keys and populated dependency chains. Malformed flags and reachable cohorts now exclude affected flags and their transitive dependents. These cases can change cached bodies and ETags even before an unsupported configuration exists, including during a rollback. Deploy this builder to every region before storing an unsupported configuration, and never deploy an older builder after one exists. Tightening these exclusion rules later requires rebuilding existing cache entries.

Cache invalidation

Django signals trigger cache updates when models change:

@receiver([post_save, post_delete], sender=FeatureFlag)
def feature_flag_changed(sender, instance, **kwargs):
    transaction.on_commit(lambda: update_team_flags_cache.delay(instance.team_id))

Models that invalidate the flags cache:

  • FeatureFlag - Flag created, updated, or deleted
  • Cohort - Cohort properties changed
  • FeatureFlagEvaluationTag - Evaluation tags changed
  • Tag - Tag renamed

Deleting a Team clears its local evaluation and team metadata entries from Redis and S3 after the deletion commits. The rebuild task cannot run for a deleted team, and the flags service falls back to the S3 copies, which have no TTL. A stale team metadata entry lets the flags service resolve the deleted team's token without a Postgres check.

The staff cache tools clear action also accepts the ids of deleted teams. Use it to purge an orphaned flags entry for a team that no longer exists. It does not clear team metadata entries, because it takes team ids and those entries are keyed by API token.

Remote config caching

Remote config uses a different caching strategy than local evaluation. It prioritizes zero-overhead cache hits by using Redis-only caching for serving, with HyperCache used only for background sync operations.

Cache lookup flow

def _get_config_via_cache(cls, token: str) -> dict:
    key = f"remote_config/{token}/config"

    data = cache.get(key)  # Direct Redis lookup
    if data:
        return data  # Cache hit - zero database calls

    # Cache miss - query database and warm cache
    remote_config = RemoteConfig.objects.select_related("team").get(team__api_token=token)
    data = remote_config.build_config()
    cache.set(key, data, timeout=CACHE_TIMEOUT)  # 1 day

    return data

No authentication

Remote config endpoints have no authentication (authentication_classes = []). The token in the URL is treated as a public identifier, not a credential. This eliminates all authentication overhead for cache hits.

CDN integration

When remote config changes, the system:

  1. Updates the database record
  2. Warms the HyperCache (Redis + S3)
  3. Updates the Redis serving cache
  4. Purges Cloudflare CDN cache

Team authentication caching

Team objects are cached by API token in posthog/models/team/team_caching.py to avoid database lookups during authentication.

Cache format

# Cache key
f"team_token:{api_token}"

# TTL: 5 days
FIVE_DAYS = 60 * 60 * 24 * 5

Usage

from posthog.models.team.team_caching import get_team_in_cache, set_team_in_cache

# Check cache first
team = get_team_in_cache(api_token)
if team is None:
    team = Team.objects.get(api_token=api_token)
    set_team_in_cache(api_token, team)

The cached team is serialized using CachingTeamSerializer and reconstructed as a Team instance on retrieval.

Read repair

Both readers write an S3 hit back into Redis so the next read of the same key is served by Redis instead of paying another S3 read.

The Django reader (HyperCache.get_from_cache_with_source) repairs on every S3 hit, synchronously, through the normal write path with the full cache TTL.

The Rust reader (rust/common/hypercache) has an opt-in variant, deliberately more conservative than Django's:

  • Repairs only on a confirmed Redis miss. A read that fell through to S3 because Redis errored or timed out does not repair, since that tier is degraded and repair writes would add load to it.
  • Writes with SET NX, so a repair never overwrites an existing entry and concurrent repairs of one cold key collapse into one write.
  • Uses a short TTL (HYPERCACHE_READ_REPAIR_TTL_SECONDS, default 600) and does not register the entry in the expiry sorted set, so the Django refresh job stays the only owner of an entry's real lifetime. The short TTL also bounds how long a resurrected entry lingers in the HyperCacheWriter::delete race, where a reader that read S3 just before the delete writes the key back afterwards.
  • Is detached and best-effort: a Redis failure cannot fail a request that already has its value.
  • Refuses etag-enabled namespaces at reader construction (with a startup warning), because the payload and its companion :etag key are written atomically by the writer and a payload-only repair would leave the pair inconsistent.

Enabled by default for the readers with no in-process cache in front of them: array/config.json in the feature-flags service, and surveys and array/config.json in hypercache-server. The feature_flags readers are excluded: they are etag-paired, and FlagDefinitionsCache already absorbs repeat reads of a cold key in process. team_metadata is excluded too: a hit there is trusted as proof of a valid token with no Postgres re-check, and team deletion clears Redis before S3, so a repair landing in that gap would resurrect a deleted team in Redis for up to the repair TTL.

Operational controls:

  • HYPERCACHE_READ_REPAIR_TTL_SECONDS=0 disables repair for a service without a code change. The deployment charts pass arbitrary env vars through .Values.env, so no chart change is needed.
  • In the feature-flags service, SKIP_WRITES=true also disables repair, keeping read-only instances read-only.
  • The posthog_hypercache_read_repair counter tracks repair outcomes. A sustained high repair rate means keys are repeatedly going cold (eviction, expiry outpacing the refresh job), which the S3 read rate alone no longer shows because repair suppresses it.

Cache TTL settings

Cache Hit TTL Miss TTL
HyperCache (default) 30 days 1 day
Remote config serving 1 day 1 day
Team authentication 5 days N/A (deleted on miss)
Rust read repair 600s N/A (no miss writes)

Performance characteristics

Scenario Latency Database calls
Local evaluation (Redis hit) ~1-2ms 0 (but auth overhead)
Remote config (Redis hit) ~1ms 0
S3 fallback ~50-100ms 0
Database fallback ~200-500ms 1+

Prometheus metrics

Metric Labels Purpose
posthog_hypercache_get_from_cache result, namespace, value Cache hit/miss tracking
posthog_hypercache_sync result, namespace, value Cache sync task outcomes
posthog_hypercache_sync_duration_seconds result, namespace, value Cache sync timing
posthog_remote_config_via_cache result Remote config cache performance
posthog_hypercache_read_repair result, namespace, value Rust reader repair outcomes
flags_flag_definitions_etag_total result Rust reader ETag read outcomes
posthog_hypercache_verify_errors_total cache_type, reason Teams the verify sweep could not check
posthog_hypercache_verify_fix_failures_total cache_type, issue_type, reason Repairs the verify sweep could not write

Result labels: hit_redis, hit_s3, hit_db, missing, batch_miss

ETag result labels: hit (client ETag matched, 304), miss (client sent a stale ETag), none (client sent none), redis_missing (the endpoint that answered held no ETag key), redis_error (the read failed, or the stored value did not decode)

redis_missing and redis_error are the pair that separates a cache-tier problem from an unreachable cluster. Keep them apart on dashboards and alerts. redis_missing reports the read, not the cause: reads go to the replica, so a key Django wrote to the primary counts here until it replicates. A sustained rise is a tier that holds nothing, and a short burst that clears on its own is lag.

hit, miss, and none partition every request. The two failure labels sit on top of that partition, and do not slice it. A failed ETag read increments redis_missing or redis_error, then falls through and increments none or miss as well. So read a failure label as a ratio over hit + miss + none, and never as a share of a stacked total. Stacked, the total exceeds the request rate, and none climbs in step with redis_missing for the same underlying cause.

Read repair result labels: success, skipped (key already existed, repair deferred to it), error

skipped also covers replica lag: reads go to the replica and repairs to the primary, so a key written to the primary but not yet replicated reads as cold and its repair is correctly refused.

The verify sweep counts a team it cannot check, and a repair it cannot write, then continues. Both carry a closed reason label, so every series is pre-created at import and an alert catches the first occurrence rather than the second. Verify reasons: dependency_unavailable (a Redis, S3 or Postgres outage, or a HyperCacheDependencyUnavailable from a load_fn), data_error (an entry the sweep cannot parse, which repeats on the same teams every run), unknown. Fix failures add update_fn_returned_false for a write path that returned False rather than raising. That covers a genuine refusal, such as a cache whose Redis URL is unset, and also a write error that update_cache caught and turned into False. The second case is reported by update_cache itself, in its own log line and, when a dependency was unavailable, in posthog_hypercache_rebuild_skipped. The label applies only where the sweep falls back to update_fn, because the direct write it normally uses raises, and those failures carry the reason of the exception. The exception class name is in the log line as error_type. A write the config vetoes is counted as neither a fix nor a failure.

The sweep's other give-ups stay log-only on purpose: a team the grace period skips, and the batch-level fallbacks that degrade a batch to per-team reads. Both slow the sweep rather than leave an entry broken, and a sweep that runs out of time is already counted by posthog_hypercache_verification_incomplete_runs_total.

Debugging

Check Redis cache

# Local evaluation flags
redis-cli get "cache/teams/{team_id}/feature_flags/flags_with_cohorts.json"

# Remote config
redis-cli get "remote_config/{api_token}/config"

# Team authentication
redis-cli get "team_token:{api_token}"

When FLAGS_REDIS_URL is set, the local-evaluation flags key is written to the dedicated instance and mirrored to the shared cache, so redis-cli returns whichever copy the cluster you point it at holds. The two can disagree. Read the flag-definitions notes under "Dedicated flags Redis" before you act on either copy.

Elevated S3 read rate on /flags/definitions

HyperCacheElevatedS3ReadRate and HyperCacheCriticalS3ReadRate fire when the reader serves flags_with_cohorts.json from S3 for too large a share of reads. Redis and the cache tier both produce that symptom, so establish which one first.

1. Separate a cluster fault from an empty tier. Split flags_flag_definitions_etag_total by result. A rise in redis_error points at the cluster; check managed-cache CPU, evictions, command latency, and memory before going further. redis_error also covers a stored ETag that did not decode, so a flat count of one or two teams is corrupt data rather than a cluster fault. A rise in redis_missing means Redis answered and the endpoint that served the read held no ETag key. That points at the tier rather than at the cluster, but the label does not prove the entry is gone: reads go to the replica, so replication lag reads as absence too. Confirm it in step 2 before you rebuild anything.

2. Compare the endpoints for one affected team. Django writes the dedicated instance and mirrors to the shared one, so the two can disagree. Which copy the reader served depends on FLAG_DEFINITIONS_DEDICATED_REDIS_ENABLED, so read flags_flag_definitions_reads_dedicated_redis first and start from the cluster it names. Reading the wrong endpoint makes a healthy mirror look broken.

Take a team id from a reader Cache hit for flag definitions record with source="s3". It logs at info, carries team_id, and names a team the alert is counting. An absent ETag writes no log record at all, so the redis_missing counter is the only signal that it happened.

Query the replica of the cluster the gauge named. That endpoint answered the read the metric counted, and that cluster's primary can hold a key the replica does not. Either reader URL can be unset, in which case reads go to the matching writer URL. The replica commands fall back the same way, so that cluster's two commands then return the same answer.

Check the payload key and its ETag key on each endpoint. Redis evicts per key, so the pair can diverge although the writer sets both with one TTL. The payload is the key the alert counts, and the ETag key explains the counters from step 1.

KEY="posthog:1:cache/teams/{team_id}/feature_flags/flags_with_cohorts.json"

# Shared replica, served to pods reporting reason="disabled" or reason="no_dedicated_client"
redis-cli -u "${REDIS_READER_URL:-$REDIS_URL}" exists "$KEY"
redis-cli -u "${REDIS_READER_URL:-$REDIS_URL}" exists "$KEY:etag"

# Shared primary, which the mirror writes
redis-cli -u "$REDIS_URL" exists "$KEY"
redis-cli -u "$REDIS_URL" exists "$KEY:etag"

# Dedicated replica, served to pods reporting reason="dedicated"
redis-cli -u "${FLAGS_REDIS_READER_URL:-$FLAGS_REDIS_URL}" exists "$KEY"
redis-cli -u "${FLAGS_REDIS_READER_URL:-$FLAGS_REDIS_URL}" exists "$KEY:etag"

# Dedicated primary, which Django writes first
redis-cli -u "$FLAGS_REDIS_URL" exists "$KEY"
redis-cli -u "$FLAGS_REDIS_URL" exists "$KEY:etag"

Read the branches below per key.

An absent payload beside a present ETag on the endpoint the reader served is the reading that every ETag-only check calls healthy. The handler reads the ETag key on every request and answers 304 before it fetches the payload, so a team whose SDKs poll with a matching If-None-Match keeps the ETag key recent while the payload ages toward eviction. Read repair is disabled for this namespace, so the S3 hit does not rewarm the payload. Rebuild it with update_flag_caches.

An absent ETag beside a present payload serves a 200 with the full payload on every poll, so it writes no source="s3" record and raises no alert. redis_missing climbing with no matching rise in S3 reads is that state. It does not repair itself: verify_team_flag_definitions compares the payload only, so the hourly verifier reads the team as clean and the ETag stays missing until the team's next flag change or its TTL refresh. The counter carries no team_id, so set TEAM_IDS_TO_TRACK to name a suspected team, or rebuild with update_flag_caches.

Absent on the replica of the cluster the reader served, and present on that cluster's primary, is replication lag rather than a lost entry. Rebuilding fixes nothing. Check replication lag on that cluster instead, and expect the alert to clear on its own.

Present on the dedicated cluster and absent on both shared endpoints isolates the fault to the mirror rather than to the writer. Absent everywhere means the entry was never built or has aged out; rebuild it with update_flag_caches and look at step 3.

A shared copy that is absent while the dedicated copy is present does not come back on its own. Read repair is disabled for ETag-enabled namespaces, the hourly verifier reads only the primary, and the refresh task selects teams by an expiry score stamped from the primary, so a team whose dedicated entry is fresh is never revisited. The team keeps reading from S3 until its next flag change, or until the primary entry nears its TTL. update_flag_caches writes both tiers and ends it sooner.

3. Confirm the writer runs. Check the success and duration signals for the flag-definitions refresh and verification tasks. Tasks that run at their normal cadence and duration while entries stay missing point back at the mirror, not at a stalled writer.

Mirror failures increment posthog_hypercache_mirror_failure and log HyperCache secondary cache op failed. Neither firing while the shared copy is absent means the mirror is not attempting the write at all, rather than attempting it and failing.

Check cache source in responses

Local evaluation responses include cache source information via Prometheus metrics. Check the posthog_hypercache_get_from_cache metric with the appropriate labels.

Force cache refresh

from posthog.models.feature_flag.local_evaluation import update_flag_caches
from posthog.models.team import Team

team = Team.objects.get(id=123)
update_flag_caches(team)

Common issues

Symptom Likely cause Solution
Stale data after flag change Signal not firing Check transaction.on_commit is used
Cache misses in production Redis unreachable, or the tier holds no entry Split the two with flags_flag_definitions_etag_total: redis_error is the cluster, redis_missing is the tier
S3 fallback errors Object storage misconfigured Verify OBJECT_STORAGE_ENABLED setting
ETag mismatches Non-deterministic JSON serialization HyperCache uses sort_keys=True

Dedicated flags Redis

The feature-flags Rust service can use a separate Redis instance for caching, isolated from the shared Django cache. This prevents flag cache operations from affecting other cache users.

Enabling dedicated Redis

FLAGS_REDIS_URL=redis://flags-redis:6379            # Separate instance for flags
FLAG_DEFINITIONS_DEDICATED_REDIS_ENABLED=false      # Cluster the /flags/definitions reader uses

When FLAGS_REDIS_URL is set, Django registers it as the flags_dedicated cache alias (FLAGS_DEDICATED_CACHE_ALIAS in posthog/caching/flags_redis_cache.py, wired up in posthog/settings/data_stores.py). Four HyperCache instances bind that alias. For flags (products/feature_flags/backend/flags_cache.py), remote config, and team metadata, Django writes to the dedicated instance and the Rust service reads them from it.

The SDK-facing flag-definitions cache (local_evaluation.py) is part-way through the same move, and its read side is switchable. Django writes it to the dedicated instance and mirrors each write to the shared default cache, covering the payload, the ETag, and the cache-miss sentinel. Deletes mirror as well.

FLAG_DEFINITIONS_DEDICATED_REDIS_ENABLED decides which cluster the Rust /flags/definitions reader uses. It defaults to false, which keeps the reader on the shared cache. Set it per fleet to move the reader to the dedicated instance.

Remove the mirror only after that move has baked. The mirror is what makes the switch reversible. With the mirror gone, a reader sent back to the shared cache reads a cluster nothing writes to. This endpoint has no database fallback on a miss.

The payload and the ETag always share one client. They therefore cannot disagree about which cluster holds the current definitions. HyperCacheReader::get_etag reads the same redis_client as the payload and derives its key from the same config. Both are set once in the constructor, from the client server.rs resolves.

Do not give the ETag a gate of its own. A stale ETag on one cluster can match a client If-None-Match while the other cluster holds a newer payload. The endpoint then answers 304 and pins the SDK to stale definitions. No metric records an error.

flags_flag_definitions_reads_dedicated_redis reports which cluster each pod resolved. Its reason label separates two cases. disabled is a pod nobody has switched. no_dedicated_client is a pod that was switched but could not build a dedicated client, so it reads the shared cache for the life of the process.

Reads on the dedicated instance go to its -ro reader endpoint. NotFound is unrecoverable, so ReadWriteClient does not consult the primary. A key the writer just wrote reads as absent until it replicates. That window serves a 200 with the full payload instead of a 304.

The flag-definitions self-heal queue follows the write side, not the read side. The Rust endpoint enqueues a rebuild request on the dedicated instance (State::flags_namespace_redis_client), and the Celery drain reads the queue from flag_definitions_hypercache.redis_url (products/feature_flags/backend/rebuild_queue.py). Both resolve from FLAGS_REDIS_URL, so the producer and the consumer move together on configuration. They can still split on connection state: a Rust process that cannot reach the dedicated cluster at startup falls back to the shared one and enqueues there for its whole life, while Celery keeps draining the dedicated one. Those teams wait for the hourly verifier. server.rs logs that startup failure at error level, and the same failure already sends the flags.json, team-metadata, and remote-config readers to the shared cluster, where Django writes nothing. Django and the Rust fleet deploy independently, so a deploy of one before the other leaves requests on the cluster the other side is not reading. Clean up on the shared cluster only, and never on the dedicated one, which holds the live queue once both sides are up. A Rust-first rollout puts the window's requests on the dedicated cluster, where the drain collects them as soon as Django deploys, so they need no cleanup. A Django-first rollout puts them on the shared cluster, where nothing reads them again. Either order also leaves the pre-move queue members and the open circuits on the shared cluster. Run DEL flag_definitions:rebuild_requests flag_definitions:rebuild_circuit there after both sides are deployed. The circuit-breaker set is included because only the drain prunes it and the key carries no TTL. The cooldown and failure-streak keys expire on their own.

FLAG_DEFINITIONS_REBUILD_ON_S3_HIT_ENABLED defaults to false. Set it to true on the Rust definitions fleet to rebuild entries that S3 serves after a confirmed Redis miss. FLAG_DEFINITIONS_SELF_HEAL_ENABLED must also be true, and the Celery rebuild drain must run. The Rust service reads both settings at startup. Restart its pods after a setting change. The service puts S3-hit requests in a lower-priority queue, so cache misses that return 503 drain first. Both queues use ZADD NX so repeated polls keep the first enqueue time. The drain skips a queued team if Redis already holds its payload and ETag.

During rollout, watch flags_flag_definitions_rebuild_requested_total by trigger and result. Also watch posthog_flag_definitions_rebuild_queue_depth and posthog_flag_definitions_rebuild_oldest_age_seconds. Set the S3-hit variable to false and restart the Rust pods to stop new S3-hit requests. Set self-heal to false and restart them to stop both triggers. The Celery drain continues to process requests that are already queued after either setting is disabled.

The Rust service only operates when FLAGS_REDIS_URL is configured. All cache update functions check this setting and skip operations if not set.

Scheduled tasks

Cache freshness is maintained through scheduled Celery tasks.

Task Schedule Purpose
refresh_expiring_flags_cache_entries Hourly at :15 Refresh caches with TTL < 24h before they expire
cleanup_stale_flags_expiry_tracking_task Daily at 3:15 AM Remove expired team entries from tracking sorted set
refresh_expiring_flag_definitions_cache_entries Hourly at :35 Refresh flag definitions caches before they expire
verify_and_fix_flag_definitions_cache_task Hourly at :50 Verify flag definitions cache against database

Refresh task

The hourly refresh job prevents cache misses by proactively refreshing entries before they expire:

# posthog/tasks/feature_flags.py
@shared_task
def refresh_expiring_flags_cache_entries():
    successful, failed = refresh_expiring_flags_caches(
        ttl_threshold_hours=settings.FLAGS_CACHE_REFRESH_TTL_THRESHOLD_HOURS,  # Default: 24
        limit=settings.FLAGS_CACHE_REFRESH_LIMIT,  # Default: 5000
    )

The task uses a Redis sorted set (flags_cache_expiry) to efficiently find expiring entries without scanning all keys.

Verification tasks

The flag definitions verification task (runs hourly at :50) compares cached flag definitions against the database and fixes discrepancies:

  1. Acquires a distributed lock so overlapping scheduled runs skip instead of duplicating work
  2. Samples teams from the cache
  3. Compares cached flags to current database state
  4. Auto-fixes mismatches by refreshing the cache
  5. Reports metrics on match/mismatch/miss rates

The task has a 35-minute soft / 40-minute hard time limit. It winds down at a batch boundary two minutes before the soft limit, recording the partial run under reason="deadline" in posthog_hypercache_verification_incomplete_runs_total; the next run restarts from the first team.

Configuration:

FLAGS_CACHE_VERIFICATION_GRACE_PERIOD_MINUTES=5  # Skip recently updated flags

For initial cache build

Scheduled tasks only maintain existing caches. For initial population, use the management command:

python manage.py warm_flags_cache [--team-ids ID1 ID2 ...]

By default, the command warms caches only for teams that have ever had a feature flag (using config.get_teams_queryset()). This includes teams whose flags have all been soft-deleted (so the cache correctly contains an empty flags list). Teams that have never had any feature flags are skipped to avoid unnecessary database queries and Redis writes. When --team-ids is provided, those specific teams are warmed regardless of the config scoping.

Signal handlers

Django signals automatically invalidate the cache when models change.

Models that trigger cache updates

Model Signal Action
FeatureFlag post_save Refresh team's flags cache
FeatureFlag post_delete Refresh team's flags cache
Team post_save Warm cache for new team
Team post_delete Clear team's flags cache
FeatureFlagEvaluationTag post_save Refresh team's flags cache
FeatureFlagEvaluationTag post_delete Refresh team's flags cache
Tag post_save Refresh caches for teams using tag

Transaction safety

All signal handlers use transaction.on_commit() to avoid race conditions:

# products/feature_flags/backend/flags_cache.py
@receiver([post_save, post_delete], sender=FeatureFlag)
def feature_flag_changed_flags_cache(sender, instance, **kwargs):
    transaction.on_commit(lambda: update_team_service_flags_cache.delay(instance.team_id))

This ensures the cache update task runs after the database transaction commits, not before.

Cohort invalidation

The local evaluation cache (for SDKs) also invalidates when cohorts change. See posthog/models/feature_flag/local_evaluation.py for the full signal handler list.

Configuration

# Required
REDIS_URL=redis://localhost:6379

# Dedicated flags Redis (optional)
FLAGS_REDIS_URL=redis://flags-redis:6379

# Cache TTL settings
FLAGS_CACHE_TTL=604800             # 7 days (default)
FLAGS_CACHE_MISS_TTL=86400         # 1 day (default)

# Scheduled task settings for the flags cache sweep
FLAGS_CACHE_REFRESH_TTL_THRESHOLD_HOURS=24  # Refresh caches expiring within 24h
FLAGS_CACHE_REFRESH_LIMIT=5000              # Max teams per refresh run
FLAGS_CACHE_VERIFICATION_GRACE_PERIOD_MINUTES=5  # Skip recently updated flags

# Scheduled task settings for the flag definitions cache sweep.
# Each one defaults to the resolved flags value above, so both sweeps move together.
# Uncomment one to move this sweep alone.
# FLAG_DEFINITIONS_CACHE_REFRESH_TTL_THRESHOLD_HOURS=12
# FLAG_DEFINITIONS_CACHE_REFRESH_LIMIT=2000

# For S3 fallback
OBJECT_STORAGE_ENABLED=true
AWS_S3_BUCKET_NAME=posthog-cache

# Remote config CDN purge (optional)
REMOTE_CONFIG_CDN_PURGE_ENDPOINT=https://api.cloudflare.com/...
REMOTE_CONFIG_CDN_PURGE_TOKEN=...
REMOTE_CONFIG_CDN_PURGE_DOMAINS=["cdn.example.com"]

Related files

  • posthog/storage/hypercache.py - Core HyperCache implementation

  • products/feature_flags/backend/local_evaluation.py - Local evaluation caching

  • products/feature_flags/backend/flags_cache.py - Flags cache, signal handlers, verification, dependency computation

  • posthog/storage/hypercache_manager.py - Batch management operations (warm, invalidate, stats)

  • posthog/caching/flags_redis_cache.py - FLAGS_DEDICATED_CACHE_ALIAS constant for the dedicated flags Redis

  • posthog/models/remote_config.py - Remote config caching

  • posthog/models/team/team_caching.py - Team authentication caching

  • products/feature_flags/backend/tasks.py - Cache update and refresh Celery tasks

  • posthog/tasks/hypercache_verification.py - Cache verification task

  • posthog/tasks/remote_config.py - Remote config sync tasks

  • posthog/tasks/scheduled.py - Task schedule definitions

  • posthog/tasks/hypercache_verification.py - Cache verification tasks

See also