Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

16.8 KB
Raw Blame

credentials-service

Serves PostHog's platform integration credentials — the OAuth app client ids/secrets and API keys PostHog itself owns, as opposed to a customer's — over HTTP, so they stop being injected as environment variables into every pod that might need one. Today those keys are duplicated across the Django stack and each temporal worker's chart values, and adding one means a PostHog/charts PR plus knowing in advance which pods need which key.

This is phase 1. The per-team Integration model, the OAuth flows and token refresh come later; nothing here touches them.

What it does and does not buy

An attacker who has compromised a pod still holds that pod's JWT signing key, and can ask for any credential on the mount. Per-request key scoping contains a leaked token, not a compromised pod. The wins are:

  1. Environment dumps stop containing credentials — /proc/self/environ, crash reports, a stray printenv in a log, and environments inherited by subprocesses.
  2. Every read is attributed, so rotating after a suspected exposure is scoped to what was actually accessed.
  3. A compromised deployment is cut off by revoking one signing key, without disturbing any other deployment.
  4. Rotation stops needing a charts PR and a rolling restart.

Nothing here stops a compromised pod reading credentials it is entitled to read, and no list is maintained pretending otherwise.

API

GET  /_liveness          200 always
GET  /_readiness         200 once the pod holds credentials
POST /v1/secrets/resolve

GET  /metrics            prometheus text, on its own port (CREDENTIALS_SERVICE_METRICS_PORT)

The request scope lives in the JWT and there is no request body. The token is the request, so a body and a claim cannot diverge, and a token lifted from a log unlocks only the fields that one call needed:

// Authorization: Bearer <token>
{
  "caller": "warehouse-sources", // the product that asked; logged, not trusted
  "keys": ["GOOGLE_ADS_APP_CLIENT_ID", "GOOGLE_ADS_APP_CLIENT_SECRET"],
  "aud": "posthog:credentials_service",
  "exp": 1786035932,
}
{
  "secrets": {
    "GOOGLE_ADS_APP_CLIENT_SECRET": {
      "state": "rotating",
      "value": "…",
      "incoming": "…",
      "version_id": "…",
      "fetched_at": "…",
    },
  },
  "missing": [], // name not on the mount, or reserved
}

An unresolvable field is reported per key in missing rather than as a 4xx over the whole batch, so a mistyped name reads as one named field a human can act on while the rest still succeeds.

Callers do not cache, so there is no freshness hint to send. A rotation reaches every caller on their next read.

Auth

Per-deployment HS256 JWT, following the recording-api scheme (PostHog/posthog#67476) — the pattern .agents/security.md names as strongly preferred, not INTERNAL_API_SECRET.

Two identities, deliberately not conflated:

What it is Trusted?
Deployment The pod set that signed the token (posthog-django, a temporal worker) Yes. It is derived from which key verified the token, never from a claim, so there is nothing in the token to forge
Caller The code path that wanted the credential (warehouse-sources, cdp), from a code enum at the call site No. Django holds one key and hosts many products, so a compromised Django pod could name any of them

Authorization hangs off the deployment. The caller claim reaches the audit log so an incident can ask "which product read this", and reaches nothing else — in particular it is never a metric label, because prom-client keeps every series in process memory for the pod's lifetime.

There is no per-deployment allowlist. Every authenticated deployment may read any credential on the mount: a list bounds nothing the signing key does not already bound, and a compromised deployment is contained by revoking its key. What a deployment actually read is in the audit log.

So one thing bounds a request: the keys claim, which is the request, for as long as the token has left to run — the verifier requires an exp.

Signing keys live in the same secret as the credentials, one flat entry per deployment (__CALLER_KEY_<DEPLOYMENT>). Deployment names are derived from the entries present, not declared in code, so onboarding a caller or revoking a compromised one is a secrets edit with no deploy. The same value goes into that deployment's own secret as CREDENTIALS_SERVICE_JWT_SECRET.

Storage and rotation

One secret holds everything: credentials-service-secrets, a flat map of KEY: value pairs. Flat and uppercase because the PostHog/secrets CLI and UI only manage [A-Z0-9_]+ keys with plain string values.

External Secrets Operator syncs that secret into a Kubernetes Secret. The service reads it from one of two sources, set by CREDENTIALS_SERVICE_SECRET_SOURCE (src/source.ts):

  • kubernetes: the service lists the Secret through the API server, then watches it (src/kubernetes.ts). Every replica gets an edit within about a second of ESO writing it.
  • directory (the default): kubelet mounts the Secret as a directory of one file per key, and the service re-reads it on a timer. Kubelet refreshes the mount on each node on its own schedule, about 60-90s. So for a minute or more after an edit, different replicas can serve different versions. Use this source for local development and tests.

Each source yields the whole Secret at one version. The service applies that one snapshot to the signing keys first and then to the credentials. So a revocation and a rotation in one edit land together, and a revoked caller never reads the new values.

The kubernetes source:

  • Lists with fieldSelector=metadata.name=<name> at boot, then watches from the listed resourceVersion with bookmarks.
  • After a normal close, it watches again from the last resourceVersion it saw.
  • After 410 Gone, it lists again and watches from the new resourceVersion.
  • After an error, it reconnects with capped exponential backoff.
  • Every CREDENTIALS_SERVICE_RELOAD_SECONDS, it lists the Secret again as a safety check. If the list finds a change that the watch did not deliver, the service applies it and replaces the watch.
  • Reads the service account token again for each request, because the projected token rotates.
  • Connects straight to KUBERNETES_SERVICE_HOST:KUBERNETES_SERVICE_PORT. It ignores HTTP_PROXY and HTTPS_PROXY, so the egress proxy never sees API server traffic.

It needs a Role with get, list and watch on secrets, restricted by resourceNames to this one Secret. With resourceNames, RBAC allows a list or watch only when it carries the field selector above.

A pod holding no credentials fails its readiness probe rather than exiting. So an empty mount, or an API server it cannot reach at boot, recovers on its own instead of crash-looping. A source that stops being readable keeps what is already held, with credentials_service_serving_stale_seconds as the only sign. A deleted Secret also keeps what is held: an empty set would answer every resolve all-missing, which callers treat as deleted credentials. credentials_service_watch_restarts_total{reason} counts watch restarts by why the previous watch ended.

credentials-service-secrets
  STRIPE_APP_SECRET_KEY                 = "<credential>"
  STRIPE_APP_SECRET_KEY_FALLBACKS       = "<staged replacement, only while rotating>"
  CREDENTIALS_RECOVERY_KEYS             = "<comma-separated key names>"
  __CALLER_KEY_POSTHOG_DJANGO           = "<new>,<old>"

Rotation rides an explicit <KEY>_FALLBACKS sibling, not AWS staging labels. AWSPREVIOUS applies to a whole secret version, so with everything in one secret, rotating Google — or adding an unrelated key — would consume the slot Stripe's in-flight rotation was using and end its overlap silently. A sibling key is unaffected by edits to anything else, and it is the convention PostHog already uses for rotatable keys (SECRET_KEY_FALLBACKS), so the rotation guard in PostHog/secrets grades these automatically.

Starting a rotation means writing the new value into the sibling, leaving the live one in place: both are then served, so a caller can use the replacement before it goes live. Completing it means moving the staged value into the key and dropping the sibling, in one write — so a sibling exists if and only if a rotation is in flight, and the live value is never briefly absent. Two rotations can be in flight at once without interfering.

The live value stops being served the moment a rotation completes. The overlap is the staging window, not a grace period afterwards, so leave the new value staged long enough for callers to pick it up before completing.

State Condition Response
steady no _FALLBACKS sibling, or it repeats the value value only
rotating the sibling holds a different, staged value value + incoming (the staged one)
recovery field named in CREDENTIALS_RECOVERY_KEYS no value; caller raises a typed error

recovery covers a credential that is known-burned with no valid replacement yet. These credentials authenticate PostHog to the third party, so there is no per-user "reconnect": the integration is down for everyone until an engineer re-provisions the app. The state buys callers an immediate, distinct error, so the outage is attributable instead of looking like a third-party problem.

What gets served

Whatever is on the mount, except entries whose name starts with __. There is no manifest in code, so adding a credential is a secrets edit and needs no deploy.

The corollary is a rule on the secret: only outbound credentials belong on it. Every entry is readable by every deployment holding a signing key. For an OAuth app secret we present to a third party that is no expansion — the pod already had its own copy in its environment. For an inbound-request authenticator such as a webhook signing secret it would be one, so those stay as plain env vars on the deployment that checks them. STRIPE_SIGNING_SECRET, which authenticates requests arriving at ee/partners/stripe/api/provisioning/, is the worked example: keep it off this secret.

__ is the one thing the mount will not serve, whatever a token asks for. The caller signing keys carry that prefix, which is why they can share a secret with the credentials they protect.

Metrics

No label value comes from a request. A key name becomes a label only once the mount is known to carry it; anything else collapses to a constant, and the caller claim is not a label at all.

Metric What it answers
credentials_service_resolve_total{deployment,key,result} who read what, and whether it resolved
credentials_service_last_resolved_timestamp{key} which credentials nothing reads any more
credentials_service_previous_version_served_total{key} how much traffic is reading a key mid-rotation
credentials_service_serving_stale_seconds how long this pod has served credentials it could not refresh
credentials_service_store_errors_total source reads that returned nothing
credentials_service_watch_restarts_total{reason} Kubernetes watch restarts, by why the previous watch ended
credentials_service_signing_keys_last_loaded_timestamp staleness means a revocation has not landed on this pod
credentials_service_signing_key_reload_failures_total reloads that kept the previous key set
credentials_service_auth_failures_total{reason} rejected tokens, by why
credentials_service_http_requests_total{method,route,status} request volume, with an unmatched path collapsed to other
credentials_service_http_request_duration_seconds{method,route,status} request latency
credentials_service_shutting_down 1 while draining

Two of these exist because a fail-open needs to be visible: the signing-key reload keeps the previous keys when an edit is malformed, and an unreadable mount keeps what is already held. Alert on the two staleness signals, or neither degradation is observable.

/metrics listens on its own port, which the chart keeps off the ingress; in-cluster Prometheus scrapes it unauthenticated, the same shape as every other service.

Isolation

This package must not depend on the rest of the monorepo. It shares the pnpm workspace only for resolution — nothing from nodejs/, frontend/ or common/. The package's own .oxlintrc.json enforces that with a no-restricted-imports rule over static import specifiers that name another part of the monorepo. It does not see require(), dynamic import(), or a new npm dependency, so those still rest on review of package.json and the lockfile.

Installs run --ignore-scripts; nothing in the tree needs an install-time script, and that is the npm-specific supply-chain path worth closing here.

Local development

pnpm --filter @posthog/credentials-service dev                # tsx watch, pretty logs
pnpm --filter @posthog/credentials-service typecheck
pnpm --filter @posthog/credentials-service test:unit
pnpm --filter @posthog/credentials-service test:integration   # boots a real server on a temp mount

The integration suite boots a real CredentialsServer against a temp-dir mount and a real socket, so it covers the wiring the unit suite fakes. Neither suite needs a database or Docker.

Point CREDENTIALS_SERVICE_SECRETS_DIR at a directory of files, one per key, to stand in for the mount.

Configuration

Variable Default Notes
CREDENTIALS_SERVICE_ENV dev Logical env, recorded on every startup log line
CREDENTIALS_SERVICE_SECRET_SOURCE directory directory or kubernetes; anything else fails boot
CREDENTIALS_SERVICE_SECRETS_DIR /etc/credentials-secrets Where the Kubernetes Secret is mounted (directory)
CREDENTIALS_SERVICE_SECRET_NAME credentials-service-secrets The Secret to watch (kubernetes)
CREDENTIALS_SERVICE_SECRET_NAMESPACE the pod's namespace Read from the service account when unset (kubernetes)
CREDENTIALS_SERVICE_RELOAD_SECONDS 30 Re-read interval; a safety re-list next to the watch
CREDENTIALS_SERVICE_METRICS_PORT 9090 Dedicated /metrics listener, kept off the ingress
CREDENTIALS_SERVICE_LOG_LEVEL by NODE_ENV debug, info, warn or error
PORT 8004
HOST 0.0.0.0
SHUTDOWN_PRESTOP_DELAY_MS 5000 Wait before draining, for the Kubernetes prestop window

The service exits at boot rather than starting degraded: a missing production variable, a numeric variable that does not parse, or a kubernetes source without KUBERNETES_SERVICE_HOST or a namespace. An empty secret mount does not exit; the pod fails its readiness probe and recovers on its own once External Secrets Operator syncs.