:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Alerts noop workers
The alerts platform registers three queues through products/alerts_platform/backend/facade/temporal.py and the shared start_temporal_worker command:
Setting in posthog/settings/temporal.py |
Queue | Workflow |
|---|---|---|
ALERTS_PLATFORM_SHARED_ORCHESTRATION_TASK_QUEUE |
alerts-platform-shared-orchestration-task-queue |
alerts-platform-orchestrate |
ALERTS_PLATFORM_EVALUATION_TASK_QUEUE |
alerts-platform-evaluation-task-queue |
alerts-platform-evaluate |
ALERTS_PLATFORM_DELIVERY_TASK_QUEUE |
alerts-platform-delivery-task-queue |
alerts-platform-deliver |
These queue names are hardcoded and stay separate even with DEBUG=True.
Shared orchestration registers the orchestration workflow, the source dispatcher and a synthetic demand-discovery activity.
The evaluation queue registers the evaluation workflow (alerts-platform-evaluate), the logs and insight source evaluations, the probe activity and the record-outcomes activity.
Each schedule tick starts orchestration, which discovers demand once and then pages source dispatchers until the demand is exhausted or its dispatch budget is spent.
Each dispatcher starts one evaluation child for its source. Evaluation runs the probe and starts its independent delivery child on the delivery queue.
Start one worker for each queue:
python manage.py start_temporal_worker --task-queue alerts-platform-shared-orchestration-task-queue --metrics-port 8104
python manage.py start_temporal_worker --task-queue alerts-platform-evaluation-task-queue --metrics-port 8102
python manage.py start_temporal_worker --task-queue alerts-platform-delivery-task-queue --metrics-port 8103
Give every worker its own --metrics-port.
The option defaults to 8001, which the shared development worker already binds, so a worker that keeps the default stops with Address already in use.
The shared development worker does not poll these queues.
See Temporal development guidance for worker setup.
Tick schedule
python manage.py schedule_temporal_workflows creates or updates alerts-platform-check-due-schedule
in every deployment that runs it. The normal deployment migration step runs this command, and bin/migrate
skips that step on hobby deploys and on local development. Only the posthog-web-django chart app enables
that migration job, and it has dev, US and EU values files, so registration reaches those three and nowhere
else. A new deployment that enables the job registers the schedule too. There is no per-region gate: the schedule is the same everywhere, and what differs between
deployments is which configurations have been backfilled.
The schedule starts alerts-platform-orchestrate with {} on the orchestration queue every minute (UTC).
There is no routing flag: the flow is tick → orchestration → evaluation → delivery.
It uses SKIP overlap, a one-minute catchup window, a 50-second workflow execution timeout,
and one workflow attempt. Creation does not trigger an immediate run; the next minute starts it.
Delivery has no schedule of its own: evaluation starts its delivery child, and the held-delivery sweep starts the ones it releases.
New schedules start unpaused. Registration updates existing schedules to this policy while retaining their state from Temporal, including manual pauses.
To stop future ticks, pause the schedule in Temporal; resume it there when ready. Pausing does not stop workflows already running.
Reverting the code that registers a schedule does not remove one already registered.
Verify the Postgres activity result and the delivery child's completion separately. Parent completion does not prove either succeeded. Schedule creation also does not prove worker availability.
Slack message formatting
- Sources can register a Slack renderer with
register_source_slack_rendererthrough the delivery facade. - Insight alert firings use the same attachment and control builders as their HogFunction templates.
- Sources without a custom renderer, and insight transitions other than firings, use the platform's default layout.
- The Slack transport preserves attachments for later message updates. If Slack rejects a chart image, it retries without the image and retains the remaining content.
- A renderer always sets
text. A push notification and a screen reader read it, and an attachment does not carry it. - A renderer that raises is counted on
alerts_platform_slack_renderer_failures_totaland the message goes out in the platform's layout. - A card that lives in an attachment also stores one stand-in block. A reader from before attachments edits
blocksand indexes its last element, so an empty list would fail that edit during a rolling deploy.
Turning a production deployment on
A production tick is a shadow run. Delivery stops at alerts-platform-deliver-preview, which records what
would have been sent and contacts no destination, so a production tick never notifies anybody and never
writes a source product's rows. What it does cost is the evaluation queries its sources run, on top of the
ones the source's own production fleet is already running for the same alerts.
The tick's work is whatever PlatformAlertConfiguration rows exist, and nothing creates those on its own:
python manage.py backfill_platform_alert_configurations [--team-id N] for logs and python manage.py backfill_platform_insight_alert_configurations [--team-id N] for insight are the only bulk writers, and both are manual. The logs alerts API also copies one alert when it is created or turned back on, but only in a team that already has an enabled logs copy.
So the order below puts the schedule in place while there is no demand, and load arrives when the backfill
is run, one cohort at a time.
- Bring up that deployment's three
alerts-platform-*workers and confirm all three are ready and polling. Do this before the code that registers the schedule reaches the deployment. They cost nothing while they idle, because no schedule is starting work for them yet. - Let the deploy carry the code in. Migration-time reconciliation registers the schedule unpaused, and with no configurations backfilled each tick discovers empty demand and exits.
- Backfill one team at a time with
--team-idand watch that team's evaluation and preview metrics before widening.
Registering the schedule before the workers poll is untidy rather than dangerous: each tick is created, nothing picks up its workflow task, and the 50-second execution timeout closes it. That repeats every minute until the workers are ready, then stops on its own.
Shared orchestration rollout and rollback
The deployment identity is temporal-worker-alerts-platform-shared-orchestration.
It uses the shared posthog-cloud image built by container-images-cd.yml, not a separate image build or repository.
Worker deployment configuration lives outside this repository. Deploying this code must be coordinated with starting the orchestration worker.
The three deployments in PostHog/charts must pass the matching alerts-platform-* queue names.
The worker command rejects an unregistered queue with ValueError at startup.
An old image with a new queue, or a new image with an old queue, cannot start.
Treat each worker's image and queue configuration as one deployment change; merging two PRs close together does not make their deployments atomic.
Schedule reconciliation routes directly to orchestration; merging the code alone does not start a worker process.
- Hold automatic promotions and syncs for all three worker deployments before either the code or chart queue changes deploys. Record the current image, queue configuration, and schedule action for rollback.
- Pause
alerts-product-check-due-scheduleif it exists, and stop manual starts and delivery-preview requests during the cutover. Before migration-time reconciliation runs with the new code, createalerts-platform-check-due-schedulein Temporal with its state explicitly paused. Use the action and policy from Tick schedule. If the new schedule already exists, pause it instead. Pause state is preserved only for the same schedule ID; pausing the old ID does not pause a newly created schedule. - Keep the old images polling all three
alerts-product-*queues until queued and running orchestration, source-dispatch, evaluation, and delivery work drains. Verify this in Temporal rather than waiting a fixed interval. Delivery can outlive its parent, and manual runs can have different timeouts. - Build the image containing the renamed worker registrations. Render each worker deployment with that image and its matching
alerts-platform-*queue before allowing its promotion or sync. Do not restart a worker with only half of this change. Keep both schedules paused while the three deployments roll out. Verify all three workers start successfully and poll their configured queues, includingalerts-platform-shared-orchestration-task-queue. - Run
python manage.py schedule_temporal_workflowswith the new code and verify the new schedule remains paused. Its action must startalerts-platform-orchestrateon the orchestration queue. Resume the new schedule only after all three workers are ready. Verify the Postgres probe result and delivery completion separately. - After the complete workflow chain succeeds, delete the paused
alerts-product-check-due-scheduleby hand and release the deployment holds. Reconciliation never deletes the old ID. Allow manual starts and delivery-preview requests again only with callers running the matching code.
For rollback:
- Pause the new schedule and stop manual starts and delivery-preview requests. Keep the current workers running until all queued and running work drains.
- Hold worker promotions and syncs. Restore each previous image together with its
alerts-product-*queue configuration, then verify all three workers are ready. - Restore the recorded schedule configuration using the previous code. Immediately before this rename, its action starts
alerts-product-orchestrateonalerts-product-shared-orchestration-task-queue. Do not restore the olderalerts-product-check-dueaction unless the selected rollback image actually registers it. If the old schedule was deleted, reconcile it only after the old workers are ready, because creation starts it unpaused. - Resume the old schedule if it is paused, verify the probe and delivery complete, then delete the paused new schedule and release the deployment holds. Restore manual starts and delivery-preview requests with callers running the previous code.
Do not reconcile with the new code after rollback: it would recreate the new schedule unpaused. Pausing or changing a schedule does not move or stop queued or running workflows.
Manual local runs
For local development, start an orchestration run by hand.
The execute_temporal_workflow and start_temporal_workflow commands do not know these workflows and reject the name, so use the Temporal CLI in the dev stack:
docker exec posthog-temporal-admin-tools-1 \
temporal workflow start --address temporal:7233 --namespace default \
--task-queue alerts-platform-shared-orchestration-task-queue \
--type alerts-platform-orchestrate \
--workflow-id "alerts-platform-orchestrate-manual-$(date +%Y%m%d%H%M%S)" \
--execution-timeout 50s \
--input '{}'
The empty --input '{}' becomes an OrchestrateInputs with every field defaulted.
Watch orchestration, its source dispatcher children, their evaluation children, and the delivery great-grandchildren in the Temporal UI at http://localhost:8081.
Evaluation and delivery accept an empty AlertsPlatformInputs dataclass; orchestration accepts OrchestrateInputs with all fields defaulted.
Orchestration pages source dispatchers, which start evaluation children with one workflow attempt, on the queue and under the execution timeout of the source's binding.
Evaluation child IDs carry the tick ID, source and page, so each tick starts distinct evaluations.
Evaluation runs a Postgres connectivity probe; delivery runs an empty activity with no I/O.
Evaluation and delivery activities each have a 10-second start-to-close timeout and a 30-second schedule-to-close timeout.
Evaluation has one attempt; delivery retains at most three attempts.
Evaluation starts one delivery child on the delivery queue and waits for confirmation that it started, without waiting for completion.
The child ID includes the evaluation run ID, so repeated runs of the same evaluation workflow ID start different children.
ParentClosePolicy.ABANDON lets delivery continue after evaluation closes.
Delivery has a one-minute execution timeout for the noop.
Real notification delivery guarantees remain undecided.
Names
The evaluation workflow is alerts-platform-evaluate (class AlertsPlatformEvaluateWorkflow), the probe activity is alerts_platform_probe_postgres_activity, and the schedule is registered by create_alerts_platform_tick_schedule.
These replace alerts-product-check-due, alerts_product_check_due_activity and create_alerts_product_check_due_schedule: discovery finds what is due and dispatchers hand it out, so this workflow only evaluates.
A workflow type rename breaks runs of the old type that are in flight at deploy time: no worker knows the old name, so they fail. Dev evaluations live under their execution timeout, and no production deployment was registering the schedule when this rename landed. A later rename has to account for whichever deployments have since opted in.
The schedule ID is alerts-platform-check-due-schedule, renamed from alerts-product-check-due-schedule.
Registration never deletes a schedule, so the rollout above deletes the old ID by hand.
Tick loop and source dispatchers
One tick is one alerts-platform-orchestrate execution. It takes an OrchestrateInputs; the schedule passes {} and every field defaults.
The first run records the tick cutoff (the scheduled start time, or the workflow start time for manual runs) and a deadline 45 seconds after the run started.
Discovery runs once per tick. The loop then starts one alerts-platform-source-dispatch child per source with demand, ID {tick_id}-{source}-p{page}, on the orchestration queue.
The tick does not record inventory. A separate schedule, alerts-platform-record-inventory-schedule, starts the alerts-platform-record-inventory workflow every 5 minutes on the orchestration queue at the lowest Temporal priority.
It counts configurations and alerts for every source in two queries, each with a 500 ms statement timeout, and pushes the alerts_platform_configurations and alerts_platform_alerts gauges to the PushGateway under the alerts_platform_inventory job.
The push replaces the job's group, so the region holds one series per label set. A failed count is logged and the next run tries again.
Dispatchers are part of the tick: the orchestrator awaits each dispatcher's report and keeps the default TERMINATE close policy on that edge.
They run on the tick's own fleet because the tick awaits them. On the evaluation queue, an evaluation fleet with no free slots
leaves the dispatcher unpicked, and the tick waits on a report that cannot arrive. A dispatcher starts no activities,
so it costs the orchestration fleet a workflow task per source per page and nothing else.
The evaluations it starts still name the evaluation queue, so moving the dispatcher does not move them.
Each dispatcher has one attempt and an execution timeout of 30 seconds, or the time left before the tick's hard stop minus one second, whichever is shorter.
The hard stop is the run's own execution timeout when it has one, and the budget plus five seconds otherwise. Both deadlines travel in the input across continued runs.
The orchestrator passes a dispatcher every remaining ID for its source. The dispatcher decides how much to take and returns the rest.
Today it takes everything: no adapter has said yet how many alerts one evaluation can hold, so nothing remains and a tick is one page.
The limit that will matter is the evaluation workflow's own history, which depends on the adapter's query shape; it arrives with the first real adapter.
It starts one evaluation child per key with ParentClosePolicy.ABANDON and one attempt.
A source with an entry in SOURCE_BINDINGS (temporal/sources.py) starts the binding's workflow on the binding's task_queue, under its evaluation_timeout.
A source with no binding starts alerts-platform-evaluate on the evaluation queue, under the 75-second NOOP_EVALUATION_TIMEOUT.
Each source has its own timeout, so no source adopts the logs ceiling. Sources share the evaluation queue while their checks finish in seconds. A source whose checks hold a worker slot for minutes needs its own queue, so that it does not hold the slots the other sources need.
The timeout has to hold every attempt a source's activities allow, because an attempt cut off here is a batch that decided something and recorded nothing.
Evaluations are abandoned rather than awaited, so it does not have to fit inside the tick.
It waits for the child to start, never for it to finish, then returns the dispatched count and the remaining IDs.
Members are not passed to evaluation yet: evaluation keeps the probe path until claims exist.
After every page the orchestrator records a TickPage (page, run ID, dispatched, remaining, failed sources, undispatched).
When nothing remains it returns OrchestrateResult(remaining=0, deadline_reached=False).
Before each page after the first, it checks the deadline. When work remains and the deadline has passed, or fewer than two seconds remain before the hard stop, it returns cleanly with the remaining count and deadline_reached=True; the next minute's tick discovers that work again.
A tick always runs its first page: the deadline is a stop rule, not an admission rule.
A tick that exits with remaining work is a load signal. A tick that hits the schedule's 50-second execution timeout is a breakage signal: a clean exit never times out.
The orchestrator calls continue_as_new only when Temporal reports is_continue_as_new_suggested(). The continued run receives the cutoff, deadline, demand and pages in its input and does not rerun discovery.
The schedule's execution timeout spans continued runs, so a rollover cannot extend the tick.
A dispatcher that overruns times out before the tick's hard stop rather than the tick being terminated mid-page. Evaluations already started by earlier dispatchers, and their delivery children, are abandoned and complete on their own.
The orchestrator settles its pages rather than failing on the first error, so one source's dispatcher failing or timing out does not stop the sources dispatching alongside it in the same minute.
A failed source is counted in the page's failed_sources, its keys in undispatched, and it is logged with its error.
Only a ChildWorkflowError counts as a source failure. Anything else, a cancellation or a bug in the orchestrator included, still fails the tick rather than being absorbed on every tick behind a report that says it completed. Nothing advanced its keys' due time, and discovery orders by that time,
so those keys are rediscovered and win a later tick; the tick reports them in remaining rather than re-paging them.
If the tick is terminated or times out anyway, Temporal terminates its in-flight dispatchers after the tick closes.
The previous workflow.patched gate around discovery is gone: the loop cannot run without discovery, and dev histories live under a minute.
Synthetic demand discovery
The first orchestration activity, alerts_platform_discover_demand_activity, accepts a timezone-aware ISO-8601 cutoff.
Scheduled runs use TemporalScheduledStartTime; manual runs use the workflow start time.
Activity retries retain the same cutoff rather than reading the activity's clock.
The activity returns an AlertDemand containing configuration IDs grouped by the shared SourceKind enum (logs and insight).
Only nonempty groups are returned. Discovery does not reserve or claim IDs.
Each source is bounded to its binding's discovery_limit, or to DISCOVERY_LIMIT_PER_SOURCE (1,000) without a binding, so the manifest stays near 55 KB per 1,000 keys, under the repository's 256 KB rule for Temporal payload fields.
The insight binding allows up to 2,000 keys (ALERTS_PLATFORM_INSIGHT_KEYS_PER_TICK), about 110 KB.
A source that admits fewer checks per tick sets a lower limit, because every key starts a workflow.
omitted_by_source counts the due IDs left out. The tick adds that count to its remaining result, and the next tick discovers that work again.
For now, logic/demand.py supplies deterministic synthetic configurations relative to that cutoff:
two eligible logs configurations and one eligible insight configuration, plus future and disabled configurations that are excluded.
There are no configuration-table reads, new database entities, or real evaluations of these IDs.
The result feeds the tick loop above. Evaluation still runs the probe/delivery smoke path without receiving synthetic IDs.
TTL claims are not implemented here.
Discovery has a five-second start-to-close timeout, a ten-second schedule-to-close timeout, and at most three attempts.
Batch keys
Discovery returns batch keys, not configuration ids. A key is (source, team_id, slot), where slot is
next_check_at floored to the minute. A key costs a fixed amount and does not grow with a team's alert
count, so the manifest bound is about how many chunks a tick starts rather than how many alerts it found.
The key names the chunk by what it holds, so the evaluation workflow id is
alerts-eval-{source}-{team_id}-{slot} and stays the same when a later tick rediscovers the same work.
A slow evaluation therefore blocks its own re-dispatch without a claim on the configuration rows.
Discovery orders by next_check_at ascending. A key the manifest bound leaves out grows more overdue
and wins a later tick; any stable ordering that is not by due time starves the same keys every tick.
Each dispatcher takes up to MAX_EVALUATIONS_PER_DISPATCH keys and hands the rest back as a later page.
The evaluation re-reads its own configurations from the key, which it has to do anyway to get thresholds
and filters, so it sees a fresher set than discovery did.
Flooring to the minute loses nothing that load spreading provides: compute_shard_offset_seconds returns
whole multiples of the 60-second schedule interval, so spreading moves an alert between minutes rather
than within one.
A source can set schedule_group on an outcome to spread by team and group instead of by configuration, through compute_team_shard_offset_seconds.
A team's alerts with one group and one cadence then share a batch key.
Logs sets the group to the query shape (window, evaluation periods, projection eligibility), so alerts that can share a cohort query land in one batch, and alerts that cannot spread across slots.
That keeps a batch under MAX_COHORTS_PER_CYCLE in the common case, where one slot per team would push a team with many shapes over it.
Source evaluation bindings
products/alerts_platform/backend/temporal/sources.py maps a SourceKind to the workflow name that evaluates it.
A source in that map gets its own workflow started by name, carrying one batch key and the tick cutoff.
A source absent from it keeps the noop alerts-platform-evaluate path, which receives no key.
The alerts product imports nothing from a source: the binding holds a name, and test_every_source_evaluation_binding_names_a_registered_workflow fails if that name is not registered on the evaluation queue.
logs is bound to logs-alert-evaluate, which is the first real source evaluation.
Logs source evaluation
logs-alert-evaluate evaluates one batch key, a team's alerts due in one minute, and previews one delivery per notification or incident edge.
The evaluation is a plain function in products/logs/backend/alert_source_cycle.py, so a test calls it without Temporal.
It writes its own state and never the logs product's rows.
The production logs-alerting-task-queue fleet evaluates these same alerts every minute against LogsAlertConfiguration,
so a write to those rows, a LogsAlertEvent row or a Kafka message here would transition an alert twice and notify a person twice for one breach.
State transitions land on PlatformAlert and schedule advancement on PlatformAlertConfiguration, which the logs fleet never reads.
Delivery stops at alerts-platform-deliver-preview, which records what would have been sent and contacts no destination.
The lifecycle decision comes from products/alerts_platform/backend/facade/lifecycle.py configured with LOGS_ALERT_POLICY,
which is the shared machine the logs product's own state machine is a thin adapter over.
Going to the shared machine directly keeps the platform's lifecycle out of a source product's import path.
It evaluates against the tick cutoff rather than the clock, so a retried attempt selects the same alerts, resolves the same windows and derives the same evaluation keys as the attempt it replaced. The due predicate is applied a second time here, because discovery ran earlier in the tick and a configuration can have been disabled or broken since.
Every check produces an outcome
A check the source cannot evaluate still records what it decided, and the two cases decide differently.
| Case | State | Schedule |
|---|---|---|
| Filter config no data satisfies | BROKEN | The next cadence step, though discovery stops selecting it |
| Query failed | The shared machine's error path decides | The next cadence step |
A skip that records nothing leaves its due time where it was, so discovery hands the same check back every tick.
That is the whole reason these exist: the work is not lost, it is repeated, and a permanently broken alert repeats it forever.
The source reports the skip; the platform stays the only writer of next_check_at.
The failure case goes through evaluate_alert_check with an errored CheckInput, so the shared machine raises
consecutive_failures and escalates to BROKEN at five, and classify_alert_error decides whether the error is
transient. A transient error advances the schedule but holds the counter, because a cluster outage must not
disable every alert that ran during it.
suppressed is the single definition of what holds a configuration back, and it is a BROKEN check_status alone.
Both discover_demand and due_checks exclude on it. Discovery has to, because a broken alert that still mints
a batch key spends the manifest bound on work its own evaluation then drops.
check_status lives on the configuration, not on an instance, for grouped and ungrouped configurations alike,
because a failed check returns no group to attribute the failure to.
A check that could not evaluate reports its failure on the outcome as a CheckFailure, with no groups, for every source.
record_outcomes writes the machine's ERRORED or BROKEN onto check_status, leaves every instance alone, and writes one history row with the empty grouping key and the nil UUID as alert_id, which stands in for "no instance" because the sort key rejects Nullable.
So a failed grouped check posts one error message, not one per group.
The check input hands the shared machine the configuration's status in place of each instance's state while it is not OK,
so the machine reads what it read when both shared one field.
A configuration's snooze_until mutes every instance, and an instance's own snooze_until mutes one group.
The check input and the read API show each instance the later of the two.
POST /api/projects/:id/platform_alerts/:id/snooze_group/ sets or clears an instance's snooze on a grouped alert the platform owns.
Unlike a snooze of the whole alert, it leaves the group's firing open and the schedule alone, so the group's incident edges still go out.
record_outcomes reads each instance's mute again when it writes a verdict, so a check that was already running when a snooze landed records its fire or resolve as muted instead of announcing it.
A configuration's grouping holds max_instances (100 by default), the most instance rows it keeps.
A source calls check.admit(keys) with its groups in priority order: a group that already has an instance is always admitted, so it can resolve, and new groups fill the remaining room front first.
The count it turned away travels on the delivery as overflowed, and the first message of that delivery says how many groups were not tracked.
record_outcomes enforces the cap again and drops a group past it, counted on alerts_platform_groups_over_cap_total.
An instance that is not firing, not muted, and that no check returned for longer than both 24 hours and its cooldown is deleted, which frees its slot.
Nothing is reaped while the configuration's checks fail, because a failed check returns no groups and every instance would look gone.
Logs is the first source that groups, by service_name and severity_text only.
A grouped logs alert leaves the batched cohort and runs GroupedAlertCheckQuery, one query per alert, with its open groups ordered first and the rest worst first.
A group with no instance whose check decides nothing is left out, so a quiet service costs no row, and an open group the query does not return is checked against a zero count so it can resolve.
Insight groups by breakdown value, with keys set to ["breakdown"].
check_alert_per_series judges every series on its own, where production's check_alert_for_insight stops at the first series that breaches, and the adapter runs each value through the shared machine against its own instance.
Only threshold alerts can group; a detector scores its series together.
alerts_platform_checks_skipped_total{source,reason} counts these by reason.
What a check leaves behind
Every check writes one row to platform_alert_events in ClickHouse for each group its outcome
reports, including a check that confirmed the alert.
An outcome carries what describes the whole check (the evaluation key, the failure count, disable,
the next due time) once, and a GroupOutcome per group for what describes one instance (its state,
firing and value). A source that does not group reports one group with the empty key.
Postgres could not take that volume without a per-check retention flag, and a TTL'd ClickHouse
table needs no such flag, so nothing has to decide which checks are worth keeping.
The row is self-sufficient.
alert_name, condition_snapshot and source_config_snapshot are read when the outcome is
recorded, so a threshold edited between a check and a retried send cannot change what a message
claims was breached, and a rename cannot make one thread contradict itself.
The snapshots are taken at write time rather than shipped with the outcome, because a source's copy
of source_config is a filter tree and shipping one per outcome would cost Temporal payload on
every batch.
The write happens after the Postgres transaction commits, not inside it.
insert_events never raises: the alert's state and schedule are already written by then, so a
ClickHouse outage costs a gap in history rather than an alert left due with its state unwritten.
alerts_platform_history_rows_dropped_total counts that gap.
platform_alert_events is a plain ReplicatedMergeTree, because every row is a distinct check
and nothing supersedes anything.
A ReplacingMergeTree would have made every count over the table wrong on any part a merge had
not reached, and ClickHouse never promises a merge will run.
ClickHouse has no unique constraint, so the insert carries an insert_deduplication_token naming
the batch by its contents.
A retried batch arrives under a token the engine has already seen and is dropped.
A reader still deduplicates on (alert_id, evaluation_key), because the token only covers a retry
of the same batch and the engine only remembers a bounded window of them.
labels lands empty and stays empty until a source groups its results.
It is the group's identity, not the alert's filter scope; service and severity live in
source_config_snapshot, which is where a message should read them.
A mute holds the announcement, not the check
A snooze and a schedule restriction both mute. Neither stops a check.
The alert evaluates on its cadence, transitions as its data says, and records what happened; only the
announcement is held. enabled=False and BROKEN are the only states that stop a check.
This is how a muted alert keeps telling the truth. Under the older behavior an incident that started and ended inside a quiet-hours window left no trace at all, and the alert's state stayed at whatever the last check before the window decided.
The semantics arrive as AlertPolicy.mute_gates_notification_only, which the platform's
PLATFORM_LOGS_ALERT_POLICY sets and production logs does not. The two stacks therefore disagree about a muted
alert on purpose, and a comparison against the logs stack has to expect it.
Three consequences worth stating:
update_last_notified_atis held with the announcement, so the cooldown keeps measuring real notifications. An alert is never gated by a send that did not happen.- A mute holds FIRE and RESOLVE only. ERROR and BROKEN describe the alert's health rather than its condition, and BROKEN stops future checks, so an announcement held there would never be released.
next_check_atstays on the cadence through a blocked window. Parking it at the end of the window is what the older skip behavior did, and it also made every restricted alert for a team come due in the same minute.
AlertCheckOutcome.muted_notification carries what was held, and
alerts_platform_notifications_muted_total{source,reason} counts it by snooze or quiet_hours.
Incident edges ignore cooldown and mute
A paging destination such as PagerDuty holds an incident open until it receives a resolve, so it needs one resolve for every trigger. Cooldown and mute hold back announcements while the state still moves, so a paging destination cannot follow announcements.
decide_incident_action in facade/lifecycle.py reads the state before and after a transition: entering a
firing is a trigger, leaving it for any reason is a resolve. It shares its firing rule with
decide_firing_episode, so a policy that parks a firing alert in SNOOZED keeps its incident open there, and
a snooze under any other policy ends the firing and resolves the incident.
The legacy logs stack's incident_edge wraps the same rule.
- The source decides the action under its own policy and puts it on
AlertDeliveryRequest.incident_actions, keyed by grouping key, because the history row does not record the policy. - The source sets an action only when the alert has a destination subscribed to its incident events, so an alert without a paging destination starts no extra delivery.
- A source sends a delivery whenever a check announces or moves a firing. A delivery that exists only for its
incident actions has
sends_messages=False, and none of its rows reach a message destination. announcement()also returns the held CHECK row of a group inincident_grouping_keys.- A mute never holds an incident edge: a fire inside quiet hours or a snooze triggers the incident.
- Delivery routes each action to the event id in
event_ids_by_incident_action, which only an incident manager destination subscribes to. A message destination never sees an incident action. - The PagerDuty transport sends a trigger or a resolve with the
dedup_key<configuration_id>:<grouping_key>:<episode_started_at>, so a resolve closes the incident of its own firing episode. It names the platform insource, because a team on the pilot also gets the HogFunction path's incident. - An insight alert announces no resolve, but it still sends incident actions for an alert the platform owns.
A PagerDuty destination on
$insight_alert_incident_openedand$insight_alert_incident_closedopens and resolves its incident. One on$insight_alert_firingkeeps its HogFunction, which only triggers.
A fire a mute swallowed is still owed an announcement.
_firing_is_unannounced in facade/lifecycle.py decides that, and its docstring holds the rule.
Without it an alert reaches the end of its quiet hours already FIRING, and renotify_while_firing
is false, so nobody is ever told.
A recovery that happened entirely inside a mute is not announced when the mute lifts, which is what
Datadog does and what a person muting an alert expects.
Production logs gets the same reset on snooze expiry, by way of the SNOOZED branch in
evaluate_alert_check; under mute semantics the state is never SNOOZED, so the reset needs its own
signal.
Evaluating and writing are separate activities
evaluate_logs_alerts_activity reads and decides; it writes nothing.
It is a sync def, so Temporal runs it on the worker's own activity executor and the thread it blocks is a slot Temporal is accounting for.
An async activity handing the work to its own thread pool releases its slot the moment the activity times out, while the thread stays on a query Temporal can no longer see.
Being sync also means Django's connection recycling is the activity's own job, which @close_db_connections does for it.
The workflow then calls alerts_platform_record_outcomes, the platform's own activity, which persists the batch.
Order matters more than the split does: Temporal holds the deliveries the batch decided on before any write
can advance a schedule past them, so an attempt lost between the two costs its queries and nothing else.
A source starts the write by name rather than importing it, so the products stay apart.
The write is safe to run twice. An attempt that commits leaves every configuration due after the cutoff, and a replay skips those rows rather than advancing them again and skipping a cycle. It runs in one transaction, so no alert is marked as notified while its schedule still says the check is due.
MAX_DELIVERIES_PER_CYCLE bounds an outcome together with the delivery it belongs to.
Recording an outcome whose preview the batch cannot carry would leave an alert firing with nothing announcing it,
and a firing alert does not fire again. Dropping the pair leaves it due, the way a truncated cohort already behaves.
alerts_platform_deliveries_deferred_total counts them.
Cohorting, the batched ClickHouse query, projection routing, the byte ceiling and ingestion-freshness gating all come from the existing logs code, so a preview says what production would have sent.
The query budget sits under the activity timeout
Four bounds, largest first: the logs binding's evaluation_timeout (75s) over the source's EVALUATION_BUDGET (62s) over EVALUATE_START_TO_CLOSE (30s) over BATCH_QUERY_BUDGET_SECONDS (25s) over MAX_QUERY_SECONDS (20s).
Temporal bounds an attempt by whichever of start-to-close and schedule-to-close expires first, so schedule-to-close is derived as start-to-close plus a queue tolerance rather than written as a literal.
A literal close to start-to-close lets queue time shorten the run below the query budget, which is the same inversion arriving by another route, on exactly the load that causes queueing.
test_the_evaluation_timeout_ladder_holds asserts the whole ladder in one place.
The order is the point. ClickHouse has to be what ends an overrunning query: with the activity timeout below the query's own limit,
the activity times out while the cluster is still running the query, the attempt's thread stays on it, and the retry starts an identical query alongside the first.
Above it, a slow query arrives as one failed cohort the batch reports and the rest of the batch continues.
A batch can run several cohort queries one after another, so a single per-query limit does not bound the batch.
Each query is given the time left before the batch deadline, capped at MAX_QUERY_SECONDS; below MIN_QUERY_SECONDS no further query starts
and the cohort keeps its due time, counted in an unqueried log line.
The budget covers the cohort queries only. The per-alert destination lookup is outside it, so a wide fan-out can still spend the activity's own timeout.
The cap reaches ClickHouse as max_execution_time on BatchedAlertCheckQuery, with timeout_overflow_mode set to throw,
because a partial count could resolve an alert that is actually breaching.
Delivery previews carry a list of group transitions with one entry and an empty grouping key. Logs does not group yet; the list is the shape that lets fan-out change the evaluation and nothing downstream.
Metrics
The path emits through Temporal's own meter, so every series carries the worker, queue and activity attributes
the runtime attaches. products/alerts_platform/backend/temporal/metrics.py holds them and a source reaches them through
facade/platform_metrics.py.
| Metric | What it answers |
|---|---|
alerts_platform_checks_total{source,outcome} |
How many checks the platform decided, and what they decided |
alerts_platform_state_transitions_total{source,from,to} |
Which transitions it reached |
alerts_platform_deliveries_previewed_total{source} |
How many deliveries it recorded instead of sending |
alerts_platform_deliveries_deferred_total{source} |
How many the payload bound left for a later tick |
alerts_platform_outcomes_recorded_total |
How many decisions reached the tables |
alerts_platform_batch_duration_ms{source} |
What one batch key costs |
alerts_platform_scheduler_lag_ms{source} |
How far past its due time a check was evaluated |
alerts_platform_discovery_omitted_total{source} |
How many due batch keys a tick left out under the discovery limit |
alerts_platform_tick_duration_ms{deadline_reached} |
How long a tick ran from its scheduled minute. Past 60 s, the schedule skips the next tick |
alerts_platform_tick_start_delay_ms |
How late a tick's first workflow task ran after its scheduled minute |
alerts_platform_tick_keys |
How many batch keys a tick's discovery found due, including those its limits left out |
alerts_platform_tick_schedule_actions{outcome} |
The tick schedule's lifetime counts of started, overlap-skipped and late-dropped ticks, pushed every 5 minutes. A rise in skipped_overlap or missed_catchup is a lost tick |
alerts_platform_checks_deferred_total{source} |
How many checks a full evaluation pool turned away |
alerts_platform_checks_unfinished_total{source} |
How many admitted insight checks reached no outcome |
alerts_platform_history_rows_dropped_total{source} |
How many check rows a failed history write lost |
Histogram buckets are registered in posthog/temporal/common/worker.py; a histogram missing from
ALERTS_PLATFORM_LATENCY_HISTOGRAM_METRICS gets Prometheus defaults instead.
Every call site goes through safe_record, so a metric failure cannot fail a check.
Seeding the shared tables
PlatformAlertConfiguration starts empty, so discovery finds nothing and no evaluation runs until configurations are copied in.
Run the copy once per environment from any pod that carries the application image and a database connection:
python manage.py backfill_platform_alert_configurations
Pass --team-id to copy one team's configurations only.
Pass --sample-percent to copy a sample of teams. Logs samples whole teams, not alerts, because one query checks a cohort of a team's alerts, so a sample of alerts spread across every team costs nearly as many queries as copying all of them. The sample is chosen by team id, so a rerun copies the same teams and a larger percentage only adds teams.
An alert the copy rejects is logged and counted as failed, and the run carries on.
It is a seed, not a sync: the logs product keeps the control plane.
Each logs check reads its logs alert's current filters, threshold, window, N-of-M, cooldown, quiet hours and snooze, so an edit counts from the next check without a rerun.
A copy whose logs alert is deleted or disabled switches itself off on its next check.
A logs alert created or turned back on is copied when the logs API commits it, but only in a team that already has an enabled logs copy. So a write in the logs product never widens a sample or undoes --disable, and those teams still wait for the next run.
A second run updates rather than duplicates, because legacy_configuration_id carries the row each copy came from.
A second run of either backfill, logs or insight, leaves an enabled copy's next_check_at alone unless it has none yet or its cadence changed, because the platform owns its schedule once the row exists. The logs product parks its own next check at the end of quiet hours, while the platform checks through them and only mutes. A new or re-enabled copy still takes the source's due time, which spreads a large copy's first checks the way the source spreads them.
Each run also copies the logs alert's snooze onto its platform alert row, so a snoozed alert stays silent, and an alert unsnoozed since the last run is unsnoozed here too.
Insight alerts are copied the same way:
python manage.py backfill_platform_insight_alert_configurations
It copies every insight alert, on every cadence.
A real-time check gets production's real-time budget: a 7-minute evaluation with one retry inside it.
An AI detector check runs on the AI queue, whose worker holds the model credentials, with production's retries: a check whose model call reaches no verdict is retried, and only its last attempt records the error.
A paused AI detector check, refused for a withdrawn rollout or consent or for no AI credits left, is not retried. It records an errored check and leaves the alert enabled, as production does.
An AI detector alert runs AI_DETECTOR_ALERT_POLICY on both stacks, so it announces a firing once and stays quiet until a clear check ends the firing.
Production marks each quiet firing check with firing_not_renotified, so the investigation safety net and the cutover do not treat it as a held notification.
Every other check gets one 12-minute attempt.
An AI detector check calls the model on both stacks while both run, and each call is billed to the project, so the backfill leaves AI detector alerts out.
Pass --include-ai-detector to copy them, with --team-id or --ids-file for an internal project, until the insight cutover.
An AI detector alert created or turned back on later waits for the next backfill run with the flag.
A copy made without the flag is switched off at its next check when its alert has since moved to the AI detector.
The AI queue's worker needs CLICKHOUSE_ALERTS_PLATFORM_INSIGHT_USER and its token file too, or those checks query as the worker's own ClickHouse user.
Run it only after the evaluation worker's chart sets CLICKHOUSE_ALERTS_PLATFORM_INSIGHT_USER and its token file.
Insight checks tag their queries with ClickHouseUser.ALERTS_PLATFORM_INSIGHT, a user of their own, so the parallel run never takes from the per-user budget of the user that production insight alerts query as.
Without that env the tag resolves to the worker's default user, which other workloads on the same servers already push against its concurrent query limit.
Logs checks use their own user, alerts_platform_logs, on the logs cluster.
Logs checks also wait for room in a fleet-wide pool of batches, ALERTS_PLATFORM_LOGS_MAX_INFLIGHT_BATCHES (default 5). A batch runs its cohort queries one after another, so this bounds the parallel run's concurrent logs queries however many teams are copied. A batch the pool turns away waits on a workflow timer and tries again for up to 20 seconds, so it usually still runs within its own minute. The wait holds no worker slot, because it is a timer in the workflow rather than a sleep in the evaluation activity. A batch still without a slot after that keeps its due time and a later tick picks it up, so a pool that is too small shows up as scheduler lag for source=logs and as alerts_platform_checks_deferred_total{source="logs"}.
Every copy adds ClickHouse load beside production's, so roll it out in steps. A full logs backfill hit ClickHouse's per-user concurrent query limit and had to be removed.
Copy one internal team with
--team-id, then a small sample with--sample-percent, for example 5. The sample is chosen by alert id, so a rerun copies the same alerts and a larger percentage only adds alerts.Watch the parallel run's ClickHouse cost in
query_log: itsclient_query_idstarts withalerts-platform-insight:.Watch scheduler lag for
source=insight, and thecapacityskip reason on the platform's skipped-check counter. Capacity skips mean ClickHouse refused the query for load.Widen the sample only while both stay flat. Two pools bound the parallel run, whatever the sample size:
ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_EVALUATIONScaps the queries that run in ClickHouse at once. A check holds one of these slots only while its query runs. A check that gets no slot within 45 seconds records nothing and stays due for a later tick.ALERTS_PLATFORM_INSIGHT_MAX_INFLIGHT_CHECKScaps the checks admitted at once, and defaults to the query cap. An admitted check also locks itself, so a later run never starts the same check while it runs. An admitted check holds an evaluation worker slot until it ends, including while it waits for a query slot, so keep this cap inside the workers' activity slots.
Each worker reads both caps from its own environment, so set them to the same values on every worker that runs insight checks, the AI detector worker included. A batch refills the check slots its finished checks free, in rounds within its tick, so the caps limit concurrency rather than checks per minute.
ALERTS_PLATFORM_INSIGHT_KEYS_PER_TICKbounds how many teams' batches start each minute, so a larger sample can need it raised. Sixty times its value must cover the batch keys due per hour. Otherwise the checks due at the top of the hour start later and later, while ClickHouse stays idle.Raise that cap from its default of 10 only while the daily count of refused queries,
exception_code = 202inquery_log, stays flat for bothalerts_platform_insightand the user that production insight alerts query as. The first shows the parallel run's own contention. The second shows whether it reaches production through the server-wide limit. Do not size it from per-second concurrency, which overcounts because short queries that run back to back inside one second read as concurrent. Code 202 also covers the server-wide limit, so a rise is a reason to look rather than proof that the cap caused it.
To stop either parallel run, logs or insight, pass --disable to its backfill, with --team-id to stop one team.
Pass --ids-file to either backfill, with one alert id per line, to copy only those alerts or, with --disable, to switch only their copies off. A rollout uses it to retire the alerts a comparison has enough evidence for and spend the load on the ones it still lacks.
It switches the copies off and keeps their rows, state and history. Checks already running finish.
Running the backfill again turns them back on at the production alert's next due time.
An hourly alert on the platform checks on a UTC grid, while production checks it at the alert's creation minute, so the two stacks check an hourly alert at different minutes.
Held deliveries
A source can hold an evaluation's delivery instead of starting it, for example while an agent investigates a firing.
It puts the request on PlatformAlertOutcome.held_delivery with a reason and a deadline, and leaves it out of the deliveries it starts.
record_outcomes writes the hold to PlatformAlertDeliveryHold in the same transaction as the outcome.
release_hold in the facade stamps a hold once, under a row lock.
It returns the request only for a release with send, and only to the releaser that won.
That releaser starts the delivery with start_deliveries, so the delivery keeps its usual id, then calls record_hold_delivery_started.
A retry from the same releaser gets the request back. Another releaser gets nothing.
reschedule_hold moves an unreleased hold's deadline.
deliver_evaluation also refuses an evaluation whose hold is unreleased or released with suppress.
alerts-platform-release-expired-holds-schedule starts alerts-platform-release-expired-holds every minute on the delivery queue, with SKIP overlap and a 5-minute execution timeout.
The workflow finds up to 200 holds past their deadline, releases each with send as the safety net, and starts their deliveries.
It re-reads each hold's deadline and configuration under the row lock, so a hold rescheduled after the find stays held. It skips a hold another transaction has locked, and a later run finds it again.
It also starts any released send whose delivery nobody recorded as started 5 minutes after the release, so a releaser that failed between the release and the start still delivers. It starts those with ALLOW_DUPLICATE_FAILED_ONLY, so a delivery that completed and lost only its start mark does not run again.
alerts_platform_holds_expired_total counts only holds a run released at their deadline, not recovered starts or retries.
It stops adding requests once they would outgrow one activity payload, and leaves the rest for the next run.
It suppresses, rather than sends, the hold of a disabled configuration, so turning the alert back on never sends an old firing. It still sends a hold whose alert went back to its source, because the request records the owner at check time.
The find step also deletes holds older than 90 days, which is how long the history rows they annotate live.
A rise in alerts_platform_holds_expired_total means a source holds deliveries and does not release them.
Postgres connectivity probe
Each evaluation activity issues one explicit SELECT 1 and checks for (1,) through Django's default main writer connection.
It inherits runtime credentials and connection/pooler settings without overrides, new aliases, or a separate pool.
It does not use replicas, persons services, or application tables.
One probe per tick means one intended activity attempt, not exactly-once SQL execution.
Connection setup and transaction control can issue additional statements.
The probe uses execute_with_timeout(1000, database="default") for a one-second transaction-local statement timeout.
The transaction commits on success and rolls back on failure, so the timeout does not persist on pooled connections.
Connection acquisition, SQL, and force-close cleanup run in the same executor thread, outside the default thread-sensitive executor.
close_db_connections closes initialized connections without a cleanup health-check query after failure.
Django OperationalError and InterfaceError become a sanitized AlertsPlatformPostgresProbeFailure activity failure.
Evaluation starts delivery after that failure or an activity start-to-close/schedule-to-close timeout.
Cancellation and unrelated errors propagate without starting delivery.
This handoff requires the parent and its worker to remain available; termination before child startup is not covered.
A successful evaluation workflow does not prove that the database probe succeeded. Activity success/failure logs and attempt duration remain separate from delivery outcomes; #97445 owns lifecycle telemetry. Activity duration includes executor wait, connection setup, transaction work, and cleanup, not just SQL execution. On activity timeout, the replay-safe workflow log records an unknown database outcome and continuation to delivery, without a completed query duration. A late activity completion does not replace that timeout observation.
The SQL timeout does not cover connection acquisition or pooler waiting. Temporal's activity bounds limit the workflow's wait, but timeout or cancellation cannot kill a running database thread or guarantee SQL has stopped. Cleanup runs when that thread finishes; this probe does not change connection defaults or add a watchdog.
Before production rollout, deployment owners must verify the runtime database identity and main writer route in dev using the deployment's credentials.
Verify a successful probe, statement timeout and transaction reset through the configured pooler, and delivery continuation after failure or timeout.
SELECT 1 alone does not verify the intended database identity, application grants, schema, or write readiness.
These tests do not replace deployment verification.
Worker registration does not deploy workers. The dev schedule sets the orchestration workflow's 50-second execution timeout; manually started workflows must set their own timeout.
Activity logs
The evaluation and delivery queues use an activity-only interceptor that emits alerts_platform_activity_started and alerts_platform_activity_finished through the shared write-only logger.
Discovery has SDK metrics and traces but does not use this logging interceptor.
The shared logger's async methods keep log processing and writes off the activity event loop.
Each retry emits its own start and finish events.
The shared logger supplies activity_id, activity_type, attempt, task_queue, workflow_id, workflow_namespace, workflow_run_id, and workflow_type.
Finish events add a monotonic duration_ms and an outcome of success, failure, or cancellation.
Start, success, and cancellation events use INFO; a failed attempt uses WARNING and includes only the exception class in exception_type.
A failed attempt does not mean the workflow has exhausted its retries.
When a valid OpenTelemetry span is active, both events include hexadecimal trace_id and span_id fields.
The interceptor does not log inputs, headers, exception messages, or locals, and does not capture exceptions separately.
Telemetry errors cannot replace an activity's result, exception, or cancellation.
Cancellation before the activity starts still propagates; cancellation while writing its finish log cannot replace the completed activity's outcome.
Evaluation and delivery workflow logging are unchanged.
Metrics
The shared worker exposes these SDK histograms with task_queue labels:
temporal_activity_schedule_to_start_latency: time from the current attempt's scheduling to its start. Earlier attempts and retry backoff are excluded.temporal_activity_execution_latency: worker-side execution time for an attempt, including interceptor overhead. This is not an end-to-end evaluation or delivery duration.
Both histograms use milliseconds and the SDK's default buckets.
Prometheus exposes their _bucket, _sum, and _count series.
Native metrics retain the temporal_ prefix. The alerts_platform_ prefix is the convention for the platform's own custom metrics.
Do not add workflow or run IDs as metric labels.
A worker killed before completion cannot emit a finish log or execution sample. Activities that never start produce no worker-side timing sample. Missing finish events are not evidence of success.
Tracing and deployment verification
All three deployments must set TEMPORAL_OTEL_PLUGIN_ENABLED=true, a nonempty OTEL_SERVICE_NAME, and the appropriate OTLP gRPC exporter configuration.
For example, configure OTEL_EXPORTER_OTLP_TRACES_ENDPOINT for the collector and any required credentials or TLS settings.
The shared startup command initializes Temporal's replay-safe provider, and the client registers OpenTelemetryPlugin(add_temporal_spans=True).
Do not add the legacy tracing interceptor or another provider.
The activity logs use the plugin's active attempt span. Orchestration, evaluation, child delivery, and their activity spans retain their trace relationships even when delivery runs after evaluation finishes. Retries have separate activity attempt spans.
Tests verify local logging, queue-labelled SDK metrics, and trace relationships without an application database or an external collector.
Charts rollout must separately configure all three deployments and verify Prometheus scraping and delivery to the tracing collector.
This change does not configure deployments, dashboards, alert rules, or SLO emission.
The alerts-platform SLO area remains reserved without changes to shared SLO handling or workflow inputs.