Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

5.4 KB
Raw Blame

Cyclotron (Node)

TypeScript job queue backed by PostgreSQL, using pg directly.

Schema

Lives in a separate database (cyclotron_node) with its own migrations in rust/cyclotron-node-migrations/. Run them with rust/bin/migrate-cyclotron-node.

Single table cyclotron_jobs with a single state BYTEA column for all job payload data (mirrors how the Kafka backend serializes everything into one blob).

Keep rows small: no file or blob storage

The cyclotron database is a queue. Rows are inserted, updated on every transition, and deleted soon after. PostgreSQL handles this churn well only while each row is small.

Do not store large or variable-size payloads in this database. This applies to cyclotron_jobs and to any new table next to it, for example email attachments, rendered bodies, exports, or files from users. Put the bytes in object storage and keep only a key (and the metadata the worker needs) in the row.

The reason:

  • PostgreSQL moves a large BYTEA or TEXT value out of line into the table's TOAST table, in chunks. A delete does not give that space back. Only vacuum reclaims it.
  • Constant insert and delete of large rows bloats the table and its TOAST table, and keeps autovacuum busy on the same database that serves dequeue.
  • Each large write also goes through the WAL, which adds write load and replication lag.
  • Dequeue, the janitor and every worker share this one database, so bloat from one feature slows every queue.

The state column is the only payload column. Keep it to what the next execution step needs to continue, not data that only passes through.

Components

Manager (producer)

Creates jobs via createJob() / bulkCreateJobs(). The bulk path uses UNNEST for efficient batch inserts.

cancelJobs() flags in-flight jobs for cancellation (by ids, by parent run, or all jobs of a function) and pulls parked rows' wake time forward. It never writes a terminal status: the owning worker observes cancel_requested_at on dequeue and performs the actual cancellation together with its lifecycle/metric writes. bulkCreateAndCheckIn() re-checks the flag inside its transaction and refuses the whole page when set, so a fan-out in progress cannot commit jobs after a cancel sweep counted zero remaining.

Worker (consumer)

Poll-based consumer using FOR UPDATE SKIP LOCKED to dequeue batches. Each dequeued job exposes an ack interface:

  • ack() — mark completed
  • fail() — mark failed
  • retry({ delayMs?, state? }) — re-queue with optional delay and updated state
  • cancel() — mark canceled
  • heartbeat() — extend the lock to prevent the janitor from reclaiming the job

Janitor (standalone service)

Runs on a timer interval as its own service (PLUGIN_SERVER_MODE=cdp-cyclotron-v2-janitor). All operations use FOR UPDATE SKIP LOCKED so multiple janitor instances are safe.

Responsibilities:

  • Cleanup — bounded DELETE of terminal jobs older than a grace period
  • Stalled job recovery — reset jobs with stale heartbeats back to available
  • Poison pill recovery — give up on jobs that have been reset too many times (janitor_touch_count), but never silently: each is recorded as a failed, replayable invocation result on hog_invocation_results (discoverable in the Invocations UI, re-runnable by the rerun tooling) before the cyclotron row is deleted. Workers reset janitor_touch_count = 0 on every deliberate release, so the budget counts CONSECUTIVE stalls — long-lived waits don't accrue touches for life.
  • Queue depth metrics — Prometheus gauges per queue

Integration

The CyclotronJobQueuePostgresV2 wrapper in job-queue/ bridges this SDK with the existing CyclotronJobInvocation types used by CDP consumers. It's enabled via CDP_CYCLOTRON_NODE_ENABLED=true and routed alongside the existing backends (kafka, delay) in CyclotronJobQueue.

No DLQ

Failed jobs are deleted by the janitor along with completed and canceled jobs. There is no dead-letter queue — a DLQ would fill the database exponentially since failed jobs often produce more failed jobs on retry, and it would be a second recovery path operators have to learn alongside rerun. Errors are captured via logs and metrics before the job reaches terminal status, and poison-pill give-ups converge on the existing rerun system: they are written as failed rows on hog_invocation_results (the table rerun already reads) and deleted only once that row is durably enqueued, so a lost invocation is always recoverable via the standard rerun tooling rather than a bespoke table.

Future work

  • Activity table — a separate table to store a high-level record of all run activities. Jobs are ephemeral (deleted after completion), so a durable activity log would give visibility into historical runs without keeping the jobs table unbounded.
  • Surfacing action_id for workflows — the column exists and parked rows are indexed by it (idx_cyclotron_jobs_action_reschedule, used by the timing-edit reschedule sweep), but scheduled workflow actions still aren't queryable or displayable in product (e.g. "show me all pending actions for this workflow step").
  • Cancellation API — API tooling to cancel scheduled jobs by query (e.g. by function_id, parent_run_id, or action_id). Especially useful for workflows where a user cancels a run and all its pending scheduled jobs need to be cleaned up.