:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(cdp): warn against storing blobs in the cyclotron database (#130949)
Meikel Ratz7302b4717960
diff --git a/nodejs/src/cdp/services/cyclotron-v2/README.md b/nodejs/src/cdp/services/cyclotron-v2/README.mdindex f5e003ac210a..87aba0e7dcfb 100644--- a/nodejs/src/cdp/services/cyclotron-v2/README.md+++ b/nodejs/src/cdp/services/cyclotron-v2/README.md@@ -11,6 +11,30 @@ Run them with `rust/bin/migrate-cyclotron-node`.Single table `cyclotron_jobs` with a single `state` BYTEA columnfor all job payload data (mirrors how the Kafka backend serializes everything into one blob).+## Keep rows small: no file or blob storage++The cyclotron database is a queue.+Rows are inserted, updated on every transition, and deleted soon after.+PostgreSQL handles this churn well only while each row is small.++Do not store large or variable-size payloads in this database.+This applies to `cyclotron_jobs` and to any new table next to it,+for example email attachments, rendered bodies, exports, or files from users.+Put the bytes in object storage and keep only a key (and the metadata the worker needs) in the row.++The reason:++- PostgreSQL moves a large `BYTEA` or `TEXT` value out of line into the table's TOAST table, in chunks.+ A delete does not give that space back. Only vacuum reclaims it.+- Constant insert and delete of large rows bloats the table and its TOAST table,+ and keeps autovacuum busy on the same database that serves dequeue.+- Each large write also goes through the WAL, which adds write load and replication lag.+- Dequeue, the janitor and every worker share this one database,+ so bloat from one feature slows every queue.++The `state` column is the only payload column.+Keep it to what the next execution step needs to continue, not data that only passes through.+## Components### Manager (producer)diff --git a/nodejs/src/cdp/services/cyclotron-v2/types.ts b/nodejs/src/cdp/services/cyclotron-v2/types.tsindex 3ca1e830eb8e..f82ff3d8c8ff 100644--- a/nodejs/src/cdp/services/cyclotron-v2/types.ts+++ b/nodejs/src/cdp/services/cyclotron-v2/types.ts@@ -26,6 +26,9 @@ export const CyclotronV2JobInitSchema = z.object({priority: z.number().int().optional(),scheduled: z.date().optional(),parentRunId: z.string().nullish(),+ // Keep `state` small. Queue rows churn, and PostgreSQL moves large BYTEA values to TOAST,+ // where only vacuum reclaims the space. Files and blobs go to object storage, never into+ // this column or a new cyclotron table. See "Keep rows small" in ./README.md.state: z.instanceof(Buffer).nullish(),distinctId: z.string().nullish(),personId: z.string().nullish(),diff --git a/rust/cyclotron-node-migrations/AGENTS.md b/rust/cyclotron-node-migrations/AGENTS.mdnew file mode 100644index 000000000000..b1d12ba94704--- /dev/null+++ b/rust/cyclotron-node-migrations/AGENTS.md@@ -0,0 +1,13 @@+# Working in `rust/cyclotron-node-migrations/`++These migrations own the `cyclotron_node` database, which backs the Node cyclotron job queue.+Read [Keep rows small](../../nodejs/src/cdp/services/cyclotron-v2/README.md#keep-rows-small-no-file-or-blob-storage) before you add a table or column here.++- **The cyclotron database holds no file or blob payloads.**+ A new table or column must not store large or variable-size bytes, such as email attachments, rendered bodies, exports or files from users.+ Put those bytes in object storage and keep only the key and the metadata in the row.+ Queue rows are inserted and deleted at a high rate, and large values in PostgreSQL go to TOAST, which only vacuum reclaims.+ That pattern bloats the database that every queue dequeues from.+- **Do not edit a migration that has been applied.**+ sqlx stores a checksum of each migration and refuses to run when a file changes.+ Add a new migration instead.diff --git a/rust/cyclotron-node-migrations/CLAUDE.md b/rust/cyclotron-node-migrations/CLAUDE.mdnew file mode 120000index 000000000000..47dc3e3d863c--- /dev/null+++ b/rust/cyclotron-node-migrations/CLAUDE.md@@ -0,0 +1 @@+AGENTS.md\ No newline at end of file