Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(cdp): warn against storing blobs in the cyclotron database (#130949)

Meikel Ratz7302b4717960
diff --git a/nodejs/src/cdp/services/cyclotron-v2/README.md b/nodejs/src/cdp/services/cyclotron-v2/README.md
index f5e003ac210a..87aba0e7dcfb 100644
--- a/nodejs/src/cdp/services/cyclotron-v2/README.md
+++ b/nodejs/src/cdp/services/cyclotron-v2/README.md
@@ -11,6 +11,30 @@ Run them with `rust/bin/migrate-cyclotron-node`.
Single table `cyclotron_jobs` with a single `state` BYTEA column
for all job payload data (mirrors how the Kafka backend serializes everything into one blob).
+## Keep rows small: no file or blob storage
+
+The cyclotron database is a queue.
+Rows are inserted, updated on every transition, and deleted soon after.
+PostgreSQL handles this churn well only while each row is small.
+
+Do not store large or variable-size payloads in this database.
+This applies to `cyclotron_jobs` and to any new table next to it,
+for example email attachments, rendered bodies, exports, or files from users.
+Put the bytes in object storage and keep only a key (and the metadata the worker needs) in the row.
+
+The reason:
+
+- PostgreSQL moves a large `BYTEA` or `TEXT` value out of line into the table's TOAST table, in chunks.
+ A delete does not give that space back. Only vacuum reclaims it.
+- Constant insert and delete of large rows bloats the table and its TOAST table,
+ and keeps autovacuum busy on the same database that serves dequeue.
+- Each large write also goes through the WAL, which adds write load and replication lag.
+- Dequeue, the janitor and every worker share this one database,
+ so bloat from one feature slows every queue.
+
+The `state` column is the only payload column.
+Keep it to what the next execution step needs to continue, not data that only passes through.
+
## Components
### Manager (producer)
diff --git a/nodejs/src/cdp/services/cyclotron-v2/types.ts b/nodejs/src/cdp/services/cyclotron-v2/types.ts
index 3ca1e830eb8e..f82ff3d8c8ff 100644
--- a/nodejs/src/cdp/services/cyclotron-v2/types.ts
+++ b/nodejs/src/cdp/services/cyclotron-v2/types.ts
@@ -26,6 +26,9 @@ export const CyclotronV2JobInitSchema = z.object({
priority: z.number().int().optional(),
scheduled: z.date().optional(),
parentRunId: z.string().nullish(),
+ // Keep `state` small. Queue rows churn, and PostgreSQL moves large BYTEA values to TOAST,
+ // where only vacuum reclaims the space. Files and blobs go to object storage, never into
+ // this column or a new cyclotron table. See "Keep rows small" in ./README.md.
state: z.instanceof(Buffer).nullish(),
distinctId: z.string().nullish(),
personId: z.string().nullish(),
diff --git a/rust/cyclotron-node-migrations/AGENTS.md b/rust/cyclotron-node-migrations/AGENTS.md
new file mode 100644
index 000000000000..b1d12ba94704
--- /dev/null
+++ b/rust/cyclotron-node-migrations/AGENTS.md
@@ -0,0 +1,13 @@
+# Working in `rust/cyclotron-node-migrations/`
+
+These migrations own the `cyclotron_node` database, which backs the Node cyclotron job queue.
+Read [Keep rows small](../../nodejs/src/cdp/services/cyclotron-v2/README.md#keep-rows-small-no-file-or-blob-storage) before you add a table or column here.
+
+- **The cyclotron database holds no file or blob payloads.**
+ A new table or column must not store large or variable-size bytes, such as email attachments, rendered bodies, exports or files from users.
+ Put those bytes in object storage and keep only the key and the metadata in the row.
+ Queue rows are inserted and deleted at a high rate, and large values in PostgreSQL go to TOAST, which only vacuum reclaims.
+ That pattern bloats the database that every queue dequeues from.
+- **Do not edit a migration that has been applied.**
+ sqlx stores a checksum of each migration and refuses to run when a file changes.
+ Add a new migration instead.
diff --git a/rust/cyclotron-node-migrations/CLAUDE.md b/rust/cyclotron-node-migrations/CLAUDE.md
new file mode 120000
index 000000000000..47dc3e3d863c
--- /dev/null
+++ b/rust/cyclotron-node-migrations/CLAUDE.md
@@ -0,0 +1 @@
+AGENTS.md
\ No newline at end of file