Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat: support custom ClickHouse cluster migrations (#17015)

Valery Meleshkin5b1ff5ffc1ad
diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md
index da853d6c75..e97c35a964 100644
--- a/.agents/AGENTS.md
+++ b/.agents/AGENTS.md
@@ -87,8 +87,9 @@ langfuse/
- High-signal shared entry points:
- Domain models: `packages/shared/src/domain/{observations,traces,scores}.ts`
- Postgres schema: `packages/shared/prisma/schema.prisma`
- - ClickHouse migrations:
- `packages/shared/clickhouse/migrations/{clustered,unclustered}/*.sql`
+ - Canonical ClickHouse migration templates (rendered for clustered and
+ unclustered installs):
+ `packages/shared/clickhouse/migrations/canonical/*.sql`
- Architecture principles live in `.agents/ARCHITECTURE_PRINCIPLES.md`.
## Core Commands
diff --git a/.agents/skills/clickhouse-best-practices/SKILL.md b/.agents/skills/clickhouse-best-practices/SKILL.md
index 87b573e219..7e47836377 100644
--- a/.agents/skills/clickhouse-best-practices/SKILL.md
+++ b/.agents/skills/clickhouse-best-practices/SKILL.md
@@ -46,28 +46,38 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization
repository layer then reads baggage via `normalizeClickHouseQueryTags(...)`
and writes it to `log_comment`. Prefer setting attribution at entry points
rather than passing tags through every repository call.
-- In `packages/shared/clickhouse/migrations/clustered/**`, every metadata
- `ALTER` (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) must end with
- `SETTINGS alter_sync = 2`, and every mutation-creating `ALTER`
- (`MATERIALIZE …`, `UPDATE`, `DELETE`) with `SETTINGS mutations_sync = 2`.
+- `packages/shared/clickhouse/migrations/canonical/**` is the single canonical
+ template tree rendered for clustered and unclustered installs. Put
+ `{CLICKHOUSE_CLUSTER_CLAUSE}` at every cluster-aware DDL position. Use
+ `{CLICKHOUSE_REPLICATION_PREFIX}` only for engines that deliberately differ
+ by mode; some tables intentionally stay non-replicated in both modes.
+- Every metadata `ALTER` (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) in a new
+ canonical migration must include
+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}`, and every
+ mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`) must include
+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2}`.
This applies to a file holding a single `ALTER` too — the race is across
migration files, not within one. `alter_sync` defaults to `1`, so the
statement returns as soon as the initiating replica has bumped the table's
metadata version in Keeper; golang-migrate then opens the next file
immediately, and its first `ALTER` on that table can land on a replica still
on the previous version. ClickHouse refuses to queue it and aborts the whole
- run with `code 517 … metadata version on replica is N, while common metadata
- is N+1`. Note that `mutations_sync` does not substitute for `alter_sync`: it
- governs when mutations finish, not metadata propagation. The matching
- `unclustered/` file runs against plain `MergeTree` and does not need (and
- should not duplicate) these settings.
+ run with `code 517` because the replica metadata version is behind the common
+ metadata version. Note that `mutations_sync` does not substitute for `alter_sync`: it
+ governs when mutations finish, not metadata propagation. The renderer omits
+ these fragments for unclustered `MergeTree` migrations. Use
+ `{CLICKHOUSE_UNCLUSTERED_ONLY:...}` only for a deliberate mode-specific
+ difference. Do not retrofit synchronization settings into already-shipped
+ migrations merely to normalize them; the historical compatibility test
+ intentionally protects their existing output.
- Never use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` /
`EXCHANGE TABLES`) in ClickHouse migrations. The atomic replace requires
`renameat2` filesystem support, which NFS-backed self-hosted deployments
(e.g. ClickHouse data on AWS EFS) lack — the migration fails and the
deployment aborts on startup (GitHub issue #14906). Redefine a plain view as
- two statements in the same migration file: `DROP VIEW IF EXISTS <name>
- [ON CLUSTER default];` then `CREATE VIEW <name> [ON CLUSTER default] AS …`.
+ two statements in the same migration file. First use
+ `DROP VIEW IF EXISTS <name> {CLICKHOUSE_CLUSTER_CLAUSE};`, then
+ `CREATE VIEW <name> {CLICKHOUSE_CLUSTER_CLAUSE} AS …`.
The migration runner passes `x-multi-statement=true` and golang-migrate
splits files on `;` without parsing SQL, so keep semicolons out of comments
and string literals. Keep every statement idempotent
@@ -78,13 +88,13 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization
- Never drop-and-recreate a materialized view whose source table receives live
inserts: every row inserted between `DROP` and `CREATE` is silently and
permanently missing from the target table. Change an MV's SELECT with
- `ALTER TABLE <mv> [ON CLUSTER default] MODIFY QUERY <select>`, which swaps
+ `ALTER TABLE <mv> {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY <select>`, which swaps
the transformation without interrupting ingestion. When the change adds
columns, `ALTER` the target table(s) first (`ADD COLUMN IF NOT EXISTS …`),
- then `MODIFY QUERY`; in `clustered/` files those target-table `ALTER`s must
- carry `SETTINGS alter_sync = 2` so no host applies the new MV query before
- its target replica has the new columns. `MODIFY QUERY` is only viable for
- TO-table MVs (all Langfuse MVs use `TO`).
+ then `MODIFY QUERY`; those target-table `ALTER`s must carry the
+ clustered-only `alter_sync` template fragment so no host applies the new MV
+ query before its target replica has the new columns. `MODIFY QUERY` is only
+ viable for TO-table MVs (all Langfuse MVs use `TO`).
---
@@ -111,7 +121,7 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization
- [ ] LowCardinality applied to appropriate string columns
- [ ] Partition key cardinality bounded (100-1,000 values)
- [ ] ReplacingMergeTree has version column if used
-- [ ] Every metadata ALTER in a clustered migration ends with `SETTINGS alter_sync = 2` — including files with a single ALTER, since the next migration file is what breaks — and every `MATERIALIZE …` / `UPDATE` / `DELETE` with `SETTINGS mutations_sync = 2`; `mutations_sync` is not a substitute for `alter_sync`; unclustered mirror has none
+- [ ] Every metadata ALTER in a new canonical migration includes `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}` — including files with a single ALTER, since the next migration file is what breaks — and every `MATERIALIZE …` / `UPDATE` / `DELETE` includes the corresponding `mutations_sync` fragment; `mutations_sync` is not a substitute for `alter_sync`; do not normalize already-shipped migration output; both rendered modes pass `prepareMigrations.test.ts`
- [ ] No `CREATE OR REPLACE VIEW/TABLE` or `EXCHANGE TABLES` in migrations (breaks NFS/EFS self-hosting); plain views are redefined via `DROP VIEW IF EXISTS` + `CREATE VIEW` in the same file
- [ ] Materialized views are never dropped and recreated while their source table takes inserts; SELECT changes go through `ALTER TABLE <mv> MODIFY QUERY` after the target-table `ALTER`s
diff --git a/.agents/skills/code-review/references/review-checklist.md b/.agents/skills/code-review/references/review-checklist.md
index 61f5c9aa25..22b3f88617 100644
--- a/.agents/skills/code-review/references/review-checklist.md
+++ b/.agents/skills/code-review/references/review-checklist.md
@@ -6,11 +6,32 @@ This is the canonical shared review checklist for Langfuse.
### ClickHouse
-- ClickHouse migrations in the `packages/shared/clickhouse/migrations/clustered` directory should include `ON CLUSTER default` and should use `Replicated` merge tree table types.
- - E.g. `ReplacingMergeTree` is likely an error while `ReplicatedReplacingMergeTree` would be correct in most cases.
-- ClickHouse migrations in the `packages/shared/clickhouse/migrations/unclustered` directory must not include `ON CLUSTER` statements and must not use `Replicated` merge tree table types.
-- Migrations in `packages/shared/clickhouse/migrations/clustered` should match their counterparts in `packages/shared/clickhouse/migrations/unclustered` aside from the restrictions listed above.
-- Every metadata `ALTER` (`ADD`/`DROP`/`MODIFY COLUMN`, `ADD`/`DROP INDEX`) in a `clustered` migration must end with `SETTINGS alter_sync = 2`, including files that contain only one `ALTER` — the race is across migration files, not within one. `alter_sync` defaults to `1`, so without it the statement returns once the initiating replica has bumped the table's metadata version, and the next migration file's first `ALTER` on that table can hit a replica still on the old version, aborting the run with `code 517 … Looks like this replica doesn't catchup with latest ALTER query updates`. `SETTINGS mutations_sync = 2` does not substitute for it: that setting governs when mutations finish, not metadata propagation. The `unclustered` counterpart runs against plain `MergeTree` and does not need (and should not duplicate) either setting.
+- `packages/shared/clickhouse/migrations/canonical` is the single canonical
+ migration-template tree for both deployment modes. Every cluster-aware DDL
+ position must use `{CLICKHOUSE_CLUSTER_CLAUSE}` instead of a literal
+ `ON CLUSTER` clause.
+- Use `{CLICKHOUSE_REPLICATION_PREFIX}` only for a table engine that should be
+ replicated in clustered mode and plain in unclustered mode. Do not apply it
+ globally: some tables intentionally use a plain engine in both modes.
+- Every metadata `ALTER` (`ADD`/`DROP`/`MODIFY COLUMN`, `ADD`/`DROP INDEX`) in a
+ new migration must add
+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}`, including files that
+ contain only one `ALTER` — the race is across migration files, not within
+ one. Mutation-generating statements use
+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2}`. `alter_sync`
+ defaults to `1`, so without it the statement returns once the initiating
+ replica has bumped the table's metadata version, and the next migration
+ file's first `ALTER` on that table can hit a replica still on the old
+ version, aborting the run with `code 517` because the replica metadata
+ version is behind the common metadata version. `mutations_sync` does not
+ substitute for `alter_sync`: it governs mutation completion, not metadata
+ propagation. The renderer omits these fragments in unclustered mode. Do not
+ retrofit these settings into already-shipped migrations solely for
+ consistency; preserve historical output unless a forward fix deliberately
+ changes it.
+- Run `prepareMigrations.test.ts` when migrations or their renderer change. It
+ must render both modes with no unresolved tokens and keep both historical
+ outputs byte-for-byte unchanged.
- Migrations must not use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` / `EXCHANGE TABLES`): the atomic replace requires `renameat2`, which fails on NFS-backed self-hosted deployments such as ClickHouse-on-EFS (GitHub issue #14906). Plain views are redefined with `DROP VIEW IF EXISTS <name>;` followed by `CREATE VIEW <name> AS ...` in the same migration file — the migration runner enables `x-multi-statement=true`, and every statement should stay idempotent (`IF EXISTS` / `IF NOT EXISTS`) so a dirty migration can be re-run.
- Materialized views must never be dropped and recreated while their source table receives inserts — rows written between `DROP` and `CREATE` are silently and permanently missing from the target table. Redefine the SELECT with `ALTER TABLE <mv> MODIFY QUERY ...` instead, altering the target table(s) first when columns change.
- For rationale and full patterns, see "Langfuse-Specific Rules" in the [`clickhouse-best-practices`](../../clickhouse-best-practices/SKILL.md) skill.
diff --git a/.env.prod.example b/.env.prod.example
index 413e2e7da6..c0ffff562e 100644
--- a/.env.prod.example
+++ b/.env.prod.example
@@ -248,6 +248,8 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true
# Clickhouse configuration
# CLICKHOUSE_URL=
+# Raw ClickHouse cluster name; do not include SQL quotes. Helm users with a
+# non-default cluster can set this through langfuse.additionalEnv.
# CLICKHOUSE_CLUSTER_NAME=default
# CLICKHOUSE_DB=default
# CLICKHOUSE_USER=
diff --git a/docker-compose.build.yml b/docker-compose.build.yml
index 400ddf2714..3a0120001a 100644
--- a/docker-compose.build.yml
+++ b/docker-compose.build.yml
@@ -41,6 +41,7 @@ services:
CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse}
CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
+ CLICKHOUSE_CLUSTER_NAME: ${CLICKHOUSE_CLUSTER_NAME:-default}
LANGFUSE_JSON_BAD_UNICODE_ESCAPE: ${LANGFUSE_JSON_BAD_UNICODE_ESCAPE:-}
LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}
LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-us-east-1}
diff --git a/docker-compose.yml b/docker-compose.yml
index f9ee89ba60..52fd3952a4 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -37,6 +37,7 @@ services:
CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}
CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEME
CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}
+ CLICKHOUSE_CLUSTER_NAME: ${CLICKHOUSE_CLUSTER_NAME:-default}
LANGFUSE_JSON_BAD_UNICODE_ESCAPE: ${LANGFUSE_JSON_BAD_UNICODE_ESCAPE:-}
LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}
LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}
diff --git a/knip.jsonc b/knip.jsonc
index 1cb5d48f1d..93e2568e41 100644
--- a/knip.jsonc
+++ b/knip.jsonc
@@ -7,6 +7,8 @@
"workspaces": {
"packages/shared": {
"ignoreFiles": [
+ // Standalone renderer invoked by ClickHouse migration shell scripts and Docker builds.
+ "clickhouse/scripts/prepare-migrations.mjs",
// Standalone helper invoked manually as documented in the adjacent framework-traces README.
"scripts/seeder/utils/framework-traces/merge-observations.ts",
],
diff --git a/package.json b/package.json
index 468acf3cbf..0ef410735a 100644
--- a/package.json
+++ b/package.json
@@ -19,6 +19,8 @@
"db:migrate": "turbo run db:migrate",
"db:seed": "turbo run db:seed",
"db:seed:examples": "turbo run db:seed:examples",
+ "ch:migrations:materialize": "pnpm --filter @langfuse/shared run ch:migrations:materialize",
+ "ch:migrations:clean": "pnpm --filter @langfuse/shared run ch:migrations:clean",
"nuke": "bash ./scripts/nuke.sh",
"scan:client-bundle": "node scripts/scan-client-bundle.mjs web/.next/static",
"seed": "pnpm --filter=shared run seed:scenario",
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.md
index adc165e5de..38f32f40d4 100644
--- a/packages/shared/AGENTS.md
+++ b/packages/shared/AGENTS.md
@@ -35,7 +35,10 @@
`src/server/query-ast/kysely/`.
- Postgres schema: `prisma/schema.prisma`
- Prisma migrations: `prisma/migrations/*`
-- ClickHouse migrations: `clickhouse/migrations/{clustered,unclustered}/*`
+- Canonical ClickHouse migration templates (rendered for clustered and
+ unclustered installs): `clickhouse/migrations/canonical/*`
+- ClickHouse SQL identifier and string quoting:
+ `src/server/clickhouse/clickhouseIdentifiers.ts`
- Seeder and support scripts: `scripts/seeder/*`, `clickhouse/scripts/*`
## Export Entry Points
@@ -75,6 +78,7 @@
the Mastra runtime and sandbox belong to the worker.
- Narrower exported subpaths also exist for targeted imports:
`@langfuse/shared/src/server/auth/apiKeys`,
+ `@langfuse/shared/src/server/clickhouse/clickhouseIdentifiers`,
`@langfuse/shared/src/server/ee/ingestionMasking`,
`@langfuse/shared/src/server/llm/llmText`, and
`@langfuse/shared/src/utils/chatml`. The experimental
@@ -106,6 +110,8 @@ the same PR.
- Prisma generate: `pnpm --filter @langfuse/shared run db:generate`
- Prisma migrate (dev): `pnpm --filter @langfuse/shared run db:migrate`
- ClickHouse reset: `pnpm --filter @langfuse/shared run ch:reset`
+- Materialize direct-migration trees: `pnpm ch:migrations:materialize`
+- Clean direct-migration trees: `pnpm ch:migrations:clean`
## Playbooks
@@ -119,7 +125,16 @@ the same PR.
### ClickHouse schema change
-1. Add migration under `clickhouse/migrations/*`.
+1. Add one canonical migration pair under `clickhouse/migrations/canonical/*`.
+ - Use `{CLICKHOUSE_CLUSTER_CLAUSE}` for every cluster-aware DDL clause.
+ - Use `{CLICKHOUSE_REPLICATION_PREFIX}` only on table engines that should
+ be replicated in clustered mode and plain in unclustered mode.
+ - Put clustered-only synchronization fragments inside
+ `{CLICKHOUSE_CLUSTERED_ONLY:...}`. Use
+ `{CLICKHOUSE_UNCLUSTERED_ONLY:...}` only when preserving a deliberate
+ mode-specific difference.
+ - Run `src/server/clickhouse/prepareMigrations.test.ts`; it validates both
+ rendered modes and protects the historical migration output.
- Redefining views or materialized views follows strict patterns (no
`CREATE OR REPLACE VIEW`; MV SELECT changes via
`ALTER TABLE … MODIFY QUERY`) — apply the "Langfuse-Specific Rules" in
diff --git a/packages/shared/clickhouse/migrations/README.md b/packages/shared/clickhouse/migrations/README.md
new file mode 100644
index 0000000000..34a2c42256
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/README.md
@@ -0,0 +1,101 @@
+# ClickHouse migrations
+
+`canonical/` is the only authored migration-template directory. Do not add a
+second source copy. `clickhouse/scripts/prepare-migrations.mjs` renders the SQL
+consumed by `golang-migrate`.
+
+The production web image contains generated `clustered/` and `unclustered/`
+directories next to `canonical/`. They are the primary migration inputs for
+default-cluster and unclustered deployments and preserve documented recovery
+commands that run `golang-migrate` directly inside the container. A custom
+cluster name is rendered from `canonical/` into a temporary directory at
+runtime. Source checkouts contain only `canonical/`, so the migration scripts
+render a temporary directory for every mode during local development.
+
+Do not edit or treat the generated `clustered/` and `unclustered/` directories
+as migration sources. Migrations 0001-0047 render byte-for-byte identically to
+the files shipped before the canonical template was introduced.
+
+Use these explicit placeholders:
+
+- `{CLICKHOUSE_CLUSTER_CLAUSE}` at every cluster-aware DDL position. It renders
+ as `ON CLUSTER default`, a safely quoted custom cluster name, or an empty
+ string for unclustered deployments.
+- `{CLICKHOUSE_REPLICATION_PREFIX}` only on engines that should render as
+ `Replicated*MergeTree` in clustered mode and `*MergeTree` in unclustered
+ mode. Some tables intentionally remain non-replicated in both modes.
+- `{CLICKHOUSE_CLUSTERED_ONLY:...}` for settings required only by replicated
+ migrations, such as `alter_sync = 2` and `mutations_sync = 2`.
+- `{CLICKHOUSE_UNCLUSTERED_ONLY:...}` only for an intentional mode-specific
+ difference. Existing uses preserve historical migration behavior.
+- `{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:...}` only in already-shipped
+ migrations whose clustered and unclustered files had unusual end-of-file
+ newlines. Do not use it in new migrations.
+
+Keep migration version numbers and filenames stable. Do not retrofit
+synchronization settings or formatting into already-shipped
+migrations merely for consistency. Their rendered output is compatibility
+history and should change only as a deliberate forward fix.
+
+After changing a migration or the renderer, run:
+
+```sh
+pnpm --filter @langfuse/shared run test prepareMigrations.test.ts
+```
+
+The test invokes the production scripts through POSIX `sh`, checks both modes,
+and locks the exact historical bytes for both generated migration trees.
+
+## Direct migrations from a source checkout
+
+Source checkouts do not contain the generated `clustered/` and `unclustered/`
+trees. Materialize them before using a direct `golang-migrate` command from an
+operations runbook:
+
+```sh
+pnpm ch:migrations:materialize
+migrate -source file://packages/shared/clickhouse/migrations/unclustered -database "<migration-url>" up
+pnpm ch:migrations:clean
+```
+
+Always run the clean command after the migration, including after a failed
+migration. Do not edit the materialized files. They are intentionally not
+ignored by Git so an incomplete cleanup or stale generated tree remains visible
+in `git status`.
+
+## Helm chart with a custom cluster name
+
+The ClickHouse cluster bundled with the current Helm chart uses the name
+`default`; it requires no additional configuration.
+
+| ClickHouse deployment | Helm setting | Migration input |
+| ------------------------------------------------- | ---------------------------------------------------- | -------------------------------- |
+| Chart-managed or external cluster named `default` | `clickhouse.cluster.enabled: true` | Delivered `clustered/` |
+| External, non-clustered server | `clickhouse.cluster.enabled: false` | Delivered `unclustered/` |
+| External cluster with another name | Enable the cluster and set `CLICKHOUSE_CLUSTER_NAME` | Runtime render from `canonical/` |
+
+For an external ClickHouse cluster with another name, enable clustered
+migrations and pass the raw cluster name to both Langfuse containers through
+the chart's global `additionalEnv`:
+
+```yaml
+clickhouse:
+ deploy: false
+ host: clickhouse.example.internal
+ cluster:
+ enabled: true
+ migration:
+ autoMigrate: true
+
+langfuse:
+ additionalEnv:
+ - name: CLICKHOUSE_CLUSTER_NAME
+ value: "virtual-cluster"
+```
+
+Do not add SQL quotes around `virtual-cluster`; the migration renderer quotes
+and URL-encodes the value. Configure the remaining ClickHouse host, ports,
+database, user, password, and TLS values through the chart as usual. On web
+startup, Langfuse renders the custom-cluster migration tree and applies it
+automatically. The same global environment variable reaches the worker for
+runtime queries.
diff --git a/packages/shared/clickhouse/migrations/canonical/0001_traces.down.sql b/packages/shared/clickhouse/migrations/canonical/0001_traces.down.sql
new file mode 100644
index 0000000000..1bf5958a67
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0001_traces.down.sql
@@ -0,0 +1 @@
+DROP TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/clustered/0001_traces.up.sql b/packages/shared/clickhouse/migrations/canonical/0001_traces.up.sql
similarity index 84%
rename from packages/shared/clickhouse/migrations/clustered/0001_traces.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0001_traces.up.sql
index dee75a8f01..4766ecf30a 100644
--- a/packages/shared/clickhouse/migrations/clustered/0001_traces.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0001_traces.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE traces ON CLUSTER default (
+CREATE TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} (
`id` String,
`timestamp` DateTime64(3),
`name` String,
@@ -20,7 +20,7 @@ CREATE TABLE traces ON CLUSTER default (
INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,
INDEX idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1
-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)
PRIMARY KEY (
project_id,
toDate(timestamp)
diff --git a/packages/shared/clickhouse/migrations/canonical/0002_observations.down.sql b/packages/shared/clickhouse/migrations/canonical/0002_observations.down.sql
new file mode 100644
index 0000000000..95ce4c33b4
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0002_observations.down.sql
@@ -0,0 +1 @@
+DROP TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/unclustered/0002_observations.up.sql b/packages/shared/clickhouse/migrations/canonical/0002_observations.up.sql
similarity index 87%
rename from packages/shared/clickhouse/migrations/unclustered/0002_observations.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0002_observations.up.sql
index cafd0c71eb..94654e8d1a 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0002_observations.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0002_observations.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE observations (
+CREATE TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} (
`id` String,
`trace_id` String,
`project_id` String,
@@ -32,7 +32,7 @@ CREATE TABLE observations (
INDEX idx_id id TYPE bloom_filter() GRANULARITY 1,
INDEX idx_trace_id trace_id TYPE bloom_filter() GRANULARITY 1,
INDEX idx_project_id project_id TYPE bloom_filter() GRANULARITY 1
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(start_time)
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(start_time)
PRIMARY KEY (
project_id,
`type`,
@@ -43,5 +43,4 @@ ORDER BY (
`type`,
toDate(start_time),
id
-);
-
+);{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:2}
diff --git a/packages/shared/clickhouse/migrations/canonical/0003_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0003_scores.down.sql
new file mode 100644
index 0000000000..7c84090900
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0003_scores.down.sql
@@ -0,0 +1 @@
+DROP TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/clustered/0003_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0003_scores.up.sql
similarity index 83%
rename from packages/shared/clickhouse/migrations/clustered/0003_scores.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0003_scores.up.sql
index 3056924974..1ae2551e18 100644
--- a/packages/shared/clickhouse/migrations/clustered/0003_scores.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0003_scores.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE scores ON CLUSTER default (
+CREATE TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} (
`id` String,
`timestamp` DateTime64(3),
`project_id` String,
@@ -19,7 +19,7 @@ CREATE TABLE scores ON CLUSTER default (
`is_deleted` UInt8,
INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,
INDEX idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1
-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)
PRIMARY KEY (
project_id,
toDate(timestamp),
diff --git a/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.down.sql
new file mode 100644
index 0000000000..4684be1b6d
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_id project_id TYPE bloom_filter() GRANULARITY 1;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.up.sql
new file mode 100644
index 0000000000..eea778e98b
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.up.sql
@@ -0,0 +1 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.down.sql
new file mode 100644
index 0000000000..4c78062f77
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_session_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.up.sql
new file mode 100644
index 0000000000..b87aba372c
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_session_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.down.sql
new file mode 100644
index 0000000000..6830e1a0c7
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_user_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}
diff --git a/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.up.sql
new file mode 100644
index 0000000000..6c1f7c7721
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_user_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.down.sql b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.down.sql
new file mode 100644
index 0000000000..a0c6b927c5
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.down.sql
@@ -0,0 +1 @@
+DROP TABLE event_log {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.up.sql b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.up.sql
similarity index 89%
rename from packages/shared/clickhouse/migrations/clustered/0007_add_event_log.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0007_add_event_log.up.sql
index 9fa430da0e..cd6a9116cd 100644
--- a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE event_log ON CLUSTER default
+CREATE TABLE event_log {CLICKHOUSE_CLUSTER_CLAUSE}
(
`id` String,
`project_id` String,
diff --git a/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.down.sql b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.down.sql
new file mode 100644
index 0000000000..841f5654cd
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.down.sql
@@ -0,0 +1,3 @@
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS environment{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS environment{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS environment{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.up.sql b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.up.sql
new file mode 100644
index 0000000000..87f031fd6a
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.down.sql b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.down.sql
new file mode 100644
index 0000000000..2086b25c25
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.down.sql
@@ -0,0 +1,7 @@
+-- Drop materialized views
+DROP VIEW IF EXISTS project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+
+-- Drop the project_environments table
+DROP TABLE IF EXISTS project_environments {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.up.sql b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.up.sql
similarity index 58%
rename from packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.up.sql
index 50c55b4690..503edfc595 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.up.sql
@@ -1,12 +1,12 @@
-- Create the project_environments table with AggregatingMergeTree engine
-CREATE TABLE project_environments (
+CREATE TABLE project_environments {CLICKHOUSE_CLUSTER_CLAUSE} (
`project_id` String,
`environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))
-) ENGINE = AggregatingMergeTree
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}AggregatingMergeTree
ORDER BY (project_id);
-- Create materialized view for traces
-CREATE MATERIALIZED VIEW project_environments_traces_mv TO project_environments AS
+CREATE MATERIALIZED VIEW project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS
SELECT
project_id,
groupUniqArray(environment) AS environments
@@ -14,7 +14,7 @@ FROM traces
GROUP BY project_id;
-- Create materialized view for observations
-CREATE MATERIALIZED VIEW project_environments_observations_mv TO project_environments AS
+CREATE MATERIALIZED VIEW project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS
SELECT
project_id,
groupUniqArray(environment) AS environments
@@ -22,7 +22,7 @@ FROM observations
GROUP BY project_id;
-- Create materialized view for scores
-CREATE MATERIALIZED VIEW project_environments_scores_mv TO project_environments AS
+CREATE MATERIALIZED VIEW project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS
SELECT
project_id,
groupUniqArray(environment) AS environments
diff --git a/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.down.sql
new file mode 100644
index 0000000000..44f388da92
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS metadata;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.up.sql
new file mode 100644
index 0000000000..0aaf63b175
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN metadata Map(LowCardinality(String), String) AFTER comment;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.down.sql b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.down.sql
new file mode 100644
index 0000000000..ec8a877ae2
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.down.sql
@@ -0,0 +1 @@
+DROP TABLE blob_storage_file_log {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.up.sql b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.up.sql
similarity index 69%
rename from packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.up.sql
index 70150bef29..1cced071d6 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.up.sql
@@ -1,5 +1,5 @@
-CREATE TABLE blob_storage_file_log
-(
+CREATE TABLE blob_storage_file_log {CLICKHOUSE_CLUSTER_CLAUSE}{CLICKHOUSE_CLUSTERED_ONLY: (}{CLICKHOUSE_UNCLUSTERED_ONLY:
+(}
`id` String,
`project_id` String,
`entity_type` String,
@@ -13,7 +13,7 @@ CREATE TABLE blob_storage_file_log
`updated_at` DateTime64(3) DEFAULT now(),
event_ts DateTime64(3),
is_deleted UInt8,
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)
ORDER BY (
project_id,
entity_type,
diff --git a/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.down.sql
new file mode 100644
index 0000000000..10f9e2d393
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS session_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.up.sql
new file mode 100644
index 0000000000..8888a63c61
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN session_id Nullable(String) AFTER trace_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.down.sql
new file mode 100644
index 0000000000..9d4b7b34aa
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.down.sql
@@ -0,0 +1,2 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.up.sql
new file mode 100644
index 0000000000..a9d2f3eca2
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:2}
diff --git a/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.down.sql b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.down.sql
new file mode 100644
index 0000000000..9bce5e87d1
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.up.sql b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.up.sql
new file mode 100644
index 0000000000..9bce5e87d1
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.down.sql
new file mode 100644
index 0000000000..10ee265522
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.up.sql
new file mode 100644
index 0000000000..a8b52facdd
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2;}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2; }
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.down.sql
new file mode 100644
index 0000000000..0a9ee001fd
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.up.sql
new file mode 100644
index 0000000000..6418e2985b
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.down.sql
new file mode 100644
index 0000000000..8e9b83379e
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS dataset_run_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.up.sql
new file mode 100644
index 0000000000..00fc84ed7f
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN dataset_run_id Nullable(String) AFTER session_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.down.sql
new file mode 100644
index 0000000000..b159c0ed5c
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.up.sql
new file mode 100644
index 0000000000..78b2f05fc3
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.down.sql b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.down.sql
new file mode 100644
index 0000000000..ea2f633d7a
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.down.sql
@@ -0,0 +1 @@
+DROP VIEW IF EXISTS analytics_traces {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.up.sql b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.up.sql
similarity index 87%
rename from packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.up.sql
index d2bdbfb947..52eee7405c 100644
--- a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.up.sql
@@ -1,4 +1,4 @@
-CREATE VIEW analytics_traces ON CLUSTER default AS
+CREATE VIEW analytics_traces {CLICKHOUSE_CLUSTER_CLAUSE} AS
SELECT
project_id,
toStartOfHour(timestamp) AS hour,
diff --git a/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.down.sql b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.down.sql
new file mode 100644
index 0000000000..c0380e0f55
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.down.sql
@@ -0,0 +1 @@
+DROP VIEW IF EXISTS analytics_observations {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.up.sql b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.up.sql
similarity index 89%
rename from packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.up.sql
index 0e5f451959..85b45c3212 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.up.sql
@@ -1,4 +1,4 @@
-CREATE VIEW analytics_observations AS
+CREATE VIEW analytics_observations {CLICKHOUSE_CLUSTER_CLAUSE} AS
SELECT
project_id,
type,
diff --git a/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.down.sql
new file mode 100644
index 0000000000..dfa3cc6c7d
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.down.sql
@@ -0,0 +1 @@
+DROP VIEW IF EXISTS analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.up.sql
similarity index 92%
rename from packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.up.sql
index eb683cd89a..6f920fffd0 100644
--- a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.up.sql
@@ -1,4 +1,4 @@
-CREATE VIEW analytics_scores ON CLUSTER default AS
+CREATE VIEW analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE} AS
SELECT
project_id,
toStartOfHour(timestamp) AS hour,
diff --git a/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.down.sql
new file mode 100644
index 0000000000..0d2d11afbc
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.down.sql
@@ -0,0 +1 @@
+DROP TABLE dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.up.sql
similarity index 88%
rename from packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.up.sql
index c8466b6c9d..f0a0143421 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE dataset_run_items (
+CREATE TABLE dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE} (
-- primary identifiers
`id` String,
`project_id` String,
@@ -33,4 +33,4 @@ CREATE TABLE dataset_run_items (
-- For dataset item lookups
INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,
) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
-ORDER BY (project_id, dataset_id, dataset_run_id, id);
\ No newline at end of file
+ORDER BY (project_id, dataset_id, dataset_run_id, id);{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.down.sql b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.down.sql
new file mode 100644
index 0000000000..623fdb4294
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.down.sql
@@ -0,0 +1,12 @@
+-- Drop materialized views first
+DROP VIEW IF EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+
+-- Drop AMT tables
+DROP TABLE IF EXISTS traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP TABLE IF EXISTS traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP TABLE IF EXISTS traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE};
+
+-- Drop the Null table
+DROP TABLE IF EXISTS traces_null {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.up.sql b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.up.sql
similarity index 93%
rename from packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.up.sql
index fd243f4f6a..246e114962 100644
--- a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.up.sql
@@ -1,6 +1,6 @@
-- Create a Null table that serves as a trigger for all materialized views.
-- We use a Null engine here to avoid storing intermediate results and save on storage.
-CREATE TABLE traces_null ON CLUSTER default
+CREATE TABLE traces_null {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -22,7 +22,7 @@ CREATE TABLE traces_null ON CLUSTER default
`bookmarked` Nullable(Bool),
`public` Nullable(Bool),
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
`observation_ids` Array(String),
`score_ids` Array(String),
`cost_details` Map(String, Decimal64(12)),
@@ -39,8 +39,8 @@ CREATE TABLE traces_null ON CLUSTER default
) Engine = Null();
-- Create the all AMT
-CREATE TABLE traces_all_amt ON CLUSTER default
-(
+CREATE TABLE traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE}
+({CLICKHOUSE_CLUSTERED_ONLY: }{CLICKHOUSE_UNCLUSTERED_ONLY: }
-- Identifiers
`project_id` String,
`id` String,
@@ -62,7 +62,7 @@ CREATE TABLE traces_all_amt ON CLUSTER default
`bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
`public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
`observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
`score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
`cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
@@ -87,7 +87,7 @@ CREATE TABLE traces_all_amt ON CLUSTER default
ORDER BY (project_id, id);
-- Create materialized view for all_amt
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv ON CLUSTER default TO traces_all_amt AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_all_amt AS
SELECT
-- Identifiers
tn.project_id as project_id,
@@ -110,7 +110,7 @@ SELECT
argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
groupUniqArrayArray(tn.observation_ids) as observation_ids,
groupUniqArrayArray(tn.score_ids) as score_ids,
sumMap(tn.cost_details) as cost_details,
@@ -126,7 +126,7 @@ FROM traces_null tn
GROUP BY project_id, id;
-- Create the 7-day TTL AMT
-CREATE TABLE traces_7d_amt ON CLUSTER default
+CREATE TABLE traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -149,7 +149,7 @@ CREATE TABLE traces_7d_amt ON CLUSTER default
`bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
`public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
`observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
`score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
`cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
@@ -174,7 +174,7 @@ CREATE TABLE traces_7d_amt ON CLUSTER default
TTL toDate(start_time) + INTERVAL 7 DAY;
-- Create materialized view for 7d_amt
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv ON CLUSTER default TO traces_7d_amt AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_7d_amt AS
SELECT
-- Identifiers
tn.project_id as project_id,
@@ -197,7 +197,7 @@ SELECT
argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
groupUniqArrayArray(tn.observation_ids) as observation_ids,
groupUniqArrayArray(tn.score_ids) as score_ids,
sumMap(tn.cost_details) as cost_details,
@@ -213,7 +213,7 @@ FROM traces_null tn
GROUP BY project_id, id;
-- Create the 30-day TTL AMT
-CREATE TABLE traces_30d_amt ON CLUSTER default
+CREATE TABLE traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -236,7 +236,7 @@ CREATE TABLE traces_30d_amt ON CLUSTER default
`bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
`public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
`observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
`score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
`cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
@@ -261,7 +261,7 @@ CREATE TABLE traces_30d_amt ON CLUSTER default
TTL toDate(start_time) + INTERVAL 30 DAY;
-- Create materialized view for 30d_amt
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv ON CLUSTER default TO traces_30d_amt AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_30d_amt AS
SELECT
-- Identifiers
tn.project_id as project_id,
@@ -284,7 +284,7 @@ SELECT
argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
- -- Aggregations -- DO NOT USE
+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}
groupUniqArrayArray(tn.observation_ids) as observation_ids,
groupUniqArrayArray(tn.score_ids) as score_ids,
sumMap(tn.cost_details) as cost_details,
@@ -297,4 +297,4 @@ SELECT
min(tn.created_at) as created_at,
max(tn.updated_at) as updated_at
FROM traces_null tn
-GROUP BY project_id, id;
+GROUP BY project_id, id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}
diff --git a/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.down.sql
new file mode 100644
index 0000000000..b263773950
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.down.sql
@@ -0,0 +1 @@
+DROP TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.up.sql
similarity index 81%
rename from packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.up.sql
index 28db500526..0c3243b791 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE dataset_run_items_rmt (
+CREATE TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} (
-- primary identifiers
`id` String,
`project_id` String,
@@ -32,5 +32,5 @@ CREATE TABLE dataset_run_items_rmt (
-- For dataset item lookups
INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
-ORDER BY (project_id, dataset_id, dataset_run_id, id);
\ No newline at end of file
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)
+ORDER BY (project_id, dataset_id, dataset_run_id, id);{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.down.sql b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.down.sql
new file mode 100644
index 0000000000..a84100e253
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.down.sql
@@ -0,0 +1,2 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_res_metadata_value{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_res_metadata_key{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.up.sql b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.up.sql
new file mode 100644
index 0000000000..40bbcc354c
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.up.sql
@@ -0,0 +1,4 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_res_metadata_key{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_res_metadata_value{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.down.sql
new file mode 100644
index 0000000000..a50050457f
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.down.sql
@@ -0,0 +1 @@
+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_trace_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.up.sql
new file mode 100644
index 0000000000..036ec05e26
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_trace_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.down.sql b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.down.sql
new file mode 100644
index 0000000000..f58c9a3532
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.down.sql
@@ -0,0 +1,21 @@
+-- Recreate materialized views for project_environments table
+CREATE MATERIALIZED VIEW project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS
+SELECT
+ project_id,
+ groupUniqArray(environment) AS environments
+FROM traces
+GROUP BY project_id;
+
+CREATE MATERIALIZED VIEW project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS
+SELECT
+ project_id,
+ groupUniqArray(environment) AS environments
+FROM observations
+GROUP BY project_id;
+
+CREATE MATERIALIZED VIEW project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS
+SELECT
+ project_id,
+ groupUniqArray(environment) AS environments
+FROM scores
+GROUP BY project_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.up.sql b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.up.sql
new file mode 100644
index 0000000000..152186dfbb
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.up.sql
@@ -0,0 +1,4 @@
+-- Drop materialized views feeding project_environments table
+DROP VIEW IF EXISTS project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.down.sql b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.down.sql
similarity index 95%
rename from packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.down.sql
index 60d3890318..9c0cf2a938 100644
--- a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.down.sql
@@ -1,5 +1,5 @@
-- Recreate materialized views derived from traces_null table
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv ON CLUSTER default TO traces_all_amt AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_all_amt AS
SELECT
-- Identifiers
tn.project_id as project_id,
@@ -37,7 +37,7 @@ SELECT
FROM traces_null tn
GROUP BY project_id, id;
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv ON CLUSTER default TO traces_7d_amt AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_7d_amt AS
SELECT
-- Identifiers
tn.project_id as project_id,
@@ -75,7 +75,7 @@ SELECT
FROM traces_null tn
GROUP BY project_id, id;
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv ON CLUSTER default TO traces_30d_amt AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_30d_amt AS
SELECT
-- Identifiers
tn.project_id as project_id,
@@ -111,4 +111,4 @@ SELECT
min(tn.created_at) as created_at,
max(tn.updated_at) as updated_at
FROM traces_null tn
-GROUP BY project_id, id;
\ No newline at end of file
+GROUP BY project_id, id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.up.sql b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.up.sql
new file mode 100644
index 0000000000..8a441e9f19
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.up.sql
@@ -0,0 +1,4 @@
+-- Drop materialized views derived from traces_null table
+DROP VIEW IF EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP VIEW IF EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.down.sql b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.down.sql
similarity index 96%
rename from packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.down.sql
index 3c51d5fd0e..5f3b85cac8 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.down.sql
@@ -1,5 +1,5 @@
-- Recreate traces_null and trace amt tables
-CREATE TABLE traces_null
+CREATE TABLE traces_null {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -37,7 +37,7 @@ CREATE TABLE traces_null
`event_ts` DateTime64(3)
) Engine = Null();
-CREATE TABLE traces_all_amt
+CREATE TABLE traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -84,7 +84,7 @@ CREATE TABLE traces_all_amt
) Engine = AggregatingMergeTree()
ORDER BY (project_id, id);
-CREATE TABLE traces_7d_amt
+CREATE TABLE traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -131,7 +131,7 @@ CREATE TABLE traces_7d_amt
ORDER BY (project_id, id)
TTL toDate(start_time) + INTERVAL 7 DAY;
-CREATE TABLE traces_30d_amt
+CREATE TABLE traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE}
(
-- Identifiers
`project_id` String,
@@ -176,4 +176,4 @@ CREATE TABLE traces_30d_amt
INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
) Engine = AggregatingMergeTree()
ORDER BY (project_id, id)
- TTL toDate(start_time) + INTERVAL 30 DAY;
\ No newline at end of file
+ TTL toDate(start_time) + INTERVAL 30 DAY;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.up.sql b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.up.sql
new file mode 100644
index 0000000000..838c5aad03
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.up.sql
@@ -0,0 +1,5 @@
+-- Drop traces_null and trace amt tables
+DROP TABLE IF EXISTS traces_null {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP TABLE IF EXISTS traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP TABLE IF EXISTS traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE};
+DROP TABLE IF EXISTS traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}
diff --git a/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.down.sql
new file mode 100644
index 0000000000..6e6c89fdde
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS execution_trace_id;
diff --git a/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.up.sql
new file mode 100644
index 0000000000..2377fa061e
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.down.sql
new file mode 100644
index 0000000000..1da1ab4f07
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.down.sql
@@ -0,0 +1,2 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS usage_pricing_tier_name{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS usage_pricing_tier_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.up.sql
new file mode 100644
index 0000000000..0a68ccc6c4
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN usage_pricing_tier_id Nullable(String){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN usage_pricing_tier_name Nullable(String){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql
new file mode 100644
index 0000000000..98a8c5d45d
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql
@@ -0,0 +1 @@
+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS dataset_item_version;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}
diff --git a/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
new file mode 100644
index 0000000000..fb4b82c6ae
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
@@ -0,0 +1 @@
+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3)){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}
diff --git a/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.down.sql
new file mode 100644
index 0000000000..8cc86f9026
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.down.sql
@@ -0,0 +1,3 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS tool_definitions SETTINGS alter_sync = 2;
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS tool_calls SETTINGS alter_sync = 2;
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS tool_call_names SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.up.sql
new file mode 100644
index 0000000000..5a7685e7fe
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.up.sql
@@ -0,0 +1,3 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS tool_definitions Map(String, String) DEFAULT map() SETTINGS alter_sync = 2;
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS tool_calls Array(String) DEFAULT [] SETTINGS alter_sync = 2;
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS tool_call_names Array(String) DEFAULT [] SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.down.sql
new file mode 100644
index 0000000000..aafb3775cd
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.down.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS long_string_value;
diff --git a/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.up.sql
new file mode 100644
index 0000000000..0c106ce47c
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.up.sql
@@ -0,0 +1 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3)){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.down.sql
new file mode 100644
index 0000000000..19d89b0b0d
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.down.sql
@@ -0,0 +1,11 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
+ DROP COLUMN IF EXISTS ingestion_sdk_version
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
+ DROP COLUMN IF EXISTS ingestion_sdk_name
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
+ DROP COLUMN IF EXISTS ingestion_api_key
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.up.sql
new file mode 100644
index 0000000000..e0627c55ad
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.up.sql
@@ -0,0 +1,11 @@
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
+ ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
+ ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
+ ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.down.sql
similarity index 84%
rename from packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.down.sql
index 1c9aa7f16f..88f241a4e7 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.down.sql
@@ -1,5 +1,5 @@
-DROP VIEW IF EXISTS analytics_scores;
-CREATE VIEW analytics_scores AS
+DROP VIEW IF EXISTS analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE};
+CREATE VIEW analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE} AS
SELECT
project_id,
toStartOfHour(timestamp) AS hour,
diff --git a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.up.sql
similarity index 87%
rename from packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.up.sql
index 2caf436211..c3942b832d 100644
--- a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.up.sql
@@ -1,5 +1,5 @@
-DROP VIEW IF EXISTS analytics_scores ON CLUSTER default;
-CREATE VIEW analytics_scores ON CLUSTER default AS
+DROP VIEW IF EXISTS analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE};
+CREATE VIEW analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE} AS
SELECT
project_id,
toStartOfHour(timestamp) AS hour,
diff --git a/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.down.sql b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.down.sql
new file mode 100644
index 0000000000..342db48a79
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.down.sql
@@ -0,0 +1,3 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_created_at;
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_created_at;
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_created_at;
diff --git a/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.up.sql b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.up.sql
new file mode 100644
index 0000000000..79fb25be2b
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.up.sql
@@ -0,0 +1,6 @@
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_created_at;
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_created_at;
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_created_at;
diff --git a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.down.sql b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.down.sql
similarity index 79%
rename from packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.down.sql
index e3177579f2..8f3ee609ad 100644
--- a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.down.sql
@@ -2,4 +2,4 @@
-- table may predate this migration (created via dev-tables.sh on v4 preview
-- deployments) and may hold production-grade event data. Rolling back drops
-- that data regardless of which path created the table.
-DROP TABLE IF EXISTS observations_batch_staging ON CLUSTER default;
+DROP TABLE IF EXISTS observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.up.sql b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.up.sql
similarity index 91%
rename from packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.up.sql
index 757131dbdd..ff47bb5dfe 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE IF NOT EXISTS observations_batch_staging
+CREATE TABLE IF NOT EXISTS observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}
(
id String,
trace_id String,
@@ -38,7 +38,7 @@ CREATE TABLE IF NOT EXISTS observations_batch_staging
s3_first_seen_timestamp DateTime64(3),
environment LowCardinality(String) DEFAULT 'default'
)
-ENGINE = ReplacingMergeTree(event_ts, is_deleted)
+ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toStartOfInterval(s3_first_seen_timestamp, INTERVAL 3 MINUTE)
PRIMARY KEY (project_id, toDate(s3_first_seen_timestamp))
ORDER BY (
diff --git a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.down.sql b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.down.sql
similarity index 82%
rename from packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0039_create_events_full.down.sql
index 9280e87133..97d3def98e 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.down.sql
@@ -2,4 +2,4 @@
-- table may predate this migration (created via dev-tables.sh on v4 preview
-- deployments) and may hold production-grade event data. Rolling back drops
-- that data regardless of which path created the table.
-DROP TABLE IF EXISTS events_full;
+DROP TABLE IF EXISTS events_full {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.up.sql b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.up.sql
similarity index 96%
rename from packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0039_create_events_full.up.sql
index c14b38ef47..bbf2a13520 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE IF NOT EXISTS events_full
+CREATE TABLE IF NOT EXISTS events_full {CLICKHOUSE_CLUSTER_CLAUSE}
(
project_id String,
trace_id String,
@@ -109,7 +109,7 @@ CREATE TABLE IF NOT EXISTS events_full
INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),
INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)
)
-ENGINE = ReplacingMergeTree(event_ts, is_deleted)
+ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toYYYYMM(start_time)
PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))
ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)
diff --git a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.down.sql b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.down.sql
similarity index 82%
rename from packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0040_create_events_core.down.sql
index c263c98542..291ef76c54 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.down.sql
@@ -2,4 +2,4 @@
-- table may predate this migration (created via dev-tables.sh on v4 preview
-- deployments) and may hold production-grade event data. Rolling back drops
-- that data regardless of which path created the table.
-DROP TABLE IF EXISTS events_core;
+DROP TABLE IF EXISTS events_core {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.up.sql b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.up.sql
similarity index 96%
rename from packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0040_create_events_core.up.sql
index fb5df27191..e5a9bde82f 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.up.sql
@@ -1,4 +1,4 @@
-CREATE TABLE IF NOT EXISTS events_core
+CREATE TABLE IF NOT EXISTS events_core {CLICKHOUSE_CLUSTER_CLAUSE}
(
project_id String,
trace_id String,
@@ -110,7 +110,7 @@ CREATE TABLE IF NOT EXISTS events_core
INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),
INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)
)
-ENGINE = ReplacingMergeTree(event_ts, is_deleted)
+ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toYYYYMM(start_time)
PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))
ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)
diff --git a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.down.sql b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.down.sql
similarity index 77%
rename from packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.down.sql
index 1e4578cc48..896dde4dc0 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.down.sql
@@ -1,4 +1,4 @@
-- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this
-- view may predate this migration (created via dev-tables.sh on v4 preview
-- deployments). Dropping it stops events_core from being populated.
-DROP VIEW IF EXISTS events_core_mv;
+DROP VIEW IF EXISTS events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.up.sql b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.up.sql
similarity index 93%
rename from packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.up.sql
index 40f95a62f2..033680f34c 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.up.sql
@@ -1,4 +1,4 @@
-CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv TO events_core AS
+CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO events_core AS
SELECT
project_id,
trace_id,
diff --git a/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.down.sql
new file mode 100644
index 0000000000..f0e7e72068
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.down.sql
@@ -0,0 +1,82 @@
+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY
+SELECT
+ project_id,
+ trace_id,
+ span_id,
+ parent_span_id,
+ start_time,
+ end_time,
+ name,
+ type,
+ environment,
+ version,
+ release,
+ trace_name,
+ user_id,
+ session_id,
+ tags,
+ level,
+ status_message,
+ completion_start_time,
+ is_app_root,
+ bookmarked,
+ public,
+ prompt_id,
+ prompt_name,
+ prompt_version,
+ model_id,
+ provided_model_name,
+ model_parameters,
+ provided_usage_details,
+ usage_details,
+ provided_cost_details,
+ cost_details,
+ usage_pricing_tier_id,
+ usage_pricing_tier_name,
+ tool_definitions,
+ tool_calls,
+ tool_call_names,
+ leftUTF8(input, 200) as input,
+ leftUTF8(output, 200) as output,
+ metadata_names,
+ arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
+ experiment_id,
+ experiment_name,
+ experiment_metadata_names,
+ experiment_metadata_values,
+ experiment_description,
+ experiment_dataset_id,
+ experiment_item_id,
+ experiment_item_version,
+ experiment_item_expected_output,
+ experiment_item_metadata_names,
+ experiment_item_metadata_values,
+ experiment_item_root_span_id,
+ source,
+ service_name,
+ service_version,
+ scope_name,
+ scope_version,
+ telemetry_sdk_language,
+ telemetry_sdk_name,
+ telemetry_sdk_version,
+ blob_storage_file_path,
+ event_bytes,
+ created_at,
+ updated_at,
+ event_ts,
+ is_deleted
+FROM events_full
+SETTINGS enable_full_text_index = 1;
+
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+
+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.up.sql
similarity index 63%
rename from packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.up.sql
index 7a43d0fb2a..3c8dd270d2 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.up.sql
@@ -1,34 +1,34 @@
-ALTER TABLE observations_batch_staging
+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE observations_batch_staging
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE observations_batch_staging
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1;
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
-ALTER TABLE events_full
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1;
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
-ALTER TABLE events_core
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1;
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
-ALTER TABLE events_core_mv MODIFY QUERY
+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY
SELECT
project_id,
trace_id,
diff --git a/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.down.sql
new file mode 100644
index 0000000000..752fe3da81
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.down.sql
@@ -0,0 +1,2 @@
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.up.sql
new file mode 100644
index 0000000000..62cd5ca4f4
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.up.sql
@@ -0,0 +1,2 @@
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.down.sql b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.down.sql
similarity index 89%
rename from packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.down.sql
index f894ad58fe..68dee80405 100644
--- a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.down.sql
@@ -1,6 +1,6 @@
-- WARNING: This recreates the table structure only. Any data held in
-- event_log before the up migration dropped it is NOT restored.
-CREATE TABLE IF NOT EXISTS event_log ON CLUSTER default
+CREATE TABLE IF NOT EXISTS event_log {CLICKHOUSE_CLUSTER_CLAUSE}
(
`id` String,
`project_id` String,
diff --git a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.up.sql b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.up.sql
similarity index 81%
rename from packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.up.sql
index 3682d6d29c..fdd0749b32 100644
--- a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.up.sql
@@ -2,4 +2,4 @@
-- were copied over by the 20250417_1737_migrate_event_log_to_blob_storage
-- background migration, which must have completed (on the latest v3 release)
-- before upgrading to v4.
-DROP TABLE IF EXISTS event_log ON CLUSTER default;
+DROP TABLE IF EXISTS event_log {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.down.sql b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.down.sql
similarity index 74%
rename from packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.down.sql
index a1de329190..d33dde9255 100644
--- a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.down.sql
@@ -2,8 +2,8 @@
-- project_environments before the up migration dropped it is NOT restored,
-- and the materialized views that used to feed the table were already dropped
-- by 0027_drop_project_environments_mvs, so it stays empty.
-CREATE TABLE IF NOT EXISTS project_environments ON CLUSTER default (
+CREATE TABLE IF NOT EXISTS project_environments {CLICKHOUSE_CLUSTER_CLAUSE} (
`project_id` String,
`environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))
-) ENGINE = ReplicatedAggregatingMergeTree
+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}AggregatingMergeTree
ORDER BY (project_id);
diff --git a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.up.sql b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.up.sql
similarity index 77%
rename from packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.up.sql
index 25cb10b8a0..aa9176e6b2 100644
--- a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.up.sql
@@ -2,4 +2,4 @@
-- 0027_drop_project_environments_mvs removed the materialized views feeding
-- it. getEnvironmentsForProject reads distinct environments from the tracing
-- and scores tables directly.
-DROP TABLE IF EXISTS project_environments ON CLUSTER default;
+DROP TABLE IF EXISTS project_environments {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.down.sql
similarity index 94%
rename from packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.down.sql
rename to packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.down.sql
index 88a8e3d72d..b3d42ae432 100644
--- a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.down.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.down.sql
@@ -1,6 +1,6 @@
-- WARNING: This recreates the table structure only. Any data held in
-- dataset_run_items before the up migration dropped it is NOT restored.
-CREATE TABLE IF NOT EXISTS dataset_run_items (
+CREATE TABLE IF NOT EXISTS dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE} (
-- primary identifiers
`id` String,
`project_id` String,
diff --git a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.up.sql
similarity index 75%
rename from packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.up.sql
index 353b69e3b4..6814a57161 100644
--- a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.up.sql
@@ -1,4 +1,4 @@
-- The dataset_run_items table was superseded by dataset_run_items_rmt
-- (created in 0024). All ingestion writes, reads and deletes target the
-- _rmt table, so this table has been frozen since the cutover.
-DROP TABLE IF EXISTS dataset_run_items ON CLUSTER default;
+DROP TABLE IF EXISTS dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE};
diff --git a/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.down.sql
new file mode 100644
index 0000000000..cd8d0615bd
--- /dev/null
+++ b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.down.sql
@@ -0,0 +1,88 @@
+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY
+SELECT
+ project_id,
+ trace_id,
+ span_id,
+ parent_span_id,
+ start_time,
+ end_time,
+ name,
+ type,
+ environment,
+ version,
+ release,
+ trace_name,
+ user_id,
+ session_id,
+ tags,
+ level,
+ status_message,
+ completion_start_time,
+ is_app_root,
+ bookmarked,
+ public,
+ prompt_id,
+ prompt_name,
+ prompt_version,
+ model_id,
+ provided_model_name,
+ model_parameters,
+ provided_usage_details,
+ usage_details,
+ provided_cost_details,
+ cost_details,
+ usage_pricing_tier_id,
+ usage_pricing_tier_name,
+ tool_definitions,
+ tool_calls,
+ tool_call_names,
+ leftUTF8(input, 200) as input,
+ leftUTF8(output, 200) as output,
+ metadata_names,
+ arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
+ experiment_id,
+ experiment_name,
+ experiment_metadata_names,
+ experiment_metadata_values,
+ experiment_description,
+ experiment_dataset_id,
+ experiment_item_id,
+ experiment_item_version,
+ experiment_item_expected_output,
+ experiment_item_metadata_names,
+ experiment_item_metadata_values,
+ experiment_item_root_span_id,
+ source,
+ service_name,
+ service_version,
+ scope_name,
+ scope_version,
+ telemetry_sdk_language,
+ telemetry_sdk_name,
+ telemetry_sdk_version,
+ blob_storage_file_path,
+ event_bytes,
+ created_at,
+ updated_at,
+ event_ts,
+ is_deleted,
+ ingestion_api_key,
+ ingestion_sdk_name,
+ ingestion_sdk_version
+FROM events_full
+SETTINGS enable_full_text_index = 1;
+
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluator_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluator_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluation_rule_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluation_rule_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
diff --git a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.up.sql
similarity index 71%
rename from packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.up.sql
rename to packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.up.sql
index 87504a51b4..a3223a4782 100644
--- a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.up.sql
+++ b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.up.sql
@@ -1,41 +1,41 @@
-- Existing parts resolve these columns through DEFAULT expressions. Skip indexes are added without materializing historical parts to keep this migration metadata-only.
-ALTER TABLE events_full ON CLUSTER default
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.01) GRANULARITY 1
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}
ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.01) GRANULARITY 1
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default
+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT metadata['evaluator_id'] AFTER metadata
- SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(metadata['evaluation_rule_id']), metadata['evaluation_rule_id'], metadata['job_configuration_id']) AFTER evaluator_id
- SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.001) GRANULARITY 1
- SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}
ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.001) GRANULARITY 1
- SETTINGS alter_sync = 2;
-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY
+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};
+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY
SELECT
project_id,
trace_id,
diff --git a/packages/shared/clickhouse/migrations/clustered/0001_traces.down.sql b/packages/shared/clickhouse/migrations/clustered/0001_traces.down.sql
deleted file mode 100644
index eb7f439d49..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0001_traces.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE traces ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0002_observations.down.sql b/packages/shared/clickhouse/migrations/clustered/0002_observations.down.sql
deleted file mode 100644
index e3aaa0066f..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0002_observations.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE observations ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0002_observations.up.sql b/packages/shared/clickhouse/migrations/clustered/0002_observations.up.sql
deleted file mode 100644
index 14d2359815..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0002_observations.up.sql
+++ /dev/null
@@ -1,47 +0,0 @@
-CREATE TABLE observations ON CLUSTER default (
- `id` String,
- `trace_id` String,
- `project_id` String,
- `type` LowCardinality(String),
- `parent_observation_id` Nullable(String),
- `start_time` DateTime64(3),
- `end_time` Nullable(DateTime64(3)),
- `name` String,
- `metadata` Map(LowCardinality(String), String),
- `level` LowCardinality(String),
- `status_message` Nullable(String),
- `version` Nullable(String),
- `input` Nullable(String) CODEC(ZSTD(3)),
- `output` Nullable(String) CODEC(ZSTD(3)),
- `provided_model_name` Nullable(String),
- `internal_model_id` Nullable(String),
- `model_parameters` Nullable(String),
- `provided_usage_details` Map(LowCardinality(String), UInt64),
- `usage_details` Map(LowCardinality(String), UInt64),
- `provided_cost_details` Map(LowCardinality(String), Decimal64(12)),
- `cost_details` Map(LowCardinality(String), Decimal64(12)),
- `total_cost` Nullable(Decimal64(12)),
- `completion_start_time` Nullable(DateTime64(3)),
- `prompt_id` Nullable(String),
- `prompt_name` Nullable(String),
- `prompt_version` Nullable(UInt16),
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now(),
- event_ts DateTime64(3),
- is_deleted UInt8,
- INDEX idx_id id TYPE bloom_filter() GRANULARITY 1,
- INDEX idx_trace_id trace_id TYPE bloom_filter() GRANULARITY 1,
- INDEX idx_project_id project_id TYPE bloom_filter() GRANULARITY 1
-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(start_time)
-PRIMARY KEY (
- project_id,
- `type`,
- toDate(start_time)
-)
-ORDER BY (
- project_id,
- `type`,
- toDate(start_time),
- id
-);
-
diff --git a/packages/shared/clickhouse/migrations/clustered/0003_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0003_scores.down.sql
deleted file mode 100644
index 973a078d62..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0003_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE scores ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.down.sql
deleted file mode 100644
index ea20f3d1b4..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_id project_id TYPE bloom_filter() GRANULARITY 1;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.up.sql
deleted file mode 100644
index 990c0e21ea..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_project_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.down.sql
deleted file mode 100644
index c9460635bd..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE traces ON CLUSTER default DROP INDEX IF EXISTS idx_session_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.up.sql
deleted file mode 100644
index ca2a595b1e..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_session_id SETTINGS mutations_sync = 2;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.down.sql
deleted file mode 100644
index 4c14ef6ff7..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE traces ON CLUSTER default DROP INDEX IF EXISTS idx_user_id;
diff --git a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.up.sql
deleted file mode 100644
index 7d9400f19e..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_user_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.down.sql b/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.down.sql
deleted file mode 100644
index 24542e21ba..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE event_log ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.down.sql b/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.down.sql
deleted file mode 100644
index 64a3c513c5..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.down.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE traces ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.up.sql b/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.up.sql
deleted file mode 100644
index 2aa2654a48..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.up.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE traces ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.down.sql b/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.down.sql
deleted file mode 100644
index 0486b8e1db..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.down.sql
+++ /dev/null
@@ -1,7 +0,0 @@
--- Drop materialized views
-DROP VIEW IF EXISTS project_environments_traces_mv ON CLUSTER default;
-DROP VIEW IF EXISTS project_environments_observations_mv ON CLUSTER default;
-DROP VIEW IF EXISTS project_environments_scores_mv ON CLUSTER default;
-
--- Drop the project_environments table
-DROP TABLE IF EXISTS project_environments ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.up.sql b/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.up.sql
deleted file mode 100644
index 6fadd913ec..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.up.sql
+++ /dev/null
@@ -1,30 +0,0 @@
--- Create the project_environments table with AggregatingMergeTree engine
-CREATE TABLE project_environments ON CLUSTER default (
- `project_id` String,
- `environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))
-) ENGINE = ReplicatedAggregatingMergeTree
-ORDER BY (project_id);
-
--- Create materialized view for traces
-CREATE MATERIALIZED VIEW project_environments_traces_mv ON CLUSTER default TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM traces
-GROUP BY project_id;
-
--- Create materialized view for observations
-CREATE MATERIALIZED VIEW project_environments_observations_mv ON CLUSTER default TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM observations
-GROUP BY project_id;
-
--- Create materialized view for scores
-CREATE MATERIALIZED VIEW project_environments_scores_mv ON CLUSTER default TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM scores
-GROUP BY project_id;
diff --git a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.down.sql
deleted file mode 100644
index c1688c2908..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS metadata;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.up.sql
deleted file mode 100644
index ecc77043c7..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN metadata Map(LowCardinality(String), String) AFTER comment;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.down.sql b/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.down.sql
deleted file mode 100644
index 29a74f1c78..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE blob_storage_file_log ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.up.sql b/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.up.sql
deleted file mode 100644
index 664281ad4c..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.up.sql
+++ /dev/null
@@ -1,21 +0,0 @@
-CREATE TABLE blob_storage_file_log ON CLUSTER default (
- `id` String,
- `project_id` String,
- `entity_type` String,
- `entity_id` String,
- `event_id` String,
-
- `bucket_name` String,
- `bucket_path` String,
-
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now(),
- event_ts DateTime64(3),
- is_deleted UInt8,
-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)
- ORDER BY (
- project_id,
- entity_type,
- entity_id,
- event_id
- );
diff --git a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.down.sql
deleted file mode 100644
index 2b8ef556f4..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS session_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.up.sql
deleted file mode 100644
index 7837f8bdfb..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN session_id Nullable(String) AFTER trace_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.down.sql
deleted file mode 100644
index 1b3d0f32bb..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.up.sql
deleted file mode 100644
index 8a8bb8becd..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
-
diff --git a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.down.sql b/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.down.sql
deleted file mode 100644
index 954e58dead..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.up.sql b/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.up.sql
deleted file mode 100644
index 954e58dead..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.down.sql
deleted file mode 100644
index bd86b182c3..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.up.sql
deleted file mode 100644
index 1b3d0f32bb..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.down.sql
deleted file mode 100644
index abba171aad..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.up.sql
deleted file mode 100644
index 2947da5227..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.down.sql
deleted file mode 100644
index 5993426be8..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS dataset_run_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.up.sql
deleted file mode 100644
index 37a3af3aa7..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN dataset_run_id Nullable(String) AFTER session_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.down.sql
deleted file mode 100644
index 714f111830..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.up.sql
deleted file mode 100644
index 9675dd2916..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.down.sql b/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.down.sql
deleted file mode 100644
index 749fd5e106..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP VIEW IF EXISTS analytics_traces ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.down.sql b/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.down.sql
deleted file mode 100644
index 176598bb76..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP VIEW IF EXISTS analytics_observations ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.up.sql b/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.up.sql
deleted file mode 100644
index 477a236683..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.up.sql
+++ /dev/null
@@ -1,18 +0,0 @@
-CREATE VIEW analytics_observations ON CLUSTER default AS
-SELECT
- project_id,
- type,
- toStartOfHour(start_time) AS hour,
- uniq(id) AS countObservations,
- max(level != 'DEFAULT') AS hasLevel,
- max(provided_model_name IS NOT NULL) AS hasProvidedModelName,
- max(length(provided_usage_details) > 0) AS hasProvidedUsageDetails,
- max(length(provided_cost_details) > 0) AS hasProvidedCostDetails,
- max(prompt_name IS NOT NULL) AS hasPromptName
-FROM
- observations
-WHERE toStartOfHour(start_time) <= toStartOfHour(subtractHours(now(), 1))
-GROUP BY
- project_id,
- type,
- hour;
diff --git a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.down.sql
deleted file mode 100644
index 277f1eeed3..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP VIEW IF EXISTS analytics_scores ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.down.sql
deleted file mode 100644
index a432f4214c..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE dataset_run_items ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.up.sql
deleted file mode 100644
index c7363a13ed..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.up.sql
+++ /dev/null
@@ -1,36 +0,0 @@
-CREATE TABLE dataset_run_items ON CLUSTER default (
- -- primary identifiers
- `id` String,
- `project_id` String,
- `dataset_run_id` String,
- `dataset_item_id` String,
- `dataset_id` String,
- `trace_id` String,
- `observation_id` Nullable(String),
-
- -- error field
- `error` Nullable(String),
-
- -- timestamps
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now(),
-
- -- denormalized immutable dataset run fields
- `dataset_run_name` String,
- `dataset_run_description` Nullable(String),
- `dataset_run_metadata` Map(LowCardinality(String), String),
- `dataset_run_created_at` DateTime64(3),
-
- -- denormalized dataset item fields (mutable, but snapshots are relevant)
- `dataset_item_input` Nullable(String) CODEC(ZSTD(3)), -- json
- `dataset_item_expected_output` Nullable(String) CODEC(ZSTD(3)), -- json
- `dataset_item_metadata` Map(LowCardinality(String), String),
-
- -- clickhouse engine fields
- `event_ts` DateTime64(3),
- `is_deleted` UInt8,
-
- -- For dataset item lookups
- INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
-ORDER BY (project_id, dataset_id, dataset_run_id, id);
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.down.sql b/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.down.sql
deleted file mode 100644
index 61ab89c3f8..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.down.sql
+++ /dev/null
@@ -1,12 +0,0 @@
--- Drop materialized views first
-DROP VIEW IF EXISTS traces_30d_amt_mv ON CLUSTER default;
-DROP VIEW IF EXISTS traces_7d_amt_mv ON CLUSTER default;
-DROP VIEW IF EXISTS traces_all_amt_mv ON CLUSTER default;
-
--- Drop AMT tables
-DROP TABLE IF EXISTS traces_30d_amt ON CLUSTER default;
-DROP TABLE IF EXISTS traces_7d_amt ON CLUSTER default;
-DROP TABLE IF EXISTS traces_all_amt ON CLUSTER default;
-
--- Drop the Null table
-DROP TABLE IF EXISTS traces_null ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.down.sql
deleted file mode 100644
index b4425304bd..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE dataset_run_items_rmt ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sql
deleted file mode 100644
index 853b0ced4b..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sql
+++ /dev/null
@@ -1,36 +0,0 @@
-CREATE TABLE dataset_run_items_rmt ON CLUSTER default (
- -- primary identifiers
- `id` String,
- `project_id` String,
- `dataset_run_id` String,
- `dataset_item_id` String,
- `dataset_id` String,
- `trace_id` String,
- `observation_id` Nullable(String),
-
- -- error field
- `error` Nullable(String),
-
- -- timestamps
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now(),
-
- -- denormalized immutable dataset run fields
- `dataset_run_name` String,
- `dataset_run_description` Nullable(String),
- `dataset_run_metadata` Map(LowCardinality(String), String),
- `dataset_run_created_at` DateTime64(3),
-
- -- denormalized dataset item fields (mutable, but snapshots are relevant)
- `dataset_item_input` Nullable(String) CODEC(ZSTD(3)), -- json
- `dataset_item_expected_output` Nullable(String) CODEC(ZSTD(3)), -- json
- `dataset_item_metadata` Map(LowCardinality(String), String),
-
- -- clickhouse engine fields
- `event_ts` DateTime64(3),
- `is_deleted` UInt8,
-
- -- For dataset item lookups
- INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,
-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)
-ORDER BY (project_id, dataset_id, dataset_run_id, id);
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.down.sql b/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.down.sql
deleted file mode 100644
index cf7d9cc96a..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_value SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_key SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.up.sql b/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.up.sql
deleted file mode 100644
index 2f672c0c71..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_key SETTINGS mutations_sync = 2;
-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_value SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.down.sql
deleted file mode 100644
index 32b5cfc9df..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default DROP INDEX IF EXISTS idx_trace_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.up.sql
deleted file mode 100644
index ccc041e82c..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_trace_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.down.sql b/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.down.sql
deleted file mode 100644
index 8c502451c9..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.down.sql
+++ /dev/null
@@ -1,21 +0,0 @@
--- Recreate materialized views for project_environments table
-CREATE MATERIALIZED VIEW project_environments_traces_mv ON CLUSTER default TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM traces
-GROUP BY project_id;
-
-CREATE MATERIALIZED VIEW project_environments_observations_mv ON CLUSTER default TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM observations
-GROUP BY project_id;
-
-CREATE MATERIALIZED VIEW project_environments_scores_mv ON CLUSTER default TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM scores
-GROUP BY project_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.up.sql b/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.up.sql
deleted file mode 100644
index 99e2d32453..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
--- Drop materialized views feeding project_environments table
-DROP VIEW IF EXISTS project_environments_traces_mv ON CLUSTER default;
-DROP VIEW IF EXISTS project_environments_observations_mv ON CLUSTER default;
-DROP VIEW IF EXISTS project_environments_scores_mv ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.up.sql b/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.up.sql
deleted file mode 100644
index 53d0c19bac..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
--- Drop materialized views derived from traces_null table
-DROP VIEW IF EXISTS traces_all_amt_mv ON CLUSTER default;
-DROP VIEW IF EXISTS traces_7d_amt_mv ON CLUSTER default;
-DROP VIEW IF EXISTS traces_30d_amt_mv ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.down.sql b/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.down.sql
deleted file mode 100644
index 53e287bd07..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.down.sql
+++ /dev/null
@@ -1,179 +0,0 @@
--- Recreate traces_null and trace amt tables
-CREATE TABLE traces_null ON CLUSTER default
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `start_time` DateTime64(3),
- `end_time` Nullable(DateTime64(3)),
- `name` Nullable(String),
-
- -- Metadata properties
- `metadata` Map(LowCardinality(String), String),
- `user_id` Nullable(String),
- `session_id` Nullable(String),
- `environment` String,
- `tags` Array(String),
- `version` Nullable(String),
- `release` Nullable(String),
-
- -- UI properties - We make them nullable to prevent absent values being interpreted as overwrites.
- `bookmarked` Nullable(Bool),
- `public` Nullable(Bool),
-
- -- Aggregations -- DO NOT USE
- `observation_ids` Array(String),
- `score_ids` Array(String),
- `cost_details` Map(String, Decimal64(12)),
- `usage_details` Map(String, UInt64),
- -- TODO: Do we want to aggregate/collect `levels` seen within the trace?
-
- -- Input/Output
- `input` String,
- `output` String,
-
- `created_at` DateTime64(3),
- `updated_at` DateTime64(3),
- `event_ts` DateTime64(3)
-) Engine = Null();
-
-CREATE TABLE traces_all_amt ON CLUSTER default
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time
- `start_time` SimpleAggregateFunction(min, DateTime64(3)),
- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),
- `name` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- Metadata properties
- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),
- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `environment` SimpleAggregateFunction(anyLast, String),
- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `version` SimpleAggregateFunction(anyLast, Nullable(String)),
- `release` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- UI properties
- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
-
- -- Aggregations -- DO NOT USE
- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),
-
- -- Input/Output -> prefer correctness via argMax
- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
-
- `created_at` SimpleAggregateFunction(min, DateTime64(3)),
- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),
-
- -- Indexes
- INDEX idx_trace_id id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
-) Engine = AggregatingMergeTree()
- ORDER BY (project_id, id);
-
-CREATE TABLE traces_7d_amt ON CLUSTER default
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time
- `start_time` SimpleAggregateFunction(min, DateTime64(3)),
- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),
- `name` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- Metadata properties
- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),
- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `environment` SimpleAggregateFunction(anyLast, String),
- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `version` SimpleAggregateFunction(anyLast, Nullable(String)),
- `release` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- UI properties
- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
-
- -- Aggregations -- DO NOT USE
- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),
-
- -- Input/Output -> prefer correctness via argMax
- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
-
- `created_at` SimpleAggregateFunction(min, DateTime64(3)),
- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),
-
- -- Indexes
- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
-) Engine = AggregatingMergeTree()
- ORDER BY (project_id, id)
- TTL toDate(start_time) + INTERVAL 7 DAY;
-
-CREATE TABLE traces_30d_amt ON CLUSTER default
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time
- `start_time` SimpleAggregateFunction(min, DateTime64(3)),
- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),
- `name` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- Metadata properties
- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),
- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `environment` SimpleAggregateFunction(anyLast, String),
- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `version` SimpleAggregateFunction(anyLast, Nullable(String)),
- `release` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- UI properties
- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
-
- -- Aggregations -- DO NOT USE
- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),
-
- -- Input/Output -> prefer correctness via argMax
- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
-
- `created_at` SimpleAggregateFunction(min, DateTime64(3)),
- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),
-
- -- Indexes
- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
-) Engine = AggregatingMergeTree()
- ORDER BY (project_id, id)
- TTL toDate(start_time) + INTERVAL 30 DAY;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.up.sql b/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.up.sql
deleted file mode 100644
index 10f1d76df7..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.up.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- Drop traces_null and trace amt tables
-DROP TABLE IF EXISTS traces_null ON CLUSTER default;
-DROP TABLE IF EXISTS traces_all_amt ON CLUSTER default;
-DROP TABLE IF EXISTS traces_7d_amt ON CLUSTER default;
-DROP TABLE IF EXISTS traces_30d_amt ON CLUSTER default;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.down.sql
deleted file mode 100644
index 644c8b9080..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS execution_trace_id;
diff --git a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql
deleted file mode 100644
index 0ea406281a..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String) SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.down.sql
deleted file mode 100644
index df06748acd..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_name SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_id SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.up.sql
deleted file mode 100644
index 5c9a17d3ec..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_id Nullable(String) SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_name Nullable(String) SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql
deleted file mode 100644
index 3a351dd8be..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default DROP COLUMN IF EXISTS dataset_item_version;
diff --git a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
deleted file mode 100644
index a06e222c2f..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3)) SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.down.sql
deleted file mode 100644
index de6cad399d..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.down.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS tool_definitions SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS tool_calls SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS tool_call_names SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.up.sql
deleted file mode 100644
index 535adb9aa8..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.up.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE observations ON CLUSTER default ADD COLUMN IF NOT EXISTS tool_definitions Map(String, String) DEFAULT map() SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default ADD COLUMN IF NOT EXISTS tool_calls Array(String) DEFAULT [] SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default ADD COLUMN IF NOT EXISTS tool_call_names Array(String) DEFAULT [] SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.down.sql
deleted file mode 100644
index c9f9555be8..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS long_string_value;
diff --git a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql
deleted file mode 100644
index 22d707f000..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3)) SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.down.sql
deleted file mode 100644
index 695a0383e5..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.down.sql
+++ /dev/null
@@ -1,11 +0,0 @@
-ALTER TABLE scores ON CLUSTER default
- DROP COLUMN IF EXISTS ingestion_sdk_version
- SETTINGS alter_sync = 2;
-
-ALTER TABLE scores ON CLUSTER default
- DROP COLUMN IF EXISTS ingestion_sdk_name
- SETTINGS alter_sync = 2;
-
-ALTER TABLE scores ON CLUSTER default
- DROP COLUMN IF EXISTS ingestion_api_key
- SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.up.sql
deleted file mode 100644
index ae8e27730f..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.up.sql
+++ /dev/null
@@ -1,11 +0,0 @@
-ALTER TABLE scores ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS alter_sync = 2;
-
-ALTER TABLE scores ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS alter_sync = 2;
-
-ALTER TABLE scores ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.down.sql
deleted file mode 100644
index 8f5c9219e2..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.down.sql
+++ /dev/null
@@ -1,22 +0,0 @@
-DROP VIEW IF EXISTS analytics_scores ON CLUSTER default;
-CREATE VIEW analytics_scores ON CLUSTER default AS
-SELECT
- project_id,
- toStartOfHour(timestamp) AS hour,
- uniq(id) AS countScores,
- max(source = 'ANNOTATION') AS hasAnnotation,
- max(source = 'API') AS hasApi,
- max(source = 'EVAL') AS hasEval,
- max(observation_id IS NOT NULL) AS hasObservationScore,
- max(session_id IS NOT NULL) AS hasSessionScore,
- max(dataset_run_id IS NOT NULL) AS hasDatasetRunScore,
- max(data_type = 'BOOLEAN') AS hasBoolScore,
- max(data_type = 'NUMERIC') AS hasNumericScore,
- max(data_type = 'CATEGORICAL') AS hasCategoricalScore,
- max(comment IS NOT NULL) AS hasComment
-FROM
- scores
-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))
-GROUP BY
- project_id,
- hour;
diff --git a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.down.sql b/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.down.sql
deleted file mode 100644
index 855010df2e..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.down.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_created_at;
-ALTER TABLE traces ON CLUSTER default DROP INDEX IF EXISTS idx_created_at;
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_created_at;
diff --git a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.up.sql b/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.up.sql
deleted file mode 100644
index 60467e64f2..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.up.sql
+++ /dev/null
@@ -1,6 +0,0 @@
-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_created_at;
-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_created_at;
-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1 SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_created_at;
diff --git a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.up.sql b/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.up.sql
deleted file mode 100644
index c43b1a8209..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.up.sql
+++ /dev/null
@@ -1,51 +0,0 @@
-CREATE TABLE IF NOT EXISTS observations_batch_staging ON CLUSTER default
-(
- id String,
- trace_id String,
- project_id String,
- type LowCardinality(String),
- parent_observation_id Nullable(String),
- start_time DateTime64(3),
- end_time Nullable(DateTime64(3)),
- name String,
- metadata Map(LowCardinality(String), String),
- level LowCardinality(String),
- status_message Nullable(String),
- version Nullable(String),
- input Nullable(String) CODEC(ZSTD(3)),
- output Nullable(String) CODEC(ZSTD(3)),
- provided_model_name Nullable(String),
- internal_model_id Nullable(String),
- model_parameters Nullable(String),
- provided_usage_details Map(LowCardinality(String), UInt64),
- usage_details Map(LowCardinality(String), UInt64),
- provided_cost_details Map(LowCardinality(String), Decimal64(12)),
- cost_details Map(LowCardinality(String), Decimal64(12)),
- total_cost Nullable(Decimal64(12)),
- usage_pricing_tier_id Nullable(String),
- usage_pricing_tier_name Nullable(String),
- tool_definitions Map(String, String),
- tool_calls Array(String),
- tool_call_names Array(String),
- completion_start_time Nullable(DateTime64(3)),
- prompt_id Nullable(String),
- prompt_name Nullable(String),
- prompt_version Nullable(UInt16),
- created_at DateTime64(3) DEFAULT now(),
- updated_at DateTime64(3) DEFAULT now(),
- event_ts DateTime64(3),
- is_deleted UInt8,
- s3_first_seen_timestamp DateTime64(3),
- environment LowCardinality(String) DEFAULT 'default'
-)
-ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)
-PARTITION BY toStartOfInterval(s3_first_seen_timestamp, INTERVAL 3 MINUTE)
-PRIMARY KEY (project_id, toDate(s3_first_seen_timestamp))
-ORDER BY (
- project_id,
- toDate(s3_first_seen_timestamp),
- trace_id,
- id
-)
-TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR
-SETTINGS ttl_only_drop_parts = 1;
diff --git a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.down.sql b/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.down.sql
deleted file mode 100644
index 3b3ae1aa49..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.down.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this
--- table may predate this migration (created via dev-tables.sh on v4 preview
--- deployments) and may hold production-grade event data. Rolling back drops
--- that data regardless of which path created the table.
-DROP TABLE IF EXISTS events_full ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.up.sql b/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.up.sql
deleted file mode 100644
index 9e8a743590..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.up.sql
+++ /dev/null
@@ -1,124 +0,0 @@
-CREATE TABLE IF NOT EXISTS events_full ON CLUSTER default
-(
- project_id String,
- trace_id String,
- span_id String,
- parent_span_id String,
-
- start_time DateTime64(6),
- end_time Nullable(DateTime64(6)),
-
- -- Core properties
- name String,
- type LowCardinality(String),
- environment LowCardinality(String) DEFAULT 'default',
- version String,
- release String,
- trace_name String,
- user_id String,
- session_id String,
- tags Array(String),
- level LowCardinality(String),
- status_message String,
- completion_start_time Nullable(DateTime64(6)),
- is_app_root Bool DEFAULT false,
-
- -- Updateable properties
- bookmarked Bool DEFAULT false,
- public Bool DEFAULT false,
-
- -- Prompt
- prompt_id String,
- prompt_name String,
- prompt_version Nullable(UInt16),
-
- -- Model
- model_id String,
- provided_model_name String,
- model_parameters String,
-
- -- Usage and Cost
- provided_usage_details Map(LowCardinality(String), UInt64),
- usage_details Map(LowCardinality(String), UInt64),
- provided_cost_details Map(LowCardinality(String), Decimal(18,12)),
- cost_details Map(LowCardinality(String), Decimal(18,12)),
- calculated_input_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0, cost_details))),
- calculated_output_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
- calculated_total_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0 OR positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
- total_cost Decimal(18, 12) ALIAS cost_details['total'],
-
- usage_pricing_tier_id Nullable(String),
- usage_pricing_tier_name Nullable(String),
-
- -- Tools
- tool_definitions Map(String, String),
- tool_calls Array(String),
- tool_call_names Array(String),
-
- -- I/O
- input String CODEC(ZSTD(3)),
- input_length UInt64 MATERIALIZED lengthUTF8(input),
- output String CODEC(ZSTD(3)),
- output_length UInt64 MATERIALIZED lengthUTF8(output),
-
- -- Metadata
- metadata_names Array(String),
- metadata_values Array(String),
-
- -- Experiment properties
- experiment_id String,
- experiment_name String,
- experiment_metadata_names Array(String),
- experiment_metadata_values Array(String),
- experiment_description String,
- experiment_dataset_id String,
- experiment_item_id String,
- experiment_item_version Nullable(DateTime64(6)),
- experiment_item_expected_output String,
- experiment_item_metadata_names Array(String),
- experiment_item_metadata_values Array(String),
- experiment_item_root_span_id String,
-
- -- Source metadata (Instrumentation)
- source LowCardinality(String),
- service_name String,
- service_version String,
- scope_name String,
- scope_version String,
- telemetry_sdk_language LowCardinality(String),
- telemetry_sdk_name String,
- telemetry_sdk_version String,
-
- -- Generic props
- blob_storage_file_path String,
- event_bytes UInt64,
- created_at DateTime64(6) DEFAULT now(),
- updated_at DateTime64(6) DEFAULT now(),
- event_ts DateTime64(6),
- is_deleted UInt8,
-
- -- Indexes
- INDEX idx_span_id span_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_user_id user_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_created_at created_at TYPE minmax GRANULARITY 1,
- INDEX idx_updated_at updated_at TYPE minmax GRANULARITY 1,
- INDEX idx_fts_input_low lower(input) TYPE text(tokenizer = splitByNonAlpha),
- INDEX idx_fts_output_low lower(output) TYPE text(tokenizer = splitByNonAlpha),
- INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),
- INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)
-)
-ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)
-PARTITION BY toYYYYMM(start_time)
-PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))
-ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)
-SAMPLE BY xxHash32(trace_id)
-SETTINGS
- index_granularity_bytes = '64Mi', -- Default 10MiB. Prevents very small granules due to large rows.
- merge_max_block_size_bytes = '64Mi',
- enable_block_number_column = 1,
- enable_block_offset_column = 1,
- prewarm_mark_cache = 1,
- prewarm_primary_key_cache = 1,
- enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.down.sql b/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.down.sql
deleted file mode 100644
index fd9ecf5910..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.down.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this
--- table may predate this migration (created via dev-tables.sh on v4 preview
--- deployments) and may hold production-grade event data. Rolling back drops
--- that data regardless of which path created the table.
-DROP TABLE IF EXISTS events_core ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.up.sql b/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.up.sql
deleted file mode 100644
index ab53c989e8..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.up.sql
+++ /dev/null
@@ -1,123 +0,0 @@
-CREATE TABLE IF NOT EXISTS events_core ON CLUSTER default
-(
- project_id String,
- trace_id String,
- span_id String,
- parent_span_id String,
-
- start_time DateTime64(6),
- end_time Nullable(DateTime64(6)),
-
- -- Core properties
- name String,
- type LowCardinality(String),
- environment LowCardinality(String) DEFAULT 'default',
- version String,
- release String,
- trace_name String,
- user_id String,
- session_id String,
- tags Array(String),
- level LowCardinality(String),
- status_message String,
- completion_start_time Nullable(DateTime64(6)),
- is_app_root Bool DEFAULT false,
-
- -- Updateable properties
- bookmarked Bool DEFAULT false,
- public Bool DEFAULT false,
-
- -- Prompt
- prompt_id String,
- prompt_name String,
- prompt_version Nullable(UInt16),
-
- -- Model
- model_id String,
- provided_model_name String,
- model_parameters String,
-
- -- Usage
- provided_usage_details Map(LowCardinality(String), UInt64),
- usage_details Map(LowCardinality(String), UInt64),
- provided_cost_details Map(LowCardinality(String), Decimal(18,12)),
- cost_details Map(LowCardinality(String), Decimal(18,12)),
- calculated_input_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0, cost_details))),
- calculated_output_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
- calculated_total_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0 OR positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
- total_cost Decimal(18, 12) ALIAS cost_details['total'],
-
- usage_pricing_tier_id Nullable(String),
- usage_pricing_tier_name Nullable(String),
-
- -- Tools
- tool_definitions Map(String, String),
- tool_calls Array(String),
- tool_call_names Array(String),
-
- -- I/O
- input String,
- input_length UInt64 MATERIALIZED lengthUTF8(input),
- output String,
- output_length UInt64 MATERIALIZED lengthUTF8(output),
-
- -- Metadata
- metadata_names Array(String),
- metadata_values Array(String),
-
- -- Experiment properties
- experiment_id String,
- experiment_name String,
- experiment_metadata_names Array(String),
- experiment_metadata_values Array(String),
- experiment_description String,
- experiment_dataset_id String,
- experiment_item_id String,
- experiment_item_version Nullable(DateTime64(6)),
- experiment_item_expected_output String,
- experiment_item_metadata_names Array(String),
- experiment_item_metadata_values Array(String),
- experiment_item_root_span_id String,
-
- -- Source metadata (Instrumentation)
- source LowCardinality(String),
- service_name String,
- service_version String,
- scope_name String,
- scope_version String,
- telemetry_sdk_language LowCardinality(String),
- telemetry_sdk_name String,
- telemetry_sdk_version String,
-
- -- Generic props
- blob_storage_file_path String,
- event_bytes UInt64,
- created_at DateTime64(6) DEFAULT now(),
- updated_at DateTime64(6) DEFAULT now(),
- event_ts DateTime64(6),
- is_deleted UInt8,
-
- -- Indexes
- INDEX idx_span_id span_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_user_id user_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_created_at created_at TYPE minmax GRANULARITY 1,
- INDEX idx_updated_at updated_at TYPE minmax GRANULARITY 1,
- INDEX idx_provided_model_name provided_model_name TYPE bloom_filter(0.01) GRANULARITY 2,
- INDEX idx_experiment_id experiment_id TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_metadata_names metadata_names TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),
- INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)
-)
-ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)
-PARTITION BY toYYYYMM(start_time)
-PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))
-ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)
-SAMPLE BY xxHash32(trace_id)
-SETTINGS
- enable_block_number_column = 1,
- enable_block_offset_column = 1,
- prewarm_mark_cache = 1,
- prewarm_primary_key_cache = 1,
- enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.down.sql b/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.down.sql
deleted file mode 100644
index dacd2eb194..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.down.sql
+++ /dev/null
@@ -1,4 +0,0 @@
--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this
--- view may predate this migration (created via dev-tables.sh on v4 preview
--- deployments). Dropping it stops events_core from being populated.
-DROP VIEW IF EXISTS events_core_mv ON CLUSTER default;
diff --git a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.up.sql b/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.up.sql
deleted file mode 100644
index 3c2db389da..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.up.sql
+++ /dev/null
@@ -1,69 +0,0 @@
-CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv ON CLUSTER default TO events_core AS
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted
-FROM events_full;
diff --git a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.down.sql
deleted file mode 100644
index 1ba2f66a48..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.down.sql
+++ /dev/null
@@ -1,82 +0,0 @@
-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted
-FROM events_full
-SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE observations_batch_staging ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE observations_batch_staging ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE observations_batch_staging ON CLUSTER default DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1, alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.up.sql
deleted file mode 100644
index b751b5a26b..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.up.sql
+++ /dev/null
@@ -1,103 +0,0 @@
-ALTER TABLE observations_batch_staging ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE observations_batch_staging ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE observations_batch_staging ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE events_full ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE events_core ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default
- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
- SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted,
- ingestion_api_key,
- ingestion_sdk_name,
- ingestion_sdk_version
-FROM events_full
-SETTINGS enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.down.sql
deleted file mode 100644
index 9f1ef0ead0..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE events_full ON CLUSTER default DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1, alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.up.sql
deleted file mode 100644
index e773b902f0..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE events_full ON CLUSTER default ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1, alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.down.sql
deleted file mode 100644
index 3291a7b3ad..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.down.sql
+++ /dev/null
@@ -1,38 +0,0 @@
--- WARNING: This recreates the table structure only. Any data held in
--- dataset_run_items before the up migration dropped it is NOT restored.
-CREATE TABLE IF NOT EXISTS dataset_run_items ON CLUSTER default (
- -- primary identifiers
- `id` String,
- `project_id` String,
- `dataset_run_id` String,
- `dataset_item_id` String,
- `dataset_id` String,
- `trace_id` String,
- `observation_id` Nullable(String),
-
- -- error field
- `error` Nullable(String),
-
- -- timestamps
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now(),
-
- -- denormalized immutable dataset run fields
- `dataset_run_name` String,
- `dataset_run_description` Nullable(String),
- `dataset_run_metadata` Map(LowCardinality(String), String),
- `dataset_run_created_at` DateTime64(3),
-
- -- denormalized dataset item fields (mutable, but snapshots are relevant)
- `dataset_item_input` Nullable(String) CODEC(ZSTD(3)), -- json
- `dataset_item_expected_output` Nullable(String) CODEC(ZSTD(3)), -- json
- `dataset_item_metadata` Map(LowCardinality(String), String),
-
- -- clickhouse engine fields
- `event_ts` DateTime64(3),
- `is_deleted` UInt8,
-
- -- For dataset item lookups
- INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
-ORDER BY (project_id, dataset_id, dataset_run_id, id);
diff --git a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.down.sql
deleted file mode 100644
index 490efbde32..0000000000
--- a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.down.sql
+++ /dev/null
@@ -1,88 +0,0 @@
-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted,
- ingestion_api_key,
- ingestion_sdk_name,
- ingestion_sdk_version
-FROM events_full
-SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE events_core ON CLUSTER default DROP INDEX IF EXISTS idx_evaluator_id SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1, alter_sync = 2;
-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1, alter_sync = 2;
-
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_evaluator_id SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS evaluator_id SETTINGS alter_sync = 2;
-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0001_traces.down.sql b/packages/shared/clickhouse/migrations/unclustered/0001_traces.down.sql
deleted file mode 100644
index 95a0b29e4b..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0001_traces.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE traces;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0001_traces.up.sql b/packages/shared/clickhouse/migrations/unclustered/0001_traces.up.sql
deleted file mode 100644
index 6e0d1c5122..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0001_traces.up.sql
+++ /dev/null
@@ -1,32 +0,0 @@
-CREATE TABLE traces (
- `id` String,
- `timestamp` DateTime64(3),
- `name` String,
- `user_id` Nullable(String),
- `metadata` Map(LowCardinality(String), String),
- `release` Nullable(String),
- `version` Nullable(String),
- `project_id` String,
- `public` Bool,
- `bookmarked` Bool,
- `tags` Array(String),
- `input` Nullable(String) CODEC(ZSTD(3)),
- `output` Nullable(String) CODEC(ZSTD(3)),
- `session_id` Nullable(String),
- `created_at` DateTime64(3) DEFAULT now(),
- updated_at DateTime64(3) DEFAULT now(),
- `event_ts` DateTime64(3),
- `is_deleted` UInt8,
- INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1,
- INDEX idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)
-PRIMARY KEY (
- project_id,
- toDate(timestamp)
-)
-ORDER BY (
- project_id,
- toDate(timestamp),
- id
-);
diff --git a/packages/shared/clickhouse/migrations/unclustered/0002_observations.down.sql b/packages/shared/clickhouse/migrations/unclustered/0002_observations.down.sql
deleted file mode 100644
index 656776f502..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0002_observations.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE observations;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0003_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0003_scores.down.sql
deleted file mode 100644
index c5ea562063..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0003_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE scores;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0003_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0003_scores.up.sql
deleted file mode 100644
index e2b4a7bd1c..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0003_scores.up.sql
+++ /dev/null
@@ -1,33 +0,0 @@
-CREATE TABLE scores (
- `id` String,
- `timestamp` DateTime64(3),
- `project_id` String,
- `trace_id` String,
- `observation_id` Nullable(String),
- `name` String,
- `value` Float64,
- `source` String,
- `comment` Nullable(String) CODEC(ZSTD(1)),
- `author_user_id` Nullable(String),
- `config_id` Nullable(String),
- `data_type` String,
- `string_value` Nullable(String),
- `queue_id` Nullable(String),
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now(),
- event_ts DateTime64(3),
- `is_deleted` UInt8,
- INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1
-) ENGINE = ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)
-PRIMARY KEY (
- project_id,
- toDate(timestamp),
- name
-)
-ORDER BY (
- project_id,
- toDate(timestamp),
- name,
- id
-)
diff --git a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.down.sql
deleted file mode 100644
index 5e36747336..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_project_id project_id TYPE bloom_filter() GRANULARITY 1;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.up.sql
deleted file mode 100644
index 1c008a4afc..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE observations DROP INDEX IF EXISTS idx_project_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.down.sql
deleted file mode 100644
index 7d2976b9d0..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE traces DROP INDEX IF EXISTS idx_session_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.up.sql
deleted file mode 100644
index 6d79dbe3b9..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE traces ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1;
-ALTER TABLE traces MATERIALIZE INDEX IF EXISTS idx_session_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.down.sql
deleted file mode 100644
index a867a7d368..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE traces DROP INDEX IF EXISTS idx_user_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.up.sql
deleted file mode 100644
index da1d3460ab..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE traces ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1;
-ALTER TABLE traces MATERIALIZE INDEX IF EXISTS idx_user_id;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.down.sql b/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.down.sql
deleted file mode 100644
index 3996548c5c..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE event_log;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.up.sql b/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.up.sql
deleted file mode 100644
index 40afc72e63..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.up.sql
+++ /dev/null
@@ -1,19 +0,0 @@
-CREATE TABLE event_log
-(
- `id` String,
- `project_id` String,
- `entity_type` String,
- `entity_id` String,
- `event_id` Nullable(String),
-
- `bucket_name` String,
- `bucket_path` String,
-
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now()
-) ENGINE = MergeTree()
- ORDER BY (
- project_id,
- entity_type,
- entity_id
- );
diff --git a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.down.sql b/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.down.sql
deleted file mode 100644
index 1c59e0df4b..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.down.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE traces DROP COLUMN IF EXISTS environment;
-ALTER TABLE observations DROP COLUMN IF EXISTS environment;
-ALTER TABLE scores DROP COLUMN IF EXISTS environment;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.up.sql b/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.up.sql
deleted file mode 100644
index b62a70f75e..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.up.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE traces ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;
-ALTER TABLE observations ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;
-ALTER TABLE scores ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.down.sql b/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.down.sql
deleted file mode 100644
index a1622de24d..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.down.sql
+++ /dev/null
@@ -1,7 +0,0 @@
--- Drop materialized views
-DROP VIEW IF EXISTS project_environments_traces_mv;
-DROP VIEW IF EXISTS project_environments_observations_mv;
-DROP VIEW IF EXISTS project_environments_scores_mv;
-
--- Drop the project_environments table
-DROP TABLE IF EXISTS project_environments;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.down.sql
deleted file mode 100644
index 1c5cc36d72..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP COLUMN IF EXISTS metadata;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.up.sql
deleted file mode 100644
index 55dbefbe95..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ADD COLUMN metadata Map(LowCardinality(String), String) AFTER comment;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.down.sql b/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.down.sql
deleted file mode 100644
index a23949b45c..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE blob_storage_file_log;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.down.sql
deleted file mode 100644
index 1967d45f8b..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP COLUMN IF EXISTS session_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.up.sql
deleted file mode 100644
index 3c8c9098c5..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ADD COLUMN session_id Nullable(String) AFTER trace_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.down.sql
deleted file mode 100644
index a5fb46bf03..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.up.sql
deleted file mode 100644
index eeb81267d8..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
-
diff --git a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.down.sql b/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.down.sql
deleted file mode 100644
index 5938637ec5..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.up.sql b/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.up.sql
deleted file mode 100644
index 5938637ec5..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.down.sql
deleted file mode 100644
index fd7ba11787..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.up.sql
deleted file mode 100644
index 3cb554b4b7..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.down.sql
deleted file mode 100644
index 9f60ffbdd3..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.up.sql
deleted file mode 100644
index a0a39e9226..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.down.sql
deleted file mode 100644
index f511371464..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP COLUMN IF EXISTS dataset_run_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.up.sql
deleted file mode 100644
index 74f42181a8..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ADD COLUMN dataset_run_id Nullable(String) AFTER session_id SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.down.sql
deleted file mode 100644
index 530007aa81..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.up.sql
deleted file mode 100644
index 4b36b72b52..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;
-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.down.sql b/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.down.sql
deleted file mode 100644
index b073974b40..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP VIEW IF EXISTS analytics_traces;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.up.sql b/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.up.sql
deleted file mode 100644
index 7562119645..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.up.sql
+++ /dev/null
@@ -1,15 +0,0 @@
-CREATE VIEW analytics_traces AS
-SELECT
- project_id,
- toStartOfHour(timestamp) AS hour,
- uniq(id) AS countTraces,
- max(user_id IS NOT NULL) AS hasUsers,
- max(session_id IS NOT NULL) AS hasSessions,
- max(if(environment != 'default', 1, 0)) AS hasEnvironments,
- max(length(tags) > 0) AS hasTags
-FROM
- traces
-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))
-GROUP BY
- project_id,
- hour;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.down.sql b/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.down.sql
deleted file mode 100644
index fd395b4199..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP VIEW IF EXISTS analytics_observations;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.down.sql
deleted file mode 100644
index b8723146c4..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP VIEW IF EXISTS analytics_scores;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.up.sql
deleted file mode 100644
index e3ce3772d5..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.up.sql
+++ /dev/null
@@ -1,21 +0,0 @@
-CREATE VIEW analytics_scores AS
-SELECT
- project_id,
- toStartOfHour(timestamp) AS hour,
- uniq(id) AS countScores,
- max(source = 'ANNOTATION') AS hasAnnotation,
- max(source = 'API') AS hasApi,
- max(source = 'EVAL') AS hasEval,
- max(observation_id IS NOT NULL) AS hasObservationScore,
- max(session_id IS NOT NULL) AS hasSessionScore,
- max(dataset_run_id IS NOT NULL) AS hasDatasetRunScore,
- max(data_type = 'BOOLEAN') AS hasBoolScore,
- max(data_type = 'NUMERIC') AS hasNumericScore,
- max(data_type = 'CATEGORICAL') AS hasCategoricalScore,
- max(comment IS NOT NULL) AS hasComment
-FROM
- scores
-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))
-GROUP BY
- project_id,
- hour;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.down.sql
deleted file mode 100644
index c45faad79c..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE dataset_run_items;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.down.sql b/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.down.sql
deleted file mode 100644
index 7042dec750..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.down.sql
+++ /dev/null
@@ -1,12 +0,0 @@
--- Drop materialized views first
-DROP VIEW IF EXISTS traces_30d_amt_mv;
-DROP VIEW IF EXISTS traces_7d_amt_mv;
-DROP VIEW IF EXISTS traces_all_amt_mv;
-
--- Drop AMT tables
-DROP TABLE IF EXISTS traces_30d_amt;
-DROP TABLE IF EXISTS traces_7d_amt;
-DROP TABLE IF EXISTS traces_all_amt;
-
--- Drop the Null table
-DROP TABLE IF EXISTS traces_null;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.up.sql b/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.up.sql
deleted file mode 100644
index b9c91a50ba..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.up.sql
+++ /dev/null
@@ -1,300 +0,0 @@
--- Create a Null table that serves as a trigger for all materialized views.
--- We use a Null engine here to avoid storing intermediate results and save on storage.
-CREATE TABLE traces_null
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `start_time` DateTime64(3),
- `end_time` Nullable(DateTime64(3)),
- `name` Nullable(String),
-
- -- Metadata properties
- `metadata` Map(LowCardinality(String), String),
- `user_id` Nullable(String),
- `session_id` Nullable(String),
- `environment` String,
- `tags` Array(String),
- `version` Nullable(String),
- `release` Nullable(String),
-
- -- UI properties - We make them nullable to prevent absent values being interpreted as overwrites.
- `bookmarked` Nullable(Bool),
- `public` Nullable(Bool),
-
- -- Aggregations
- `observation_ids` Array(String),
- `score_ids` Array(String),
- `cost_details` Map(String, Decimal64(12)),
- `usage_details` Map(String, UInt64),
- -- TODO: Do we want to aggregate/collect `levels` seen within the trace?
-
- -- Input/Output
- `input` String,
- `output` String,
-
- `created_at` DateTime64(3),
- `updated_at` DateTime64(3),
- `event_ts` DateTime64(3)
-) Engine = Null();
-
--- Create the all AMT
-CREATE TABLE traces_all_amt
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time
- `start_time` SimpleAggregateFunction(min, DateTime64(3)),
- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),
- `name` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- Metadata properties
- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),
- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `environment` SimpleAggregateFunction(anyLast, String),
- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `version` SimpleAggregateFunction(anyLast, Nullable(String)),
- `release` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- UI properties
- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
-
- -- Aggregations
- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),
-
- -- Input/Output -> prefer correctness via argMax
- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
-
- `created_at` SimpleAggregateFunction(min, DateTime64(3)),
- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),
-
- -- Indexes
- INDEX idx_trace_id id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
-) Engine = AggregatingMergeTree()
- ORDER BY (project_id, id);
-
--- Create materialized view for all_amt
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv TO traces_all_amt AS
-SELECT
- -- Identifiers
- tn.project_id as project_id,
- tn.id as id,
- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time
- min(tn.start_time) as start_time,
- max(coalesce(tn.end_time, tn.start_time)) as end_time,
- anyLast(tn.name) as name,
-
- -- Metadata properties
- maxMap(tn.metadata) as metadata,
- anyLast(tn.user_id) as user_id,
- anyLast(tn.session_id) as session_id,
- anyLast(tn.environment) as environment,
- groupUniqArrayArray(tn.tags) as tags,
- anyLast(tn.version) as version,
- anyLast(tn.release) as release,
-
- -- UI properties
- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
-
- -- Aggregations
- groupUniqArrayArray(tn.observation_ids) as observation_ids,
- groupUniqArrayArray(tn.score_ids) as score_ids,
- sumMap(tn.cost_details) as cost_details,
- sumMap(tn.usage_details) as usage_details,
-
- -- Input/Output
- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,
- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,
-
- min(tn.created_at) as created_at,
- max(tn.updated_at) as updated_at
-FROM traces_null tn
-GROUP BY project_id, id;
-
--- Create the 7-day TTL AMT
-CREATE TABLE traces_7d_amt
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time
- `start_time` SimpleAggregateFunction(min, DateTime64(3)),
- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),
- `name` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- Metadata properties
- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),
- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `environment` SimpleAggregateFunction(anyLast, String),
- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `version` SimpleAggregateFunction(anyLast, Nullable(String)),
- `release` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- UI properties
- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
-
- -- Aggregations
- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),
-
- -- Input/Output -> prefer correctness via argMax
- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
-
- `created_at` SimpleAggregateFunction(min, DateTime64(3)),
- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),
-
- -- Indexes
- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
-) Engine = AggregatingMergeTree()
- ORDER BY (project_id, id)
- TTL toDate(start_time) + INTERVAL 7 DAY;
-
--- Create materialized view for 7d_amt
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv TO traces_7d_amt AS
-SELECT
- -- Identifiers
- tn.project_id as project_id,
- tn.id as id,
- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time
- min(tn.start_time) as start_time,
- max(coalesce(tn.end_time, tn.start_time)) as end_time,
- anyLast(tn.name) as name,
-
- -- Metadata properties
- maxMap(tn.metadata) as metadata,
- anyLast(tn.user_id) as user_id,
- anyLast(tn.session_id) as session_id,
- anyLast(tn.environment) as environment,
- groupUniqArrayArray(tn.tags) as tags,
- anyLast(tn.version) as version,
- anyLast(tn.release) as release,
-
- -- UI properties
- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
-
- -- Aggregations
- groupUniqArrayArray(tn.observation_ids) as observation_ids,
- groupUniqArrayArray(tn.score_ids) as score_ids,
- sumMap(tn.cost_details) as cost_details,
- sumMap(tn.usage_details) as usage_details,
-
- -- Input/Output
- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,
- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,
-
- min(tn.created_at) as created_at,
- max(tn.updated_at) as updated_at
-FROM traces_null tn
-GROUP BY project_id, id;
-
--- Create the 30-day TTL AMT
-CREATE TABLE traces_30d_amt
-(
- -- Identifiers
- `project_id` String,
- `id` String,
- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time
- `start_time` SimpleAggregateFunction(min, DateTime64(3)),
- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),
- `name` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- Metadata properties
- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),
- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),
- `environment` SimpleAggregateFunction(anyLast, String),
- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `version` SimpleAggregateFunction(anyLast, Nullable(String)),
- `release` SimpleAggregateFunction(anyLast, Nullable(String)),
-
- -- UI properties
- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),
-
- -- Aggregations
- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),
- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),
- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),
-
- -- Input/Output -> prefer correctness via argMax
- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),
-
- `created_at` SimpleAggregateFunction(min, DateTime64(3)),
- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),
-
- -- Indexes
- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,
- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1
-) Engine = AggregatingMergeTree()
- ORDER BY (project_id, id)
- TTL toDate(start_time) + INTERVAL 30 DAY;
-
--- Create materialized view for 30d_amt
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv TO traces_30d_amt AS
-SELECT
- -- Identifiers
- tn.project_id as project_id,
- tn.id as id,
- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time
- min(tn.start_time) as start_time,
- max(coalesce(tn.end_time, tn.start_time)) as end_time,
- anyLast(tn.name) as name,
-
- -- Metadata properties
- maxMap(tn.metadata) as metadata,
- anyLast(tn.user_id) as user_id,
- anyLast(tn.session_id) as session_id,
- anyLast(tn.environment) as environment,
- groupUniqArrayArray(tn.tags) as tags,
- anyLast(tn.version) as version,
- anyLast(tn.release) as release,
-
- -- UI properties
- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
-
- -- Aggregations
- groupUniqArrayArray(tn.observation_ids) as observation_ids,
- groupUniqArrayArray(tn.score_ids) as score_ids,
- sumMap(tn.cost_details) as cost_details,
- sumMap(tn.usage_details) as usage_details,
-
- -- Input/Output
- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,
- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,
-
- min(tn.created_at) as created_at,
- max(tn.updated_at) as updated_at
-FROM traces_null tn
-GROUP BY project_id, id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.down.sql
deleted file mode 100644
index 017daef96f..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-DROP TABLE dataset_run_items_rmt;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.down.sql b/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.down.sql
deleted file mode 100644
index b8762b84b7..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE observations DROP INDEX IF EXISTS idx_res_metadata_value;
-ALTER TABLE observations DROP INDEX IF EXISTS idx_res_metadata_key;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.up.sql b/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.up.sql
deleted file mode 100644
index 5e65eff735..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1;
-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1;
-ALTER TABLE observations MATERIALIZE INDEX IF EXISTS idx_res_metadata_key;
-ALTER TABLE observations MATERIALIZE INDEX IF EXISTS idx_res_metadata_value;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.down.sql
deleted file mode 100644
index 080d700fa6..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE dataset_run_items_rmt DROP INDEX IF EXISTS idx_trace_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.up.sql
deleted file mode 100644
index 696dbd74ff..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE dataset_run_items_rmt ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1;
-ALTER TABLE dataset_run_items_rmt MATERIALIZE INDEX IF EXISTS idx_trace_id;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.down.sql b/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.down.sql
deleted file mode 100644
index 49e56f9e89..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.down.sql
+++ /dev/null
@@ -1,21 +0,0 @@
--- Recreate materialized views for project_environments table
-CREATE MATERIALIZED VIEW project_environments_traces_mv TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM traces
-GROUP BY project_id;
-
-CREATE MATERIALIZED VIEW project_environments_observations_mv TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM observations
-GROUP BY project_id;
-
-CREATE MATERIALIZED VIEW project_environments_scores_mv TO project_environments AS
-SELECT
- project_id,
- groupUniqArray(environment) AS environments
-FROM scores
-GROUP BY project_id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.up.sql b/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.up.sql
deleted file mode 100644
index a4fe385fa5..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
--- Drop materialized views feeding project_environments table
-DROP VIEW IF EXISTS project_environments_traces_mv;
-DROP VIEW IF EXISTS project_environments_observations_mv;
-DROP VIEW IF EXISTS project_environments_scores_mv;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.down.sql b/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.down.sql
deleted file mode 100644
index 0d1a7a6e32..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.down.sql
+++ /dev/null
@@ -1,114 +0,0 @@
--- Recreate materialized views derived from traces_null table
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv TO traces_all_amt AS
-SELECT
- -- Identifiers
- tn.project_id as project_id,
- tn.id as id,
- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time
- min(tn.start_time) as start_time,
- max(coalesce(tn.end_time, tn.start_time)) as end_time,
- anyLast(tn.name) as name,
-
- -- Metadata properties
- maxMap(tn.metadata) as metadata,
- anyLast(tn.user_id) as user_id,
- anyLast(tn.session_id) as session_id,
- anyLast(tn.environment) as environment,
- groupUniqArrayArray(tn.tags) as tags,
- anyLast(tn.version) as version,
- anyLast(tn.release) as release,
-
- -- UI properties
- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
-
- -- Aggregations -- DO NOT USE
- groupUniqArrayArray(tn.observation_ids) as observation_ids,
- groupUniqArrayArray(tn.score_ids) as score_ids,
- sumMap(tn.cost_details) as cost_details,
- sumMap(tn.usage_details) as usage_details,
-
- -- Input/Output
- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,
- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,
-
- min(tn.created_at) as created_at,
- max(tn.updated_at) as updated_at
-FROM traces_null tn
-GROUP BY project_id, id;
-
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv TO traces_7d_amt AS
-SELECT
- -- Identifiers
- tn.project_id as project_id,
- tn.id as id,
- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time
- min(tn.start_time) as start_time,
- max(coalesce(tn.end_time, tn.start_time)) as end_time,
- anyLast(tn.name) as name,
-
- -- Metadata properties
- maxMap(tn.metadata) as metadata,
- anyLast(tn.user_id) as user_id,
- anyLast(tn.session_id) as session_id,
- anyLast(tn.environment) as environment,
- groupUniqArrayArray(tn.tags) as tags,
- anyLast(tn.version) as version,
- anyLast(tn.release) as release,
-
- -- UI properties
- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
-
- -- Aggregations -- DO NOT USE
- groupUniqArrayArray(tn.observation_ids) as observation_ids,
- groupUniqArrayArray(tn.score_ids) as score_ids,
- sumMap(tn.cost_details) as cost_details,
- sumMap(tn.usage_details) as usage_details,
-
- -- Input/Output
- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,
- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,
-
- min(tn.created_at) as created_at,
- max(tn.updated_at) as updated_at
-FROM traces_null tn
-GROUP BY project_id, id;
-
-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv TO traces_30d_amt AS
-SELECT
- -- Identifiers
- tn.project_id as project_id,
- tn.id as id,
- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time
- min(tn.start_time) as start_time,
- max(coalesce(tn.end_time, tn.start_time)) as end_time,
- anyLast(tn.name) as name,
-
- -- Metadata properties
- maxMap(tn.metadata) as metadata,
- anyLast(tn.user_id) as user_id,
- anyLast(tn.session_id) as session_id,
- anyLast(tn.environment) as environment,
- groupUniqArrayArray(tn.tags) as tags,
- anyLast(tn.version) as version,
- anyLast(tn.release) as release,
-
- -- UI properties
- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,
- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,
-
- -- Aggregations -- DO NOT USE
- groupUniqArrayArray(tn.observation_ids) as observation_ids,
- groupUniqArrayArray(tn.score_ids) as score_ids,
- sumMap(tn.cost_details) as cost_details,
- sumMap(tn.usage_details) as usage_details,
-
- -- Input/Output
- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,
- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,
-
- min(tn.created_at) as created_at,
- max(tn.updated_at) as updated_at
-FROM traces_null tn
-GROUP BY project_id, id;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.up.sql b/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.up.sql
deleted file mode 100644
index 2f95c940f4..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
--- Drop materialized views derived from traces_null table
-DROP VIEW IF EXISTS traces_all_amt_mv;
-DROP VIEW IF EXISTS traces_7d_amt_mv;
-DROP VIEW IF EXISTS traces_30d_amt_mv;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.up.sql b/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.up.sql
deleted file mode 100644
index fc99886fc1..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.up.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- Drop traces_null and trace amt tables
-DROP TABLE IF EXISTS traces_null;
-DROP TABLE IF EXISTS traces_all_amt;
-DROP TABLE IF EXISTS traces_7d_amt;
-DROP TABLE IF EXISTS traces_30d_amt;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.down.sql
deleted file mode 100644
index ec8e08c7ae..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP COLUMN IF EXISTS execution_trace_id;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.up.sql
deleted file mode 100644
index 96115a56e6..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String);
diff --git a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.down.sql
deleted file mode 100644
index 9dc3a6e6db..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE observations DROP COLUMN IF EXISTS usage_pricing_tier_name;
-ALTER TABLE observations DROP COLUMN IF EXISTS usage_pricing_tier_id;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.up.sql
deleted file mode 100644
index b9b8e23508..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE observations ADD COLUMN usage_pricing_tier_id Nullable(String);
-ALTER TABLE observations ADD COLUMN usage_pricing_tier_name Nullable(String);
diff --git a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql b/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql
deleted file mode 100644
index 1752676856..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE dataset_run_items_rmt DROP COLUMN IF EXISTS dataset_item_version;
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
deleted file mode 100644
index 18b0ada406..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE dataset_run_items_rmt ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3));
\ No newline at end of file
diff --git a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.down.sql
deleted file mode 100644
index f5e64ad73c..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.down.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE observations DROP COLUMN IF EXISTS tool_definitions SETTINGS alter_sync = 2;
-ALTER TABLE observations DROP COLUMN IF EXISTS tool_calls SETTINGS alter_sync = 2;
-ALTER TABLE observations DROP COLUMN IF EXISTS tool_call_names SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.up.sql
deleted file mode 100644
index 17627f1137..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.up.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE observations ADD COLUMN IF NOT EXISTS tool_definitions Map(String, String) DEFAULT map() SETTINGS alter_sync = 2;
-ALTER TABLE observations ADD COLUMN IF NOT EXISTS tool_calls Array(String) DEFAULT [] SETTINGS alter_sync = 2;
-ALTER TABLE observations ADD COLUMN IF NOT EXISTS tool_call_names Array(String) DEFAULT [] SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.down.sql
deleted file mode 100644
index b10e392480..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.down.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores DROP COLUMN IF EXISTS long_string_value;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.up.sql
deleted file mode 100644
index 7dfe2da5cf..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.up.sql
+++ /dev/null
@@ -1 +0,0 @@
-ALTER TABLE scores ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3));
diff --git a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.down.sql
deleted file mode 100644
index 162a83db91..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.down.sql
+++ /dev/null
@@ -1,8 +0,0 @@
-ALTER TABLE scores
- DROP COLUMN IF EXISTS ingestion_sdk_version;
-
-ALTER TABLE scores
- DROP COLUMN IF EXISTS ingestion_sdk_name;
-
-ALTER TABLE scores
- DROP COLUMN IF EXISTS ingestion_api_key;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.up.sql
deleted file mode 100644
index f6b12e724b..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.up.sql
+++ /dev/null
@@ -1,8 +0,0 @@
-ALTER TABLE scores
- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT '';
-
-ALTER TABLE scores
- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown';
-
-ALTER TABLE scores
- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown';
diff --git a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.up.sql
deleted file mode 100644
index 4cf20e3eb6..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.up.sql
+++ /dev/null
@@ -1,23 +0,0 @@
-DROP VIEW IF EXISTS analytics_scores;
-CREATE VIEW analytics_scores AS
-SELECT
- project_id,
- toStartOfHour(timestamp) AS hour,
- uniq(id) AS countScores,
- max(source = 'ANNOTATION') AS hasAnnotation,
- max(source = 'API') AS hasApi,
- max(source = 'EVAL') AS hasEval,
- max(observation_id IS NOT NULL) AS hasObservationScore,
- max(session_id IS NOT NULL) AS hasSessionScore,
- max(dataset_run_id IS NOT NULL) AS hasDatasetRunScore,
- max(data_type = 'BOOLEAN') AS hasBoolScore,
- max(data_type = 'NUMERIC') AS hasNumericScore,
- max(data_type = 'CATEGORICAL') AS hasCategoricalScore,
- max(comment IS NOT NULL) AS hasComment,
- sumMap(map(concat(if(ingestion_sdk_name = '', 'unknown', ingestion_sdk_name), '@', if(ingestion_sdk_version = '', 'unknown', ingestion_sdk_version)), toUInt64(1))) AS ingested_sdks
-FROM
- scores
-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))
-GROUP BY
- project_id,
- hour;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.down.sql b/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.down.sql
deleted file mode 100644
index 61d605ecb7..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.down.sql
+++ /dev/null
@@ -1,3 +0,0 @@
-ALTER TABLE observations DROP INDEX IF EXISTS idx_created_at;
-ALTER TABLE traces DROP INDEX IF EXISTS idx_created_at;
-ALTER TABLE scores DROP INDEX IF EXISTS idx_created_at;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.up.sql b/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.up.sql
deleted file mode 100644
index 6b74d0f599..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.up.sql
+++ /dev/null
@@ -1,6 +0,0 @@
-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1;
-ALTER TABLE observations MATERIALIZE INDEX IF EXISTS idx_created_at;
-ALTER TABLE traces ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1;
-ALTER TABLE traces MATERIALIZE INDEX IF EXISTS idx_created_at;
-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1;
-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_created_at;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.down.sql b/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.down.sql
deleted file mode 100644
index 1f7193f389..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.down.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this
--- table may predate this migration (created via dev-tables.sh on v4 preview
--- deployments) and may hold production-grade event data. Rolling back drops
--- that data regardless of which path created the table.
-DROP TABLE IF EXISTS observations_batch_staging;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.down.sql
deleted file mode 100644
index 87863a03ac..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.down.sql
+++ /dev/null
@@ -1,82 +0,0 @@
-ALTER TABLE events_core_mv MODIFY QUERY
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted
-FROM events_full
-SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE events_core DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE events_full DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE observations_batch_staging DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1;
-ALTER TABLE observations_batch_staging DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1;
-ALTER TABLE observations_batch_staging DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.down.sql
deleted file mode 100644
index 53a2eec2e8..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.down.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE events_full DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.up.sql
deleted file mode 100644
index 847ebc0f53..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.up.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-ALTER TABLE events_full ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.down.sql b/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.down.sql
deleted file mode 100644
index 939378be98..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.down.sql
+++ /dev/null
@@ -1,21 +0,0 @@
--- WARNING: This recreates the table structure only. Any data held in
--- event_log before the up migration dropped it is NOT restored.
-CREATE TABLE IF NOT EXISTS event_log
-(
- `id` String,
- `project_id` String,
- `entity_type` String,
- `entity_id` String,
- `event_id` Nullable(String),
-
- `bucket_name` String,
- `bucket_path` String,
-
- `created_at` DateTime64(3) DEFAULT now(),
- `updated_at` DateTime64(3) DEFAULT now()
-) ENGINE = MergeTree()
- ORDER BY (
- project_id,
- entity_type,
- entity_id
- );
diff --git a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.up.sql b/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.up.sql
deleted file mode 100644
index 1fdc2a46fc..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.up.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- The event_log table was superseded by blob_storage_file_log. Its contents
--- were copied over by the 20250417_1737_migrate_event_log_to_blob_storage
--- background migration, which must have completed (on the latest v3 release)
--- before upgrading to v4.
-DROP TABLE IF EXISTS event_log;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.down.sql b/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.down.sql
deleted file mode 100644
index fdf1f4db39..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.down.sql
+++ /dev/null
@@ -1,9 +0,0 @@
--- WARNING: This recreates the table structure only. Any data held in
--- project_environments before the up migration dropped it is NOT restored,
--- and the materialized views that used to feed the table were already dropped
--- by 0027_drop_project_environments_mvs, so it stays empty.
-CREATE TABLE IF NOT EXISTS project_environments (
- `project_id` String,
- `environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))
-) ENGINE = AggregatingMergeTree
-ORDER BY (project_id);
diff --git a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.up.sql b/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.up.sql
deleted file mode 100644
index 18a2795bc0..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.up.sql
+++ /dev/null
@@ -1,5 +0,0 @@
--- The project_environments table has been unused since
--- 0027_drop_project_environments_mvs removed the materialized views feeding
--- it. getEnvironmentsForProject reads distinct environments from the tracing
--- and scores tables directly.
-DROP TABLE IF EXISTS project_environments;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.up.sql
deleted file mode 100644
index 5e05c6cec7..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.up.sql
+++ /dev/null
@@ -1,4 +0,0 @@
--- The dataset_run_items table was superseded by dataset_run_items_rmt
--- (created in 0024). All ingestion writes, reads and deletes target the
--- _rmt table, so this table has been frozen since the cutover.
-DROP TABLE IF EXISTS dataset_run_items;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.down.sql
deleted file mode 100644
index 6f1ed617d1..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.down.sql
+++ /dev/null
@@ -1,88 +0,0 @@
-ALTER TABLE events_core_mv MODIFY QUERY
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted,
- ingestion_api_key,
- ingestion_sdk_name,
- ingestion_sdk_version
-FROM events_full
-SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE events_full DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE events_core DROP INDEX IF EXISTS idx_evaluator_id SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1;
-
-ALTER TABLE scores DROP INDEX IF EXISTS idx_evaluator_id;
-ALTER TABLE scores DROP INDEX IF EXISTS idx_evaluation_rule_id;
-ALTER TABLE scores DROP COLUMN IF EXISTS evaluator_id;
-ALTER TABLE scores DROP COLUMN IF EXISTS evaluation_rule_id;
diff --git a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.up.sql
deleted file mode 100644
index 6a979c92f4..0000000000
--- a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.up.sql
+++ /dev/null
@@ -1,109 +0,0 @@
--- Existing parts resolve these columns through DEFAULT expressions. Skip indexes are added without materializing historical parts to keep this migration metadata-only.
-ALTER TABLE events_full
- ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full
- ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_full
- ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
- ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
- ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
- ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
- ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.01) GRANULARITY 1
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE events_core
- ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.01) GRANULARITY 1
- SETTINGS enable_full_text_index = 1;
-ALTER TABLE scores
- ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT metadata['evaluator_id'] AFTER metadata;
-ALTER TABLE scores
- ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(metadata['evaluation_rule_id']), metadata['evaluation_rule_id'], metadata['job_configuration_id']) AFTER evaluator_id;
-ALTER TABLE scores
- ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.001) GRANULARITY 1;
-ALTER TABLE scores
- ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.001) GRANULARITY 1;
-ALTER TABLE events_core_mv MODIFY QUERY
-SELECT
- project_id,
- trace_id,
- span_id,
- parent_span_id,
- start_time,
- end_time,
- name,
- type,
- environment,
- version,
- release,
- trace_name,
- user_id,
- session_id,
- tags,
- level,
- status_message,
- completion_start_time,
- is_app_root,
- bookmarked,
- public,
- prompt_id,
- prompt_name,
- prompt_version,
- model_id,
- provided_model_name,
- model_parameters,
- provided_usage_details,
- usage_details,
- provided_cost_details,
- cost_details,
- usage_pricing_tier_id,
- usage_pricing_tier_name,
- tool_definitions,
- tool_calls,
- tool_call_names,
- leftUTF8(input, 200) as input,
- leftUTF8(output, 200) as output,
- metadata_names,
- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
- evaluator_id,
- evaluation_rule_id,
- evaluator_execution_is_test,
- experiment_id,
- experiment_name,
- experiment_metadata_names,
- experiment_metadata_values,
- experiment_description,
- experiment_dataset_id,
- experiment_item_id,
- experiment_item_version,
- experiment_item_expected_output,
- experiment_item_metadata_names,
- experiment_item_metadata_values,
- experiment_item_root_span_id,
- source,
- service_name,
- service_version,
- scope_name,
- scope_version,
- telemetry_sdk_language,
- telemetry_sdk_name,
- telemetry_sdk_version,
- blob_storage_file_path,
- event_bytes,
- created_at,
- updated_at,
- event_ts,
- is_deleted,
- ingestion_api_key,
- ingestion_sdk_name,
- ingestion_sdk_version
-FROM events_full
-SETTINGS enable_full_text_index = 1;
diff --git a/packages/shared/clickhouse/scripts/down.sh b/packages/shared/clickhouse/scripts/down.sh
index 31332a7498..ff56897c56 100644
--- a/packages/shared/clickhouse/scripts/down.sh
+++ b/packages/shared/clickhouse/scripts/down.sh
@@ -1,7 +1,9 @@
-#!/bin/bash
+#!/bin/sh
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
# Load environment variables
-[ -f ../../.env ] && source ../../.env
+[ -f ../../.env ] && . ../../.env
# Check if CLICKHOUSE_URL is configured
if [ -z "${CLICKHOUSE_URL}" ]; then
@@ -10,7 +12,7 @@ if [ -z "${CLICKHOUSE_URL}" ]; then
fi
# Check if golang-migrate is installed
-if ! command -v migrate &> /dev/null
+if ! command -v migrate >/dev/null 2>&1
then
echo "Error: golang-migrate is not installed or not in PATH."
echo "Please install golang-migrate via 'brew install golang-migrate' to run this script."
@@ -18,41 +20,80 @@ then
exit 1
fi
+require_node() {
+ if ! command -v node >/dev/null 2>&1
+ then
+ echo "Error: node is not installed or not in PATH."
+ exit 1
+ fi
+}
+
# Ensure CLICKHOUSE_DB is set
-if [ -z "${CLICKHOUSE_DB}" ]; then
- export CLICKHOUSE_DB="default"
-fi
+export CLICKHOUSE_DB="${CLICKHOUSE_DB:-default}"
# Ensure CLICKHOUSE_CLUSTER_NAME is set
-if [ -z "${CLICKHOUSE_CLUSTER_NAME}" ]; then
- export CLICKHOUSE_CLUSTER_NAME="default"
-fi
+export CLICKHOUSE_CLUSTER_NAME="${CLICKHOUSE_CLUSTER_NAME:-default}"
-# Construct the database URL
-if [ "$CLICKHOUSE_CLUSTER_ENABLED" == "false" ] ; then
- if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-migrations-table-engine=MergeTree"
+MIGRATION_MODE="clustered"
+MIGRATIONS_TABLE_ENGINE="ReplicatedMergeTree"
+CLICKHOUSE_CLUSTER_QUERY=""
+if [ "$CLICKHOUSE_CLUSTER_ENABLED" = "false" ]; then
+ MIGRATION_MODE="unclustered"
+ MIGRATIONS_TABLE_ENGINE="MergeTree"
+else
+ if [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then
+ CLICKHOUSE_CLUSTER_NAME_ENCODED="default"
else
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-migrations-table-engine=MergeTree"
+ require_node
+ if ! CLICKHOUSE_CLUSTER_NAME_ENCODED=$(node "$SCRIPT_DIR/prepare-migrations.mjs" encode-cluster-name "$CLICKHOUSE_CLUSTER_NAME"); then
+ exit 1
+ fi
fi
+ CLICKHOUSE_CLUSTER_QUERY="&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME_ENCODED}"
+fi
- # If SKIP_CONFIRM is set, automatically answer the confirmation prompt. Otherwise run interactively.
- if [ "$SKIP_CONFIRM" = "1" ] || [ "$SKIP_CONFIRM" = "true" ]; then
- printf 'y\n' | migrate -source file://clickhouse/migrations/unclustered -database "$DATABASE_URL" down
- else
- migrate -source file://clickhouse/migrations/unclustered -database "$DATABASE_URL" down
+CANONICAL_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/canonical"
+if [ ! -d "$CANONICAL_MIGRATIONS_DIRECTORY" ]; then
+ echo "Error: canonical ClickHouse migration templates are missing."
+ exit 1
+fi
+
+DELIVERED_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/$MIGRATION_MODE"
+USE_DELIVERED_MIGRATIONS="false"
+if [ -f "$DELIVERED_MIGRATIONS_DIRECTORY/0001_traces.up.sql" ]; then
+ if [ "$MIGRATION_MODE" = "unclustered" ] || [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then
+ USE_DELIVERED_MIGRATIONS="true"
fi
+fi
+
+MIGRATIONS_DIRECTORY_IS_TEMP="false"
+if [ "$USE_DELIVERED_MIGRATIONS" = "true" ]; then
+ MIGRATIONS_DIRECTORY="$DELIVERED_MIGRATIONS_DIRECTORY"
else
- if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"
- else
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"
+ require_node
+ if ! MIGRATIONS_DIRECTORY=$(node "$SCRIPT_DIR/prepare-migrations.mjs" render "$CANONICAL_MIGRATIONS_DIRECTORY" "$MIGRATION_MODE" "$CLICKHOUSE_CLUSTER_NAME"); then
+ exit 1
fi
+ MIGRATIONS_DIRECTORY_IS_TEMP="true"
+fi
- # If SKIP_CONFIRM is set, automatically answer the confirmation prompt. Otherwise run interactively.
- if [ "$SKIP_CONFIRM" = "1" ] || [ "$SKIP_CONFIRM" = "true" ]; then
- printf 'y\n' | migrate -source file://clickhouse/migrations/clustered -database "$DATABASE_URL" down
- else
- migrate -source file://clickhouse/migrations/clustered -database "$DATABASE_URL" down
+cleanup_migrations() {
+ if [ "$MIGRATIONS_DIRECTORY_IS_TEMP" = "true" ] && [ -n "${MIGRATIONS_DIRECTORY:-}" ]; then
+ rm -rf -- "$MIGRATIONS_DIRECTORY"
fi
-fi
\ No newline at end of file
+}
+trap cleanup_migrations 0
+
+# Construct the database URL
+DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true"
+if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then
+ DATABASE_URL="${DATABASE_URL}&secure=true&skip_verify=true"
+fi
+DATABASE_URL="${DATABASE_URL}${CLICKHOUSE_CLUSTER_QUERY}&x-migrations-table-engine=${MIGRATIONS_TABLE_ENGINE}"
+
+# If SKIP_CONFIRM is set, automatically answer the confirmation prompt. Otherwise run interactively.
+if [ "$SKIP_CONFIRM" = "1" ] || [ "$SKIP_CONFIRM" = "true" ]; then
+ printf 'y\n' | migrate -source "file://${MIGRATIONS_DIRECTORY}" -database "$DATABASE_URL" down
+else
+ migrate -source "file://${MIGRATIONS_DIRECTORY}" -database "$DATABASE_URL" down
+fi
diff --git a/packages/shared/clickhouse/scripts/prepare-migrations.mjs b/packages/shared/clickhouse/scripts/prepare-migrations.mjs
new file mode 100644
index 0000000000..335b94065e
--- /dev/null
+++ b/packages/shared/clickhouse/scripts/prepare-migrations.mjs
@@ -0,0 +1,337 @@
+#!/usr/bin/env node
+
+import {
+ cpSync,
+ existsSync,
+ mkdtempSync,
+ mkdirSync,
+ readFileSync,
+ readdirSync,
+ rmSync,
+ statSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { basename, join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const clusterClausePlaceholder = "{CLICKHOUSE_CLUSTER_CLAUSE}";
+const replicationPrefixPlaceholder = "{CLICKHOUSE_REPLICATION_PREFIX}";
+const historicalFinalNewlinesPattern =
+ /\{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:(0|2|clustered)\}\n$/;
+const anyHistoricalFinalNewlinesPattern =
+ /\{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:[^{}]*\}/;
+const clusteredOnlyPattern = /\{CLICKHOUSE_CLUSTERED_ONLY:([^{}]*)\}/g;
+const unclusteredOnlyPattern = /\{CLICKHOUSE_UNCLUSTERED_ONLY:([^{}]*)\}/g;
+const anyPlaceholderPattern = /\{CLICKHOUSE_[A-Z_]+(?::[^{}]*)?\}/;
+const historicalByteCompatibilityVersion = 47;
+const deliveredModes = ["clustered", "unclustered"];
+
+function countOccurrences(value, search) {
+ return value.split(search).length - 1;
+}
+
+function quoteClickHouseString(value) {
+ if (value.includes("\0")) {
+ throw new Error("ClickHouse cluster names must not contain NUL bytes");
+ }
+
+ return `'${value
+ .replaceAll("\\", "\\\\")
+ .replaceAll("'", "\\'")
+ .replaceAll("\n", "\\n")
+ .replaceAll("\r", "\\r")
+ .replaceAll("\t", "\\t")}'`;
+}
+
+function renderMigration(
+ source,
+ mode,
+ clusterName,
+ preserveHistoricalUnclusteredWhitespace,
+) {
+ const placeholderCount = countOccurrences(source, clusterClausePlaceholder);
+ const historicalFinalNewlines = source.match(historicalFinalNewlinesPattern);
+
+ if (placeholderCount === 0) {
+ throw new Error(`migration does not contain ${clusterClausePlaceholder}`);
+ }
+
+ if (/\bON\s+CLUSTER\b/i.test(source)) {
+ throw new Error(
+ `migration contains a hardcoded ON CLUSTER clause instead of ${clusterClausePlaceholder}`,
+ );
+ }
+
+ if (/\bReplicated[A-Za-z]*MergeTree\b/.test(source)) {
+ throw new Error(
+ `migration contains a hardcoded replicated engine instead of ${replicationPrefixPlaceholder}`,
+ );
+ }
+
+ if (
+ countOccurrences(source, ` ${clusterClausePlaceholder}`) !==
+ placeholderCount
+ ) {
+ throw new Error(
+ `${clusterClausePlaceholder} must include its preceding space`,
+ );
+ }
+
+ if (
+ anyHistoricalFinalNewlinesPattern.test(source) &&
+ !historicalFinalNewlines
+ ) {
+ throw new Error(
+ "CLICKHOUSE_HISTORICAL_FINAL_NEWLINES must appear once at the end of a migration and use 0, 2, or clustered",
+ );
+ }
+
+ const clusterClause =
+ mode === "clustered"
+ ? clusterName === "default"
+ ? "ON CLUSTER default"
+ : `ON CLUSTER ${quoteClickHouseString(clusterName)}`
+ : "";
+
+ const replicationPrefix = mode === "clustered" ? "Replicated" : "";
+ let rendered = source
+ .replaceAll(
+ ` ${clusterClausePlaceholder}`,
+ mode === "clustered" ? ` ${clusterClause}` : "",
+ )
+ .replaceAll(replicationPrefixPlaceholder, replicationPrefix)
+ .replace(clusteredOnlyPattern, mode === "clustered" ? "$1" : "");
+
+ if (mode === "unclustered" && preserveHistoricalUnclusteredWhitespace) {
+ rendered = rendered.replace(/[ \t]+\n/g, "\n").replace(/\n[ \t]*;/g, ";");
+ }
+
+ rendered = rendered
+ .replace(unclusteredOnlyPattern, mode === "unclustered" ? "$1" : "")
+ .replace(historicalFinalNewlinesPattern, (_, finalNewlines) => {
+ if (finalNewlines === "2") return "\n\n";
+ if (finalNewlines === "clustered" && mode === "clustered") return "\n";
+ return "";
+ });
+
+ if (anyPlaceholderPattern.test(rendered)) {
+ throw new Error("migration contains an unrendered ClickHouse placeholder");
+ }
+
+ const renderedClusterClauseCount = countOccurrences(rendered, "ON CLUSTER");
+ if (mode === "clustered" && renderedClusterClauseCount !== placeholderCount) {
+ throw new Error("not every cluster clause placeholder was rendered");
+ }
+
+ if (mode === "unclustered") {
+ if (renderedClusterClauseCount > 0) {
+ throw new Error("unclustered migration contains an ON CLUSTER clause");
+ }
+ if (/\bReplicated[A-Za-z]*MergeTree\b/.test(rendered)) {
+ throw new Error(
+ "unclustered migration contains a replicated table engine",
+ );
+ }
+ }
+
+ return rendered;
+}
+
+function validateMigrationFileSet(files) {
+ if (files.length === 0) {
+ throw new Error("no ClickHouse migration SQL files found");
+ }
+
+ const fileSet = new Set(files);
+ const filesByVersion = new Map();
+
+ for (const file of files) {
+ const match = /^(\d{4})_.+\.(up|down)\.sql$/.exec(file);
+ if (!match) {
+ throw new Error(`invalid ClickHouse migration filename: ${file}`);
+ }
+
+ const version = Number(match[1]);
+ if (version < 1) {
+ throw new Error(`invalid ClickHouse migration version in ${file}`);
+ }
+ filesByVersion.set(version, (filesByVersion.get(version) ?? 0) + 1);
+ const counterpart =
+ match[2] === "up"
+ ? file.replace(/\.up\.sql$/, ".down.sql")
+ : file.replace(/\.down\.sql$/, ".up.sql");
+ if (!fileSet.has(counterpart)) {
+ throw new Error(`missing migration counterpart for ${file}`);
+ }
+ }
+
+ const highestVersion = Math.max(...filesByVersion.keys());
+ for (let version = 1; version <= highestVersion; version += 1) {
+ const fileCount = filesByVersion.get(version);
+ if (!fileCount) {
+ throw new Error(`missing ClickHouse migration version ${version}`);
+ }
+ if (fileCount !== 2) {
+ throw new Error(
+ `ClickHouse migration version ${version} must have one up and one down file`,
+ );
+ }
+ }
+}
+
+export function encodeClusterName(clusterName) {
+ const clusterExpression =
+ clusterName === "default"
+ ? clusterName
+ : quoteClickHouseString(clusterName);
+
+ return encodeURIComponent(clusterExpression).replaceAll("'", "%27");
+}
+
+export function prepareMigrations(sourceDirectory, mode, clusterName) {
+ if (mode !== "clustered" && mode !== "unclustered") {
+ throw new Error(`unsupported ClickHouse migration mode: ${mode}`);
+ }
+ if (mode === "clustered" && clusterName.length === 0) {
+ throw new Error("ClickHouse cluster name must not be empty");
+ }
+
+ const sourcePath = resolve(sourceDirectory);
+ if (!statSync(sourcePath).isDirectory()) {
+ throw new Error(
+ `ClickHouse migration source is not a directory: ${sourcePath}`,
+ );
+ }
+
+ const migrationFiles = readdirSync(sourcePath, { withFileTypes: true })
+ .filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))
+ .map((entry) => entry.name)
+ .sort();
+ validateMigrationFileSet(migrationFiles);
+
+ const targetDirectory = mkdtempSync(
+ join(tmpdir(), `langfuse-clickhouse-${mode}-migrations-`),
+ );
+
+ try {
+ for (const migrationFile of migrationFiles) {
+ const source = readFileSync(join(sourcePath, migrationFile), "utf8");
+ const migrationVersion = Number(migrationFile.slice(0, 4));
+ const rendered = renderMigration(
+ source,
+ mode,
+ clusterName,
+ migrationVersion <= historicalByteCompatibilityVersion,
+ );
+ writeFileSync(join(targetDirectory, basename(migrationFile)), rendered);
+ }
+ } catch (error) {
+ rmSync(targetDirectory, { force: true, recursive: true });
+ throw error;
+ }
+
+ return targetDirectory;
+}
+
+function requireCanonicalMigrationDirectory(migrationsDirectory) {
+ const migrationsPath = resolve(migrationsDirectory);
+ const canonicalPath = join(migrationsPath, "canonical");
+
+ if (!existsSync(canonicalPath) || !statSync(canonicalPath).isDirectory()) {
+ throw new Error(
+ `ClickHouse migration directory must contain canonical/: ${migrationsPath}`,
+ );
+ }
+
+ return { canonicalPath, migrationsPath };
+}
+
+export function materializeMigrations(migrationsDirectory) {
+ const { canonicalPath, migrationsPath } =
+ requireCanonicalMigrationDirectory(migrationsDirectory);
+ const renderedDirectories = [];
+
+ try {
+ for (const mode of deliveredModes) {
+ renderedDirectories.push({
+ mode,
+ path: prepareMigrations(canonicalPath, mode, "default"),
+ });
+ }
+
+ mkdirSync(migrationsPath, { recursive: true });
+ for (const rendered of renderedDirectories) {
+ const targetDirectory = join(migrationsPath, rendered.mode);
+ rmSync(targetDirectory, { force: true, recursive: true });
+ cpSync(rendered.path, targetDirectory, { recursive: true });
+ }
+ } finally {
+ for (const rendered of renderedDirectories) {
+ rmSync(rendered.path, { force: true, recursive: true });
+ }
+ }
+
+ return deliveredModes.map((mode) => join(migrationsPath, mode));
+}
+
+export function cleanMaterializedMigrations(migrationsDirectory) {
+ const { migrationsPath } =
+ requireCanonicalMigrationDirectory(migrationsDirectory);
+
+ for (const mode of deliveredModes) {
+ rmSync(join(migrationsPath, mode), { force: true, recursive: true });
+ }
+}
+
+function printUsage() {
+ console.error(
+ "Usage: prepare-migrations.mjs render <source-directory> <clustered|unclustered> <cluster-name> | materialize <migrations-directory> | clean <migrations-directory> | encode-cluster-name <cluster-name>",
+ );
+}
+
+function main() {
+ const [command, ...args] = process.argv.slice(2);
+
+ if (command === "render" && args.length === 3) {
+ const [sourceDirectory, mode, clusterName] = args;
+ process.stdout.write(
+ `${prepareMigrations(sourceDirectory, mode, clusterName)}\n`,
+ );
+ return;
+ }
+
+ if (command === "encode-cluster-name" && args.length === 1) {
+ process.stdout.write(`${encodeClusterName(args[0])}\n`);
+ return;
+ }
+
+ if (command === "materialize" && args.length === 1) {
+ for (const directory of materializeMigrations(args[0])) {
+ process.stdout.write(`${directory}\n`);
+ }
+ return;
+ }
+
+ if (command === "clean" && args.length === 1) {
+ cleanMaterializedMigrations(args[0]);
+ return;
+ }
+
+ printUsage();
+ process.exitCode = 1;
+}
+
+if (
+ process.argv[1] &&
+ resolve(process.argv[1]) === fileURLToPath(import.meta.url)
+) {
+ try {
+ main();
+ } catch (error) {
+ console.error(
+ `Error: ${error instanceof Error ? error.message : String(error)}`,
+ );
+ process.exitCode = 1;
+ }
+}
diff --git a/packages/shared/clickhouse/scripts/up.sh b/packages/shared/clickhouse/scripts/up.sh
index 608a73f4d3..c9e0d8f03e 100644
--- a/packages/shared/clickhouse/scripts/up.sh
+++ b/packages/shared/clickhouse/scripts/up.sh
@@ -1,7 +1,9 @@
-#!/bin/bash
+#!/bin/sh
+
+SCRIPT_DIR=$(CDPATH= cd -- "$(dirname "$0")" && pwd)
# Load environment variables
-[ -f ../../.env ] && source ../../.env
+[ -f ../../.env ] && . ../../.env
# Check if CLICKHOUSE_URL is configured
if [ -z "${CLICKHOUSE_URL}" ]; then
@@ -32,7 +34,7 @@ if [ -z "${CLICKHOUSE_PASSWORD}" ]; then
fi
# Check if golang-migrate is installed
-if ! command -v migrate &> /dev/null
+if ! command -v migrate >/dev/null 2>&1
then
echo "Error: golang-migrate is not installed or not in PATH."
echo "Please install golang-migrate via 'brew install golang-migrate' to run this script."
@@ -40,33 +42,76 @@ then
exit 1
fi
+require_node() {
+ if ! command -v node >/dev/null 2>&1
+ then
+ echo "Error: node is not installed or not in PATH."
+ exit 1
+ fi
+}
+
# Ensure CLICKHOUSE_DB is set
-if [ -z "${CLICKHOUSE_DB}" ]; then
- export CLICKHOUSE_DB="default"
-fi
+export CLICKHOUSE_DB="${CLICKHOUSE_DB:-default}"
# Ensure CLICKHOUSE_CLUSTER_NAME is set
-if [ -z "${CLICKHOUSE_CLUSTER_NAME}" ]; then
- export CLICKHOUSE_CLUSTER_NAME="default"
-fi
+export CLICKHOUSE_CLUSTER_NAME="${CLICKHOUSE_CLUSTER_NAME:-default}"
-# Construct the database URL
-if [ "$CLICKHOUSE_CLUSTER_ENABLED" == "false" ] ; then
- if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-migrations-table-engine=MergeTree"
+MIGRATION_MODE="clustered"
+MIGRATIONS_TABLE_ENGINE="ReplicatedMergeTree"
+CLICKHOUSE_CLUSTER_QUERY=""
+if [ "$CLICKHOUSE_CLUSTER_ENABLED" = "false" ]; then
+ MIGRATION_MODE="unclustered"
+ MIGRATIONS_TABLE_ENGINE="MergeTree"
+else
+ if [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then
+ CLICKHOUSE_CLUSTER_NAME_ENCODED="default"
else
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-migrations-table-engine=MergeTree"
+ require_node
+ if ! CLICKHOUSE_CLUSTER_NAME_ENCODED=$(node "$SCRIPT_DIR/prepare-migrations.mjs" encode-cluster-name "$CLICKHOUSE_CLUSTER_NAME"); then
+ exit 1
+ fi
fi
+ CLICKHOUSE_CLUSTER_QUERY="&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME_ENCODED}"
+fi
+
+CANONICAL_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/canonical"
+if [ ! -d "$CANONICAL_MIGRATIONS_DIRECTORY" ]; then
+ echo "Error: canonical ClickHouse migration templates are missing."
+ exit 1
+fi
- # Execute the up command
- migrate -source file://clickhouse/migrations/unclustered -database "$DATABASE_URL" up
+DELIVERED_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/$MIGRATION_MODE"
+USE_DELIVERED_MIGRATIONS="false"
+if [ -f "$DELIVERED_MIGRATIONS_DIRECTORY/0001_traces.up.sql" ]; then
+ if [ "$MIGRATION_MODE" = "unclustered" ] || [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then
+ USE_DELIVERED_MIGRATIONS="true"
+ fi
+fi
+
+MIGRATIONS_DIRECTORY_IS_TEMP="false"
+if [ "$USE_DELIVERED_MIGRATIONS" = "true" ]; then
+ MIGRATIONS_DIRECTORY="$DELIVERED_MIGRATIONS_DIRECTORY"
else
-if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"
- else
- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"
+ require_node
+ if ! MIGRATIONS_DIRECTORY=$(node "$SCRIPT_DIR/prepare-migrations.mjs" render "$CANONICAL_MIGRATIONS_DIRECTORY" "$MIGRATION_MODE" "$CLICKHOUSE_CLUSTER_NAME"); then
+ exit 1
fi
+ MIGRATIONS_DIRECTORY_IS_TEMP="true"
+fi
- # Execute the up command
- migrate -source file://clickhouse/migrations/clustered -database "$DATABASE_URL" up
+cleanup_migrations() {
+ if [ "$MIGRATIONS_DIRECTORY_IS_TEMP" = "true" ] && [ -n "${MIGRATIONS_DIRECTORY:-}" ]; then
+ rm -rf -- "$MIGRATIONS_DIRECTORY"
+ fi
+}
+trap cleanup_migrations 0
+
+# Construct the database URL
+DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true"
+if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then
+ DATABASE_URL="${DATABASE_URL}&secure=true&skip_verify=true"
fi
+DATABASE_URL="${DATABASE_URL}${CLICKHOUSE_CLUSTER_QUERY}&x-migrations-table-engine=${MIGRATIONS_TABLE_ENGINE}"
+
+# Execute the up command against delivered or custom-rendered migrations.
+migrate -source "file://${MIGRATIONS_DIRECTORY}" -database "$DATABASE_URL" up
diff --git a/packages/shared/package.json b/packages/shared/package.json
index 4bccbbe9f8..a5e73bbb85 100644
--- a/packages/shared/package.json
+++ b/packages/shared/package.json
@@ -31,6 +31,10 @@
"import": "./dist/src/server/index.js",
"require": "./dist/src/server/index.js"
},
+ "./src/server/clickhouse/clickhouseIdentifiers": {
+ "import": "./dist/src/server/clickhouse/clickhouseIdentifiers.js",
+ "require": "./dist/src/server/clickhouse/clickhouseIdentifiers.js"
+ },
"./src/server/llm/llmText": {
"import": "./dist/src/server/llm/llmText.js",
"require": "./dist/src/server/llm/llmText.js"
@@ -137,8 +141,10 @@
"db:generate": "dotenv -e ../../.env -- npx prisma generate --no-hints",
"db:seed:examples": "dotenv -e ../../.env -- npx prisma db seed -- --environment examples",
"db:seed:load": "dotenv -e ../../.env -- npx prisma db seed -- --environment load",
- "ch:up": "bash clickhouse/scripts/up.sh",
- "ch:down": "bash clickhouse/scripts/down.sh",
+ "ch:migrations:materialize": "node clickhouse/scripts/prepare-migrations.mjs materialize clickhouse/migrations",
+ "ch:migrations:clean": "node clickhouse/scripts/prepare-migrations.mjs clean clickhouse/migrations",
+ "ch:up": "sh clickhouse/scripts/up.sh",
+ "ch:down": "sh clickhouse/scripts/down.sh",
"ch:drop": "bash clickhouse/scripts/drop.sh",
"ch:dev-tables": "bash clickhouse/scripts/dev-tables.sh",
"ch:reset": "pnpm run ch:down && pnpm run ch:up && pnpm run ch:seed && pnpm run ch:dev-tables",
diff --git a/packages/shared/src/server/clickhouse/clickhouseIdentifiers.ts b/packages/shared/src/server/clickhouse/clickhouseIdentifiers.ts
new file mode 100644
index 0000000000..db3c45fc11
--- /dev/null
+++ b/packages/shared/src/server/clickhouse/clickhouseIdentifiers.ts
@@ -0,0 +1,49 @@
+/**
+ * Quoting helpers for ClickHouse identifiers and string literals that have to
+ * be interpolated into SQL text rather than bound as query parameters.
+ */
+
+export function assertClickhouseIdentifier(value: string, label: string): void {
+ if (value.length === 0 || value.includes("\0")) {
+ throw new Error(`Invalid ClickHouse ${label}: ${value}`);
+ }
+}
+
+export function quoteClickhouseIdentifier(
+ value: string,
+ label: string,
+): string {
+ assertClickhouseIdentifier(value, label);
+ const escaped = value.replace(/\\/g, "\\\\").replace(/`/g, "\\`");
+ return `\`${escaped}\``;
+}
+
+export function quoteClickhouseString(value: string): string {
+ if (value.includes("\0")) {
+ throw new Error("Invalid ClickHouse string: NUL bytes are not supported");
+ }
+
+ return `'${value
+ .replace(/\\/g, "\\\\")
+ .replace(/'/g, "\\'")
+ .replace(/\n/g, "\\n")
+ .replace(/\r/g, "\\r")
+ .replace(/\t/g, "\\t")}'`;
+}
+
+/**
+ * Qualifies a table with its database.
+ *
+ * Required for `SYSTEM ... ON CLUSTER` statements: `AddDefaultDatabaseVisitor`
+ * only fills in the initiator's current database for `ASTAlterQuery`,
+ * `ASTQueryWithTableAndOutput` and `ASTRenameQuery`. `ASTSystemQuery` is not
+ * visited, so an unqualified name reaches each host's DDL worker unqualified
+ * and resolves against that server's default database (`default`) instead of
+ * `CLICKHOUSE_DB`.
+ */
+export function qualifiedClickhouseTableName(
+ database: string,
+ table: string,
+): string {
+ return `${quoteClickhouseIdentifier(database, "database")}.${quoteClickhouseIdentifier(table, "table")}`;
+}
diff --git a/packages/shared/src/server/clickhouse/fixtures/fake-migrate.sh b/packages/shared/src/server/clickhouse/fixtures/fake-migrate.sh
new file mode 100755
index 0000000000..e207fc6ac1
--- /dev/null
+++ b/packages/shared/src/server/clickhouse/fixtures/fake-migrate.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+set -eu
+
+: > "$CAPTURED_ARGS_PATH"
+for argument in "$@"; do
+ printf "%s\n" "$argument" >> "$CAPTURED_ARGS_PATH"
+done
+
+source_uri=""
+while [ "$#" -gt 0 ]; do
+ if [ "$1" = "-source" ]; then
+ source_uri="$2"
+ shift 2
+ else
+ shift
+ fi
+done
+
+source_directory="${source_uri#file://}"
+printf "%s" "$source_directory" > "$CAPTURED_SOURCE_PATH"
+cp -R "$source_directory/." "$CAPTURED_MIGRATIONS_DIRECTORY"
+
+if [ "${CAPTURE_STDIN:-false}" = "true" ]; then
+ cat > "$CAPTURED_STDIN_PATH"
+fi
+
+exit "${MIGRATE_EXIT_CODE:-0}"
diff --git a/packages/shared/src/server/clickhouse/prepareMigrations.test.ts b/packages/shared/src/server/clickhouse/prepareMigrations.test.ts
new file mode 100644
index 0000000000..7fbfff1b5c
--- /dev/null
+++ b/packages/shared/src/server/clickhouse/prepareMigrations.test.ts
@@ -0,0 +1,411 @@
+import { spawnSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import {
+ cpSync,
+ existsSync,
+ mkdtempSync,
+ mkdirSync,
+ readFileSync,
+ readdirSync,
+ rmSync,
+ symlinkSync,
+ writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join, resolve } from "node:path";
+import { afterEach, describe, expect, it } from "vitest";
+
+const sharedDirectory = resolve(__dirname, "../../..");
+const scriptsDirectory = join(sharedDirectory, "clickhouse/scripts");
+const fakeMigratePath = join(__dirname, "fixtures/fake-migrate.sh");
+const migrationSourceDirectory = join(
+ sharedDirectory,
+ "clickhouse/migrations/canonical",
+);
+const historicalMigrationCount = 94;
+const historicalMigrationVersion = 47;
+// Compatibility baselines for migrations 0001-0047 as shipped in each mode.
+const historicalClusteredHash =
+ "1f1cceaf0d89af2c78ec6f527dc9722900a572aa743f7650a83b277c7ae49fc2";
+const historicalUnclusteredHash =
+ "26de8baf42d0690445c9e8ec11951b1c85696b63d8bd47845c695803fced0aa3";
+const temporaryDirectories: string[] = [];
+
+type MigrationMode = "clustered" | "unclustered";
+type MigrationScript = "up" | "down";
+type MigrationLayout = "delivered" | "source";
+
+type MigrationRun = {
+ args: string[];
+ capturedMigrationsDirectory: string;
+ databaseUrl: string;
+ renderedMigrationsDirectory: string;
+ status: number | null;
+ stdin: string;
+};
+
+afterEach(() => {
+ for (const directory of temporaryDirectories.splice(0)) {
+ rmSync(directory, { force: true, recursive: true });
+ }
+});
+
+function createTemporaryDirectory(prefix: string): string {
+ const directory = mkdtempSync(join(tmpdir(), prefix));
+ temporaryDirectories.push(directory);
+ return directory;
+}
+
+function runMigrationScript({
+ clusterName,
+ migrationLayout = "source",
+ migrateExitCode = 0,
+ mode,
+ script = "up",
+ ssl = false,
+}: {
+ clusterName?: string;
+ migrationLayout?: MigrationLayout;
+ migrateExitCode?: number;
+ mode: MigrationMode;
+ script?: MigrationScript;
+ ssl?: boolean;
+}): MigrationRun {
+ const root = createTemporaryDirectory("langfuse-clickhouse-migration-test-");
+ const binaryDirectory = join(root, "bin");
+ const workingDirectory = join(root, "shared");
+ const temporaryDirectory = join(root, "tmp");
+ const capturedMigrationsDirectory = join(root, "captured-migrations");
+ const capturedArgsPath = join(root, "captured-args");
+ const capturedSourcePath = join(root, "captured-source");
+ const capturedStdinPath = join(root, "captured-stdin");
+
+ mkdirSync(binaryDirectory);
+ mkdirSync(workingDirectory);
+ mkdirSync(temporaryDirectory);
+ mkdirSync(capturedMigrationsDirectory);
+ const clickhouseDirectory = join(workingDirectory, "clickhouse");
+ if (migrationLayout === "source") {
+ symlinkSync(join(sharedDirectory, "clickhouse"), clickhouseDirectory);
+ } else {
+ const migrationsDirectory = join(clickhouseDirectory, "migrations");
+ mkdirSync(migrationsDirectory, { recursive: true });
+ cpSync(scriptsDirectory, join(clickhouseDirectory, "scripts"), {
+ recursive: true,
+ });
+ symlinkSync(
+ migrationSourceDirectory,
+ join(migrationsDirectory, "canonical"),
+ );
+
+ for (const deliveredMode of ["clustered", "unclustered"] as const) {
+ const rendered = spawnSync(
+ process.execPath,
+ [
+ join(scriptsDirectory, "prepare-migrations.mjs"),
+ "render",
+ migrationSourceDirectory,
+ deliveredMode,
+ "default",
+ ],
+ { encoding: "utf8" },
+ );
+ expect(rendered.status, rendered.stderr).toBe(0);
+ const renderedDirectory = rendered.stdout.trim();
+ temporaryDirectories.push(renderedDirectory);
+ cpSync(renderedDirectory, join(migrationsDirectory, deliveredMode), {
+ recursive: true,
+ });
+ }
+ }
+
+ symlinkSync(fakeMigratePath, join(binaryDirectory, "migrate"));
+
+ const env: NodeJS.ProcessEnv = {
+ ...process.env,
+ CAPTURED_ARGS_PATH: capturedArgsPath,
+ CAPTURED_MIGRATIONS_DIRECTORY: capturedMigrationsDirectory,
+ CAPTURED_SOURCE_PATH: capturedSourcePath,
+ CAPTURED_STDIN_PATH: capturedStdinPath,
+ CAPTURE_STDIN: script === "down" ? "true" : "false",
+ CLICKHOUSE_CLUSTER_ENABLED: mode === "clustered" ? "true" : "false",
+ CLICKHOUSE_DB: "default",
+ CLICKHOUSE_MIGRATION_SSL: ssl ? "true" : "false",
+ CLICKHOUSE_MIGRATION_URL: "clickhouse://localhost:9000",
+ CLICKHOUSE_PASSWORD: "password",
+ CLICKHOUSE_URL: "http://localhost:8123",
+ CLICKHOUSE_USER: "user",
+ MIGRATE_EXIT_CODE: String(migrateExitCode),
+ PATH: [
+ binaryDirectory,
+ ...(migrationLayout === "source" ? [dirname(process.execPath)] : []),
+ "/usr/bin",
+ "/bin",
+ ].join(":"),
+ SKIP_CONFIRM: script === "down" ? "true" : "false",
+ TMPDIR: temporaryDirectory,
+ };
+ if (clusterName === undefined) {
+ delete env.CLICKHOUSE_CLUSTER_NAME;
+ } else {
+ env.CLICKHOUSE_CLUSTER_NAME = clusterName;
+ }
+
+ const migrationScriptPath =
+ migrationLayout === "delivered"
+ ? join(clickhouseDirectory, "scripts", `${script}.sh`)
+ : join(scriptsDirectory, `${script}.sh`);
+ const execution = spawnSync("sh", [migrationScriptPath], {
+ cwd: workingDirectory,
+ env,
+ encoding: "utf8",
+ });
+ expect(execution.status, execution.stderr).toBe(migrateExitCode);
+
+ const args = readFileSync(capturedArgsPath, "utf8").trimEnd().split("\n");
+ const renderedMigrationsDirectory = readFileSync(capturedSourcePath, "utf8");
+ const databaseArgumentIndex = args.indexOf("-database");
+
+ return {
+ args,
+ capturedMigrationsDirectory,
+ databaseUrl: args[databaseArgumentIndex + 1] ?? "",
+ renderedMigrationsDirectory,
+ status: execution.status,
+ stdin: existsSync(capturedStdinPath)
+ ? readFileSync(capturedStdinPath, "utf8")
+ : "",
+ };
+}
+
+function readMigration(run: MigrationRun, file: string): string {
+ return readFileSync(join(run.capturedMigrationsDirectory, file), "utf8");
+}
+
+function readHistoricalSql(run: MigrationRun): string {
+ return historicalMigrationFiles(run.capturedMigrationsDirectory)
+ .map((file) => readMigration(run, file))
+ .join("\n");
+}
+
+function countMatches(value: string, pattern: RegExp): number {
+ return [...value.matchAll(pattern)].length;
+}
+
+function historicalMigrationFiles(directory: string): string[] {
+ return readdirSync(directory)
+ .filter(
+ (file) =>
+ file.endsWith(".sql") &&
+ Number(file.slice(0, 4)) <= historicalMigrationVersion,
+ )
+ .sort();
+}
+
+function hashHistoricalMigrations(directory: string): string {
+ const hash = createHash("sha256");
+ const files = historicalMigrationFiles(directory);
+ expect(files).toHaveLength(historicalMigrationCount);
+
+ for (const file of files) {
+ hash.update(file);
+ hash.update("\0");
+ hash.update(readFileSync(join(directory, file), "utf8"));
+ hash.update("\0");
+ }
+
+ return hash.digest("hex");
+}
+
+describe("ClickHouse migration preparation", () => {
+ it("renders and URL-encodes a custom cluster name through POSIX sh", () => {
+ const clusterName = "eu-west' & blue";
+ const run = runMigrationScript({ clusterName, mode: "clustered" });
+ const sql = readMigration(run, "0001_traces.up.sql");
+ const allSql = readHistoricalSql(run);
+
+ expect(sql).toContain("ON CLUSTER 'eu-west\\' & blue'");
+ expect(sql).toContain("ENGINE = ReplicatedReplacingMergeTree");
+ expect(sql).not.toContain("ON CLUSTER default");
+ expect(countMatches(allSql, /ON CLUSTER 'eu-west\\' & blue'/g)).toBe(204);
+ expect(run.databaseUrl).toContain(
+ `&x-cluster-name=${encodeURIComponent(`'eu-west\\' & blue'`).replaceAll("'", "%27")}`,
+ );
+ expect(run.databaseUrl).toContain(
+ "&x-migrations-table-engine=ReplicatedMergeTree",
+ );
+ expect(run.args.at(-1)).toBe("up");
+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(false);
+ });
+
+ it("preserves the historical default-cluster migration bytes", () => {
+ const run = runMigrationScript({ mode: "clustered", ssl: true });
+ const allSql = readHistoricalSql(run);
+
+ expect(hashHistoricalMigrations(run.capturedMigrationsDirectory)).toBe(
+ historicalClusteredHash,
+ );
+ expect(countMatches(allSql, /\bON CLUSTER default\b/g)).toBe(204);
+ expect(countMatches(allSql, /\balter_sync = 2\b/g)).toBe(85);
+ expect(countMatches(allSql, /\bmutations_sync = 2\b/g)).toBe(19);
+ expect(countMatches(allSql, /\bReplicated[A-Za-z]*MergeTree\b/g)).toBe(10);
+ expect(run.databaseUrl).toContain("&x-cluster-name=default");
+ expect(run.databaseUrl).toContain("&secure=true&skip_verify=true");
+ });
+
+ it("preserves the historical unclustered migration bytes", () => {
+ const run = runMigrationScript({
+ clusterName: "ignored-cluster",
+ mode: "unclustered",
+ });
+ const allSql = readHistoricalSql(run);
+
+ expect(hashHistoricalMigrations(run.capturedMigrationsDirectory)).toBe(
+ historicalUnclusteredHash,
+ );
+ expect(allSql).not.toMatch(/\bON\s+CLUSTER\b/);
+ expect(allSql).not.toMatch(/\bReplicated[A-Za-z]*MergeTree\b/);
+ expect(countMatches(allSql, /\balter_sync = 2\b/g)).toBe(6);
+ expect(countMatches(allSql, /\bmutations_sync = 2\b/g)).toBe(18);
+ expect(readMigration(run, "0015_add_scores_trace_index.up.sql")).toContain(
+ "SETTINGS mutations_sync = 2",
+ );
+ expect(readMigration(run, "0033_add_tool_call_columns.up.sql")).toContain(
+ "SETTINGS alter_sync = 2",
+ );
+ expect(
+ readMigration(run, "0047_add_eval_execution_columns.up.sql"),
+ ).not.toMatch(/(?:alter|mutations)_sync/);
+ expect(run.databaseUrl).not.toContain("x-cluster-name");
+ expect(run.databaseUrl).toContain("&x-migrations-table-engine=MergeTree");
+ });
+
+ it.each(["clustered", "unclustered"] as const)(
+ "uses the delivered %s migrations without runtime rendering",
+ (mode) => {
+ const run = runMigrationScript({ migrationLayout: "delivered", mode });
+
+ expect(run.renderedMigrationsDirectory).toContain(`/migrations/${mode}`);
+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(true);
+ expect(hashHistoricalMigrations(run.capturedMigrationsDirectory)).toBe(
+ mode === "clustered"
+ ? historicalClusteredHash
+ : historicalUnclusteredHash,
+ );
+ },
+ );
+
+ it("materializes and cleans the delivered migration trees", () => {
+ const targetDirectory = createTemporaryDirectory(
+ "langfuse-clickhouse-materialized-migrations-test-",
+ );
+ symlinkSync(migrationSourceDirectory, join(targetDirectory, "canonical"));
+ const materialize = spawnSync(
+ process.execPath,
+ [
+ join(scriptsDirectory, "prepare-migrations.mjs"),
+ "materialize",
+ targetDirectory,
+ ],
+ { encoding: "utf8" },
+ );
+
+ expect(materialize.status, materialize.stderr).toBe(0);
+ for (const mode of ["clustered", "unclustered"] as const) {
+ const modeDirectory = join(targetDirectory, mode);
+ expect(existsSync(modeDirectory)).toBe(true);
+ expect(hashHistoricalMigrations(modeDirectory)).toBe(
+ mode === "clustered"
+ ? historicalClusteredHash
+ : historicalUnclusteredHash,
+ );
+ expect(
+ readFileSync(
+ join(modeDirectory, "0047_add_eval_execution_columns.up.sql"),
+ "utf8",
+ ),
+ ).not.toMatch(/\{CLICKHOUSE_[A-Z_]+/);
+ }
+
+ const clean = spawnSync(
+ process.execPath,
+ [
+ join(scriptsDirectory, "prepare-migrations.mjs"),
+ "clean",
+ targetDirectory,
+ ],
+ { encoding: "utf8" },
+ );
+
+ expect(clean.status, clean.stderr).toBe(0);
+ expect(existsSync(join(targetDirectory, "clustered"))).toBe(false);
+ expect(existsSync(join(targetDirectory, "unclustered"))).toBe(false);
+ });
+
+ it("renders down migrations, confirms non-interactively, and cleans up", () => {
+ const run = runMigrationScript({ mode: "unclustered", script: "down" });
+
+ expect(run.args.at(-1)).toBe("down");
+ expect(run.stdin).toBe("y\n");
+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(false);
+ });
+
+ it("propagates migrate failures and still cleans up", () => {
+ const run = runMigrationScript({
+ migrateExitCode: 23,
+ mode: "clustered",
+ });
+
+ expect(run.status).toBe(23);
+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(false);
+ });
+
+ it("rejects hardcoded cluster clauses in canonical migrations", () => {
+ const sourceDirectory = createTemporaryDirectory(
+ "langfuse-clickhouse-invalid-migrations-",
+ );
+ const invalidSql =
+ "CREATE TABLE test {CLICKHOUSE_CLUSTER_CLAUSE} () ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}MergeTree; ON CLUSTER default;";
+ writeFileSync(join(sourceDirectory, "0001_test.up.sql"), invalidSql);
+ writeFileSync(join(sourceDirectory, "0001_test.down.sql"), invalidSql);
+
+ const result = spawnSync(
+ "node",
+ [
+ join(scriptsDirectory, "prepare-migrations.mjs"),
+ "render",
+ sourceDirectory,
+ "clustered",
+ "default",
+ ],
+ { encoding: "utf8" },
+ );
+
+ expect(result.status).toBe(1);
+ expect(result.stderr).toContain("hardcoded ON CLUSTER clause");
+ });
+
+ it("keeps every canonical migration renderable in both modes", () => {
+ for (const mode of ["clustered", "unclustered"] as const) {
+ const result = spawnSync(
+ "node",
+ [
+ join(scriptsDirectory, "prepare-migrations.mjs"),
+ "render",
+ migrationSourceDirectory,
+ mode,
+ "default",
+ ],
+ { encoding: "utf8" },
+ );
+
+ expect(result.status, result.stderr).toBe(0);
+ const renderedDirectory = result.stdout.trim();
+ temporaryDirectories.push(renderedDirectory);
+ const renderedSql = historicalMigrationFiles(renderedDirectory)
+ .map((file) => readFileSync(join(renderedDirectory, file), "utf8"))
+ .join("\n");
+ expect(renderedSql).not.toMatch(/\{CLICKHOUSE_[A-Z_]+/);
+ }
+ });
+});
diff --git a/packages/shared/src/server/clickhouse/queryTracking.test.ts b/packages/shared/src/server/clickhouse/queryTracking.test.ts
index 9fca4f5656..1185d29ffd 100644
--- a/packages/shared/src/server/clickhouse/queryTracking.test.ts
+++ b/packages/shared/src/server/clickhouse/queryTracking.test.ts
@@ -1,4 +1,4 @@
-import { describe, expect, it, vi } from "vitest";
+import { beforeEach, describe, expect, it, vi } from "vitest";
const mocks = vi.hoisted(() => ({
env: {
@@ -13,9 +13,30 @@ vi.mock("../repositories", () => ({ queryClickhouse: vi.fn() }));
import { systemTableRef } from "./queryTracking";
describe("systemTableRef", () => {
+ beforeEach(() => {
+ mocks.env.CLICKHOUSE_CLUSTER_ENABLED = "true";
+ mocks.env.CLICKHOUSE_CLUSTER_NAME = "default";
+ });
+
it("queries all query log tables on every replica in clustered mode", () => {
expect(systemTableRef("system.query_log")).toBe(
"clusterAllReplicas('default', merge(system, '^query_log*'))",
);
});
+
+ it("escapes custom cluster names before interpolating them into SQL", () => {
+ mocks.env.CLICKHOUSE_CLUSTER_NAME = "eu-west'\\blue\n";
+
+ expect(systemTableRef("system.processes")).toBe(
+ "clusterAllReplicas('eu-west\\'\\\\blue\\n', 'system.processes')",
+ );
+ });
+
+ it("rejects cluster names containing NUL bytes", () => {
+ mocks.env.CLICKHOUSE_CLUSTER_NAME = "invalid\0cluster";
+
+ expect(() => systemTableRef("system.processes")).toThrow(
+ "Invalid ClickHouse string",
+ );
+ });
});
diff --git a/packages/shared/src/server/clickhouse/queryTracking.ts b/packages/shared/src/server/clickhouse/queryTracking.ts
index 543866edfe..4052d82862 100644
--- a/packages/shared/src/server/clickhouse/queryTracking.ts
+++ b/packages/shared/src/server/clickhouse/queryTracking.ts
@@ -1,5 +1,6 @@
import { env } from "../../env";
import { queryClickhouse } from "../repositories";
+import { quoteClickhouseString } from "./clickhouseIdentifiers";
// ============================================================================
// Types
@@ -24,10 +25,11 @@ export function systemTableRef(
table: "system.processes" | "system.query_log",
): string {
if (env.CLICKHOUSE_CLUSTER_ENABLED === "true") {
+ const clusterName = quoteClickhouseString(env.CLICKHOUSE_CLUSTER_NAME);
if (table === "system.query_log") {
- return `clusterAllReplicas('${env.CLICKHOUSE_CLUSTER_NAME}', merge(system, '^query_log*'))`;
+ return `clusterAllReplicas(${clusterName}, merge(system, '^query_log*'))`;
}
- return `clusterAllReplicas('${env.CLICKHOUSE_CLUSTER_NAME}', '${table}')`;
+ return `clusterAllReplicas(${clusterName}, '${table}')`;
}
return table;
}
diff --git a/packages/shared/src/server/index.ts b/packages/shared/src/server/index.ts
index a13eb35342..8b56a4ce87 100644
--- a/packages/shared/src/server/index.ts
+++ b/packages/shared/src/server/index.ts
@@ -76,6 +76,7 @@ export {
} from "./clickhouse/compatibility";
export * from "./clickhouse/schemaUtils";
export * from "./clickhouse/schema";
+export * from "./clickhouse/clickhouseIdentifiers";
export * from "./clickhouse/queryTracking";
export * from "./clickhouse/queryTags";
export * from "./repositories/definitions";
diff --git a/scripts/release-preflight.sh b/scripts/release-preflight.sh
index 9e8dbd2331..7a0a69a97f 100755
--- a/scripts/release-preflight.sh
+++ b/scripts/release-preflight.sh
@@ -86,12 +86,19 @@ collect_prisma_migrations_not_in_production() {
| sort -u
}
-collect_clickhouse_migrations_not_in_production() {
- git diff --name-only origin/production..HEAD -- packages/shared/clickhouse/migrations \
- | sed -E 's#.*/([0-9]+_[^.]+)\.(up|down)\.sql#\1#' \
+collect_clickhouse_migration_names_at_ref() {
+ local ref="$1"
+ git ls-tree -r --name-only "$ref" -- packages/shared/clickhouse/migrations \
+ | sed -nE 's#.*/([0-9]+_[^.]+)\.(up|down)\.sql$#\1#p' \
| sort -u
}
+collect_clickhouse_migrations_not_in_production() {
+ comm -23 \
+ <(collect_clickhouse_migration_names_at_ref HEAD) \
+ <(collect_clickhouse_migration_names_at_ref origin/production)
+}
+
confirm_migrations_are_applied() {
local prisma_migrations="$1"
local clickhouse_migrations="$2"
diff --git a/web/Dockerfile b/web/Dockerfile
index 1691bb57b5..87597bc791 100644
--- a/web/Dockerfile
+++ b/web/Dockerfile
@@ -124,6 +124,12 @@ ENV SENTRY_RELEASE=$NEXT_PUBLIC_BUILD_ID
# Copy source code of isolated subworkspace
COPY --from=pruner /app/out/full/ .
+# Keep the documented direct golang-migrate recovery paths available in the
+# production image while maintaining canonical/ as the only authored tree.
+RUN set -eu; \
+ node ./packages/shared/clickhouse/scripts/prepare-migrations.mjs materialize ./packages/shared/clickhouse/migrations; \
+ chmod 755 ./packages/shared/clickhouse/migrations/clustered ./packages/shared/clickhouse/migrations/unclustered
+
# remove middleware.ts if it exists - not needed in self-hosted environments
RUN rm -f ./web/src/middleware.ts
diff --git a/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts b/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts
index d25f5030ab..bf033c0716 100644
--- a/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts
+++ b/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts
@@ -1,4 +1,9 @@
-import { logger, queryClickhouse, sleep } from "@langfuse/shared/src/server";
+import {
+ logger,
+ queryClickhouse,
+ quoteClickhouseString,
+ sleep,
+} from "@langfuse/shared/src/server";
import { env } from "../env";
import {
BaseChunkTodo,
@@ -145,7 +150,7 @@ export default class BackfillEventsFullFromObservations extends ChunkedClickhous
sum(future_parts) AS total_future_parts,
countIf(is_session_expired) AS expired_replicas,
countIf(is_readonly) AS readonly_replicas
- FROM clusterAllReplicas('${cluster}', 'system.replicas')
+ FROM clusterAllReplicas(${quoteClickhouseString(cluster)}, 'system.replicas')
WHERE database = currentDatabase()
AND table = '${PID_TID_SORTING_TABLE}'
`,
diff --git a/worker/src/utils/clickhouseIdentifiers.ts b/worker/src/utils/clickhouseIdentifiers.ts
index 9d21b9d9f7..0467bda9f8 100644
--- a/worker/src/utils/clickhouseIdentifiers.ts
+++ b/worker/src/utils/clickhouseIdentifiers.ts
@@ -1,41 +1,6 @@
-/**
- * Quoting helpers for ClickHouse identifiers that have to be interpolated into
- * SQL text rather than bound as query parameters — table/database/cluster names
- * in DDL and SYSTEM statements cannot be parameterized.
- */
-
-export function assertClickhouseIdentifier(value: string, label: string): void {
- if (value.length === 0 || value.includes("\0")) {
- throw new Error(`Invalid ClickHouse ${label}: ${value}`);
- }
-}
-
-export function quoteClickhouseIdentifier(
- value: string,
- label: string,
-): string {
- assertClickhouseIdentifier(value, label);
- const escaped = value.replace(/\\/g, "\\\\").replace(/`/g, "\\`");
- return `\`${escaped}\``;
-}
-
-export function quoteClickhouseString(value: string): string {
- return `'${value.replace(/\\/g, "\\\\").replace(/'/g, "\\'")}'`;
-}
-
-/**
- * Qualifies a table with its database.
- *
- * Required for `SYSTEM ... ON CLUSTER` statements: `AddDefaultDatabaseVisitor`
- * only fills in the initiator's current database for `ASTAlterQuery`,
- * `ASTQueryWithTableAndOutput` and `ASTRenameQuery`. `ASTSystemQuery` is not
- * visited, so an unqualified name reaches each host's DDL worker unqualified
- * and resolves against that server's default database (`default`) instead of
- * `CLICKHOUSE_DB`.
- */
-export function qualifiedClickhouseTableName(
- database: string,
- table: string,
-): string {
- return `${quoteClickhouseIdentifier(database, "database")}.${quoteClickhouseIdentifier(table, "table")}`;
-}
+export {
+ assertClickhouseIdentifier,
+ qualifiedClickhouseTableName,
+ quoteClickhouseIdentifier,
+ quoteClickhouseString,
+} from "@langfuse/shared/src/server/clickhouse/clickhouseIdentifiers";