🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat: support custom ClickHouse cluster migrations (#17015)
Valery Meleshkin5b1ff5ffc1ad
diff --git a/.agents/AGENTS.md b/.agents/AGENTS.mdindex da853d6c75..e97c35a964 100644--- a/.agents/AGENTS.md+++ b/.agents/AGENTS.md@@ -87,8 +87,9 @@ langfuse/- High-signal shared entry points:- Domain models: `packages/shared/src/domain/{observations,traces,scores}.ts`- Postgres schema: `packages/shared/prisma/schema.prisma`- - ClickHouse migrations:- `packages/shared/clickhouse/migrations/{clustered,unclustered}/*.sql`+ - Canonical ClickHouse migration templates (rendered for clustered and+ unclustered installs):+ `packages/shared/clickhouse/migrations/canonical/*.sql`- Architecture principles live in `.agents/ARCHITECTURE_PRINCIPLES.md`.## Core Commandsdiff --git a/.agents/skills/clickhouse-best-practices/SKILL.md b/.agents/skills/clickhouse-best-practices/SKILL.mdindex 87b573e219..7e47836377 100644--- a/.agents/skills/clickhouse-best-practices/SKILL.md+++ b/.agents/skills/clickhouse-best-practices/SKILL.md@@ -46,28 +46,38 @@ Comprehensive guidance for ClickHouse covering schema design, query optimizationrepository layer then reads baggage via `normalizeClickHouseQueryTags(...)`and writes it to `log_comment`. Prefer setting attribution at entry pointsrather than passing tags through every repository call.-- In `packages/shared/clickhouse/migrations/clustered/**`, every metadata- `ALTER` (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) must end with- `SETTINGS alter_sync = 2`, and every mutation-creating `ALTER`- (`MATERIALIZE …`, `UPDATE`, `DELETE`) with `SETTINGS mutations_sync = 2`.+- `packages/shared/clickhouse/migrations/canonical/**` is the single canonical+ template tree rendered for clustered and unclustered installs. Put+ `{CLICKHOUSE_CLUSTER_CLAUSE}` at every cluster-aware DDL position. Use+ `{CLICKHOUSE_REPLICATION_PREFIX}` only for engines that deliberately differ+ by mode; some tables intentionally stay non-replicated in both modes.+- Every metadata `ALTER` (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) in a new+ canonical migration must include+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}`, and every+ mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`) must include+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2}`.This applies to a file holding a single `ALTER` too — the race is acrossmigration files, not within one. `alter_sync` defaults to `1`, so thestatement returns as soon as the initiating replica has bumped the table'smetadata version in Keeper; golang-migrate then opens the next fileimmediately, and its first `ALTER` on that table can land on a replica stillon the previous version. ClickHouse refuses to queue it and aborts the whole- run with `code 517 … metadata version on replica is N, while common metadata- is N+1`. Note that `mutations_sync` does not substitute for `alter_sync`: it- governs when mutations finish, not metadata propagation. The matching- `unclustered/` file runs against plain `MergeTree` and does not need (and- should not duplicate) these settings.+ run with `code 517` because the replica metadata version is behind the common+ metadata version. Note that `mutations_sync` does not substitute for `alter_sync`: it+ governs when mutations finish, not metadata propagation. The renderer omits+ these fragments for unclustered `MergeTree` migrations. Use+ `{CLICKHOUSE_UNCLUSTERED_ONLY:...}` only for a deliberate mode-specific+ difference. Do not retrofit synchronization settings into already-shipped+ migrations merely to normalize them; the historical compatibility test+ intentionally protects their existing output.- Never use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` /`EXCHANGE TABLES`) in ClickHouse migrations. The atomic replace requires`renameat2` filesystem support, which NFS-backed self-hosted deployments(e.g. ClickHouse data on AWS EFS) lack — the migration fails and thedeployment aborts on startup (GitHub issue #14906). Redefine a plain view as- two statements in the same migration file: `DROP VIEW IF EXISTS <name>- [ON CLUSTER default];` then `CREATE VIEW <name> [ON CLUSTER default] AS …`.+ two statements in the same migration file. First use+ `DROP VIEW IF EXISTS <name> {CLICKHOUSE_CLUSTER_CLAUSE};`, then+ `CREATE VIEW <name> {CLICKHOUSE_CLUSTER_CLAUSE} AS …`.The migration runner passes `x-multi-statement=true` and golang-migratesplits files on `;` without parsing SQL, so keep semicolons out of commentsand string literals. Keep every statement idempotent@@ -78,13 +88,13 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization- Never drop-and-recreate a materialized view whose source table receives liveinserts: every row inserted between `DROP` and `CREATE` is silently andpermanently missing from the target table. Change an MV's SELECT with- `ALTER TABLE <mv> [ON CLUSTER default] MODIFY QUERY <select>`, which swaps+ `ALTER TABLE <mv> {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY <select>`, which swapsthe transformation without interrupting ingestion. When the change addscolumns, `ALTER` the target table(s) first (`ADD COLUMN IF NOT EXISTS …`),- then `MODIFY QUERY`; in `clustered/` files those target-table `ALTER`s must- carry `SETTINGS alter_sync = 2` so no host applies the new MV query before- its target replica has the new columns. `MODIFY QUERY` is only viable for- TO-table MVs (all Langfuse MVs use `TO`).+ then `MODIFY QUERY`; those target-table `ALTER`s must carry the+ clustered-only `alter_sync` template fragment so no host applies the new MV+ query before its target replica has the new columns. `MODIFY QUERY` is only+ viable for TO-table MVs (all Langfuse MVs use `TO`).---@@ -111,7 +121,7 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization- [ ] LowCardinality applied to appropriate string columns- [ ] Partition key cardinality bounded (100-1,000 values)- [ ] ReplacingMergeTree has version column if used-- [ ] Every metadata ALTER in a clustered migration ends with `SETTINGS alter_sync = 2` — including files with a single ALTER, since the next migration file is what breaks — and every `MATERIALIZE …` / `UPDATE` / `DELETE` with `SETTINGS mutations_sync = 2`; `mutations_sync` is not a substitute for `alter_sync`; unclustered mirror has none+- [ ] Every metadata ALTER in a new canonical migration includes `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}` — including files with a single ALTER, since the next migration file is what breaks — and every `MATERIALIZE …` / `UPDATE` / `DELETE` includes the corresponding `mutations_sync` fragment; `mutations_sync` is not a substitute for `alter_sync`; do not normalize already-shipped migration output; both rendered modes pass `prepareMigrations.test.ts`- [ ] No `CREATE OR REPLACE VIEW/TABLE` or `EXCHANGE TABLES` in migrations (breaks NFS/EFS self-hosting); plain views are redefined via `DROP VIEW IF EXISTS` + `CREATE VIEW` in the same file- [ ] Materialized views are never dropped and recreated while their source table takes inserts; SELECT changes go through `ALTER TABLE <mv> MODIFY QUERY` after the target-table `ALTER`sdiff --git a/.agents/skills/code-review/references/review-checklist.md b/.agents/skills/code-review/references/review-checklist.mdindex 61f5c9aa25..22b3f88617 100644--- a/.agents/skills/code-review/references/review-checklist.md+++ b/.agents/skills/code-review/references/review-checklist.md@@ -6,11 +6,32 @@ This is the canonical shared review checklist for Langfuse.### ClickHouse-- ClickHouse migrations in the `packages/shared/clickhouse/migrations/clustered` directory should include `ON CLUSTER default` and should use `Replicated` merge tree table types.- - E.g. `ReplacingMergeTree` is likely an error while `ReplicatedReplacingMergeTree` would be correct in most cases.-- ClickHouse migrations in the `packages/shared/clickhouse/migrations/unclustered` directory must not include `ON CLUSTER` statements and must not use `Replicated` merge tree table types.-- Migrations in `packages/shared/clickhouse/migrations/clustered` should match their counterparts in `packages/shared/clickhouse/migrations/unclustered` aside from the restrictions listed above.-- Every metadata `ALTER` (`ADD`/`DROP`/`MODIFY COLUMN`, `ADD`/`DROP INDEX`) in a `clustered` migration must end with `SETTINGS alter_sync = 2`, including files that contain only one `ALTER` — the race is across migration files, not within one. `alter_sync` defaults to `1`, so without it the statement returns once the initiating replica has bumped the table's metadata version, and the next migration file's first `ALTER` on that table can hit a replica still on the old version, aborting the run with `code 517 … Looks like this replica doesn't catchup with latest ALTER query updates`. `SETTINGS mutations_sync = 2` does not substitute for it: that setting governs when mutations finish, not metadata propagation. The `unclustered` counterpart runs against plain `MergeTree` and does not need (and should not duplicate) either setting.+- `packages/shared/clickhouse/migrations/canonical` is the single canonical+ migration-template tree for both deployment modes. Every cluster-aware DDL+ position must use `{CLICKHOUSE_CLUSTER_CLAUSE}` instead of a literal+ `ON CLUSTER` clause.+- Use `{CLICKHOUSE_REPLICATION_PREFIX}` only for a table engine that should be+ replicated in clustered mode and plain in unclustered mode. Do not apply it+ globally: some tables intentionally use a plain engine in both modes.+- Every metadata `ALTER` (`ADD`/`DROP`/`MODIFY COLUMN`, `ADD`/`DROP INDEX`) in a+ new migration must add+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}`, including files that+ contain only one `ALTER` — the race is across migration files, not within+ one. Mutation-generating statements use+ `{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2}`. `alter_sync`+ defaults to `1`, so without it the statement returns once the initiating+ replica has bumped the table's metadata version, and the next migration+ file's first `ALTER` on that table can hit a replica still on the old+ version, aborting the run with `code 517` because the replica metadata+ version is behind the common metadata version. `mutations_sync` does not+ substitute for `alter_sync`: it governs mutation completion, not metadata+ propagation. The renderer omits these fragments in unclustered mode. Do not+ retrofit these settings into already-shipped migrations solely for+ consistency; preserve historical output unless a forward fix deliberately+ changes it.+- Run `prepareMigrations.test.ts` when migrations or their renderer change. It+ must render both modes with no unresolved tokens and keep both historical+ outputs byte-for-byte unchanged.- Migrations must not use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` / `EXCHANGE TABLES`): the atomic replace requires `renameat2`, which fails on NFS-backed self-hosted deployments such as ClickHouse-on-EFS (GitHub issue #14906). Plain views are redefined with `DROP VIEW IF EXISTS <name>;` followed by `CREATE VIEW <name> AS ...` in the same migration file — the migration runner enables `x-multi-statement=true`, and every statement should stay idempotent (`IF EXISTS` / `IF NOT EXISTS`) so a dirty migration can be re-run.- Materialized views must never be dropped and recreated while their source table receives inserts — rows written between `DROP` and `CREATE` are silently and permanently missing from the target table. Redefine the SELECT with `ALTER TABLE <mv> MODIFY QUERY ...` instead, altering the target table(s) first when columns change.- For rationale and full patterns, see "Langfuse-Specific Rules" in the [`clickhouse-best-practices`](../../clickhouse-best-practices/SKILL.md) skill.diff --git a/.env.prod.example b/.env.prod.exampleindex 413e2e7da6..c0ffff562e 100644--- a/.env.prod.example+++ b/.env.prod.example@@ -248,6 +248,8 @@ LANGFUSE_ENABLE_BLOB_STORAGE_FILE_LOG=true# Clickhouse configuration# CLICKHOUSE_URL=+# Raw ClickHouse cluster name; do not include SQL quotes. Helm users with a+# non-default cluster can set this through langfuse.additionalEnv.# CLICKHOUSE_CLUSTER_NAME=default# CLICKHOUSE_DB=default# CLICKHOUSE_USER=diff --git a/docker-compose.build.yml b/docker-compose.build.ymlindex 400ddf2714..3a0120001a 100644--- a/docker-compose.build.yml+++ b/docker-compose.build.yml@@ -41,6 +41,7 @@ services:CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse}CLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}+ CLICKHOUSE_CLUSTER_NAME: ${CLICKHOUSE_CLUSTER_NAME:-default}LANGFUSE_JSON_BAD_UNICODE_ESCAPE: ${LANGFUSE_JSON_BAD_UNICODE_ESCAPE:-}LANGFUSE_S3_EVENT_UPLOAD_BUCKET: ${LANGFUSE_S3_EVENT_UPLOAD_BUCKET:-langfuse}LANGFUSE_S3_EVENT_UPLOAD_REGION: ${LANGFUSE_S3_EVENT_UPLOAD_REGION:-us-east-1}diff --git a/docker-compose.yml b/docker-compose.ymlindex f9ee89ba60..52fd3952a4 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -37,6 +37,7 @@ services:CLICKHOUSE_USER: ${CLICKHOUSE_USER:-clickhouse}CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse} # CHANGEMECLICKHOUSE_CLUSTER_ENABLED: ${CLICKHOUSE_CLUSTER_ENABLED:-false}+ CLICKHOUSE_CLUSTER_NAME: ${CLICKHOUSE_CLUSTER_NAME:-default}LANGFUSE_JSON_BAD_UNICODE_ESCAPE: ${LANGFUSE_JSON_BAD_UNICODE_ESCAPE:-}LANGFUSE_USE_AZURE_BLOB: ${LANGFUSE_USE_AZURE_BLOB:-false}LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE: ${LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE:-false}diff --git a/knip.jsonc b/knip.jsoncindex 1cb5d48f1d..93e2568e41 100644--- a/knip.jsonc+++ b/knip.jsonc@@ -7,6 +7,8 @@"workspaces": {"packages/shared": {"ignoreFiles": [+ // Standalone renderer invoked by ClickHouse migration shell scripts and Docker builds.+ "clickhouse/scripts/prepare-migrations.mjs",// Standalone helper invoked manually as documented in the adjacent framework-traces README."scripts/seeder/utils/framework-traces/merge-observations.ts",],diff --git a/package.json b/package.jsonindex 468acf3cbf..0ef410735a 100644--- a/package.json+++ b/package.json@@ -19,6 +19,8 @@"db:migrate": "turbo run db:migrate","db:seed": "turbo run db:seed","db:seed:examples": "turbo run db:seed:examples",+ "ch:migrations:materialize": "pnpm --filter @langfuse/shared run ch:migrations:materialize",+ "ch:migrations:clean": "pnpm --filter @langfuse/shared run ch:migrations:clean","nuke": "bash ./scripts/nuke.sh","scan:client-bundle": "node scripts/scan-client-bundle.mjs web/.next/static","seed": "pnpm --filter=shared run seed:scenario",diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.mdindex adc165e5de..38f32f40d4 100644--- a/packages/shared/AGENTS.md+++ b/packages/shared/AGENTS.md@@ -35,7 +35,10 @@`src/server/query-ast/kysely/`.- Postgres schema: `prisma/schema.prisma`- Prisma migrations: `prisma/migrations/*`-- ClickHouse migrations: `clickhouse/migrations/{clustered,unclustered}/*`+- Canonical ClickHouse migration templates (rendered for clustered and+ unclustered installs): `clickhouse/migrations/canonical/*`+- ClickHouse SQL identifier and string quoting:+ `src/server/clickhouse/clickhouseIdentifiers.ts`- Seeder and support scripts: `scripts/seeder/*`, `clickhouse/scripts/*`## Export Entry Points@@ -75,6 +78,7 @@the Mastra runtime and sandbox belong to the worker.- Narrower exported subpaths also exist for targeted imports:`@langfuse/shared/src/server/auth/apiKeys`,+ `@langfuse/shared/src/server/clickhouse/clickhouseIdentifiers`,`@langfuse/shared/src/server/ee/ingestionMasking`,`@langfuse/shared/src/server/llm/llmText`, and`@langfuse/shared/src/utils/chatml`. The experimental@@ -106,6 +110,8 @@ the same PR.- Prisma generate: `pnpm --filter @langfuse/shared run db:generate`- Prisma migrate (dev): `pnpm --filter @langfuse/shared run db:migrate`- ClickHouse reset: `pnpm --filter @langfuse/shared run ch:reset`+- Materialize direct-migration trees: `pnpm ch:migrations:materialize`+- Clean direct-migration trees: `pnpm ch:migrations:clean`## Playbooks@@ -119,7 +125,16 @@ the same PR.### ClickHouse schema change-1. Add migration under `clickhouse/migrations/*`.+1. Add one canonical migration pair under `clickhouse/migrations/canonical/*`.+ - Use `{CLICKHOUSE_CLUSTER_CLAUSE}` for every cluster-aware DDL clause.+ - Use `{CLICKHOUSE_REPLICATION_PREFIX}` only on table engines that should+ be replicated in clustered mode and plain in unclustered mode.+ - Put clustered-only synchronization fragments inside+ `{CLICKHOUSE_CLUSTERED_ONLY:...}`. Use+ `{CLICKHOUSE_UNCLUSTERED_ONLY:...}` only when preserving a deliberate+ mode-specific difference.+ - Run `src/server/clickhouse/prepareMigrations.test.ts`; it validates both+ rendered modes and protects the historical migration output.- Redefining views or materialized views follows strict patterns (no`CREATE OR REPLACE VIEW`; MV SELECT changes via`ALTER TABLE … MODIFY QUERY`) — apply the "Langfuse-Specific Rules" indiff --git a/packages/shared/clickhouse/migrations/README.md b/packages/shared/clickhouse/migrations/README.mdnew file mode 100644index 0000000000..34a2c42256--- /dev/null+++ b/packages/shared/clickhouse/migrations/README.md@@ -0,0 +1,101 @@+# ClickHouse migrations++`canonical/` is the only authored migration-template directory. Do not add a+second source copy. `clickhouse/scripts/prepare-migrations.mjs` renders the SQL+consumed by `golang-migrate`.++The production web image contains generated `clustered/` and `unclustered/`+directories next to `canonical/`. They are the primary migration inputs for+default-cluster and unclustered deployments and preserve documented recovery+commands that run `golang-migrate` directly inside the container. A custom+cluster name is rendered from `canonical/` into a temporary directory at+runtime. Source checkouts contain only `canonical/`, so the migration scripts+render a temporary directory for every mode during local development.++Do not edit or treat the generated `clustered/` and `unclustered/` directories+as migration sources. Migrations 0001-0047 render byte-for-byte identically to+the files shipped before the canonical template was introduced.++Use these explicit placeholders:++- `{CLICKHOUSE_CLUSTER_CLAUSE}` at every cluster-aware DDL position. It renders+ as `ON CLUSTER default`, a safely quoted custom cluster name, or an empty+ string for unclustered deployments.+- `{CLICKHOUSE_REPLICATION_PREFIX}` only on engines that should render as+ `Replicated*MergeTree` in clustered mode and `*MergeTree` in unclustered+ mode. Some tables intentionally remain non-replicated in both modes.+- `{CLICKHOUSE_CLUSTERED_ONLY:...}` for settings required only by replicated+ migrations, such as `alter_sync = 2` and `mutations_sync = 2`.+- `{CLICKHOUSE_UNCLUSTERED_ONLY:...}` only for an intentional mode-specific+ difference. Existing uses preserve historical migration behavior.+- `{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:...}` only in already-shipped+ migrations whose clustered and unclustered files had unusual end-of-file+ newlines. Do not use it in new migrations.++Keep migration version numbers and filenames stable. Do not retrofit+synchronization settings or formatting into already-shipped+migrations merely for consistency. Their rendered output is compatibility+history and should change only as a deliberate forward fix.++After changing a migration or the renderer, run:++```sh+pnpm --filter @langfuse/shared run test prepareMigrations.test.ts+```++The test invokes the production scripts through POSIX `sh`, checks both modes,+and locks the exact historical bytes for both generated migration trees.++## Direct migrations from a source checkout++Source checkouts do not contain the generated `clustered/` and `unclustered/`+trees. Materialize them before using a direct `golang-migrate` command from an+operations runbook:++```sh+pnpm ch:migrations:materialize+migrate -source file://packages/shared/clickhouse/migrations/unclustered -database "<migration-url>" up+pnpm ch:migrations:clean+```++Always run the clean command after the migration, including after a failed+migration. Do not edit the materialized files. They are intentionally not+ignored by Git so an incomplete cleanup or stale generated tree remains visible+in `git status`.++## Helm chart with a custom cluster name++The ClickHouse cluster bundled with the current Helm chart uses the name+`default`; it requires no additional configuration.++| ClickHouse deployment | Helm setting | Migration input |+| ------------------------------------------------- | ---------------------------------------------------- | -------------------------------- |+| Chart-managed or external cluster named `default` | `clickhouse.cluster.enabled: true` | Delivered `clustered/` |+| External, non-clustered server | `clickhouse.cluster.enabled: false` | Delivered `unclustered/` |+| External cluster with another name | Enable the cluster and set `CLICKHOUSE_CLUSTER_NAME` | Runtime render from `canonical/` |++For an external ClickHouse cluster with another name, enable clustered+migrations and pass the raw cluster name to both Langfuse containers through+the chart's global `additionalEnv`:++```yaml+clickhouse:+ deploy: false+ host: clickhouse.example.internal+ cluster:+ enabled: true+ migration:+ autoMigrate: true++langfuse:+ additionalEnv:+ - name: CLICKHOUSE_CLUSTER_NAME+ value: "virtual-cluster"+```++Do not add SQL quotes around `virtual-cluster`; the migration renderer quotes+and URL-encodes the value. Configure the remaining ClickHouse host, ports,+database, user, password, and TLS values through the chart as usual. On web+startup, Langfuse renders the custom-cluster migration tree and applies it+automatically. The same global environment variable reaches the worker for+runtime queries.diff --git a/packages/shared/clickhouse/migrations/canonical/0001_traces.down.sql b/packages/shared/clickhouse/migrations/canonical/0001_traces.down.sqlnew file mode 100644index 0000000000..1bf5958a67--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0001_traces.down.sql@@ -0,0 +1 @@+DROP TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/clustered/0001_traces.up.sql b/packages/shared/clickhouse/migrations/canonical/0001_traces.up.sqlsimilarity index 84%rename from packages/shared/clickhouse/migrations/clustered/0001_traces.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0001_traces.up.sqlindex dee75a8f01..4766ecf30a 100644--- a/packages/shared/clickhouse/migrations/clustered/0001_traces.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0001_traces.up.sql@@ -1,4 +1,4 @@-CREATE TABLE traces ON CLUSTER default (+CREATE TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} (`id` String,`timestamp` DateTime64(3),`name` String,@@ -20,7 +20,7 @@ CREATE TABLE traces ON CLUSTER default (INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,INDEX idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1,INDEX idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)PRIMARY KEY (project_id,toDate(timestamp)diff --git a/packages/shared/clickhouse/migrations/canonical/0002_observations.down.sql b/packages/shared/clickhouse/migrations/canonical/0002_observations.down.sqlnew file mode 100644index 0000000000..95ce4c33b4--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0002_observations.down.sql@@ -0,0 +1 @@+DROP TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/unclustered/0002_observations.up.sql b/packages/shared/clickhouse/migrations/canonical/0002_observations.up.sqlsimilarity index 87%rename from packages/shared/clickhouse/migrations/unclustered/0002_observations.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0002_observations.up.sqlindex cafd0c71eb..94654e8d1a 100644--- a/packages/shared/clickhouse/migrations/unclustered/0002_observations.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0002_observations.up.sql@@ -1,4 +1,4 @@-CREATE TABLE observations (+CREATE TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} (`id` String,`trace_id` String,`project_id` String,@@ -32,7 +32,7 @@ CREATE TABLE observations (INDEX idx_id id TYPE bloom_filter() GRANULARITY 1,INDEX idx_trace_id trace_id TYPE bloom_filter() GRANULARITY 1,INDEX idx_project_id project_id TYPE bloom_filter() GRANULARITY 1-) ENGINE = ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(start_time)+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(start_time)PRIMARY KEY (project_id,`type`,@@ -43,5 +43,4 @@ ORDER BY (`type`,toDate(start_time),id-);-+);{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:2}diff --git a/packages/shared/clickhouse/migrations/canonical/0003_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0003_scores.down.sqlnew file mode 100644index 0000000000..7c84090900--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0003_scores.down.sql@@ -0,0 +1 @@+DROP TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/clustered/0003_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0003_scores.up.sqlsimilarity index 83%rename from packages/shared/clickhouse/migrations/clustered/0003_scores.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0003_scores.up.sqlindex 3056924974..1ae2551e18 100644--- a/packages/shared/clickhouse/migrations/clustered/0003_scores.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0003_scores.up.sql@@ -1,4 +1,4 @@-CREATE TABLE scores ON CLUSTER default (+CREATE TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} (`id` String,`timestamp` DateTime64(3),`project_id` String,@@ -19,7 +19,7 @@ CREATE TABLE scores ON CLUSTER default (`is_deleted` UInt8,INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,INDEX idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)PRIMARY KEY (project_id,toDate(timestamp),diff --git a/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.down.sqlnew file mode 100644index 0000000000..4684be1b6d--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.down.sql@@ -0,0 +1 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_id project_id TYPE bloom_filter() GRANULARITY 1;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.up.sqlnew file mode 100644index 0000000000..eea778e98b--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0004_drop_observations_index.up.sql@@ -0,0 +1 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.down.sqlnew file mode 100644index 0000000000..4c78062f77--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.down.sql@@ -0,0 +1 @@+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_session_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.up.sqlnew file mode 100644index 0000000000..b87aba372c--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0005_add_session_id_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_session_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.down.sqlnew file mode 100644index 0000000000..6830e1a0c7--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.down.sql@@ -0,0 +1 @@+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_user_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}diff --git a/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.up.sqlnew file mode 100644index 0000000000..6c1f7c7721--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0006_add_user_id_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_user_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.down.sql b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.down.sqlnew file mode 100644index 0000000000..a0c6b927c5--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.down.sql@@ -0,0 +1 @@+DROP TABLE event_log {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.up.sql b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.up.sqlsimilarity index 89%rename from packages/shared/clickhouse/migrations/clustered/0007_add_event_log.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0007_add_event_log.up.sqlindex 9fa430da0e..cd6a9116cd 100644--- a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0007_add_event_log.up.sql@@ -1,4 +1,4 @@-CREATE TABLE event_log ON CLUSTER default+CREATE TABLE event_log {CLICKHOUSE_CLUSTER_CLAUSE}(`id` String,`project_id` String,diff --git a/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.down.sql b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.down.sqlnew file mode 100644index 0000000000..841f5654cd--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.down.sql@@ -0,0 +1,3 @@+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS environment{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS environment{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS environment{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.up.sql b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.up.sqlnew file mode 100644index 0000000000..87f031fd6a--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0008_add_environments_column.up.sql@@ -0,0 +1,3 @@+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.down.sql b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.down.sqlnew file mode 100644index 0000000000..2086b25c25--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.down.sql@@ -0,0 +1,7 @@+-- Drop materialized views+DROP VIEW IF EXISTS project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE};++-- Drop the project_environments table+DROP TABLE IF EXISTS project_environments {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.up.sql b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.up.sqlsimilarity index 58%rename from packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.up.sqlindex 50c55b4690..503edfc595 100644--- a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0009_add_project_environments.up.sql@@ -1,12 +1,12 @@-- Create the project_environments table with AggregatingMergeTree engine-CREATE TABLE project_environments (+CREATE TABLE project_environments {CLICKHOUSE_CLUSTER_CLAUSE} (`project_id` String,`environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))-) ENGINE = AggregatingMergeTree+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}AggregatingMergeTreeORDER BY (project_id);-- Create materialized view for traces-CREATE MATERIALIZED VIEW project_environments_traces_mv TO project_environments AS+CREATE MATERIALIZED VIEW project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments ASSELECTproject_id,groupUniqArray(environment) AS environments@@ -14,7 +14,7 @@ FROM tracesGROUP BY project_id;-- Create materialized view for observations-CREATE MATERIALIZED VIEW project_environments_observations_mv TO project_environments AS+CREATE MATERIALIZED VIEW project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments ASSELECTproject_id,groupUniqArray(environment) AS environments@@ -22,7 +22,7 @@ FROM observationsGROUP BY project_id;-- Create materialized view for scores-CREATE MATERIALIZED VIEW project_environments_scores_mv TO project_environments AS+CREATE MATERIALIZED VIEW project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments ASSELECTproject_id,groupUniqArray(environment) AS environmentsdiff --git a/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.down.sqlnew file mode 100644index 0000000000..44f388da92--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS metadata;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.up.sqlnew file mode 100644index 0000000000..0aaf63b175--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0010_add_metadata_column_on_scores.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN metadata Map(LowCardinality(String), String) AFTER comment;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.down.sql b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.down.sqlnew file mode 100644index 0000000000..ec8a877ae2--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.down.sql@@ -0,0 +1 @@+DROP TABLE blob_storage_file_log {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.up.sql b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.up.sqlsimilarity index 69%rename from packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.up.sqlindex 70150bef29..1cced071d6 100644--- a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0011_add_blob_storage_file_log.up.sql@@ -1,5 +1,5 @@-CREATE TABLE blob_storage_file_log-(+CREATE TABLE blob_storage_file_log {CLICKHOUSE_CLUSTER_CLAUSE}{CLICKHOUSE_CLUSTERED_ONLY: (}{CLICKHOUSE_UNCLUSTERED_ONLY:+(}`id` String,`project_id` String,`entity_type` String,@@ -13,7 +13,7 @@ CREATE TABLE blob_storage_file_log`updated_at` DateTime64(3) DEFAULT now(),event_ts DateTime64(3),is_deleted UInt8,-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)ORDER BY (project_id,entity_type,diff --git a/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.down.sqlnew file mode 100644index 0000000000..10f9e2d393--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS session_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.up.sqlnew file mode 100644index 0000000000..8888a63c61--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0012_add_session_id_column_scores.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN session_id Nullable(String) AFTER trace_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.down.sqlnew file mode 100644index 0000000000..9d4b7b34aa--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.down.sql@@ -0,0 +1,2 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.up.sqlnew file mode 100644index 0000000000..a9d2f3eca2--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0013_drop_scores_trace_id_index.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:2}diff --git a/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.down.sql b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.down.sqlnew file mode 100644index 0000000000..9bce5e87d1--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.up.sql b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.up.sqlnew file mode 100644index 0000000000..9bce5e87d1--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0014_scores_modify_nullable_trace_id_column.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.down.sqlnew file mode 100644index 0000000000..10ee265522--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.up.sqlnew file mode 100644index 0000000000..a8b52facdd--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0015_add_scores_trace_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2;}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2; }+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.down.sqlnew file mode 100644index 0000000000..0a9ee001fd--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.up.sqlnew file mode 100644index 0000000000..6418e2985b--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0016_add_scores_session_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.down.sqlnew file mode 100644index 0000000000..8e9b83379e--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS dataset_run_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.up.sqlnew file mode 100644index 0000000000..00fc84ed7f--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0017_add_run_id_column_scores.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN dataset_run_id Nullable(String) AFTER session_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.down.sqlnew file mode 100644index 0000000000..b159c0ed5c--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.up.sqlnew file mode 100644index 0000000000..78b2f05fc3--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0018_add_scores_run_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2}{CLICKHOUSE_UNCLUSTERED_ONLY: SETTINGS mutations_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.down.sql b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.down.sqlnew file mode 100644index 0000000000..ea2f633d7a--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.down.sql@@ -0,0 +1 @@+DROP VIEW IF EXISTS analytics_traces {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.up.sql b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.up.sqlsimilarity index 87%rename from packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.up.sqlindex d2bdbfb947..52eee7405c 100644--- a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0019_analytics_traces.up.sql@@ -1,4 +1,4 @@-CREATE VIEW analytics_traces ON CLUSTER default AS+CREATE VIEW analytics_traces {CLICKHOUSE_CLUSTER_CLAUSE} ASSELECTproject_id,toStartOfHour(timestamp) AS hour,diff --git a/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.down.sql b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.down.sqlnew file mode 100644index 0000000000..c0380e0f55--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.down.sql@@ -0,0 +1 @@+DROP VIEW IF EXISTS analytics_observations {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.up.sql b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.up.sqlsimilarity index 89%rename from packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.up.sqlindex 0e5f451959..85b45c3212 100644--- a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0020_analytics_observations.up.sql@@ -1,4 +1,4 @@-CREATE VIEW analytics_observations AS+CREATE VIEW analytics_observations {CLICKHOUSE_CLUSTER_CLAUSE} ASSELECTproject_id,type,diff --git a/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.down.sqlnew file mode 100644index 0000000000..dfa3cc6c7d--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.down.sql@@ -0,0 +1 @@+DROP VIEW IF EXISTS analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.up.sqlsimilarity index 92%rename from packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.up.sqlindex eb683cd89a..6f920fffd0 100644--- a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0021_analytics_scores.up.sql@@ -1,4 +1,4 @@-CREATE VIEW analytics_scores ON CLUSTER default AS+CREATE VIEW analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE} ASSELECTproject_id,toStartOfHour(timestamp) AS hour,diff --git a/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.down.sqlnew file mode 100644index 0000000000..0d2d11afbc--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.down.sql@@ -0,0 +1 @@+DROP TABLE dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.up.sqlsimilarity index 88%rename from packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.up.sqlindex c8466b6c9d..f0a0143421 100644--- a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0022_dataset_run_items.up.sql@@ -1,4 +1,4 @@-CREATE TABLE dataset_run_items (+CREATE TABLE dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE} (-- primary identifiers`id` String,`project_id` String,@@ -33,4 +33,4 @@ CREATE TABLE dataset_run_items (-- For dataset item lookupsINDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,) ENGINE = ReplacingMergeTree(event_ts, is_deleted)-ORDER BY (project_id, dataset_id, dataset_run_id, id);\ No newline at end of file+ORDER BY (project_id, dataset_id, dataset_run_id, id);{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.down.sql b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.down.sqlnew file mode 100644index 0000000000..623fdb4294--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.down.sql@@ -0,0 +1,12 @@+-- Drop materialized views first+DROP VIEW IF EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};++-- Drop AMT tables+DROP TABLE IF EXISTS traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE};+DROP TABLE IF EXISTS traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE};+DROP TABLE IF EXISTS traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE};++-- Drop the Null table+DROP TABLE IF EXISTS traces_null {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.up.sql b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.up.sqlsimilarity index 93%rename from packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.up.sqlindex fd243f4f6a..246e114962 100644--- a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0023_traces_aggregating_merge_trees.up.sql@@ -1,6 +1,6 @@-- Create a Null table that serves as a trigger for all materialized views.-- We use a Null engine here to avoid storing intermediate results and save on storage.-CREATE TABLE traces_null ON CLUSTER default+CREATE TABLE traces_null {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -22,7 +22,7 @@ CREATE TABLE traces_null ON CLUSTER default`bookmarked` Nullable(Bool),`public` Nullable(Bool),- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}`observation_ids` Array(String),`score_ids` Array(String),`cost_details` Map(String, Decimal64(12)),@@ -39,8 +39,8 @@ CREATE TABLE traces_null ON CLUSTER default) Engine = Null();-- Create the all AMT-CREATE TABLE traces_all_amt ON CLUSTER default-(+CREATE TABLE traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE}+({CLICKHOUSE_CLUSTERED_ONLY: }{CLICKHOUSE_UNCLUSTERED_ONLY: }-- Identifiers`project_id` String,`id` String,@@ -62,7 +62,7 @@ CREATE TABLE traces_all_amt ON CLUSTER default`bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),`public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}`observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),`score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),`cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),@@ -87,7 +87,7 @@ CREATE TABLE traces_all_amt ON CLUSTER defaultORDER BY (project_id, id);-- Create materialized view for all_amt-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv ON CLUSTER default TO traces_all_amt AS+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_all_amt ASSELECT-- Identifierstn.project_id as project_id,@@ -110,7 +110,7 @@ SELECTargMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}groupUniqArrayArray(tn.observation_ids) as observation_ids,groupUniqArrayArray(tn.score_ids) as score_ids,sumMap(tn.cost_details) as cost_details,@@ -126,7 +126,7 @@ FROM traces_null tnGROUP BY project_id, id;-- Create the 7-day TTL AMT-CREATE TABLE traces_7d_amt ON CLUSTER default+CREATE TABLE traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -149,7 +149,7 @@ CREATE TABLE traces_7d_amt ON CLUSTER default`bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),`public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}`observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),`score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),`cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),@@ -174,7 +174,7 @@ CREATE TABLE traces_7d_amt ON CLUSTER defaultTTL toDate(start_time) + INTERVAL 7 DAY;-- Create materialized view for 7d_amt-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv ON CLUSTER default TO traces_7d_amt AS+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_7d_amt ASSELECT-- Identifierstn.project_id as project_id,@@ -197,7 +197,7 @@ SELECTargMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}groupUniqArrayArray(tn.observation_ids) as observation_ids,groupUniqArrayArray(tn.score_ids) as score_ids,sumMap(tn.cost_details) as cost_details,@@ -213,7 +213,7 @@ FROM traces_null tnGROUP BY project_id, id;-- Create the 30-day TTL AMT-CREATE TABLE traces_30d_amt ON CLUSTER default+CREATE TABLE traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -236,7 +236,7 @@ CREATE TABLE traces_30d_amt ON CLUSTER default`bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),`public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}`observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),`score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),`cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),@@ -261,7 +261,7 @@ CREATE TABLE traces_30d_amt ON CLUSTER defaultTTL toDate(start_time) + INTERVAL 30 DAY;-- Create materialized view for 30d_amt-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv ON CLUSTER default TO traces_30d_amt AS+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_30d_amt ASSELECT-- Identifierstn.project_id as project_id,@@ -284,7 +284,7 @@ SELECTargMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,- -- Aggregations -- DO NOT USE+ -- Aggregations{CLICKHOUSE_CLUSTERED_ONLY: -- DO NOT USE}groupUniqArrayArray(tn.observation_ids) as observation_ids,groupUniqArrayArray(tn.score_ids) as score_ids,sumMap(tn.cost_details) as cost_details,@@ -297,4 +297,4 @@ SELECTmin(tn.created_at) as created_at,max(tn.updated_at) as updated_atFROM traces_null tn-GROUP BY project_id, id;+GROUP BY project_id, id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}diff --git a/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.down.sqlnew file mode 100644index 0000000000..b263773950--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.down.sql@@ -0,0 +1 @@+DROP TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.up.sqlsimilarity index 81%rename from packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.up.sqlindex 28db500526..0c3243b791 100644--- a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0024_dataset_run_items.up.sql@@ -1,4 +1,4 @@-CREATE TABLE dataset_run_items_rmt (+CREATE TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} (-- primary identifiers`id` String,`project_id` String,@@ -32,5 +32,5 @@ CREATE TABLE dataset_run_items_rmt (-- For dataset item lookupsINDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)-ORDER BY (project_id, dataset_id, dataset_run_id, id);\ No newline at end of file+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)+ORDER BY (project_id, dataset_id, dataset_run_id, id);{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.down.sql b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.down.sqlnew file mode 100644index 0000000000..a84100e253--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.down.sql@@ -0,0 +1,2 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_res_metadata_value{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_res_metadata_key{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.up.sql b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.up.sqlnew file mode 100644index 0000000000..40bbcc354c--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0025_add_observations_metadata_indexes.up.sql@@ -0,0 +1,4 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_res_metadata_key{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_res_metadata_value{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.down.sqlnew file mode 100644index 0000000000..a50050457f--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.down.sql@@ -0,0 +1 @@+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_trace_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.up.sqlnew file mode 100644index 0000000000..036ec05e26--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0026_add_trace_id_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_trace_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS mutations_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.down.sql b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.down.sqlnew file mode 100644index 0000000000..f58c9a3532--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.down.sql@@ -0,0 +1,21 @@+-- Recreate materialized views for project_environments table+CREATE MATERIALIZED VIEW project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS+SELECT+ project_id,+ groupUniqArray(environment) AS environments+FROM traces+GROUP BY project_id;++CREATE MATERIALIZED VIEW project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS+SELECT+ project_id,+ groupUniqArray(environment) AS environments+FROM observations+GROUP BY project_id;++CREATE MATERIALIZED VIEW project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO project_environments AS+SELECT+ project_id,+ groupUniqArray(environment) AS environments+FROM scores+GROUP BY project_id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.up.sql b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.up.sqlnew file mode 100644index 0000000000..152186dfbb--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0027_drop_project_environments_mvs.up.sql@@ -0,0 +1,4 @@+-- Drop materialized views feeding project_environments table+DROP VIEW IF EXISTS project_environments_traces_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS project_environments_observations_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS project_environments_scores_mv {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.down.sql b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.down.sqlsimilarity index 95%rename from packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.down.sqlindex 60d3890318..9c0cf2a938 100644--- a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.down.sql@@ -1,5 +1,5 @@-- Recreate materialized views derived from traces_null table-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv ON CLUSTER default TO traces_all_amt AS+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_all_amt ASSELECT-- Identifierstn.project_id as project_id,@@ -37,7 +37,7 @@ SELECTFROM traces_null tnGROUP BY project_id, id;-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv ON CLUSTER default TO traces_7d_amt AS+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_7d_amt ASSELECT-- Identifierstn.project_id as project_id,@@ -75,7 +75,7 @@ SELECTFROM traces_null tnGROUP BY project_id, id;-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv ON CLUSTER default TO traces_30d_amt AS+CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO traces_30d_amt ASSELECT-- Identifierstn.project_id as project_id,@@ -111,4 +111,4 @@ SELECTmin(tn.created_at) as created_at,max(tn.updated_at) as updated_atFROM traces_null tn-GROUP BY project_id, id;\ No newline at end of file+GROUP BY project_id, id;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.up.sql b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.up.sqlnew file mode 100644index 0000000000..8a441e9f19--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0028_drop_traces_null_mvs.up.sql@@ -0,0 +1,4 @@+-- Drop materialized views derived from traces_null table+DROP VIEW IF EXISTS traces_all_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS traces_7d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};+DROP VIEW IF EXISTS traces_30d_amt_mv {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.down.sql b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.down.sqlsimilarity index 96%rename from packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.down.sqlindex 3c51d5fd0e..5f3b85cac8 100644--- a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.down.sql@@ -1,5 +1,5 @@-- Recreate traces_null and trace amt tables-CREATE TABLE traces_null+CREATE TABLE traces_null {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -37,7 +37,7 @@ CREATE TABLE traces_null`event_ts` DateTime64(3)) Engine = Null();-CREATE TABLE traces_all_amt+CREATE TABLE traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -84,7 +84,7 @@ CREATE TABLE traces_all_amt) Engine = AggregatingMergeTree()ORDER BY (project_id, id);-CREATE TABLE traces_7d_amt+CREATE TABLE traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -131,7 +131,7 @@ CREATE TABLE traces_7d_amtORDER BY (project_id, id)TTL toDate(start_time) + INTERVAL 7 DAY;-CREATE TABLE traces_30d_amt+CREATE TABLE traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE}(-- Identifiers`project_id` String,@@ -176,4 +176,4 @@ CREATE TABLE traces_30d_amtINDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1) Engine = AggregatingMergeTree()ORDER BY (project_id, id)- TTL toDate(start_time) + INTERVAL 30 DAY;\ No newline at end of file+ TTL toDate(start_time) + INTERVAL 30 DAY;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.up.sql b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.up.sqlnew file mode 100644index 0000000000..838c5aad03--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0029_drop_traces_null_and_amt_tables.up.sql@@ -0,0 +1,5 @@+-- Drop traces_null and trace amt tables+DROP TABLE IF EXISTS traces_null {CLICKHOUSE_CLUSTER_CLAUSE};+DROP TABLE IF EXISTS traces_all_amt {CLICKHOUSE_CLUSTER_CLAUSE};+DROP TABLE IF EXISTS traces_7d_amt {CLICKHOUSE_CLUSTER_CLAUSE};+DROP TABLE IF EXISTS traces_30d_amt {CLICKHOUSE_CLUSTER_CLAUSE};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:0}diff --git a/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.down.sqlnew file mode 100644index 0000000000..6e6c89fdde--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS execution_trace_id;diff --git a/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.up.sqlnew file mode 100644index 0000000000..2377fa061e--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0030_add_eval_execution_trace_id_to_scores.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.down.sqlnew file mode 100644index 0000000000..1da1ab4f07--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.down.sql@@ -0,0 +1,2 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS usage_pricing_tier_name{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS usage_pricing_tier_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.up.sqlnew file mode 100644index 0000000000..0a68ccc6c4--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0031_add_usage_pricing_tier_columns.up.sql@@ -0,0 +1,2 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN usage_pricing_tier_id Nullable(String){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN usage_pricing_tier_name Nullable(String){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.down.sqlnew file mode 100644index 0000000000..98a8c5d45d--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql@@ -0,0 +1 @@+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS dataset_item_version;{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}diff --git a/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.up.sqlnew file mode 100644index 0000000000..fb4b82c6ae--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql@@ -0,0 +1 @@+ALTER TABLE dataset_run_items_rmt {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3)){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:clustered}diff --git a/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.down.sqlnew file mode 100644index 0000000000..8cc86f9026--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.down.sql@@ -0,0 +1,3 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS tool_definitions SETTINGS alter_sync = 2;+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS tool_calls SETTINGS alter_sync = 2;+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS tool_call_names SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.up.sqlnew file mode 100644index 0000000000..5a7685e7fe--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0033_add_tool_call_columns.up.sql@@ -0,0 +1,3 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS tool_definitions Map(String, String) DEFAULT map() SETTINGS alter_sync = 2;+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS tool_calls Array(String) DEFAULT [] SETTINGS alter_sync = 2;+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS tool_call_names Array(String) DEFAULT [] SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.down.sqlnew file mode 100644index 0000000000..aafb3775cd--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.down.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS long_string_value;diff --git a/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.up.sqlnew file mode 100644index 0000000000..0c106ce47c--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0034_add_long_string_value_to_scores.up.sql@@ -0,0 +1 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3)){CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.down.sqlnew file mode 100644index 0000000000..19d89b0b0d--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.down.sql@@ -0,0 +1,11 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}+ DROP COLUMN IF EXISTS ingestion_sdk_version+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};++ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}+ DROP COLUMN IF EXISTS ingestion_sdk_name+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};++ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}+ DROP COLUMN IF EXISTS ingestion_api_key+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.up.sqlnew file mode 100644index 0000000000..e0627c55ad--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0035_add_ingestion_attribution_columns.up.sql@@ -0,0 +1,11 @@+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}+ ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};++ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}+ ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};++ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}+ ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.down.sqlsimilarity index 84%rename from packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.down.sqlindex 1c9aa7f16f..88f241a4e7 100644--- a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.down.sql@@ -1,5 +1,5 @@-DROP VIEW IF EXISTS analytics_scores;-CREATE VIEW analytics_scores AS+DROP VIEW IF EXISTS analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE};+CREATE VIEW analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE} ASSELECTproject_id,toStartOfHour(timestamp) AS hour,diff --git a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.up.sqlsimilarity index 87%rename from packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.up.sqlindex 2caf436211..c3942b832d 100644--- a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0036_add_ingested_sdks_to_analytics_scores.up.sql@@ -1,5 +1,5 @@-DROP VIEW IF EXISTS analytics_scores ON CLUSTER default;-CREATE VIEW analytics_scores ON CLUSTER default AS+DROP VIEW IF EXISTS analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE};+CREATE VIEW analytics_scores {CLICKHOUSE_CLUSTER_CLAUSE} ASSELECTproject_id,toStartOfHour(timestamp) AS hour,diff --git a/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.down.sql b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.down.sqlnew file mode 100644index 0000000000..342db48a79--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.down.sql@@ -0,0 +1,3 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_created_at;+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_created_at;+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_created_at;diff --git a/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.up.sql b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.up.sqlnew file mode 100644index 0000000000..79fb25be2b--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0037_add_created_at_indexes.up.sql@@ -0,0 +1,6 @@+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE observations {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_created_at;+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE traces {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_created_at;+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} MATERIALIZE INDEX IF EXISTS idx_created_at;diff --git a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.down.sql b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.down.sqlsimilarity index 79%rename from packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.down.sqlindex e3177579f2..8f3ee609ad 100644--- a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.down.sql@@ -2,4 +2,4 @@-- table may predate this migration (created via dev-tables.sh on v4 preview-- deployments) and may hold production-grade event data. Rolling back drops-- that data regardless of which path created the table.-DROP TABLE IF EXISTS observations_batch_staging ON CLUSTER default;+DROP TABLE IF EXISTS observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.up.sql b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.up.sqlsimilarity index 91%rename from packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.up.sqlindex 757131dbdd..ff47bb5dfe 100644--- a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0038_create_observations_batch_staging.up.sql@@ -1,4 +1,4 @@-CREATE TABLE IF NOT EXISTS observations_batch_staging+CREATE TABLE IF NOT EXISTS observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}(id String,trace_id String,@@ -38,7 +38,7 @@ CREATE TABLE IF NOT EXISTS observations_batch_stagings3_first_seen_timestamp DateTime64(3),environment LowCardinality(String) DEFAULT 'default')-ENGINE = ReplacingMergeTree(event_ts, is_deleted)+ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)PARTITION BY toStartOfInterval(s3_first_seen_timestamp, INTERVAL 3 MINUTE)PRIMARY KEY (project_id, toDate(s3_first_seen_timestamp))ORDER BY (diff --git a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.down.sql b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.down.sqlsimilarity index 82%rename from packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0039_create_events_full.down.sqlindex 9280e87133..97d3def98e 100644--- a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.down.sql@@ -2,4 +2,4 @@-- table may predate this migration (created via dev-tables.sh on v4 preview-- deployments) and may hold production-grade event data. Rolling back drops-- that data regardless of which path created the table.-DROP TABLE IF EXISTS events_full;+DROP TABLE IF EXISTS events_full {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.up.sql b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.up.sqlsimilarity index 96%rename from packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0039_create_events_full.up.sqlindex c14b38ef47..bbf2a13520 100644--- a/packages/shared/clickhouse/migrations/unclustered/0039_create_events_full.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0039_create_events_full.up.sql@@ -1,4 +1,4 @@-CREATE TABLE IF NOT EXISTS events_full+CREATE TABLE IF NOT EXISTS events_full {CLICKHOUSE_CLUSTER_CLAUSE}(project_id String,trace_id String,@@ -109,7 +109,7 @@ CREATE TABLE IF NOT EXISTS events_fullINDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha))-ENGINE = ReplacingMergeTree(event_ts, is_deleted)+ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)PARTITION BY toYYYYMM(start_time)PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)diff --git a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.down.sql b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.down.sqlsimilarity index 82%rename from packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0040_create_events_core.down.sqlindex c263c98542..291ef76c54 100644--- a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.down.sql@@ -2,4 +2,4 @@-- table may predate this migration (created via dev-tables.sh on v4 preview-- deployments) and may hold production-grade event data. Rolling back drops-- that data regardless of which path created the table.-DROP TABLE IF EXISTS events_core;+DROP TABLE IF EXISTS events_core {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.up.sql b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.up.sqlsimilarity index 96%rename from packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0040_create_events_core.up.sqlindex fb5df27191..e5a9bde82f 100644--- a/packages/shared/clickhouse/migrations/unclustered/0040_create_events_core.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0040_create_events_core.up.sql@@ -1,4 +1,4 @@-CREATE TABLE IF NOT EXISTS events_core+CREATE TABLE IF NOT EXISTS events_core {CLICKHOUSE_CLUSTER_CLAUSE}(project_id String,trace_id String,@@ -110,7 +110,7 @@ CREATE TABLE IF NOT EXISTS events_coreINDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha))-ENGINE = ReplacingMergeTree(event_ts, is_deleted)+ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}ReplacingMergeTree(event_ts, is_deleted)PARTITION BY toYYYYMM(start_time)PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)diff --git a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.down.sql b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.down.sqlsimilarity index 77%rename from packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.down.sqlindex 1e4578cc48..896dde4dc0 100644--- a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.down.sql@@ -1,4 +1,4 @@-- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this-- view may predate this migration (created via dev-tables.sh on v4 preview-- deployments). Dropping it stops events_core from being populated.-DROP VIEW IF EXISTS events_core_mv;+DROP VIEW IF EXISTS events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.up.sql b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.up.sqlsimilarity index 93%rename from packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.up.sqlindex 40f95a62f2..033680f34c 100644--- a/packages/shared/clickhouse/migrations/unclustered/0041_create_events_core_mv.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0041_create_events_core_mv.up.sql@@ -1,4 +1,4 @@-CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv TO events_core AS+CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} TO events_core ASSELECTproject_id,trace_id,diff --git a/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.down.sqlnew file mode 100644index 0000000000..f0e7e72068--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.down.sql@@ -0,0 +1,82 @@+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY+SELECT+ project_id,+ trace_id,+ span_id,+ parent_span_id,+ start_time,+ end_time,+ name,+ type,+ environment,+ version,+ release,+ trace_name,+ user_id,+ session_id,+ tags,+ level,+ status_message,+ completion_start_time,+ is_app_root,+ bookmarked,+ public,+ prompt_id,+ prompt_name,+ prompt_version,+ model_id,+ provided_model_name,+ model_parameters,+ provided_usage_details,+ usage_details,+ provided_cost_details,+ cost_details,+ usage_pricing_tier_id,+ usage_pricing_tier_name,+ tool_definitions,+ tool_calls,+ tool_call_names,+ leftUTF8(input, 200) as input,+ leftUTF8(output, 200) as output,+ metadata_names,+ arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,+ experiment_id,+ experiment_name,+ experiment_metadata_names,+ experiment_metadata_values,+ experiment_description,+ experiment_dataset_id,+ experiment_item_id,+ experiment_item_version,+ experiment_item_expected_output,+ experiment_item_metadata_names,+ experiment_item_metadata_values,+ experiment_item_root_span_id,+ source,+ service_name,+ service_version,+ scope_name,+ scope_version,+ telemetry_sdk_language,+ telemetry_sdk_name,+ telemetry_sdk_version,+ blob_storage_file_path,+ event_bytes,+ created_at,+ updated_at,+ event_ts,+ is_deleted+FROM events_full+SETTINGS enable_full_text_index = 1;++ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};++ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};++ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.up.sqlsimilarity index 63%rename from packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.up.sqlindex 7a43d0fb2a..3c8dd270d2 100644--- a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0042_add_events_ingestion_attribution_columns.up.sql@@ -1,34 +1,34 @@-ALTER TABLE observations_batch_staging+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS enable_full_text_index = 1;-ALTER TABLE observations_batch_staging+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1;-ALTER TABLE observations_batch_staging+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE observations_batch_staging {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1;+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};-ALTER TABLE events_full+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1;+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};-ALTER TABLE events_core+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1;+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};-ALTER TABLE events_core_mv MODIFY QUERY+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERYSELECTproject_id,trace_id,diff --git a/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.down.sql b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.down.sqlnew file mode 100644index 0000000000..752fe3da81--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.down.sql@@ -0,0 +1,2 @@+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.up.sql b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.up.sqlnew file mode 100644index 0000000000..62cd5ca4f4--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0043_add_events_metadata_ngram_index.up.sql@@ -0,0 +1,2 @@+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.down.sql b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.down.sqlsimilarity index 89%rename from packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.down.sqlindex f894ad58fe..68dee80405 100644--- a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.down.sql@@ -1,6 +1,6 @@-- WARNING: This recreates the table structure only. Any data held in-- event_log before the up migration dropped it is NOT restored.-CREATE TABLE IF NOT EXISTS event_log ON CLUSTER default+CREATE TABLE IF NOT EXISTS event_log {CLICKHOUSE_CLUSTER_CLAUSE}(`id` String,`project_id` String,diff --git a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.up.sql b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.up.sqlsimilarity index 81%rename from packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.up.sqlindex 3682d6d29c..fdd0749b32 100644--- a/packages/shared/clickhouse/migrations/clustered/0044_drop_event_log.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0044_drop_event_log.up.sql@@ -2,4 +2,4 @@-- were copied over by the 20250417_1737_migrate_event_log_to_blob_storage-- background migration, which must have completed (on the latest v3 release)-- before upgrading to v4.-DROP TABLE IF EXISTS event_log ON CLUSTER default;+DROP TABLE IF EXISTS event_log {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.down.sql b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.down.sqlsimilarity index 74%rename from packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.down.sqlindex a1de329190..d33dde9255 100644--- a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.down.sql@@ -2,8 +2,8 @@-- project_environments before the up migration dropped it is NOT restored,-- and the materialized views that used to feed the table were already dropped-- by 0027_drop_project_environments_mvs, so it stays empty.-CREATE TABLE IF NOT EXISTS project_environments ON CLUSTER default (+CREATE TABLE IF NOT EXISTS project_environments {CLICKHOUSE_CLUSTER_CLAUSE} (`project_id` String,`environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))-) ENGINE = ReplicatedAggregatingMergeTree+) ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}AggregatingMergeTreeORDER BY (project_id);diff --git a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.up.sql b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.up.sqlsimilarity index 77%rename from packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.up.sqlindex 25cb10b8a0..aa9176e6b2 100644--- a/packages/shared/clickhouse/migrations/clustered/0045_drop_project_environments.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0045_drop_project_environments.up.sql@@ -2,4 +2,4 @@-- 0027_drop_project_environments_mvs removed the materialized views feeding-- it. getEnvironmentsForProject reads distinct environments from the tracing-- and scores tables directly.-DROP TABLE IF EXISTS project_environments ON CLUSTER default;+DROP TABLE IF EXISTS project_environments {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.down.sqlsimilarity index 94%rename from packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.down.sqlrename to packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.down.sqlindex 88a8e3d72d..b3d42ae432 100644--- a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.down.sql+++ b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.down.sql@@ -1,6 +1,6 @@-- WARNING: This recreates the table structure only. Any data held in-- dataset_run_items before the up migration dropped it is NOT restored.-CREATE TABLE IF NOT EXISTS dataset_run_items (+CREATE TABLE IF NOT EXISTS dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE} (-- primary identifiers`id` String,`project_id` String,diff --git a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.up.sqlsimilarity index 75%rename from packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.up.sqlindex 353b69e3b4..6814a57161 100644--- a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0046_drop_dataset_run_items.up.sql@@ -1,4 +1,4 @@-- The dataset_run_items table was superseded by dataset_run_items_rmt-- (created in 0024). All ingestion writes, reads and deletes target the-- _rmt table, so this table has been frozen since the cutover.-DROP TABLE IF EXISTS dataset_run_items ON CLUSTER default;+DROP TABLE IF EXISTS dataset_run_items {CLICKHOUSE_CLUSTER_CLAUSE};diff --git a/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.down.sql b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.down.sqlnew file mode 100644index 0000000000..cd8d0615bd--- /dev/null+++ b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.down.sql@@ -0,0 +1,88 @@+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERY+SELECT+ project_id,+ trace_id,+ span_id,+ parent_span_id,+ start_time,+ end_time,+ name,+ type,+ environment,+ version,+ release,+ trace_name,+ user_id,+ session_id,+ tags,+ level,+ status_message,+ completion_start_time,+ is_app_root,+ bookmarked,+ public,+ prompt_id,+ prompt_name,+ prompt_version,+ model_id,+ provided_model_name,+ model_parameters,+ provided_usage_details,+ usage_details,+ provided_cost_details,+ cost_details,+ usage_pricing_tier_id,+ usage_pricing_tier_name,+ tool_definitions,+ tool_calls,+ tool_call_names,+ leftUTF8(input, 200) as input,+ leftUTF8(output, 200) as output,+ metadata_names,+ arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,+ experiment_id,+ experiment_name,+ experiment_metadata_names,+ experiment_metadata_values,+ experiment_description,+ experiment_dataset_id,+ experiment_item_id,+ experiment_item_version,+ experiment_item_expected_output,+ experiment_item_metadata_names,+ experiment_item_metadata_values,+ experiment_item_root_span_id,+ source,+ service_name,+ service_version,+ scope_name,+ scope_version,+ telemetry_sdk_language,+ telemetry_sdk_name,+ telemetry_sdk_version,+ blob_storage_file_path,+ event_bytes,+ created_at,+ updated_at,+ event_ts,+ is_deleted,+ ingestion_api_key,+ ingestion_sdk_name,+ ingestion_sdk_version+FROM events_full+SETTINGS enable_full_text_index = 1;++ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};++ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluator_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};++ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluator_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP INDEX IF EXISTS idx_evaluation_rule_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluator_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE} DROP COLUMN IF EXISTS evaluation_rule_id{CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};diff --git a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.up.sql b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.up.sqlsimilarity index 71%rename from packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.up.sqlrename to packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.up.sqlindex 87504a51b4..a3223a4782 100644--- a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.up.sql+++ b/packages/shared/clickhouse/migrations/canonical/0047_add_eval_execution_columns.up.sql@@ -1,41 +1,41 @@-- Existing parts resolve these columns through DEFAULT expressions. Skip indexes are added without materializing historical parts to keep this migration metadata-only.-ALTER TABLE events_full ON CLUSTER default+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_full {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.01) GRANULARITY 1- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE events_core {CLICKHOUSE_CLUSTER_CLAUSE}ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.01) GRANULARITY 1- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE scores ON CLUSTER default+ SETTINGS enable_full_text_index = 1{CLICKHOUSE_CLUSTERED_ONLY:, alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT metadata['evaluator_id'] AFTER metadata- SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(metadata['evaluation_rule_id']), metadata['evaluation_rule_id'], metadata['job_configuration_id']) AFTER evaluator_id- SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.001) GRANULARITY 1- SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE scores {CLICKHOUSE_CLUSTER_CLAUSE}ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.001) GRANULARITY 1- SETTINGS alter_sync = 2;-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY+ {CLICKHOUSE_CLUSTERED_ONLY: SETTINGS alter_sync = 2};+ALTER TABLE events_core_mv {CLICKHOUSE_CLUSTER_CLAUSE} MODIFY QUERYSELECTproject_id,trace_id,diff --git a/packages/shared/clickhouse/migrations/clustered/0001_traces.down.sql b/packages/shared/clickhouse/migrations/clustered/0001_traces.down.sqldeleted file mode 100644index eb7f439d49..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0001_traces.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE traces ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0002_observations.down.sql b/packages/shared/clickhouse/migrations/clustered/0002_observations.down.sqldeleted file mode 100644index e3aaa0066f..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0002_observations.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE observations ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0002_observations.up.sql b/packages/shared/clickhouse/migrations/clustered/0002_observations.up.sqldeleted file mode 100644index 14d2359815..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0002_observations.up.sql+++ /dev/null@@ -1,47 +0,0 @@-CREATE TABLE observations ON CLUSTER default (- `id` String,- `trace_id` String,- `project_id` String,- `type` LowCardinality(String),- `parent_observation_id` Nullable(String),- `start_time` DateTime64(3),- `end_time` Nullable(DateTime64(3)),- `name` String,- `metadata` Map(LowCardinality(String), String),- `level` LowCardinality(String),- `status_message` Nullable(String),- `version` Nullable(String),- `input` Nullable(String) CODEC(ZSTD(3)),- `output` Nullable(String) CODEC(ZSTD(3)),- `provided_model_name` Nullable(String),- `internal_model_id` Nullable(String),- `model_parameters` Nullable(String),- `provided_usage_details` Map(LowCardinality(String), UInt64),- `usage_details` Map(LowCardinality(String), UInt64),- `provided_cost_details` Map(LowCardinality(String), Decimal64(12)),- `cost_details` Map(LowCardinality(String), Decimal64(12)),- `total_cost` Nullable(Decimal64(12)),- `completion_start_time` Nullable(DateTime64(3)),- `prompt_id` Nullable(String),- `prompt_name` Nullable(String),- `prompt_version` Nullable(UInt16),- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now(),- event_ts DateTime64(3),- is_deleted UInt8,- INDEX idx_id id TYPE bloom_filter() GRANULARITY 1,- INDEX idx_trace_id trace_id TYPE bloom_filter() GRANULARITY 1,- INDEX idx_project_id project_id TYPE bloom_filter() GRANULARITY 1-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(start_time)-PRIMARY KEY (- project_id,- `type`,- toDate(start_time)-)-ORDER BY (- project_id,- `type`,- toDate(start_time),- id-);-diff --git a/packages/shared/clickhouse/migrations/clustered/0003_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0003_scores.down.sqldeleted file mode 100644index 973a078d62..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0003_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE scores ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.down.sqldeleted file mode 100644index ea20f3d1b4..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_id project_id TYPE bloom_filter() GRANULARITY 1;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.up.sqldeleted file mode 100644index 990c0e21ea..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0004_drop_observations_index.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_project_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.down.sqldeleted file mode 100644index c9460635bd..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE traces ON CLUSTER default DROP INDEX IF EXISTS idx_session_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.up.sqldeleted file mode 100644index ca2a595b1e..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0005_add_session_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_session_id SETTINGS mutations_sync = 2;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.down.sqldeleted file mode 100644index 4c14ef6ff7..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE traces ON CLUSTER default DROP INDEX IF EXISTS idx_user_id;diff --git a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.up.sqldeleted file mode 100644index 7d9400f19e..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0006_add_user_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_user_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.down.sql b/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.down.sqldeleted file mode 100644index 24542e21ba..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0007_add_event_log.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE event_log ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.down.sql b/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.down.sqldeleted file mode 100644index 64a3c513c5..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.down.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE traces ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS environment SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.up.sql b/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.up.sqldeleted file mode 100644index 2aa2654a48..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0008_add_environments_column.up.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE traces ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id SETTINGS alter_sync = 2;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.down.sql b/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.down.sqldeleted file mode 100644index 0486b8e1db..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.down.sql+++ /dev/null@@ -1,7 +0,0 @@--- Drop materialized views-DROP VIEW IF EXISTS project_environments_traces_mv ON CLUSTER default;-DROP VIEW IF EXISTS project_environments_observations_mv ON CLUSTER default;-DROP VIEW IF EXISTS project_environments_scores_mv ON CLUSTER default;---- Drop the project_environments table-DROP TABLE IF EXISTS project_environments ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.up.sql b/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.up.sqldeleted file mode 100644index 6fadd913ec..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0009_add_project_environments.up.sql+++ /dev/null@@ -1,30 +0,0 @@--- Create the project_environments table with AggregatingMergeTree engine-CREATE TABLE project_environments ON CLUSTER default (- `project_id` String,- `environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))-) ENGINE = ReplicatedAggregatingMergeTree-ORDER BY (project_id);---- Create materialized view for traces-CREATE MATERIALIZED VIEW project_environments_traces_mv ON CLUSTER default TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM traces-GROUP BY project_id;---- Create materialized view for observations-CREATE MATERIALIZED VIEW project_environments_observations_mv ON CLUSTER default TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM observations-GROUP BY project_id;---- Create materialized view for scores-CREATE MATERIALIZED VIEW project_environments_scores_mv ON CLUSTER default TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM scores-GROUP BY project_id;diff --git a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.down.sqldeleted file mode 100644index c1688c2908..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS metadata;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.up.sqldeleted file mode 100644index ecc77043c7..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0010_add_metadata_column_on_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN metadata Map(LowCardinality(String), String) AFTER comment;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.down.sql b/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.down.sqldeleted file mode 100644index 29a74f1c78..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE blob_storage_file_log ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.up.sql b/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.up.sqldeleted file mode 100644index 664281ad4c..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0011_add_blob_storage_file_log.up.sql+++ /dev/null@@ -1,21 +0,0 @@-CREATE TABLE blob_storage_file_log ON CLUSTER default (- `id` String,- `project_id` String,- `entity_type` String,- `entity_id` String,- `event_id` String,-- `bucket_name` String,- `bucket_path` String,-- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now(),- event_ts DateTime64(3),- is_deleted UInt8,-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)- ORDER BY (- project_id,- entity_type,- entity_id,- event_id- );diff --git a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.down.sqldeleted file mode 100644index 2b8ef556f4..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS session_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.up.sqldeleted file mode 100644index 7837f8bdfb..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0012_add_session_id_column_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN session_id Nullable(String) AFTER trace_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.down.sqldeleted file mode 100644index 1b3d0f32bb..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.up.sqldeleted file mode 100644index 8a8bb8becd..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0013_drop_scores_trace_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;-diff --git a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.down.sql b/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.down.sqldeleted file mode 100644index 954e58dead..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.up.sql b/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.up.sqldeleted file mode 100644index 954e58dead..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0014_scores_modify_nullable_trace_id_column.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.down.sqldeleted file mode 100644index bd86b182c3..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.up.sqldeleted file mode 100644index 1b3d0f32bb..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0015_add_scores_trace_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.down.sqldeleted file mode 100644index abba171aad..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.up.sqldeleted file mode 100644index 2947da5227..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0016_add_scores_session_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.down.sqldeleted file mode 100644index 5993426be8..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS dataset_run_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.up.sqldeleted file mode 100644index 37a3af3aa7..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0017_add_run_id_column_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN dataset_run_id Nullable(String) AFTER session_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.down.sqldeleted file mode 100644index 714f111830..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.up.sqldeleted file mode 100644index 9675dd2916..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0018_add_scores_run_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.down.sql b/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.down.sqldeleted file mode 100644index 749fd5e106..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0019_analytics_traces.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP VIEW IF EXISTS analytics_traces ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.down.sql b/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.down.sqldeleted file mode 100644index 176598bb76..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP VIEW IF EXISTS analytics_observations ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.up.sql b/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.up.sqldeleted file mode 100644index 477a236683..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0020_analytics_observations.up.sql+++ /dev/null@@ -1,18 +0,0 @@-CREATE VIEW analytics_observations ON CLUSTER default AS-SELECT- project_id,- type,- toStartOfHour(start_time) AS hour,- uniq(id) AS countObservations,- max(level != 'DEFAULT') AS hasLevel,- max(provided_model_name IS NOT NULL) AS hasProvidedModelName,- max(length(provided_usage_details) > 0) AS hasProvidedUsageDetails,- max(length(provided_cost_details) > 0) AS hasProvidedCostDetails,- max(prompt_name IS NOT NULL) AS hasPromptName-FROM- observations-WHERE toStartOfHour(start_time) <= toStartOfHour(subtractHours(now(), 1))-GROUP BY- project_id,- type,- hour;diff --git a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.down.sqldeleted file mode 100644index 277f1eeed3..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0021_analytics_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP VIEW IF EXISTS analytics_scores ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.down.sqldeleted file mode 100644index a432f4214c..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE dataset_run_items ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.up.sqldeleted file mode 100644index c7363a13ed..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0022_dataset_run_items.up.sql+++ /dev/null@@ -1,36 +0,0 @@-CREATE TABLE dataset_run_items ON CLUSTER default (- -- primary identifiers- `id` String,- `project_id` String,- `dataset_run_id` String,- `dataset_item_id` String,- `dataset_id` String,- `trace_id` String,- `observation_id` Nullable(String),-- -- error field- `error` Nullable(String),-- -- timestamps- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now(),-- -- denormalized immutable dataset run fields- `dataset_run_name` String,- `dataset_run_description` Nullable(String),- `dataset_run_metadata` Map(LowCardinality(String), String),- `dataset_run_created_at` DateTime64(3),-- -- denormalized dataset item fields (mutable, but snapshots are relevant)- `dataset_item_input` Nullable(String) CODEC(ZSTD(3)), -- json- `dataset_item_expected_output` Nullable(String) CODEC(ZSTD(3)), -- json- `dataset_item_metadata` Map(LowCardinality(String), String),-- -- clickhouse engine fields- `event_ts` DateTime64(3),- `is_deleted` UInt8,-- -- For dataset item lookups- INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)-ORDER BY (project_id, dataset_id, dataset_run_id, id);\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.down.sql b/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.down.sqldeleted file mode 100644index 61ab89c3f8..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0023_traces_aggregating_merge_trees.down.sql+++ /dev/null@@ -1,12 +0,0 @@--- Drop materialized views first-DROP VIEW IF EXISTS traces_30d_amt_mv ON CLUSTER default;-DROP VIEW IF EXISTS traces_7d_amt_mv ON CLUSTER default;-DROP VIEW IF EXISTS traces_all_amt_mv ON CLUSTER default;---- Drop AMT tables-DROP TABLE IF EXISTS traces_30d_amt ON CLUSTER default;-DROP TABLE IF EXISTS traces_7d_amt ON CLUSTER default;-DROP TABLE IF EXISTS traces_all_amt ON CLUSTER default;---- Drop the Null table-DROP TABLE IF EXISTS traces_null ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.down.sqldeleted file mode 100644index b4425304bd..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE dataset_run_items_rmt ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sqldeleted file mode 100644index 853b0ced4b..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sql+++ /dev/null@@ -1,36 +0,0 @@-CREATE TABLE dataset_run_items_rmt ON CLUSTER default (- -- primary identifiers- `id` String,- `project_id` String,- `dataset_run_id` String,- `dataset_item_id` String,- `dataset_id` String,- `trace_id` String,- `observation_id` Nullable(String),-- -- error field- `error` Nullable(String),-- -- timestamps- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now(),-- -- denormalized immutable dataset run fields- `dataset_run_name` String,- `dataset_run_description` Nullable(String),- `dataset_run_metadata` Map(LowCardinality(String), String),- `dataset_run_created_at` DateTime64(3),-- -- denormalized dataset item fields (mutable, but snapshots are relevant)- `dataset_item_input` Nullable(String) CODEC(ZSTD(3)), -- json- `dataset_item_expected_output` Nullable(String) CODEC(ZSTD(3)), -- json- `dataset_item_metadata` Map(LowCardinality(String), String),-- -- clickhouse engine fields- `event_ts` DateTime64(3),- `is_deleted` UInt8,-- -- For dataset item lookups- INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,-) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)-ORDER BY (project_id, dataset_id, dataset_run_id, id);\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.down.sql b/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.down.sqldeleted file mode 100644index cf7d9cc96a..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_value SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_res_metadata_key SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.up.sql b/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.up.sqldeleted file mode 100644index 2f672c0c71..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0025_add_observations_metadata_indexes.up.sql+++ /dev/null@@ -1,4 +0,0 @@-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_key SETTINGS mutations_sync = 2;-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_res_metadata_value SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.down.sqldeleted file mode 100644index 32b5cfc9df..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE dataset_run_items_rmt ON CLUSTER default DROP INDEX IF EXISTS idx_trace_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.up.sqldeleted file mode 100644index ccc041e82c..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0026_add_trace_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE dataset_run_items_rmt ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_trace_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.down.sql b/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.down.sqldeleted file mode 100644index 8c502451c9..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.down.sql+++ /dev/null@@ -1,21 +0,0 @@--- Recreate materialized views for project_environments table-CREATE MATERIALIZED VIEW project_environments_traces_mv ON CLUSTER default TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM traces-GROUP BY project_id;--CREATE MATERIALIZED VIEW project_environments_observations_mv ON CLUSTER default TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM observations-GROUP BY project_id;--CREATE MATERIALIZED VIEW project_environments_scores_mv ON CLUSTER default TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM scores-GROUP BY project_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.up.sql b/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.up.sqldeleted file mode 100644index 99e2d32453..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0027_drop_project_environments_mvs.up.sql+++ /dev/null@@ -1,4 +0,0 @@--- Drop materialized views feeding project_environments table-DROP VIEW IF EXISTS project_environments_traces_mv ON CLUSTER default;-DROP VIEW IF EXISTS project_environments_observations_mv ON CLUSTER default;-DROP VIEW IF EXISTS project_environments_scores_mv ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.up.sql b/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.up.sqldeleted file mode 100644index 53d0c19bac..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0028_drop_traces_null_mvs.up.sql+++ /dev/null@@ -1,4 +0,0 @@--- Drop materialized views derived from traces_null table-DROP VIEW IF EXISTS traces_all_amt_mv ON CLUSTER default;-DROP VIEW IF EXISTS traces_7d_amt_mv ON CLUSTER default;-DROP VIEW IF EXISTS traces_30d_amt_mv ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.down.sql b/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.down.sqldeleted file mode 100644index 53e287bd07..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.down.sql+++ /dev/null@@ -1,179 +0,0 @@--- Recreate traces_null and trace amt tables-CREATE TABLE traces_null ON CLUSTER default-(- -- Identifiers- `project_id` String,- `id` String,- `start_time` DateTime64(3),- `end_time` Nullable(DateTime64(3)),- `name` Nullable(String),-- -- Metadata properties- `metadata` Map(LowCardinality(String), String),- `user_id` Nullable(String),- `session_id` Nullable(String),- `environment` String,- `tags` Array(String),- `version` Nullable(String),- `release` Nullable(String),-- -- UI properties - We make them nullable to prevent absent values being interpreted as overwrites.- `bookmarked` Nullable(Bool),- `public` Nullable(Bool),-- -- Aggregations -- DO NOT USE- `observation_ids` Array(String),- `score_ids` Array(String),- `cost_details` Map(String, Decimal64(12)),- `usage_details` Map(String, UInt64),- -- TODO: Do we want to aggregate/collect `levels` seen within the trace?-- -- Input/Output- `input` String,- `output` String,-- `created_at` DateTime64(3),- `updated_at` DateTime64(3),- `event_ts` DateTime64(3)-) Engine = Null();--CREATE TABLE traces_all_amt ON CLUSTER default-(- -- Identifiers- `project_id` String,- `id` String,- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time- `start_time` SimpleAggregateFunction(min, DateTime64(3)),- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),- `name` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- Metadata properties- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `environment` SimpleAggregateFunction(anyLast, String),- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `version` SimpleAggregateFunction(anyLast, Nullable(String)),- `release` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- UI properties- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),-- -- Aggregations -- DO NOT USE- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),-- -- Input/Output -> prefer correctness via argMax- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),-- `created_at` SimpleAggregateFunction(min, DateTime64(3)),- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),-- -- Indexes- INDEX idx_trace_id id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1-) Engine = AggregatingMergeTree()- ORDER BY (project_id, id);--CREATE TABLE traces_7d_amt ON CLUSTER default-(- -- Identifiers- `project_id` String,- `id` String,- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time- `start_time` SimpleAggregateFunction(min, DateTime64(3)),- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),- `name` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- Metadata properties- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `environment` SimpleAggregateFunction(anyLast, String),- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `version` SimpleAggregateFunction(anyLast, Nullable(String)),- `release` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- UI properties- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),-- -- Aggregations -- DO NOT USE- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),-- -- Input/Output -> prefer correctness via argMax- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),-- `created_at` SimpleAggregateFunction(min, DateTime64(3)),- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),-- -- Indexes- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1-) Engine = AggregatingMergeTree()- ORDER BY (project_id, id)- TTL toDate(start_time) + INTERVAL 7 DAY;--CREATE TABLE traces_30d_amt ON CLUSTER default-(- -- Identifiers- `project_id` String,- `id` String,- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time- `start_time` SimpleAggregateFunction(min, DateTime64(3)),- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),- `name` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- Metadata properties- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `environment` SimpleAggregateFunction(anyLast, String),- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `version` SimpleAggregateFunction(anyLast, Nullable(String)),- `release` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- UI properties- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),-- -- Aggregations -- DO NOT USE- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),-- -- Input/Output -> prefer correctness via argMax- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),-- `created_at` SimpleAggregateFunction(min, DateTime64(3)),- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),-- -- Indexes- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1-) Engine = AggregatingMergeTree()- ORDER BY (project_id, id)- TTL toDate(start_time) + INTERVAL 30 DAY;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.up.sql b/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.up.sqldeleted file mode 100644index 10f1d76df7..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0029_drop_traces_null_and_amt_tables.up.sql+++ /dev/null@@ -1,5 +0,0 @@--- Drop traces_null and trace amt tables-DROP TABLE IF EXISTS traces_null ON CLUSTER default;-DROP TABLE IF EXISTS traces_all_amt ON CLUSTER default;-DROP TABLE IF EXISTS traces_7d_amt ON CLUSTER default;-DROP TABLE IF EXISTS traces_30d_amt ON CLUSTER default;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.down.sqldeleted file mode 100644index 644c8b9080..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS execution_trace_id;diff --git a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sqldeleted file mode 100644index 0ea406281a..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String) SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.down.sqldeleted file mode 100644index df06748acd..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_name SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS usage_pricing_tier_id SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.up.sqldeleted file mode 100644index 5c9a17d3ec..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0031_add_usage_pricing_tier_columns.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_id Nullable(String) SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default ADD COLUMN usage_pricing_tier_name Nullable(String) SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sqldeleted file mode 100644index 3a351dd8be..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE dataset_run_items_rmt ON CLUSTER default DROP COLUMN IF EXISTS dataset_item_version;diff --git a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sqldeleted file mode 100644index a06e222c2f..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3)) SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.down.sqldeleted file mode 100644index de6cad399d..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.down.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS tool_definitions SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS tool_calls SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default DROP COLUMN IF EXISTS tool_call_names SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.up.sqldeleted file mode 100644index 535adb9aa8..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0033_add_tool_call_columns.up.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE observations ON CLUSTER default ADD COLUMN IF NOT EXISTS tool_definitions Map(String, String) DEFAULT map() SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default ADD COLUMN IF NOT EXISTS tool_calls Array(String) DEFAULT [] SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default ADD COLUMN IF NOT EXISTS tool_call_names Array(String) DEFAULT [] SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.down.sqldeleted file mode 100644index c9f9555be8..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS long_string_value;diff --git a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sqldeleted file mode 100644index 22d707f000..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3)) SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.down.sqldeleted file mode 100644index 695a0383e5..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.down.sql+++ /dev/null@@ -1,11 +0,0 @@-ALTER TABLE scores ON CLUSTER default- DROP COLUMN IF EXISTS ingestion_sdk_version- SETTINGS alter_sync = 2;--ALTER TABLE scores ON CLUSTER default- DROP COLUMN IF EXISTS ingestion_sdk_name- SETTINGS alter_sync = 2;--ALTER TABLE scores ON CLUSTER default- DROP COLUMN IF EXISTS ingestion_api_key- SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.up.sqldeleted file mode 100644index ae8e27730f..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0035_add_ingestion_attribution_columns.up.sql+++ /dev/null@@ -1,11 +0,0 @@-ALTER TABLE scores ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS alter_sync = 2;--ALTER TABLE scores ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS alter_sync = 2;--ALTER TABLE scores ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.down.sqldeleted file mode 100644index 8f5c9219e2..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0036_add_ingested_sdks_to_analytics_scores.down.sql+++ /dev/null@@ -1,22 +0,0 @@-DROP VIEW IF EXISTS analytics_scores ON CLUSTER default;-CREATE VIEW analytics_scores ON CLUSTER default AS-SELECT- project_id,- toStartOfHour(timestamp) AS hour,- uniq(id) AS countScores,- max(source = 'ANNOTATION') AS hasAnnotation,- max(source = 'API') AS hasApi,- max(source = 'EVAL') AS hasEval,- max(observation_id IS NOT NULL) AS hasObservationScore,- max(session_id IS NOT NULL) AS hasSessionScore,- max(dataset_run_id IS NOT NULL) AS hasDatasetRunScore,- max(data_type = 'BOOLEAN') AS hasBoolScore,- max(data_type = 'NUMERIC') AS hasNumericScore,- max(data_type = 'CATEGORICAL') AS hasCategoricalScore,- max(comment IS NOT NULL) AS hasComment-FROM- scores-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))-GROUP BY- project_id,- hour;diff --git a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.down.sql b/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.down.sqldeleted file mode 100644index 855010df2e..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.down.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE observations ON CLUSTER default DROP INDEX IF EXISTS idx_created_at;-ALTER TABLE traces ON CLUSTER default DROP INDEX IF EXISTS idx_created_at;-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_created_at;diff --git a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.up.sql b/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.up.sqldeleted file mode 100644index 60467e64f2..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0037_add_created_at_indexes.up.sql+++ /dev/null@@ -1,6 +0,0 @@-ALTER TABLE observations ON CLUSTER default ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE observations ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_created_at;-ALTER TABLE traces ON CLUSTER default ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE traces ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_created_at;-ALTER TABLE scores ON CLUSTER default ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1 SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default MATERIALIZE INDEX IF EXISTS idx_created_at;diff --git a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.up.sql b/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.up.sqldeleted file mode 100644index c43b1a8209..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0038_create_observations_batch_staging.up.sql+++ /dev/null@@ -1,51 +0,0 @@-CREATE TABLE IF NOT EXISTS observations_batch_staging ON CLUSTER default-(- id String,- trace_id String,- project_id String,- type LowCardinality(String),- parent_observation_id Nullable(String),- start_time DateTime64(3),- end_time Nullable(DateTime64(3)),- name String,- metadata Map(LowCardinality(String), String),- level LowCardinality(String),- status_message Nullable(String),- version Nullable(String),- input Nullable(String) CODEC(ZSTD(3)),- output Nullable(String) CODEC(ZSTD(3)),- provided_model_name Nullable(String),- internal_model_id Nullable(String),- model_parameters Nullable(String),- provided_usage_details Map(LowCardinality(String), UInt64),- usage_details Map(LowCardinality(String), UInt64),- provided_cost_details Map(LowCardinality(String), Decimal64(12)),- cost_details Map(LowCardinality(String), Decimal64(12)),- total_cost Nullable(Decimal64(12)),- usage_pricing_tier_id Nullable(String),- usage_pricing_tier_name Nullable(String),- tool_definitions Map(String, String),- tool_calls Array(String),- tool_call_names Array(String),- completion_start_time Nullable(DateTime64(3)),- prompt_id Nullable(String),- prompt_name Nullable(String),- prompt_version Nullable(UInt16),- created_at DateTime64(3) DEFAULT now(),- updated_at DateTime64(3) DEFAULT now(),- event_ts DateTime64(3),- is_deleted UInt8,- s3_first_seen_timestamp DateTime64(3),- environment LowCardinality(String) DEFAULT 'default'-)-ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)-PARTITION BY toStartOfInterval(s3_first_seen_timestamp, INTERVAL 3 MINUTE)-PRIMARY KEY (project_id, toDate(s3_first_seen_timestamp))-ORDER BY (- project_id,- toDate(s3_first_seen_timestamp),- trace_id,- id-)-TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR-SETTINGS ttl_only_drop_parts = 1;diff --git a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.down.sql b/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.down.sqldeleted file mode 100644index 3b3ae1aa49..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.down.sql+++ /dev/null@@ -1,5 +0,0 @@--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this--- table may predate this migration (created via dev-tables.sh on v4 preview--- deployments) and may hold production-grade event data. Rolling back drops--- that data regardless of which path created the table.-DROP TABLE IF EXISTS events_full ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.up.sql b/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.up.sqldeleted file mode 100644index 9e8a743590..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0039_create_events_full.up.sql+++ /dev/null@@ -1,124 +0,0 @@-CREATE TABLE IF NOT EXISTS events_full ON CLUSTER default-(- project_id String,- trace_id String,- span_id String,- parent_span_id String,-- start_time DateTime64(6),- end_time Nullable(DateTime64(6)),-- -- Core properties- name String,- type LowCardinality(String),- environment LowCardinality(String) DEFAULT 'default',- version String,- release String,- trace_name String,- user_id String,- session_id String,- tags Array(String),- level LowCardinality(String),- status_message String,- completion_start_time Nullable(DateTime64(6)),- is_app_root Bool DEFAULT false,-- -- Updateable properties- bookmarked Bool DEFAULT false,- public Bool DEFAULT false,-- -- Prompt- prompt_id String,- prompt_name String,- prompt_version Nullable(UInt16),-- -- Model- model_id String,- provided_model_name String,- model_parameters String,-- -- Usage and Cost- provided_usage_details Map(LowCardinality(String), UInt64),- usage_details Map(LowCardinality(String), UInt64),- provided_cost_details Map(LowCardinality(String), Decimal(18,12)),- cost_details Map(LowCardinality(String), Decimal(18,12)),- calculated_input_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0, cost_details))),- calculated_output_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'output') > 0, cost_details))),- calculated_total_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0 OR positionCaseInsensitive(x.1, 'output') > 0, cost_details))),- total_cost Decimal(18, 12) ALIAS cost_details['total'],-- usage_pricing_tier_id Nullable(String),- usage_pricing_tier_name Nullable(String),-- -- Tools- tool_definitions Map(String, String),- tool_calls Array(String),- tool_call_names Array(String),-- -- I/O- input String CODEC(ZSTD(3)),- input_length UInt64 MATERIALIZED lengthUTF8(input),- output String CODEC(ZSTD(3)),- output_length UInt64 MATERIALIZED lengthUTF8(output),-- -- Metadata- metadata_names Array(String),- metadata_values Array(String),-- -- Experiment properties- experiment_id String,- experiment_name String,- experiment_metadata_names Array(String),- experiment_metadata_values Array(String),- experiment_description String,- experiment_dataset_id String,- experiment_item_id String,- experiment_item_version Nullable(DateTime64(6)),- experiment_item_expected_output String,- experiment_item_metadata_names Array(String),- experiment_item_metadata_values Array(String),- experiment_item_root_span_id String,-- -- Source metadata (Instrumentation)- source LowCardinality(String),- service_name String,- service_version String,- scope_name String,- scope_version String,- telemetry_sdk_language LowCardinality(String),- telemetry_sdk_name String,- telemetry_sdk_version String,-- -- Generic props- blob_storage_file_path String,- event_bytes UInt64,- created_at DateTime64(6) DEFAULT now(),- updated_at DateTime64(6) DEFAULT now(),- event_ts DateTime64(6),- is_deleted UInt8,-- -- Indexes- INDEX idx_span_id span_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_user_id user_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_created_at created_at TYPE minmax GRANULARITY 1,- INDEX idx_updated_at updated_at TYPE minmax GRANULARITY 1,- INDEX idx_fts_input_low lower(input) TYPE text(tokenizer = splitByNonAlpha),- INDEX idx_fts_output_low lower(output) TYPE text(tokenizer = splitByNonAlpha),- INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),- INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)-)-ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)-PARTITION BY toYYYYMM(start_time)-PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))-ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)-SAMPLE BY xxHash32(trace_id)-SETTINGS- index_granularity_bytes = '64Mi', -- Default 10MiB. Prevents very small granules due to large rows.- merge_max_block_size_bytes = '64Mi',- enable_block_number_column = 1,- enable_block_offset_column = 1,- prewarm_mark_cache = 1,- prewarm_primary_key_cache = 1,- enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.down.sql b/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.down.sqldeleted file mode 100644index fd9ecf5910..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.down.sql+++ /dev/null@@ -1,5 +0,0 @@--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this--- table may predate this migration (created via dev-tables.sh on v4 preview--- deployments) and may hold production-grade event data. Rolling back drops--- that data regardless of which path created the table.-DROP TABLE IF EXISTS events_core ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.up.sql b/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.up.sqldeleted file mode 100644index ab53c989e8..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0040_create_events_core.up.sql+++ /dev/null@@ -1,123 +0,0 @@-CREATE TABLE IF NOT EXISTS events_core ON CLUSTER default-(- project_id String,- trace_id String,- span_id String,- parent_span_id String,-- start_time DateTime64(6),- end_time Nullable(DateTime64(6)),-- -- Core properties- name String,- type LowCardinality(String),- environment LowCardinality(String) DEFAULT 'default',- version String,- release String,- trace_name String,- user_id String,- session_id String,- tags Array(String),- level LowCardinality(String),- status_message String,- completion_start_time Nullable(DateTime64(6)),- is_app_root Bool DEFAULT false,-- -- Updateable properties- bookmarked Bool DEFAULT false,- public Bool DEFAULT false,-- -- Prompt- prompt_id String,- prompt_name String,- prompt_version Nullable(UInt16),-- -- Model- model_id String,- provided_model_name String,- model_parameters String,-- -- Usage- provided_usage_details Map(LowCardinality(String), UInt64),- usage_details Map(LowCardinality(String), UInt64),- provided_cost_details Map(LowCardinality(String), Decimal(18,12)),- cost_details Map(LowCardinality(String), Decimal(18,12)),- calculated_input_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0, cost_details))),- calculated_output_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'output') > 0, cost_details))),- calculated_total_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0 OR positionCaseInsensitive(x.1, 'output') > 0, cost_details))),- total_cost Decimal(18, 12) ALIAS cost_details['total'],-- usage_pricing_tier_id Nullable(String),- usage_pricing_tier_name Nullable(String),-- -- Tools- tool_definitions Map(String, String),- tool_calls Array(String),- tool_call_names Array(String),-- -- I/O- input String,- input_length UInt64 MATERIALIZED lengthUTF8(input),- output String,- output_length UInt64 MATERIALIZED lengthUTF8(output),-- -- Metadata- metadata_names Array(String),- metadata_values Array(String),-- -- Experiment properties- experiment_id String,- experiment_name String,- experiment_metadata_names Array(String),- experiment_metadata_values Array(String),- experiment_description String,- experiment_dataset_id String,- experiment_item_id String,- experiment_item_version Nullable(DateTime64(6)),- experiment_item_expected_output String,- experiment_item_metadata_names Array(String),- experiment_item_metadata_values Array(String),- experiment_item_root_span_id String,-- -- Source metadata (Instrumentation)- source LowCardinality(String),- service_name String,- service_version String,- scope_name String,- scope_version String,- telemetry_sdk_language LowCardinality(String),- telemetry_sdk_name String,- telemetry_sdk_version String,-- -- Generic props- blob_storage_file_path String,- event_bytes UInt64,- created_at DateTime64(6) DEFAULT now(),- updated_at DateTime64(6) DEFAULT now(),- event_ts DateTime64(6),- is_deleted UInt8,-- -- Indexes- INDEX idx_span_id span_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_user_id user_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_created_at created_at TYPE minmax GRANULARITY 1,- INDEX idx_updated_at updated_at TYPE minmax GRANULARITY 1,- INDEX idx_provided_model_name provided_model_name TYPE bloom_filter(0.01) GRANULARITY 2,- INDEX idx_experiment_id experiment_id TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_metadata_names metadata_names TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),- INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)-)-ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)-PARTITION BY toYYYYMM(start_time)-PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))-ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)-SAMPLE BY xxHash32(trace_id)-SETTINGS- enable_block_number_column = 1,- enable_block_offset_column = 1,- prewarm_mark_cache = 1,- prewarm_primary_key_cache = 1,- enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.down.sql b/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.down.sqldeleted file mode 100644index dacd2eb194..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.down.sql+++ /dev/null@@ -1,4 +0,0 @@--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this--- view may predate this migration (created via dev-tables.sh on v4 preview--- deployments). Dropping it stops events_core from being populated.-DROP VIEW IF EXISTS events_core_mv ON CLUSTER default;diff --git a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.up.sql b/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.up.sqldeleted file mode 100644index 3c2db389da..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0041_create_events_core_mv.up.sql+++ /dev/null@@ -1,69 +0,0 @@-CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv ON CLUSTER default TO events_core AS-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted-FROM events_full;diff --git a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.down.sqldeleted file mode 100644index 1ba2f66a48..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.down.sql+++ /dev/null@@ -1,82 +0,0 @@-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted-FROM events_full-SETTINGS enable_full_text_index = 1;--ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE observations_batch_staging ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE observations_batch_staging ON CLUSTER default DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE observations_batch_staging ON CLUSTER default DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1, alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.up.sqldeleted file mode 100644index b751b5a26b..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0042_add_events_ingestion_attribution_columns.up.sql+++ /dev/null@@ -1,103 +0,0 @@-ALTER TABLE observations_batch_staging ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE observations_batch_staging ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE observations_batch_staging ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE events_full ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE events_core ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'- SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted,- ingestion_api_key,- ingestion_sdk_name,- ingestion_sdk_version-FROM events_full-SETTINGS enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.down.sql b/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.down.sqldeleted file mode 100644index 9f1ef0ead0..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE events_full ON CLUSTER default DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1, alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.up.sql b/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.up.sqldeleted file mode 100644index e773b902f0..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0043_add_events_metadata_ngram_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE events_full ON CLUSTER default ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1, alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.down.sqldeleted file mode 100644index 3291a7b3ad..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0046_drop_dataset_run_items.down.sql+++ /dev/null@@ -1,38 +0,0 @@--- WARNING: This recreates the table structure only. Any data held in--- dataset_run_items before the up migration dropped it is NOT restored.-CREATE TABLE IF NOT EXISTS dataset_run_items ON CLUSTER default (- -- primary identifiers- `id` String,- `project_id` String,- `dataset_run_id` String,- `dataset_item_id` String,- `dataset_id` String,- `trace_id` String,- `observation_id` Nullable(String),-- -- error field- `error` Nullable(String),-- -- timestamps- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now(),-- -- denormalized immutable dataset run fields- `dataset_run_name` String,- `dataset_run_description` Nullable(String),- `dataset_run_metadata` Map(LowCardinality(String), String),- `dataset_run_created_at` DateTime64(3),-- -- denormalized dataset item fields (mutable, but snapshots are relevant)- `dataset_item_input` Nullable(String) CODEC(ZSTD(3)), -- json- `dataset_item_expected_output` Nullable(String) CODEC(ZSTD(3)), -- json- `dataset_item_metadata` Map(LowCardinality(String), String),-- -- clickhouse engine fields- `event_ts` DateTime64(3),- `is_deleted` UInt8,-- -- For dataset item lookups- INDEX idx_dataset_item dataset_item_id TYPE bloom_filter(0.001) GRANULARITY 1,-) ENGINE = ReplacingMergeTree(event_ts, is_deleted)-ORDER BY (project_id, dataset_id, dataset_run_id, id);diff --git a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.down.sql b/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.down.sqldeleted file mode 100644index 490efbde32..0000000000--- a/packages/shared/clickhouse/migrations/clustered/0047_add_eval_execution_columns.down.sql+++ /dev/null@@ -1,88 +0,0 @@-ALTER TABLE events_core_mv ON CLUSTER default MODIFY QUERY-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted,- ingestion_api_key,- ingestion_sdk_name,- ingestion_sdk_version-FROM events_full-SETTINGS enable_full_text_index = 1;--ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_full ON CLUSTER default DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE events_core ON CLUSTER default DROP INDEX IF EXISTS idx_evaluator_id SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1, alter_sync = 2;-ALTER TABLE events_core ON CLUSTER default DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1, alter_sync = 2;--ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_evaluator_id SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS evaluator_id SETTINGS alter_sync = 2;-ALTER TABLE scores ON CLUSTER default DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0001_traces.down.sql b/packages/shared/clickhouse/migrations/unclustered/0001_traces.down.sqldeleted file mode 100644index 95a0b29e4b..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0001_traces.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE traces;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0001_traces.up.sql b/packages/shared/clickhouse/migrations/unclustered/0001_traces.up.sqldeleted file mode 100644index 6e0d1c5122..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0001_traces.up.sql+++ /dev/null@@ -1,32 +0,0 @@-CREATE TABLE traces (- `id` String,- `timestamp` DateTime64(3),- `name` String,- `user_id` Nullable(String),- `metadata` Map(LowCardinality(String), String),- `release` Nullable(String),- `version` Nullable(String),- `project_id` String,- `public` Bool,- `bookmarked` Bool,- `tags` Array(String),- `input` Nullable(String) CODEC(ZSTD(3)),- `output` Nullable(String) CODEC(ZSTD(3)),- `session_id` Nullable(String),- `created_at` DateTime64(3) DEFAULT now(),- updated_at DateTime64(3) DEFAULT now(),- `event_ts` DateTime64(3),- `is_deleted` UInt8,- INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1,- INDEX idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1-) ENGINE = ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)-PRIMARY KEY (- project_id,- toDate(timestamp)-)-ORDER BY (- project_id,- toDate(timestamp),- id-);diff --git a/packages/shared/clickhouse/migrations/unclustered/0002_observations.down.sql b/packages/shared/clickhouse/migrations/unclustered/0002_observations.down.sqldeleted file mode 100644index 656776f502..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0002_observations.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE observations;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0003_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0003_scores.down.sqldeleted file mode 100644index c5ea562063..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0003_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE scores;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0003_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0003_scores.up.sqldeleted file mode 100644index e2b4a7bd1c..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0003_scores.up.sql+++ /dev/null@@ -1,33 +0,0 @@-CREATE TABLE scores (- `id` String,- `timestamp` DateTime64(3),- `project_id` String,- `trace_id` String,- `observation_id` Nullable(String),- `name` String,- `value` Float64,- `source` String,- `comment` Nullable(String) CODEC(ZSTD(1)),- `author_user_id` Nullable(String),- `config_id` Nullable(String),- `data_type` String,- `string_value` Nullable(String),- `queue_id` Nullable(String),- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now(),- event_ts DateTime64(3),- `is_deleted` UInt8,- INDEX idx_id id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1-) ENGINE = ReplacingMergeTree(event_ts, is_deleted) Partition by toYYYYMM(timestamp)-PRIMARY KEY (- project_id,- toDate(timestamp),- name-)-ORDER BY (- project_id,- toDate(timestamp),- name,- id-)diff --git a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.down.sqldeleted file mode 100644index 5e36747336..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_project_id project_id TYPE bloom_filter() GRANULARITY 1;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.up.sqldeleted file mode 100644index 1c008a4afc..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0004_drop_observations_index.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE observations DROP INDEX IF EXISTS idx_project_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.down.sqldeleted file mode 100644index 7d2976b9d0..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE traces DROP INDEX IF EXISTS idx_session_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.up.sqldeleted file mode 100644index 6d79dbe3b9..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0005_add_session_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE traces ADD INDEX IF NOT EXISTS idx_session_id session_id TYPE bloom_filter() GRANULARITY 1;-ALTER TABLE traces MATERIALIZE INDEX IF EXISTS idx_session_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.down.sqldeleted file mode 100644index a867a7d368..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE traces DROP INDEX IF EXISTS idx_user_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.up.sqldeleted file mode 100644index da1d3460ab..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0006_add_user_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE traces ADD INDEX IF NOT EXISTS idx_user_id user_id TYPE bloom_filter() GRANULARITY 1;-ALTER TABLE traces MATERIALIZE INDEX IF EXISTS idx_user_id;diff --git a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.down.sql b/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.down.sqldeleted file mode 100644index 3996548c5c..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE event_log;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.up.sql b/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.up.sqldeleted file mode 100644index 40afc72e63..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0007_add_event_log.up.sql+++ /dev/null@@ -1,19 +0,0 @@-CREATE TABLE event_log-(- `id` String,- `project_id` String,- `entity_type` String,- `entity_id` String,- `event_id` Nullable(String),-- `bucket_name` String,- `bucket_path` String,-- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now()-) ENGINE = MergeTree()- ORDER BY (- project_id,- entity_type,- entity_id- );diff --git a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.down.sql b/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.down.sqldeleted file mode 100644index 1c59e0df4b..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.down.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE traces DROP COLUMN IF EXISTS environment;-ALTER TABLE observations DROP COLUMN IF EXISTS environment;-ALTER TABLE scores DROP COLUMN IF EXISTS environment;diff --git a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.up.sql b/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.up.sqldeleted file mode 100644index b62a70f75e..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0008_add_environments_column.up.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE traces ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;-ALTER TABLE observations ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;-ALTER TABLE scores ADD COLUMN environment LowCardinality(String) DEFAULT 'default' AFTER project_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.down.sql b/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.down.sqldeleted file mode 100644index a1622de24d..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0009_add_project_environments.down.sql+++ /dev/null@@ -1,7 +0,0 @@--- Drop materialized views-DROP VIEW IF EXISTS project_environments_traces_mv;-DROP VIEW IF EXISTS project_environments_observations_mv;-DROP VIEW IF EXISTS project_environments_scores_mv;---- Drop the project_environments table-DROP TABLE IF EXISTS project_environments;diff --git a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.down.sqldeleted file mode 100644index 1c5cc36d72..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP COLUMN IF EXISTS metadata;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.up.sqldeleted file mode 100644index 55dbefbe95..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0010_add_metadata_column_on_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ADD COLUMN metadata Map(LowCardinality(String), String) AFTER comment;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.down.sql b/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.down.sqldeleted file mode 100644index a23949b45c..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0011_add_blob_storage_file_log.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE blob_storage_file_log;diff --git a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.down.sqldeleted file mode 100644index 1967d45f8b..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP COLUMN IF EXISTS session_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.up.sqldeleted file mode 100644index 3c8c9098c5..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0012_add_session_id_column_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ADD COLUMN session_id Nullable(String) AFTER trace_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.down.sqldeleted file mode 100644index a5fb46bf03..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.up.sqldeleted file mode 100644index eeb81267d8..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0013_drop_scores_trace_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;-diff --git a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.down.sql b/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.down.sqldeleted file mode 100644index 5938637ec5..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.up.sql b/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.up.sqldeleted file mode 100644index 5938637ec5..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0014_scores_modify_nullable_trace_id_column.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores MODIFY COLUMN trace_id Nullable(String) SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.down.sqldeleted file mode 100644index fd7ba11787..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.up.sqldeleted file mode 100644index 3cb554b4b7..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0015_add_scores_trace_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_trace_observation (project_id, trace_id, observation_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_trace_observation SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.down.sqldeleted file mode 100644index 9f60ffbdd3..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.up.sqldeleted file mode 100644index a0a39e9226..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0016_add_scores_session_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_session (project_id, session_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_session SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.down.sqldeleted file mode 100644index f511371464..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP COLUMN IF EXISTS dataset_run_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.up.sqldeleted file mode 100644index 74f42181a8..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0017_add_run_id_column_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ADD COLUMN dataset_run_id Nullable(String) AFTER session_id SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.down.sqldeleted file mode 100644index 530007aa81..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.up.sqldeleted file mode 100644index 4b36b72b52..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0018_add_scores_run_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_project_dataset_run (project_id, dataset_run_id) TYPE bloom_filter(0.001) GRANULARITY 1 SETTINGS mutations_sync = 2;-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_project_dataset_run SETTINGS mutations_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.down.sql b/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.down.sqldeleted file mode 100644index b073974b40..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP VIEW IF EXISTS analytics_traces;diff --git a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.up.sql b/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.up.sqldeleted file mode 100644index 7562119645..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0019_analytics_traces.up.sql+++ /dev/null@@ -1,15 +0,0 @@-CREATE VIEW analytics_traces AS-SELECT- project_id,- toStartOfHour(timestamp) AS hour,- uniq(id) AS countTraces,- max(user_id IS NOT NULL) AS hasUsers,- max(session_id IS NOT NULL) AS hasSessions,- max(if(environment != 'default', 1, 0)) AS hasEnvironments,- max(length(tags) > 0) AS hasTags-FROM- traces-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))-GROUP BY- project_id,- hour;diff --git a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.down.sql b/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.down.sqldeleted file mode 100644index fd395b4199..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0020_analytics_observations.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP VIEW IF EXISTS analytics_observations;diff --git a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.down.sqldeleted file mode 100644index b8723146c4..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP VIEW IF EXISTS analytics_scores;diff --git a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.up.sqldeleted file mode 100644index e3ce3772d5..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0021_analytics_scores.up.sql+++ /dev/null@@ -1,21 +0,0 @@-CREATE VIEW analytics_scores AS-SELECT- project_id,- toStartOfHour(timestamp) AS hour,- uniq(id) AS countScores,- max(source = 'ANNOTATION') AS hasAnnotation,- max(source = 'API') AS hasApi,- max(source = 'EVAL') AS hasEval,- max(observation_id IS NOT NULL) AS hasObservationScore,- max(session_id IS NOT NULL) AS hasSessionScore,- max(dataset_run_id IS NOT NULL) AS hasDatasetRunScore,- max(data_type = 'BOOLEAN') AS hasBoolScore,- max(data_type = 'NUMERIC') AS hasNumericScore,- max(data_type = 'CATEGORICAL') AS hasCategoricalScore,- max(comment IS NOT NULL) AS hasComment-FROM- scores-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))-GROUP BY- project_id,- hour;diff --git a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.down.sqldeleted file mode 100644index c45faad79c..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0022_dataset_run_items.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE dataset_run_items;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.down.sql b/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.down.sqldeleted file mode 100644index 7042dec750..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.down.sql+++ /dev/null@@ -1,12 +0,0 @@--- Drop materialized views first-DROP VIEW IF EXISTS traces_30d_amt_mv;-DROP VIEW IF EXISTS traces_7d_amt_mv;-DROP VIEW IF EXISTS traces_all_amt_mv;---- Drop AMT tables-DROP TABLE IF EXISTS traces_30d_amt;-DROP TABLE IF EXISTS traces_7d_amt;-DROP TABLE IF EXISTS traces_all_amt;---- Drop the Null table-DROP TABLE IF EXISTS traces_null;diff --git a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.up.sql b/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.up.sqldeleted file mode 100644index b9c91a50ba..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0023_traces_aggregating_merge_trees.up.sql+++ /dev/null@@ -1,300 +0,0 @@--- Create a Null table that serves as a trigger for all materialized views.--- We use a Null engine here to avoid storing intermediate results and save on storage.-CREATE TABLE traces_null-(- -- Identifiers- `project_id` String,- `id` String,- `start_time` DateTime64(3),- `end_time` Nullable(DateTime64(3)),- `name` Nullable(String),-- -- Metadata properties- `metadata` Map(LowCardinality(String), String),- `user_id` Nullable(String),- `session_id` Nullable(String),- `environment` String,- `tags` Array(String),- `version` Nullable(String),- `release` Nullable(String),-- -- UI properties - We make them nullable to prevent absent values being interpreted as overwrites.- `bookmarked` Nullable(Bool),- `public` Nullable(Bool),-- -- Aggregations- `observation_ids` Array(String),- `score_ids` Array(String),- `cost_details` Map(String, Decimal64(12)),- `usage_details` Map(String, UInt64),- -- TODO: Do we want to aggregate/collect `levels` seen within the trace?-- -- Input/Output- `input` String,- `output` String,-- `created_at` DateTime64(3),- `updated_at` DateTime64(3),- `event_ts` DateTime64(3)-) Engine = Null();---- Create the all AMT-CREATE TABLE traces_all_amt-(- -- Identifiers- `project_id` String,- `id` String,- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time- `start_time` SimpleAggregateFunction(min, DateTime64(3)),- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),- `name` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- Metadata properties- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `environment` SimpleAggregateFunction(anyLast, String),- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `version` SimpleAggregateFunction(anyLast, Nullable(String)),- `release` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- UI properties- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),-- -- Aggregations- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),-- -- Input/Output -> prefer correctness via argMax- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),-- `created_at` SimpleAggregateFunction(min, DateTime64(3)),- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),-- -- Indexes- INDEX idx_trace_id id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1-) Engine = AggregatingMergeTree()- ORDER BY (project_id, id);---- Create materialized view for all_amt-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv TO traces_all_amt AS-SELECT- -- Identifiers- tn.project_id as project_id,- tn.id as id,- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time- min(tn.start_time) as start_time,- max(coalesce(tn.end_time, tn.start_time)) as end_time,- anyLast(tn.name) as name,-- -- Metadata properties- maxMap(tn.metadata) as metadata,- anyLast(tn.user_id) as user_id,- anyLast(tn.session_id) as session_id,- anyLast(tn.environment) as environment,- groupUniqArrayArray(tn.tags) as tags,- anyLast(tn.version) as version,- anyLast(tn.release) as release,-- -- UI properties- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,-- -- Aggregations- groupUniqArrayArray(tn.observation_ids) as observation_ids,- groupUniqArrayArray(tn.score_ids) as score_ids,- sumMap(tn.cost_details) as cost_details,- sumMap(tn.usage_details) as usage_details,-- -- Input/Output- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,-- min(tn.created_at) as created_at,- max(tn.updated_at) as updated_at-FROM traces_null tn-GROUP BY project_id, id;---- Create the 7-day TTL AMT-CREATE TABLE traces_7d_amt-(- -- Identifiers- `project_id` String,- `id` String,- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time- `start_time` SimpleAggregateFunction(min, DateTime64(3)),- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),- `name` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- Metadata properties- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `environment` SimpleAggregateFunction(anyLast, String),- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `version` SimpleAggregateFunction(anyLast, Nullable(String)),- `release` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- UI properties- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),-- -- Aggregations- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),-- -- Input/Output -> prefer correctness via argMax- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),-- `created_at` SimpleAggregateFunction(min, DateTime64(3)),- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),-- -- Indexes- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1-) Engine = AggregatingMergeTree()- ORDER BY (project_id, id)- TTL toDate(start_time) + INTERVAL 7 DAY;---- Create materialized view for 7d_amt-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv TO traces_7d_amt AS-SELECT- -- Identifiers- tn.project_id as project_id,- tn.id as id,- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time- min(tn.start_time) as start_time,- max(coalesce(tn.end_time, tn.start_time)) as end_time,- anyLast(tn.name) as name,-- -- Metadata properties- maxMap(tn.metadata) as metadata,- anyLast(tn.user_id) as user_id,- anyLast(tn.session_id) as session_id,- anyLast(tn.environment) as environment,- groupUniqArrayArray(tn.tags) as tags,- anyLast(tn.version) as version,- anyLast(tn.release) as release,-- -- UI properties- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,-- -- Aggregations- groupUniqArrayArray(tn.observation_ids) as observation_ids,- groupUniqArrayArray(tn.score_ids) as score_ids,- sumMap(tn.cost_details) as cost_details,- sumMap(tn.usage_details) as usage_details,-- -- Input/Output- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,-- min(tn.created_at) as created_at,- max(tn.updated_at) as updated_at-FROM traces_null tn-GROUP BY project_id, id;---- Create the 30-day TTL AMT-CREATE TABLE traces_30d_amt-(- -- Identifiers- `project_id` String,- `id` String,- `timestamp` SimpleAggregateFunction(min, DateTime64(3)), -- Backward compatibility: redundant with start_time- `start_time` SimpleAggregateFunction(min, DateTime64(3)),- `end_time` SimpleAggregateFunction(max, Nullable(DateTime64(3))),- `name` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- Metadata properties- `metadata` SimpleAggregateFunction(maxMap, Map(String, String)),- `user_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `session_id` SimpleAggregateFunction(anyLast, Nullable(String)),- `environment` SimpleAggregateFunction(anyLast, String),- `tags` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `version` SimpleAggregateFunction(anyLast, Nullable(String)),- `release` SimpleAggregateFunction(anyLast, Nullable(String)),-- -- UI properties- `bookmarked` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),- `public` AggregateFunction(argMax, Nullable(Bool), DateTime64(3)),-- -- Aggregations- `observation_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `score_ids` SimpleAggregateFunction(groupUniqArrayArray, Array(String)),- `cost_details` SimpleAggregateFunction(sumMap, Map(String, Decimal(38, 12))),- `usage_details` SimpleAggregateFunction(sumMap, Map(String, UInt64)),-- -- Input/Output -> prefer correctness via argMax- `input` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),- `output` AggregateFunction(argMax, String, DateTime64(3)) CODEC (ZSTD(3)),-- `created_at` SimpleAggregateFunction(min, DateTime64(3)),- `updated_at` SimpleAggregateFunction(max, DateTime64(3)),-- -- Indexes- INDEX idx_user_id user_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_session_id session_id TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_name name TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_version version TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_release release TYPE bloom_filter(0.001) GRANULARITY 1,- INDEX idx_tags tags TYPE bloom_filter(0.001) GRANULARITY 1-) Engine = AggregatingMergeTree()- ORDER BY (project_id, id)- TTL toDate(start_time) + INTERVAL 30 DAY;---- Create materialized view for 30d_amt-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv TO traces_30d_amt AS-SELECT- -- Identifiers- tn.project_id as project_id,- tn.id as id,- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time- min(tn.start_time) as start_time,- max(coalesce(tn.end_time, tn.start_time)) as end_time,- anyLast(tn.name) as name,-- -- Metadata properties- maxMap(tn.metadata) as metadata,- anyLast(tn.user_id) as user_id,- anyLast(tn.session_id) as session_id,- anyLast(tn.environment) as environment,- groupUniqArrayArray(tn.tags) as tags,- anyLast(tn.version) as version,- anyLast(tn.release) as release,-- -- UI properties- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,-- -- Aggregations- groupUniqArrayArray(tn.observation_ids) as observation_ids,- groupUniqArrayArray(tn.score_ids) as score_ids,- sumMap(tn.cost_details) as cost_details,- sumMap(tn.usage_details) as usage_details,-- -- Input/Output- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,-- min(tn.created_at) as created_at,- max(tn.updated_at) as updated_at-FROM traces_null tn-GROUP BY project_id, id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.down.sql b/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.down.sqldeleted file mode 100644index 017daef96f..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0024_dataset_run_items.down.sql+++ /dev/null@@ -1 +0,0 @@-DROP TABLE dataset_run_items_rmt;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.down.sql b/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.down.sqldeleted file mode 100644index b8762b84b7..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE observations DROP INDEX IF EXISTS idx_res_metadata_value;-ALTER TABLE observations DROP INDEX IF EXISTS idx_res_metadata_key;diff --git a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.up.sql b/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.up.sqldeleted file mode 100644index 5e65eff735..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0025_add_observations_metadata_indexes.up.sql+++ /dev/null@@ -1,4 +0,0 @@-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_res_metadata_key mapKeys(metadata) TYPE bloom_filter(0.01) GRANULARITY 1;-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_res_metadata_value mapValues(metadata) TYPE bloom_filter(0.01) GRANULARITY 1;-ALTER TABLE observations MATERIALIZE INDEX IF EXISTS idx_res_metadata_key;-ALTER TABLE observations MATERIALIZE INDEX IF EXISTS idx_res_metadata_value;diff --git a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.down.sqldeleted file mode 100644index 080d700fa6..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE dataset_run_items_rmt DROP INDEX IF EXISTS idx_trace_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.up.sqldeleted file mode 100644index 696dbd74ff..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0026_add_trace_id_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE dataset_run_items_rmt ADD INDEX IF NOT EXISTS idx_trace_id trace_id TYPE bloom_filter(0.001) GRANULARITY 1;-ALTER TABLE dataset_run_items_rmt MATERIALIZE INDEX IF EXISTS idx_trace_id;diff --git a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.down.sql b/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.down.sqldeleted file mode 100644index 49e56f9e89..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.down.sql+++ /dev/null@@ -1,21 +0,0 @@--- Recreate materialized views for project_environments table-CREATE MATERIALIZED VIEW project_environments_traces_mv TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM traces-GROUP BY project_id;--CREATE MATERIALIZED VIEW project_environments_observations_mv TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM observations-GROUP BY project_id;--CREATE MATERIALIZED VIEW project_environments_scores_mv TO project_environments AS-SELECT- project_id,- groupUniqArray(environment) AS environments-FROM scores-GROUP BY project_id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.up.sql b/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.up.sqldeleted file mode 100644index a4fe385fa5..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0027_drop_project_environments_mvs.up.sql+++ /dev/null@@ -1,4 +0,0 @@--- Drop materialized views feeding project_environments table-DROP VIEW IF EXISTS project_environments_traces_mv;-DROP VIEW IF EXISTS project_environments_observations_mv;-DROP VIEW IF EXISTS project_environments_scores_mv;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.down.sql b/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.down.sqldeleted file mode 100644index 0d1a7a6e32..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.down.sql+++ /dev/null@@ -1,114 +0,0 @@--- Recreate materialized views derived from traces_null table-CREATE MATERIALIZED VIEW IF NOT EXISTS traces_all_amt_mv TO traces_all_amt AS-SELECT- -- Identifiers- tn.project_id as project_id,- tn.id as id,- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time- min(tn.start_time) as start_time,- max(coalesce(tn.end_time, tn.start_time)) as end_time,- anyLast(tn.name) as name,-- -- Metadata properties- maxMap(tn.metadata) as metadata,- anyLast(tn.user_id) as user_id,- anyLast(tn.session_id) as session_id,- anyLast(tn.environment) as environment,- groupUniqArrayArray(tn.tags) as tags,- anyLast(tn.version) as version,- anyLast(tn.release) as release,-- -- UI properties- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,-- -- Aggregations -- DO NOT USE- groupUniqArrayArray(tn.observation_ids) as observation_ids,- groupUniqArrayArray(tn.score_ids) as score_ids,- sumMap(tn.cost_details) as cost_details,- sumMap(tn.usage_details) as usage_details,-- -- Input/Output- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,-- min(tn.created_at) as created_at,- max(tn.updated_at) as updated_at-FROM traces_null tn-GROUP BY project_id, id;--CREATE MATERIALIZED VIEW IF NOT EXISTS traces_7d_amt_mv TO traces_7d_amt AS-SELECT- -- Identifiers- tn.project_id as project_id,- tn.id as id,- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time- min(tn.start_time) as start_time,- max(coalesce(tn.end_time, tn.start_time)) as end_time,- anyLast(tn.name) as name,-- -- Metadata properties- maxMap(tn.metadata) as metadata,- anyLast(tn.user_id) as user_id,- anyLast(tn.session_id) as session_id,- anyLast(tn.environment) as environment,- groupUniqArrayArray(tn.tags) as tags,- anyLast(tn.version) as version,- anyLast(tn.release) as release,-- -- UI properties- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,-- -- Aggregations -- DO NOT USE- groupUniqArrayArray(tn.observation_ids) as observation_ids,- groupUniqArrayArray(tn.score_ids) as score_ids,- sumMap(tn.cost_details) as cost_details,- sumMap(tn.usage_details) as usage_details,-- -- Input/Output- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,-- min(tn.created_at) as created_at,- max(tn.updated_at) as updated_at-FROM traces_null tn-GROUP BY project_id, id;--CREATE MATERIALIZED VIEW IF NOT EXISTS traces_30d_amt_mv TO traces_30d_amt AS-SELECT- -- Identifiers- tn.project_id as project_id,- tn.id as id,- min(tn.start_time) as timestamp, -- Backward compatibility: redundant with start_time- min(tn.start_time) as start_time,- max(coalesce(tn.end_time, tn.start_time)) as end_time,- anyLast(tn.name) as name,-- -- Metadata properties- maxMap(tn.metadata) as metadata,- anyLast(tn.user_id) as user_id,- anyLast(tn.session_id) as session_id,- anyLast(tn.environment) as environment,- groupUniqArrayArray(tn.tags) as tags,- anyLast(tn.version) as version,- anyLast(tn.release) as release,-- -- UI properties- argMaxState(tn.bookmarked, if(tn.bookmarked is not null, tn.event_ts, toDateTime64(0, 3))) as bookmarked,- argMaxState(tn.public, if(tn.public is not null, tn.event_ts, toDateTime64(0, 3))) as public,-- -- Aggregations -- DO NOT USE- groupUniqArrayArray(tn.observation_ids) as observation_ids,- groupUniqArrayArray(tn.score_ids) as score_ids,- sumMap(tn.cost_details) as cost_details,- sumMap(tn.usage_details) as usage_details,-- -- Input/Output- argMaxState(tn.input, if(tn.input <> '', tn.event_ts, toDateTime64(0, 3))) as input,- argMaxState(tn.output, if(tn.output <> '', tn.event_ts, toDateTime64(0, 3))) as output,-- min(tn.created_at) as created_at,- max(tn.updated_at) as updated_at-FROM traces_null tn-GROUP BY project_id, id;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.up.sql b/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.up.sqldeleted file mode 100644index 2f95c940f4..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0028_drop_traces_null_mvs.up.sql+++ /dev/null@@ -1,4 +0,0 @@--- Drop materialized views derived from traces_null table-DROP VIEW IF EXISTS traces_all_amt_mv;-DROP VIEW IF EXISTS traces_7d_amt_mv;-DROP VIEW IF EXISTS traces_30d_amt_mv;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.up.sql b/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.up.sqldeleted file mode 100644index fc99886fc1..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0029_drop_traces_null_and_amt_tables.up.sql+++ /dev/null@@ -1,5 +0,0 @@--- Drop traces_null and trace amt tables-DROP TABLE IF EXISTS traces_null;-DROP TABLE IF EXISTS traces_all_amt;-DROP TABLE IF EXISTS traces_7d_amt;-DROP TABLE IF EXISTS traces_30d_amt;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.down.sqldeleted file mode 100644index ec8e08c7ae..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP COLUMN IF EXISTS execution_trace_id;diff --git a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.up.sqldeleted file mode 100644index 96115a56e6..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0030_add_eval_execution_trace_id_to_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String);diff --git a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.down.sqldeleted file mode 100644index 9dc3a6e6db..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE observations DROP COLUMN IF EXISTS usage_pricing_tier_name;-ALTER TABLE observations DROP COLUMN IF EXISTS usage_pricing_tier_id;diff --git a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.up.sqldeleted file mode 100644index b9b8e23508..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0031_add_usage_pricing_tier_columns.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE observations ADD COLUMN usage_pricing_tier_id Nullable(String);-ALTER TABLE observations ADD COLUMN usage_pricing_tier_name Nullable(String);diff --git a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql b/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sqldeleted file mode 100644index 1752676856..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE dataset_run_items_rmt DROP COLUMN IF EXISTS dataset_item_version;\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sqldeleted file mode 100644index 18b0ada406..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE dataset_run_items_rmt ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3));\ No newline at end of filediff --git a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.down.sqldeleted file mode 100644index f5e64ad73c..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.down.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE observations DROP COLUMN IF EXISTS tool_definitions SETTINGS alter_sync = 2;-ALTER TABLE observations DROP COLUMN IF EXISTS tool_calls SETTINGS alter_sync = 2;-ALTER TABLE observations DROP COLUMN IF EXISTS tool_call_names SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.up.sqldeleted file mode 100644index 17627f1137..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0033_add_tool_call_columns.up.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE observations ADD COLUMN IF NOT EXISTS tool_definitions Map(String, String) DEFAULT map() SETTINGS alter_sync = 2;-ALTER TABLE observations ADD COLUMN IF NOT EXISTS tool_calls Array(String) DEFAULT [] SETTINGS alter_sync = 2;-ALTER TABLE observations ADD COLUMN IF NOT EXISTS tool_call_names Array(String) DEFAULT [] SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.down.sql b/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.down.sqldeleted file mode 100644index b10e392480..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.down.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores DROP COLUMN IF EXISTS long_string_value;diff --git a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.up.sqldeleted file mode 100644index 7dfe2da5cf..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0034_add_long_string_value_to_scores.up.sql+++ /dev/null@@ -1 +0,0 @@-ALTER TABLE scores ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3));diff --git a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.down.sqldeleted file mode 100644index 162a83db91..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.down.sql+++ /dev/null@@ -1,8 +0,0 @@-ALTER TABLE scores- DROP COLUMN IF EXISTS ingestion_sdk_version;--ALTER TABLE scores- DROP COLUMN IF EXISTS ingestion_sdk_name;--ALTER TABLE scores- DROP COLUMN IF EXISTS ingestion_api_key;diff --git a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.up.sqldeleted file mode 100644index f6b12e724b..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0035_add_ingestion_attribution_columns.up.sql+++ /dev/null@@ -1,8 +0,0 @@-ALTER TABLE scores- ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT '';--ALTER TABLE scores- ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown';--ALTER TABLE scores- ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown';diff --git a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.up.sql b/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.up.sqldeleted file mode 100644index 4cf20e3eb6..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0036_add_ingested_sdks_to_analytics_scores.up.sql+++ /dev/null@@ -1,23 +0,0 @@-DROP VIEW IF EXISTS analytics_scores;-CREATE VIEW analytics_scores AS-SELECT- project_id,- toStartOfHour(timestamp) AS hour,- uniq(id) AS countScores,- max(source = 'ANNOTATION') AS hasAnnotation,- max(source = 'API') AS hasApi,- max(source = 'EVAL') AS hasEval,- max(observation_id IS NOT NULL) AS hasObservationScore,- max(session_id IS NOT NULL) AS hasSessionScore,- max(dataset_run_id IS NOT NULL) AS hasDatasetRunScore,- max(data_type = 'BOOLEAN') AS hasBoolScore,- max(data_type = 'NUMERIC') AS hasNumericScore,- max(data_type = 'CATEGORICAL') AS hasCategoricalScore,- max(comment IS NOT NULL) AS hasComment,- sumMap(map(concat(if(ingestion_sdk_name = '', 'unknown', ingestion_sdk_name), '@', if(ingestion_sdk_version = '', 'unknown', ingestion_sdk_version)), toUInt64(1))) AS ingested_sdks-FROM- scores-WHERE toStartOfHour(timestamp) <= toStartOfHour(subtractHours(now(), 1))-GROUP BY- project_id,- hour;diff --git a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.down.sql b/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.down.sqldeleted file mode 100644index 61d605ecb7..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.down.sql+++ /dev/null@@ -1,3 +0,0 @@-ALTER TABLE observations DROP INDEX IF EXISTS idx_created_at;-ALTER TABLE traces DROP INDEX IF EXISTS idx_created_at;-ALTER TABLE scores DROP INDEX IF EXISTS idx_created_at;diff --git a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.up.sql b/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.up.sqldeleted file mode 100644index 6b74d0f599..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0037_add_created_at_indexes.up.sql+++ /dev/null@@ -1,6 +0,0 @@-ALTER TABLE observations ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1;-ALTER TABLE observations MATERIALIZE INDEX IF EXISTS idx_created_at;-ALTER TABLE traces ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1;-ALTER TABLE traces MATERIALIZE INDEX IF EXISTS idx_created_at;-ALTER TABLE scores ADD INDEX IF NOT EXISTS idx_created_at created_at TYPE minmax GRANULARITY 1;-ALTER TABLE scores MATERIALIZE INDEX IF EXISTS idx_created_at;diff --git a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.down.sql b/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.down.sqldeleted file mode 100644index 1f7193f389..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0038_create_observations_batch_staging.down.sql+++ /dev/null@@ -1,5 +0,0 @@--- WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this--- table may predate this migration (created via dev-tables.sh on v4 preview--- deployments) and may hold production-grade event data. Rolling back drops--- that data regardless of which path created the table.-DROP TABLE IF EXISTS observations_batch_staging;diff --git a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.down.sqldeleted file mode 100644index 87863a03ac..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0042_add_events_ingestion_attribution_columns.down.sql+++ /dev/null@@ -1,82 +0,0 @@-ALTER TABLE events_core_mv MODIFY QUERY-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted-FROM events_full-SETTINGS enable_full_text_index = 1;--ALTER TABLE events_core DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1;--ALTER TABLE events_full DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1;--ALTER TABLE observations_batch_staging DROP COLUMN IF EXISTS ingestion_sdk_version SETTINGS enable_full_text_index = 1;-ALTER TABLE observations_batch_staging DROP COLUMN IF EXISTS ingestion_sdk_name SETTINGS enable_full_text_index = 1;-ALTER TABLE observations_batch_staging DROP COLUMN IF EXISTS ingestion_api_key SETTINGS enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.down.sql b/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.down.sqldeleted file mode 100644index 53a2eec2e8..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.down.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE events_full DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP INDEX IF EXISTS idx_ngram_metadata_values SETTINGS enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.up.sql b/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.up.sqldeleted file mode 100644index 847ebc0f53..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0043_add_events_metadata_ngram_index.up.sql+++ /dev/null@@ -1,2 +0,0 @@-ALTER TABLE events_full ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core ADD INDEX IF NOT EXISTS idx_ngram_metadata_values arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.down.sql b/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.down.sqldeleted file mode 100644index 939378be98..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.down.sql+++ /dev/null@@ -1,21 +0,0 @@--- WARNING: This recreates the table structure only. Any data held in--- event_log before the up migration dropped it is NOT restored.-CREATE TABLE IF NOT EXISTS event_log-(- `id` String,- `project_id` String,- `entity_type` String,- `entity_id` String,- `event_id` Nullable(String),-- `bucket_name` String,- `bucket_path` String,-- `created_at` DateTime64(3) DEFAULT now(),- `updated_at` DateTime64(3) DEFAULT now()-) ENGINE = MergeTree()- ORDER BY (- project_id,- entity_type,- entity_id- );diff --git a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.up.sql b/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.up.sqldeleted file mode 100644index 1fdc2a46fc..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0044_drop_event_log.up.sql+++ /dev/null@@ -1,5 +0,0 @@--- The event_log table was superseded by blob_storage_file_log. Its contents--- were copied over by the 20250417_1737_migrate_event_log_to_blob_storage--- background migration, which must have completed (on the latest v3 release)--- before upgrading to v4.-DROP TABLE IF EXISTS event_log;diff --git a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.down.sql b/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.down.sqldeleted file mode 100644index fdf1f4db39..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.down.sql+++ /dev/null@@ -1,9 +0,0 @@--- WARNING: This recreates the table structure only. Any data held in--- project_environments before the up migration dropped it is NOT restored,--- and the materialized views that used to feed the table were already dropped--- by 0027_drop_project_environments_mvs, so it stays empty.-CREATE TABLE IF NOT EXISTS project_environments (- `project_id` String,- `environments` SimpleAggregateFunction(groupUniqArrayArray, Array(String))-) ENGINE = AggregatingMergeTree-ORDER BY (project_id);diff --git a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.up.sql b/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.up.sqldeleted file mode 100644index 18a2795bc0..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0045_drop_project_environments.up.sql+++ /dev/null@@ -1,5 +0,0 @@--- The project_environments table has been unused since--- 0027_drop_project_environments_mvs removed the materialized views feeding--- it. getEnvironmentsForProject reads distinct environments from the tracing--- and scores tables directly.-DROP TABLE IF EXISTS project_environments;diff --git a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.up.sql b/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.up.sqldeleted file mode 100644index 5e05c6cec7..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0046_drop_dataset_run_items.up.sql+++ /dev/null@@ -1,4 +0,0 @@--- The dataset_run_items table was superseded by dataset_run_items_rmt--- (created in 0024). All ingestion writes, reads and deletes target the--- _rmt table, so this table has been frozen since the cutover.-DROP TABLE IF EXISTS dataset_run_items;diff --git a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.down.sql b/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.down.sqldeleted file mode 100644index 6f1ed617d1..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.down.sql+++ /dev/null@@ -1,88 +0,0 @@-ALTER TABLE events_core_mv MODIFY QUERY-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted,- ingestion_api_key,- ingestion_sdk_name,- ingestion_sdk_version-FROM events_full-SETTINGS enable_full_text_index = 1;--ALTER TABLE events_full DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1;--ALTER TABLE events_core DROP INDEX IF EXISTS idx_evaluator_id SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP INDEX IF EXISTS idx_evaluation_rule_id SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP COLUMN IF EXISTS evaluator_execution_is_test SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP COLUMN IF EXISTS evaluator_id SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core DROP COLUMN IF EXISTS evaluation_rule_id SETTINGS enable_full_text_index = 1;--ALTER TABLE scores DROP INDEX IF EXISTS idx_evaluator_id;-ALTER TABLE scores DROP INDEX IF EXISTS idx_evaluation_rule_id;-ALTER TABLE scores DROP COLUMN IF EXISTS evaluator_id;-ALTER TABLE scores DROP COLUMN IF EXISTS evaluation_rule_id;diff --git a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.up.sql b/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.up.sqldeleted file mode 100644index 6a979c92f4..0000000000--- a/packages/shared/clickhouse/migrations/unclustered/0047_add_eval_execution_columns.up.sql+++ /dev/null@@ -1,109 +0,0 @@--- Existing parts resolve these columns through DEFAULT expressions. Skip indexes are added without materializing historical parts to keep this migration metadata-only.-ALTER TABLE events_full- ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full- ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_full- ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core- ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_id')) AFTER metadata_values- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core- ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id'))), arrayElement(metadata_values, indexOf(metadata_names, 'evaluation_rule_id')), arrayElement(metadata_values, indexOf(metadata_names, 'job_configuration_id'))) AFTER evaluator_id- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core- ADD COLUMN IF NOT EXISTS evaluator_execution_is_test Bool DEFAULT has(metadata_names, 'evaluator_test') AND arrayElement(metadata_values, indexOf(metadata_names, 'evaluator_test')) = 'true' AFTER evaluation_rule_id- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core- ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.01) GRANULARITY 1- SETTINGS enable_full_text_index = 1;-ALTER TABLE events_core- ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.01) GRANULARITY 1- SETTINGS enable_full_text_index = 1;-ALTER TABLE scores- ADD COLUMN IF NOT EXISTS evaluator_id String DEFAULT metadata['evaluator_id'] AFTER metadata;-ALTER TABLE scores- ADD COLUMN IF NOT EXISTS evaluation_rule_id String DEFAULT if(notEmpty(metadata['evaluation_rule_id']), metadata['evaluation_rule_id'], metadata['job_configuration_id']) AFTER evaluator_id;-ALTER TABLE scores- ADD INDEX IF NOT EXISTS idx_evaluator_id evaluator_id TYPE bloom_filter(0.001) GRANULARITY 1;-ALTER TABLE scores- ADD INDEX IF NOT EXISTS idx_evaluation_rule_id evaluation_rule_id TYPE bloom_filter(0.001) GRANULARITY 1;-ALTER TABLE events_core_mv MODIFY QUERY-SELECT- project_id,- trace_id,- span_id,- parent_span_id,- start_time,- end_time,- name,- type,- environment,- version,- release,- trace_name,- user_id,- session_id,- tags,- level,- status_message,- completion_start_time,- is_app_root,- bookmarked,- public,- prompt_id,- prompt_name,- prompt_version,- model_id,- provided_model_name,- model_parameters,- provided_usage_details,- usage_details,- provided_cost_details,- cost_details,- usage_pricing_tier_id,- usage_pricing_tier_name,- tool_definitions,- tool_calls,- tool_call_names,- leftUTF8(input, 200) as input,- leftUTF8(output, 200) as output,- metadata_names,- arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,- evaluator_id,- evaluation_rule_id,- evaluator_execution_is_test,- experiment_id,- experiment_name,- experiment_metadata_names,- experiment_metadata_values,- experiment_description,- experiment_dataset_id,- experiment_item_id,- experiment_item_version,- experiment_item_expected_output,- experiment_item_metadata_names,- experiment_item_metadata_values,- experiment_item_root_span_id,- source,- service_name,- service_version,- scope_name,- scope_version,- telemetry_sdk_language,- telemetry_sdk_name,- telemetry_sdk_version,- blob_storage_file_path,- event_bytes,- created_at,- updated_at,- event_ts,- is_deleted,- ingestion_api_key,- ingestion_sdk_name,- ingestion_sdk_version-FROM events_full-SETTINGS enable_full_text_index = 1;diff --git a/packages/shared/clickhouse/scripts/down.sh b/packages/shared/clickhouse/scripts/down.shindex 31332a7498..ff56897c56 100644--- a/packages/shared/clickhouse/scripts/down.sh+++ b/packages/shared/clickhouse/scripts/down.sh@@ -1,7 +1,9 @@-#!/bin/bash+#!/bin/sh++SCRIPT_DIR=$(CDPATH= cd -- "$(dirname "$0")" && pwd)# Load environment variables-[ -f ../../.env ] && source ../../.env+[ -f ../../.env ] && . ../../.env# Check if CLICKHOUSE_URL is configuredif [ -z "${CLICKHOUSE_URL}" ]; then@@ -10,7 +12,7 @@ if [ -z "${CLICKHOUSE_URL}" ]; thenfi# Check if golang-migrate is installed-if ! command -v migrate &> /dev/null+if ! command -v migrate >/dev/null 2>&1thenecho "Error: golang-migrate is not installed or not in PATH."echo "Please install golang-migrate via 'brew install golang-migrate' to run this script."@@ -18,41 +20,80 @@ thenexit 1fi+require_node() {+ if ! command -v node >/dev/null 2>&1+ then+ echo "Error: node is not installed or not in PATH."+ exit 1+ fi+}+# Ensure CLICKHOUSE_DB is set-if [ -z "${CLICKHOUSE_DB}" ]; then- export CLICKHOUSE_DB="default"-fi+export CLICKHOUSE_DB="${CLICKHOUSE_DB:-default}"# Ensure CLICKHOUSE_CLUSTER_NAME is set-if [ -z "${CLICKHOUSE_CLUSTER_NAME}" ]; then- export CLICKHOUSE_CLUSTER_NAME="default"-fi+export CLICKHOUSE_CLUSTER_NAME="${CLICKHOUSE_CLUSTER_NAME:-default}"-# Construct the database URL-if [ "$CLICKHOUSE_CLUSTER_ENABLED" == "false" ] ; then- if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-migrations-table-engine=MergeTree"+MIGRATION_MODE="clustered"+MIGRATIONS_TABLE_ENGINE="ReplicatedMergeTree"+CLICKHOUSE_CLUSTER_QUERY=""+if [ "$CLICKHOUSE_CLUSTER_ENABLED" = "false" ]; then+ MIGRATION_MODE="unclustered"+ MIGRATIONS_TABLE_ENGINE="MergeTree"+else+ if [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then+ CLICKHOUSE_CLUSTER_NAME_ENCODED="default"else- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-migrations-table-engine=MergeTree"+ require_node+ if ! CLICKHOUSE_CLUSTER_NAME_ENCODED=$(node "$SCRIPT_DIR/prepare-migrations.mjs" encode-cluster-name "$CLICKHOUSE_CLUSTER_NAME"); then+ exit 1+ fifi+ CLICKHOUSE_CLUSTER_QUERY="&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME_ENCODED}"+fi- # If SKIP_CONFIRM is set, automatically answer the confirmation prompt. Otherwise run interactively.- if [ "$SKIP_CONFIRM" = "1" ] || [ "$SKIP_CONFIRM" = "true" ]; then- printf 'y\n' | migrate -source file://clickhouse/migrations/unclustered -database "$DATABASE_URL" down- else- migrate -source file://clickhouse/migrations/unclustered -database "$DATABASE_URL" down+CANONICAL_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/canonical"+if [ ! -d "$CANONICAL_MIGRATIONS_DIRECTORY" ]; then+ echo "Error: canonical ClickHouse migration templates are missing."+ exit 1+fi++DELIVERED_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/$MIGRATION_MODE"+USE_DELIVERED_MIGRATIONS="false"+if [ -f "$DELIVERED_MIGRATIONS_DIRECTORY/0001_traces.up.sql" ]; then+ if [ "$MIGRATION_MODE" = "unclustered" ] || [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then+ USE_DELIVERED_MIGRATIONS="true"fi+fi++MIGRATIONS_DIRECTORY_IS_TEMP="false"+if [ "$USE_DELIVERED_MIGRATIONS" = "true" ]; then+ MIGRATIONS_DIRECTORY="$DELIVERED_MIGRATIONS_DIRECTORY"else- if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"- else- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"+ require_node+ if ! MIGRATIONS_DIRECTORY=$(node "$SCRIPT_DIR/prepare-migrations.mjs" render "$CANONICAL_MIGRATIONS_DIRECTORY" "$MIGRATION_MODE" "$CLICKHOUSE_CLUSTER_NAME"); then+ exit 1fi+ MIGRATIONS_DIRECTORY_IS_TEMP="true"+fi- # If SKIP_CONFIRM is set, automatically answer the confirmation prompt. Otherwise run interactively.- if [ "$SKIP_CONFIRM" = "1" ] || [ "$SKIP_CONFIRM" = "true" ]; then- printf 'y\n' | migrate -source file://clickhouse/migrations/clustered -database "$DATABASE_URL" down- else- migrate -source file://clickhouse/migrations/clustered -database "$DATABASE_URL" down+cleanup_migrations() {+ if [ "$MIGRATIONS_DIRECTORY_IS_TEMP" = "true" ] && [ -n "${MIGRATIONS_DIRECTORY:-}" ]; then+ rm -rf -- "$MIGRATIONS_DIRECTORY"fi-fi\ No newline at end of file+}+trap cleanup_migrations 0++# Construct the database URL+DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true"+if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then+ DATABASE_URL="${DATABASE_URL}&secure=true&skip_verify=true"+fi+DATABASE_URL="${DATABASE_URL}${CLICKHOUSE_CLUSTER_QUERY}&x-migrations-table-engine=${MIGRATIONS_TABLE_ENGINE}"++# If SKIP_CONFIRM is set, automatically answer the confirmation prompt. Otherwise run interactively.+if [ "$SKIP_CONFIRM" = "1" ] || [ "$SKIP_CONFIRM" = "true" ]; then+ printf 'y\n' | migrate -source "file://${MIGRATIONS_DIRECTORY}" -database "$DATABASE_URL" down+else+ migrate -source "file://${MIGRATIONS_DIRECTORY}" -database "$DATABASE_URL" down+fidiff --git a/packages/shared/clickhouse/scripts/prepare-migrations.mjs b/packages/shared/clickhouse/scripts/prepare-migrations.mjsnew file mode 100644index 0000000000..335b94065e--- /dev/null+++ b/packages/shared/clickhouse/scripts/prepare-migrations.mjs@@ -0,0 +1,337 @@+#!/usr/bin/env node++import {+ cpSync,+ existsSync,+ mkdtempSync,+ mkdirSync,+ readFileSync,+ readdirSync,+ rmSync,+ statSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { basename, join, resolve } from "node:path";+import { fileURLToPath } from "node:url";++const clusterClausePlaceholder = "{CLICKHOUSE_CLUSTER_CLAUSE}";+const replicationPrefixPlaceholder = "{CLICKHOUSE_REPLICATION_PREFIX}";+const historicalFinalNewlinesPattern =+ /\{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:(0|2|clustered)\}\n$/;+const anyHistoricalFinalNewlinesPattern =+ /\{CLICKHOUSE_HISTORICAL_FINAL_NEWLINES:[^{}]*\}/;+const clusteredOnlyPattern = /\{CLICKHOUSE_CLUSTERED_ONLY:([^{}]*)\}/g;+const unclusteredOnlyPattern = /\{CLICKHOUSE_UNCLUSTERED_ONLY:([^{}]*)\}/g;+const anyPlaceholderPattern = /\{CLICKHOUSE_[A-Z_]+(?::[^{}]*)?\}/;+const historicalByteCompatibilityVersion = 47;+const deliveredModes = ["clustered", "unclustered"];++function countOccurrences(value, search) {+ return value.split(search).length - 1;+}++function quoteClickHouseString(value) {+ if (value.includes("\0")) {+ throw new Error("ClickHouse cluster names must not contain NUL bytes");+ }++ return `'${value+ .replaceAll("\\", "\\\\")+ .replaceAll("'", "\\'")+ .replaceAll("\n", "\\n")+ .replaceAll("\r", "\\r")+ .replaceAll("\t", "\\t")}'`;+}++function renderMigration(+ source,+ mode,+ clusterName,+ preserveHistoricalUnclusteredWhitespace,+) {+ const placeholderCount = countOccurrences(source, clusterClausePlaceholder);+ const historicalFinalNewlines = source.match(historicalFinalNewlinesPattern);++ if (placeholderCount === 0) {+ throw new Error(`migration does not contain ${clusterClausePlaceholder}`);+ }++ if (/\bON\s+CLUSTER\b/i.test(source)) {+ throw new Error(+ `migration contains a hardcoded ON CLUSTER clause instead of ${clusterClausePlaceholder}`,+ );+ }++ if (/\bReplicated[A-Za-z]*MergeTree\b/.test(source)) {+ throw new Error(+ `migration contains a hardcoded replicated engine instead of ${replicationPrefixPlaceholder}`,+ );+ }++ if (+ countOccurrences(source, ` ${clusterClausePlaceholder}`) !==+ placeholderCount+ ) {+ throw new Error(+ `${clusterClausePlaceholder} must include its preceding space`,+ );+ }++ if (+ anyHistoricalFinalNewlinesPattern.test(source) &&+ !historicalFinalNewlines+ ) {+ throw new Error(+ "CLICKHOUSE_HISTORICAL_FINAL_NEWLINES must appear once at the end of a migration and use 0, 2, or clustered",+ );+ }++ const clusterClause =+ mode === "clustered"+ ? clusterName === "default"+ ? "ON CLUSTER default"+ : `ON CLUSTER ${quoteClickHouseString(clusterName)}`+ : "";++ const replicationPrefix = mode === "clustered" ? "Replicated" : "";+ let rendered = source+ .replaceAll(+ ` ${clusterClausePlaceholder}`,+ mode === "clustered" ? ` ${clusterClause}` : "",+ )+ .replaceAll(replicationPrefixPlaceholder, replicationPrefix)+ .replace(clusteredOnlyPattern, mode === "clustered" ? "$1" : "");++ if (mode === "unclustered" && preserveHistoricalUnclusteredWhitespace) {+ rendered = rendered.replace(/[ \t]+\n/g, "\n").replace(/\n[ \t]*;/g, ";");+ }++ rendered = rendered+ .replace(unclusteredOnlyPattern, mode === "unclustered" ? "$1" : "")+ .replace(historicalFinalNewlinesPattern, (_, finalNewlines) => {+ if (finalNewlines === "2") return "\n\n";+ if (finalNewlines === "clustered" && mode === "clustered") return "\n";+ return "";+ });++ if (anyPlaceholderPattern.test(rendered)) {+ throw new Error("migration contains an unrendered ClickHouse placeholder");+ }++ const renderedClusterClauseCount = countOccurrences(rendered, "ON CLUSTER");+ if (mode === "clustered" && renderedClusterClauseCount !== placeholderCount) {+ throw new Error("not every cluster clause placeholder was rendered");+ }++ if (mode === "unclustered") {+ if (renderedClusterClauseCount > 0) {+ throw new Error("unclustered migration contains an ON CLUSTER clause");+ }+ if (/\bReplicated[A-Za-z]*MergeTree\b/.test(rendered)) {+ throw new Error(+ "unclustered migration contains a replicated table engine",+ );+ }+ }++ return rendered;+}++function validateMigrationFileSet(files) {+ if (files.length === 0) {+ throw new Error("no ClickHouse migration SQL files found");+ }++ const fileSet = new Set(files);+ const filesByVersion = new Map();++ for (const file of files) {+ const match = /^(\d{4})_.+\.(up|down)\.sql$/.exec(file);+ if (!match) {+ throw new Error(`invalid ClickHouse migration filename: ${file}`);+ }++ const version = Number(match[1]);+ if (version < 1) {+ throw new Error(`invalid ClickHouse migration version in ${file}`);+ }+ filesByVersion.set(version, (filesByVersion.get(version) ?? 0) + 1);+ const counterpart =+ match[2] === "up"+ ? file.replace(/\.up\.sql$/, ".down.sql")+ : file.replace(/\.down\.sql$/, ".up.sql");+ if (!fileSet.has(counterpart)) {+ throw new Error(`missing migration counterpart for ${file}`);+ }+ }++ const highestVersion = Math.max(...filesByVersion.keys());+ for (let version = 1; version <= highestVersion; version += 1) {+ const fileCount = filesByVersion.get(version);+ if (!fileCount) {+ throw new Error(`missing ClickHouse migration version ${version}`);+ }+ if (fileCount !== 2) {+ throw new Error(+ `ClickHouse migration version ${version} must have one up and one down file`,+ );+ }+ }+}++export function encodeClusterName(clusterName) {+ const clusterExpression =+ clusterName === "default"+ ? clusterName+ : quoteClickHouseString(clusterName);++ return encodeURIComponent(clusterExpression).replaceAll("'", "%27");+}++export function prepareMigrations(sourceDirectory, mode, clusterName) {+ if (mode !== "clustered" && mode !== "unclustered") {+ throw new Error(`unsupported ClickHouse migration mode: ${mode}`);+ }+ if (mode === "clustered" && clusterName.length === 0) {+ throw new Error("ClickHouse cluster name must not be empty");+ }++ const sourcePath = resolve(sourceDirectory);+ if (!statSync(sourcePath).isDirectory()) {+ throw new Error(+ `ClickHouse migration source is not a directory: ${sourcePath}`,+ );+ }++ const migrationFiles = readdirSync(sourcePath, { withFileTypes: true })+ .filter((entry) => entry.isFile() && entry.name.endsWith(".sql"))+ .map((entry) => entry.name)+ .sort();+ validateMigrationFileSet(migrationFiles);++ const targetDirectory = mkdtempSync(+ join(tmpdir(), `langfuse-clickhouse-${mode}-migrations-`),+ );++ try {+ for (const migrationFile of migrationFiles) {+ const source = readFileSync(join(sourcePath, migrationFile), "utf8");+ const migrationVersion = Number(migrationFile.slice(0, 4));+ const rendered = renderMigration(+ source,+ mode,+ clusterName,+ migrationVersion <= historicalByteCompatibilityVersion,+ );+ writeFileSync(join(targetDirectory, basename(migrationFile)), rendered);+ }+ } catch (error) {+ rmSync(targetDirectory, { force: true, recursive: true });+ throw error;+ }++ return targetDirectory;+}++function requireCanonicalMigrationDirectory(migrationsDirectory) {+ const migrationsPath = resolve(migrationsDirectory);+ const canonicalPath = join(migrationsPath, "canonical");++ if (!existsSync(canonicalPath) || !statSync(canonicalPath).isDirectory()) {+ throw new Error(+ `ClickHouse migration directory must contain canonical/: ${migrationsPath}`,+ );+ }++ return { canonicalPath, migrationsPath };+}++export function materializeMigrations(migrationsDirectory) {+ const { canonicalPath, migrationsPath } =+ requireCanonicalMigrationDirectory(migrationsDirectory);+ const renderedDirectories = [];++ try {+ for (const mode of deliveredModes) {+ renderedDirectories.push({+ mode,+ path: prepareMigrations(canonicalPath, mode, "default"),+ });+ }++ mkdirSync(migrationsPath, { recursive: true });+ for (const rendered of renderedDirectories) {+ const targetDirectory = join(migrationsPath, rendered.mode);+ rmSync(targetDirectory, { force: true, recursive: true });+ cpSync(rendered.path, targetDirectory, { recursive: true });+ }+ } finally {+ for (const rendered of renderedDirectories) {+ rmSync(rendered.path, { force: true, recursive: true });+ }+ }++ return deliveredModes.map((mode) => join(migrationsPath, mode));+}++export function cleanMaterializedMigrations(migrationsDirectory) {+ const { migrationsPath } =+ requireCanonicalMigrationDirectory(migrationsDirectory);++ for (const mode of deliveredModes) {+ rmSync(join(migrationsPath, mode), { force: true, recursive: true });+ }+}++function printUsage() {+ console.error(+ "Usage: prepare-migrations.mjs render <source-directory> <clustered|unclustered> <cluster-name> | materialize <migrations-directory> | clean <migrations-directory> | encode-cluster-name <cluster-name>",+ );+}++function main() {+ const [command, ...args] = process.argv.slice(2);++ if (command === "render" && args.length === 3) {+ const [sourceDirectory, mode, clusterName] = args;+ process.stdout.write(+ `${prepareMigrations(sourceDirectory, mode, clusterName)}\n`,+ );+ return;+ }++ if (command === "encode-cluster-name" && args.length === 1) {+ process.stdout.write(`${encodeClusterName(args[0])}\n`);+ return;+ }++ if (command === "materialize" && args.length === 1) {+ for (const directory of materializeMigrations(args[0])) {+ process.stdout.write(`${directory}\n`);+ }+ return;+ }++ if (command === "clean" && args.length === 1) {+ cleanMaterializedMigrations(args[0]);+ return;+ }++ printUsage();+ process.exitCode = 1;+}++if (+ process.argv[1] &&+ resolve(process.argv[1]) === fileURLToPath(import.meta.url)+) {+ try {+ main();+ } catch (error) {+ console.error(+ `Error: ${error instanceof Error ? error.message : String(error)}`,+ );+ process.exitCode = 1;+ }+}diff --git a/packages/shared/clickhouse/scripts/up.sh b/packages/shared/clickhouse/scripts/up.shindex 608a73f4d3..c9e0d8f03e 100644--- a/packages/shared/clickhouse/scripts/up.sh+++ b/packages/shared/clickhouse/scripts/up.sh@@ -1,7 +1,9 @@-#!/bin/bash+#!/bin/sh++SCRIPT_DIR=$(CDPATH= cd -- "$(dirname "$0")" && pwd)# Load environment variables-[ -f ../../.env ] && source ../../.env+[ -f ../../.env ] && . ../../.env# Check if CLICKHOUSE_URL is configuredif [ -z "${CLICKHOUSE_URL}" ]; then@@ -32,7 +34,7 @@ if [ -z "${CLICKHOUSE_PASSWORD}" ]; thenfi# Check if golang-migrate is installed-if ! command -v migrate &> /dev/null+if ! command -v migrate >/dev/null 2>&1thenecho "Error: golang-migrate is not installed or not in PATH."echo "Please install golang-migrate via 'brew install golang-migrate' to run this script."@@ -40,33 +42,76 @@ thenexit 1fi+require_node() {+ if ! command -v node >/dev/null 2>&1+ then+ echo "Error: node is not installed or not in PATH."+ exit 1+ fi+}+# Ensure CLICKHOUSE_DB is set-if [ -z "${CLICKHOUSE_DB}" ]; then- export CLICKHOUSE_DB="default"-fi+export CLICKHOUSE_DB="${CLICKHOUSE_DB:-default}"# Ensure CLICKHOUSE_CLUSTER_NAME is set-if [ -z "${CLICKHOUSE_CLUSTER_NAME}" ]; then- export CLICKHOUSE_CLUSTER_NAME="default"-fi+export CLICKHOUSE_CLUSTER_NAME="${CLICKHOUSE_CLUSTER_NAME:-default}"-# Construct the database URL-if [ "$CLICKHOUSE_CLUSTER_ENABLED" == "false" ] ; then- if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-migrations-table-engine=MergeTree"+MIGRATION_MODE="clustered"+MIGRATIONS_TABLE_ENGINE="ReplicatedMergeTree"+CLICKHOUSE_CLUSTER_QUERY=""+if [ "$CLICKHOUSE_CLUSTER_ENABLED" = "false" ]; then+ MIGRATION_MODE="unclustered"+ MIGRATIONS_TABLE_ENGINE="MergeTree"+else+ if [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then+ CLICKHOUSE_CLUSTER_NAME_ENCODED="default"else- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-migrations-table-engine=MergeTree"+ require_node+ if ! CLICKHOUSE_CLUSTER_NAME_ENCODED=$(node "$SCRIPT_DIR/prepare-migrations.mjs" encode-cluster-name "$CLICKHOUSE_CLUSTER_NAME"); then+ exit 1+ fifi+ CLICKHOUSE_CLUSTER_QUERY="&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME_ENCODED}"+fi++CANONICAL_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/canonical"+if [ ! -d "$CANONICAL_MIGRATIONS_DIRECTORY" ]; then+ echo "Error: canonical ClickHouse migration templates are missing."+ exit 1+fi- # Execute the up command- migrate -source file://clickhouse/migrations/unclustered -database "$DATABASE_URL" up+DELIVERED_MIGRATIONS_DIRECTORY="$SCRIPT_DIR/../migrations/$MIGRATION_MODE"+USE_DELIVERED_MIGRATIONS="false"+if [ -f "$DELIVERED_MIGRATIONS_DIRECTORY/0001_traces.up.sql" ]; then+ if [ "$MIGRATION_MODE" = "unclustered" ] || [ "$CLICKHOUSE_CLUSTER_NAME" = "default" ]; then+ USE_DELIVERED_MIGRATIONS="true"+ fi+fi++MIGRATIONS_DIRECTORY_IS_TEMP="false"+if [ "$USE_DELIVERED_MIGRATIONS" = "true" ]; then+ MIGRATIONS_DIRECTORY="$DELIVERED_MIGRATIONS_DIRECTORY"else-if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&secure=true&skip_verify=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"- else- DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true&x-cluster-name=${CLICKHOUSE_CLUSTER_NAME}&x-migrations-table-engine=ReplicatedMergeTree"+ require_node+ if ! MIGRATIONS_DIRECTORY=$(node "$SCRIPT_DIR/prepare-migrations.mjs" render "$CANONICAL_MIGRATIONS_DIRECTORY" "$MIGRATION_MODE" "$CLICKHOUSE_CLUSTER_NAME"); then+ exit 1fi+ MIGRATIONS_DIRECTORY_IS_TEMP="true"+fi- # Execute the up command- migrate -source file://clickhouse/migrations/clustered -database "$DATABASE_URL" up+cleanup_migrations() {+ if [ "$MIGRATIONS_DIRECTORY_IS_TEMP" = "true" ] && [ -n "${MIGRATIONS_DIRECTORY:-}" ]; then+ rm -rf -- "$MIGRATIONS_DIRECTORY"+ fi+}+trap cleanup_migrations 0++# Construct the database URL+DATABASE_URL="${CLICKHOUSE_MIGRATION_URL}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${CLICKHOUSE_DB}&x-multi-statement=true"+if [ "$CLICKHOUSE_MIGRATION_SSL" = true ] ; then+ DATABASE_URL="${DATABASE_URL}&secure=true&skip_verify=true"fi+DATABASE_URL="${DATABASE_URL}${CLICKHOUSE_CLUSTER_QUERY}&x-migrations-table-engine=${MIGRATIONS_TABLE_ENGINE}"++# Execute the up command against delivered or custom-rendered migrations.+migrate -source "file://${MIGRATIONS_DIRECTORY}" -database "$DATABASE_URL" updiff --git a/packages/shared/package.json b/packages/shared/package.jsonindex 4bccbbe9f8..a5e73bbb85 100644--- a/packages/shared/package.json+++ b/packages/shared/package.json@@ -31,6 +31,10 @@"import": "./dist/src/server/index.js","require": "./dist/src/server/index.js"},+ "./src/server/clickhouse/clickhouseIdentifiers": {+ "import": "./dist/src/server/clickhouse/clickhouseIdentifiers.js",+ "require": "./dist/src/server/clickhouse/clickhouseIdentifiers.js"+ },"./src/server/llm/llmText": {"import": "./dist/src/server/llm/llmText.js","require": "./dist/src/server/llm/llmText.js"@@ -137,8 +141,10 @@"db:generate": "dotenv -e ../../.env -- npx prisma generate --no-hints","db:seed:examples": "dotenv -e ../../.env -- npx prisma db seed -- --environment examples","db:seed:load": "dotenv -e ../../.env -- npx prisma db seed -- --environment load",- "ch:up": "bash clickhouse/scripts/up.sh",- "ch:down": "bash clickhouse/scripts/down.sh",+ "ch:migrations:materialize": "node clickhouse/scripts/prepare-migrations.mjs materialize clickhouse/migrations",+ "ch:migrations:clean": "node clickhouse/scripts/prepare-migrations.mjs clean clickhouse/migrations",+ "ch:up": "sh clickhouse/scripts/up.sh",+ "ch:down": "sh clickhouse/scripts/down.sh","ch:drop": "bash clickhouse/scripts/drop.sh","ch:dev-tables": "bash clickhouse/scripts/dev-tables.sh","ch:reset": "pnpm run ch:down && pnpm run ch:up && pnpm run ch:seed && pnpm run ch:dev-tables",diff --git a/packages/shared/src/server/clickhouse/clickhouseIdentifiers.ts b/packages/shared/src/server/clickhouse/clickhouseIdentifiers.tsnew file mode 100644index 0000000000..db3c45fc11--- /dev/null+++ b/packages/shared/src/server/clickhouse/clickhouseIdentifiers.ts@@ -0,0 +1,49 @@+/**+ * Quoting helpers for ClickHouse identifiers and string literals that have to+ * be interpolated into SQL text rather than bound as query parameters.+ */++export function assertClickhouseIdentifier(value: string, label: string): void {+ if (value.length === 0 || value.includes("\0")) {+ throw new Error(`Invalid ClickHouse ${label}: ${value}`);+ }+}++export function quoteClickhouseIdentifier(+ value: string,+ label: string,+): string {+ assertClickhouseIdentifier(value, label);+ const escaped = value.replace(/\\/g, "\\\\").replace(/`/g, "\\`");+ return `\`${escaped}\``;+}++export function quoteClickhouseString(value: string): string {+ if (value.includes("\0")) {+ throw new Error("Invalid ClickHouse string: NUL bytes are not supported");+ }++ return `'${value+ .replace(/\\/g, "\\\\")+ .replace(/'/g, "\\'")+ .replace(/\n/g, "\\n")+ .replace(/\r/g, "\\r")+ .replace(/\t/g, "\\t")}'`;+}++/**+ * Qualifies a table with its database.+ *+ * Required for `SYSTEM ... ON CLUSTER` statements: `AddDefaultDatabaseVisitor`+ * only fills in the initiator's current database for `ASTAlterQuery`,+ * `ASTQueryWithTableAndOutput` and `ASTRenameQuery`. `ASTSystemQuery` is not+ * visited, so an unqualified name reaches each host's DDL worker unqualified+ * and resolves against that server's default database (`default`) instead of+ * `CLICKHOUSE_DB`.+ */+export function qualifiedClickhouseTableName(+ database: string,+ table: string,+): string {+ return `${quoteClickhouseIdentifier(database, "database")}.${quoteClickhouseIdentifier(table, "table")}`;+}diff --git a/packages/shared/src/server/clickhouse/fixtures/fake-migrate.sh b/packages/shared/src/server/clickhouse/fixtures/fake-migrate.shnew file mode 100755index 0000000000..e207fc6ac1--- /dev/null+++ b/packages/shared/src/server/clickhouse/fixtures/fake-migrate.sh@@ -0,0 +1,27 @@+#!/bin/sh+set -eu++: > "$CAPTURED_ARGS_PATH"+for argument in "$@"; do+ printf "%s\n" "$argument" >> "$CAPTURED_ARGS_PATH"+done++source_uri=""+while [ "$#" -gt 0 ]; do+ if [ "$1" = "-source" ]; then+ source_uri="$2"+ shift 2+ else+ shift+ fi+done++source_directory="${source_uri#file://}"+printf "%s" "$source_directory" > "$CAPTURED_SOURCE_PATH"+cp -R "$source_directory/." "$CAPTURED_MIGRATIONS_DIRECTORY"++if [ "${CAPTURE_STDIN:-false}" = "true" ]; then+ cat > "$CAPTURED_STDIN_PATH"+fi++exit "${MIGRATE_EXIT_CODE:-0}"diff --git a/packages/shared/src/server/clickhouse/prepareMigrations.test.ts b/packages/shared/src/server/clickhouse/prepareMigrations.test.tsnew file mode 100644index 0000000000..7fbfff1b5c--- /dev/null+++ b/packages/shared/src/server/clickhouse/prepareMigrations.test.ts@@ -0,0 +1,411 @@+import { spawnSync } from "node:child_process";+import { createHash } from "node:crypto";+import {+ cpSync,+ existsSync,+ mkdtempSync,+ mkdirSync,+ readFileSync,+ readdirSync,+ rmSync,+ symlinkSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { dirname, join, resolve } from "node:path";+import { afterEach, describe, expect, it } from "vitest";++const sharedDirectory = resolve(__dirname, "../../..");+const scriptsDirectory = join(sharedDirectory, "clickhouse/scripts");+const fakeMigratePath = join(__dirname, "fixtures/fake-migrate.sh");+const migrationSourceDirectory = join(+ sharedDirectory,+ "clickhouse/migrations/canonical",+);+const historicalMigrationCount = 94;+const historicalMigrationVersion = 47;+// Compatibility baselines for migrations 0001-0047 as shipped in each mode.+const historicalClusteredHash =+ "1f1cceaf0d89af2c78ec6f527dc9722900a572aa743f7650a83b277c7ae49fc2";+const historicalUnclusteredHash =+ "26de8baf42d0690445c9e8ec11951b1c85696b63d8bd47845c695803fced0aa3";+const temporaryDirectories: string[] = [];++type MigrationMode = "clustered" | "unclustered";+type MigrationScript = "up" | "down";+type MigrationLayout = "delivered" | "source";++type MigrationRun = {+ args: string[];+ capturedMigrationsDirectory: string;+ databaseUrl: string;+ renderedMigrationsDirectory: string;+ status: number | null;+ stdin: string;+};++afterEach(() => {+ for (const directory of temporaryDirectories.splice(0)) {+ rmSync(directory, { force: true, recursive: true });+ }+});++function createTemporaryDirectory(prefix: string): string {+ const directory = mkdtempSync(join(tmpdir(), prefix));+ temporaryDirectories.push(directory);+ return directory;+}++function runMigrationScript({+ clusterName,+ migrationLayout = "source",+ migrateExitCode = 0,+ mode,+ script = "up",+ ssl = false,+}: {+ clusterName?: string;+ migrationLayout?: MigrationLayout;+ migrateExitCode?: number;+ mode: MigrationMode;+ script?: MigrationScript;+ ssl?: boolean;+}): MigrationRun {+ const root = createTemporaryDirectory("langfuse-clickhouse-migration-test-");+ const binaryDirectory = join(root, "bin");+ const workingDirectory = join(root, "shared");+ const temporaryDirectory = join(root, "tmp");+ const capturedMigrationsDirectory = join(root, "captured-migrations");+ const capturedArgsPath = join(root, "captured-args");+ const capturedSourcePath = join(root, "captured-source");+ const capturedStdinPath = join(root, "captured-stdin");++ mkdirSync(binaryDirectory);+ mkdirSync(workingDirectory);+ mkdirSync(temporaryDirectory);+ mkdirSync(capturedMigrationsDirectory);+ const clickhouseDirectory = join(workingDirectory, "clickhouse");+ if (migrationLayout === "source") {+ symlinkSync(join(sharedDirectory, "clickhouse"), clickhouseDirectory);+ } else {+ const migrationsDirectory = join(clickhouseDirectory, "migrations");+ mkdirSync(migrationsDirectory, { recursive: true });+ cpSync(scriptsDirectory, join(clickhouseDirectory, "scripts"), {+ recursive: true,+ });+ symlinkSync(+ migrationSourceDirectory,+ join(migrationsDirectory, "canonical"),+ );++ for (const deliveredMode of ["clustered", "unclustered"] as const) {+ const rendered = spawnSync(+ process.execPath,+ [+ join(scriptsDirectory, "prepare-migrations.mjs"),+ "render",+ migrationSourceDirectory,+ deliveredMode,+ "default",+ ],+ { encoding: "utf8" },+ );+ expect(rendered.status, rendered.stderr).toBe(0);+ const renderedDirectory = rendered.stdout.trim();+ temporaryDirectories.push(renderedDirectory);+ cpSync(renderedDirectory, join(migrationsDirectory, deliveredMode), {+ recursive: true,+ });+ }+ }++ symlinkSync(fakeMigratePath, join(binaryDirectory, "migrate"));++ const env: NodeJS.ProcessEnv = {+ ...process.env,+ CAPTURED_ARGS_PATH: capturedArgsPath,+ CAPTURED_MIGRATIONS_DIRECTORY: capturedMigrationsDirectory,+ CAPTURED_SOURCE_PATH: capturedSourcePath,+ CAPTURED_STDIN_PATH: capturedStdinPath,+ CAPTURE_STDIN: script === "down" ? "true" : "false",+ CLICKHOUSE_CLUSTER_ENABLED: mode === "clustered" ? "true" : "false",+ CLICKHOUSE_DB: "default",+ CLICKHOUSE_MIGRATION_SSL: ssl ? "true" : "false",+ CLICKHOUSE_MIGRATION_URL: "clickhouse://localhost:9000",+ CLICKHOUSE_PASSWORD: "password",+ CLICKHOUSE_URL: "http://localhost:8123",+ CLICKHOUSE_USER: "user",+ MIGRATE_EXIT_CODE: String(migrateExitCode),+ PATH: [+ binaryDirectory,+ ...(migrationLayout === "source" ? [dirname(process.execPath)] : []),+ "/usr/bin",+ "/bin",+ ].join(":"),+ SKIP_CONFIRM: script === "down" ? "true" : "false",+ TMPDIR: temporaryDirectory,+ };+ if (clusterName === undefined) {+ delete env.CLICKHOUSE_CLUSTER_NAME;+ } else {+ env.CLICKHOUSE_CLUSTER_NAME = clusterName;+ }++ const migrationScriptPath =+ migrationLayout === "delivered"+ ? join(clickhouseDirectory, "scripts", `${script}.sh`)+ : join(scriptsDirectory, `${script}.sh`);+ const execution = spawnSync("sh", [migrationScriptPath], {+ cwd: workingDirectory,+ env,+ encoding: "utf8",+ });+ expect(execution.status, execution.stderr).toBe(migrateExitCode);++ const args = readFileSync(capturedArgsPath, "utf8").trimEnd().split("\n");+ const renderedMigrationsDirectory = readFileSync(capturedSourcePath, "utf8");+ const databaseArgumentIndex = args.indexOf("-database");++ return {+ args,+ capturedMigrationsDirectory,+ databaseUrl: args[databaseArgumentIndex + 1] ?? "",+ renderedMigrationsDirectory,+ status: execution.status,+ stdin: existsSync(capturedStdinPath)+ ? readFileSync(capturedStdinPath, "utf8")+ : "",+ };+}++function readMigration(run: MigrationRun, file: string): string {+ return readFileSync(join(run.capturedMigrationsDirectory, file), "utf8");+}++function readHistoricalSql(run: MigrationRun): string {+ return historicalMigrationFiles(run.capturedMigrationsDirectory)+ .map((file) => readMigration(run, file))+ .join("\n");+}++function countMatches(value: string, pattern: RegExp): number {+ return [...value.matchAll(pattern)].length;+}++function historicalMigrationFiles(directory: string): string[] {+ return readdirSync(directory)+ .filter(+ (file) =>+ file.endsWith(".sql") &&+ Number(file.slice(0, 4)) <= historicalMigrationVersion,+ )+ .sort();+}++function hashHistoricalMigrations(directory: string): string {+ const hash = createHash("sha256");+ const files = historicalMigrationFiles(directory);+ expect(files).toHaveLength(historicalMigrationCount);++ for (const file of files) {+ hash.update(file);+ hash.update("\0");+ hash.update(readFileSync(join(directory, file), "utf8"));+ hash.update("\0");+ }++ return hash.digest("hex");+}++describe("ClickHouse migration preparation", () => {+ it("renders and URL-encodes a custom cluster name through POSIX sh", () => {+ const clusterName = "eu-west' & blue";+ const run = runMigrationScript({ clusterName, mode: "clustered" });+ const sql = readMigration(run, "0001_traces.up.sql");+ const allSql = readHistoricalSql(run);++ expect(sql).toContain("ON CLUSTER 'eu-west\\' & blue'");+ expect(sql).toContain("ENGINE = ReplicatedReplacingMergeTree");+ expect(sql).not.toContain("ON CLUSTER default");+ expect(countMatches(allSql, /ON CLUSTER 'eu-west\\' & blue'/g)).toBe(204);+ expect(run.databaseUrl).toContain(+ `&x-cluster-name=${encodeURIComponent(`'eu-west\\' & blue'`).replaceAll("'", "%27")}`,+ );+ expect(run.databaseUrl).toContain(+ "&x-migrations-table-engine=ReplicatedMergeTree",+ );+ expect(run.args.at(-1)).toBe("up");+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(false);+ });++ it("preserves the historical default-cluster migration bytes", () => {+ const run = runMigrationScript({ mode: "clustered", ssl: true });+ const allSql = readHistoricalSql(run);++ expect(hashHistoricalMigrations(run.capturedMigrationsDirectory)).toBe(+ historicalClusteredHash,+ );+ expect(countMatches(allSql, /\bON CLUSTER default\b/g)).toBe(204);+ expect(countMatches(allSql, /\balter_sync = 2\b/g)).toBe(85);+ expect(countMatches(allSql, /\bmutations_sync = 2\b/g)).toBe(19);+ expect(countMatches(allSql, /\bReplicated[A-Za-z]*MergeTree\b/g)).toBe(10);+ expect(run.databaseUrl).toContain("&x-cluster-name=default");+ expect(run.databaseUrl).toContain("&secure=true&skip_verify=true");+ });++ it("preserves the historical unclustered migration bytes", () => {+ const run = runMigrationScript({+ clusterName: "ignored-cluster",+ mode: "unclustered",+ });+ const allSql = readHistoricalSql(run);++ expect(hashHistoricalMigrations(run.capturedMigrationsDirectory)).toBe(+ historicalUnclusteredHash,+ );+ expect(allSql).not.toMatch(/\bON\s+CLUSTER\b/);+ expect(allSql).not.toMatch(/\bReplicated[A-Za-z]*MergeTree\b/);+ expect(countMatches(allSql, /\balter_sync = 2\b/g)).toBe(6);+ expect(countMatches(allSql, /\bmutations_sync = 2\b/g)).toBe(18);+ expect(readMigration(run, "0015_add_scores_trace_index.up.sql")).toContain(+ "SETTINGS mutations_sync = 2",+ );+ expect(readMigration(run, "0033_add_tool_call_columns.up.sql")).toContain(+ "SETTINGS alter_sync = 2",+ );+ expect(+ readMigration(run, "0047_add_eval_execution_columns.up.sql"),+ ).not.toMatch(/(?:alter|mutations)_sync/);+ expect(run.databaseUrl).not.toContain("x-cluster-name");+ expect(run.databaseUrl).toContain("&x-migrations-table-engine=MergeTree");+ });++ it.each(["clustered", "unclustered"] as const)(+ "uses the delivered %s migrations without runtime rendering",+ (mode) => {+ const run = runMigrationScript({ migrationLayout: "delivered", mode });++ expect(run.renderedMigrationsDirectory).toContain(`/migrations/${mode}`);+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(true);+ expect(hashHistoricalMigrations(run.capturedMigrationsDirectory)).toBe(+ mode === "clustered"+ ? historicalClusteredHash+ : historicalUnclusteredHash,+ );+ },+ );++ it("materializes and cleans the delivered migration trees", () => {+ const targetDirectory = createTemporaryDirectory(+ "langfuse-clickhouse-materialized-migrations-test-",+ );+ symlinkSync(migrationSourceDirectory, join(targetDirectory, "canonical"));+ const materialize = spawnSync(+ process.execPath,+ [+ join(scriptsDirectory, "prepare-migrations.mjs"),+ "materialize",+ targetDirectory,+ ],+ { encoding: "utf8" },+ );++ expect(materialize.status, materialize.stderr).toBe(0);+ for (const mode of ["clustered", "unclustered"] as const) {+ const modeDirectory = join(targetDirectory, mode);+ expect(existsSync(modeDirectory)).toBe(true);+ expect(hashHistoricalMigrations(modeDirectory)).toBe(+ mode === "clustered"+ ? historicalClusteredHash+ : historicalUnclusteredHash,+ );+ expect(+ readFileSync(+ join(modeDirectory, "0047_add_eval_execution_columns.up.sql"),+ "utf8",+ ),+ ).not.toMatch(/\{CLICKHOUSE_[A-Z_]+/);+ }++ const clean = spawnSync(+ process.execPath,+ [+ join(scriptsDirectory, "prepare-migrations.mjs"),+ "clean",+ targetDirectory,+ ],+ { encoding: "utf8" },+ );++ expect(clean.status, clean.stderr).toBe(0);+ expect(existsSync(join(targetDirectory, "clustered"))).toBe(false);+ expect(existsSync(join(targetDirectory, "unclustered"))).toBe(false);+ });++ it("renders down migrations, confirms non-interactively, and cleans up", () => {+ const run = runMigrationScript({ mode: "unclustered", script: "down" });++ expect(run.args.at(-1)).toBe("down");+ expect(run.stdin).toBe("y\n");+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(false);+ });++ it("propagates migrate failures and still cleans up", () => {+ const run = runMigrationScript({+ migrateExitCode: 23,+ mode: "clustered",+ });++ expect(run.status).toBe(23);+ expect(existsSync(run.renderedMigrationsDirectory)).toBe(false);+ });++ it("rejects hardcoded cluster clauses in canonical migrations", () => {+ const sourceDirectory = createTemporaryDirectory(+ "langfuse-clickhouse-invalid-migrations-",+ );+ const invalidSql =+ "CREATE TABLE test {CLICKHOUSE_CLUSTER_CLAUSE} () ENGINE = {CLICKHOUSE_REPLICATION_PREFIX}MergeTree; ON CLUSTER default;";+ writeFileSync(join(sourceDirectory, "0001_test.up.sql"), invalidSql);+ writeFileSync(join(sourceDirectory, "0001_test.down.sql"), invalidSql);++ const result = spawnSync(+ "node",+ [+ join(scriptsDirectory, "prepare-migrations.mjs"),+ "render",+ sourceDirectory,+ "clustered",+ "default",+ ],+ { encoding: "utf8" },+ );++ expect(result.status).toBe(1);+ expect(result.stderr).toContain("hardcoded ON CLUSTER clause");+ });++ it("keeps every canonical migration renderable in both modes", () => {+ for (const mode of ["clustered", "unclustered"] as const) {+ const result = spawnSync(+ "node",+ [+ join(scriptsDirectory, "prepare-migrations.mjs"),+ "render",+ migrationSourceDirectory,+ mode,+ "default",+ ],+ { encoding: "utf8" },+ );++ expect(result.status, result.stderr).toBe(0);+ const renderedDirectory = result.stdout.trim();+ temporaryDirectories.push(renderedDirectory);+ const renderedSql = historicalMigrationFiles(renderedDirectory)+ .map((file) => readFileSync(join(renderedDirectory, file), "utf8"))+ .join("\n");+ expect(renderedSql).not.toMatch(/\{CLICKHOUSE_[A-Z_]+/);+ }+ });+});diff --git a/packages/shared/src/server/clickhouse/queryTracking.test.ts b/packages/shared/src/server/clickhouse/queryTracking.test.tsindex 9fca4f5656..1185d29ffd 100644--- a/packages/shared/src/server/clickhouse/queryTracking.test.ts+++ b/packages/shared/src/server/clickhouse/queryTracking.test.ts@@ -1,4 +1,4 @@-import { describe, expect, it, vi } from "vitest";+import { beforeEach, describe, expect, it, vi } from "vitest";const mocks = vi.hoisted(() => ({env: {@@ -13,9 +13,30 @@ vi.mock("../repositories", () => ({ queryClickhouse: vi.fn() }));import { systemTableRef } from "./queryTracking";describe("systemTableRef", () => {+ beforeEach(() => {+ mocks.env.CLICKHOUSE_CLUSTER_ENABLED = "true";+ mocks.env.CLICKHOUSE_CLUSTER_NAME = "default";+ });+it("queries all query log tables on every replica in clustered mode", () => {expect(systemTableRef("system.query_log")).toBe("clusterAllReplicas('default', merge(system, '^query_log*'))",);});++ it("escapes custom cluster names before interpolating them into SQL", () => {+ mocks.env.CLICKHOUSE_CLUSTER_NAME = "eu-west'\\blue\n";++ expect(systemTableRef("system.processes")).toBe(+ "clusterAllReplicas('eu-west\\'\\\\blue\\n', 'system.processes')",+ );+ });++ it("rejects cluster names containing NUL bytes", () => {+ mocks.env.CLICKHOUSE_CLUSTER_NAME = "invalid\0cluster";++ expect(() => systemTableRef("system.processes")).toThrow(+ "Invalid ClickHouse string",+ );+ });});diff --git a/packages/shared/src/server/clickhouse/queryTracking.ts b/packages/shared/src/server/clickhouse/queryTracking.tsindex 543866edfe..4052d82862 100644--- a/packages/shared/src/server/clickhouse/queryTracking.ts+++ b/packages/shared/src/server/clickhouse/queryTracking.ts@@ -1,5 +1,6 @@import { env } from "../../env";import { queryClickhouse } from "../repositories";+import { quoteClickhouseString } from "./clickhouseIdentifiers";// ============================================================================// Types@@ -24,10 +25,11 @@ export function systemTableRef(table: "system.processes" | "system.query_log",): string {if (env.CLICKHOUSE_CLUSTER_ENABLED === "true") {+ const clusterName = quoteClickhouseString(env.CLICKHOUSE_CLUSTER_NAME);if (table === "system.query_log") {- return `clusterAllReplicas('${env.CLICKHOUSE_CLUSTER_NAME}', merge(system, '^query_log*'))`;+ return `clusterAllReplicas(${clusterName}, merge(system, '^query_log*'))`;}- return `clusterAllReplicas('${env.CLICKHOUSE_CLUSTER_NAME}', '${table}')`;+ return `clusterAllReplicas(${clusterName}, '${table}')`;}return table;}diff --git a/packages/shared/src/server/index.ts b/packages/shared/src/server/index.tsindex a13eb35342..8b56a4ce87 100644--- a/packages/shared/src/server/index.ts+++ b/packages/shared/src/server/index.ts@@ -76,6 +76,7 @@ export {} from "./clickhouse/compatibility";export * from "./clickhouse/schemaUtils";export * from "./clickhouse/schema";+export * from "./clickhouse/clickhouseIdentifiers";export * from "./clickhouse/queryTracking";export * from "./clickhouse/queryTags";export * from "./repositories/definitions";diff --git a/scripts/release-preflight.sh b/scripts/release-preflight.shindex 9e8dbd2331..7a0a69a97f 100755--- a/scripts/release-preflight.sh+++ b/scripts/release-preflight.sh@@ -86,12 +86,19 @@ collect_prisma_migrations_not_in_production() {| sort -u}-collect_clickhouse_migrations_not_in_production() {- git diff --name-only origin/production..HEAD -- packages/shared/clickhouse/migrations \- | sed -E 's#.*/([0-9]+_[^.]+)\.(up|down)\.sql#\1#' \+collect_clickhouse_migration_names_at_ref() {+ local ref="$1"+ git ls-tree -r --name-only "$ref" -- packages/shared/clickhouse/migrations \+ | sed -nE 's#.*/([0-9]+_[^.]+)\.(up|down)\.sql$#\1#p' \| sort -u}+collect_clickhouse_migrations_not_in_production() {+ comm -23 \+ <(collect_clickhouse_migration_names_at_ref HEAD) \+ <(collect_clickhouse_migration_names_at_ref origin/production)+}+confirm_migrations_are_applied() {local prisma_migrations="$1"local clickhouse_migrations="$2"diff --git a/web/Dockerfile b/web/Dockerfileindex 1691bb57b5..87597bc791 100644--- a/web/Dockerfile+++ b/web/Dockerfile@@ -124,6 +124,12 @@ ENV SENTRY_RELEASE=$NEXT_PUBLIC_BUILD_ID# Copy source code of isolated subworkspaceCOPY --from=pruner /app/out/full/ .+# Keep the documented direct golang-migrate recovery paths available in the+# production image while maintaining canonical/ as the only authored tree.+RUN set -eu; \+ node ./packages/shared/clickhouse/scripts/prepare-migrations.mjs materialize ./packages/shared/clickhouse/migrations; \+ chmod 755 ./packages/shared/clickhouse/migrations/clustered ./packages/shared/clickhouse/migrations/unclustered+# remove middleware.ts if it exists - not needed in self-hosted environmentsRUN rm -f ./web/src/middleware.tsdiff --git a/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts b/worker/src/backgroundMigrations/backfillEventsFullFromObservations.tsindex d25f5030ab..bf033c0716 100644--- a/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts+++ b/worker/src/backgroundMigrations/backfillEventsFullFromObservations.ts@@ -1,4 +1,9 @@-import { logger, queryClickhouse, sleep } from "@langfuse/shared/src/server";+import {+ logger,+ queryClickhouse,+ quoteClickhouseString,+ sleep,+} from "@langfuse/shared/src/server";import { env } from "../env";import {BaseChunkTodo,@@ -145,7 +150,7 @@ export default class BackfillEventsFullFromObservations extends ChunkedClickhoussum(future_parts) AS total_future_parts,countIf(is_session_expired) AS expired_replicas,countIf(is_readonly) AS readonly_replicas- FROM clusterAllReplicas('${cluster}', 'system.replicas')+ FROM clusterAllReplicas(${quoteClickhouseString(cluster)}, 'system.replicas')WHERE database = currentDatabase()AND table = '${PID_TID_SORTING_TABLE}'`,diff --git a/worker/src/utils/clickhouseIdentifiers.ts b/worker/src/utils/clickhouseIdentifiers.tsindex 9d21b9d9f7..0467bda9f8 100644--- a/worker/src/utils/clickhouseIdentifiers.ts+++ b/worker/src/utils/clickhouseIdentifiers.ts@@ -1,41 +1,6 @@-/**- * Quoting helpers for ClickHouse identifiers that have to be interpolated into- * SQL text rather than bound as query parameters — table/database/cluster names- * in DDL and SYSTEM statements cannot be parameterized.- */--export function assertClickhouseIdentifier(value: string, label: string): void {- if (value.length === 0 || value.includes("\0")) {- throw new Error(`Invalid ClickHouse ${label}: ${value}`);- }-}--export function quoteClickhouseIdentifier(- value: string,- label: string,-): string {- assertClickhouseIdentifier(value, label);- const escaped = value.replace(/\\/g, "\\\\").replace(/`/g, "\\`");- return `\`${escaped}\``;-}--export function quoteClickhouseString(value: string): string {- return `'${value.replace(/\\/g, "\\\\").replace(/'/g, "\\'")}'`;-}--/**- * Qualifies a table with its database.- *- * Required for `SYSTEM ... ON CLUSTER` statements: `AddDefaultDatabaseVisitor`- * only fills in the initiator's current database for `ASTAlterQuery`,- * `ASTQueryWithTableAndOutput` and `ASTRenameQuery`. `ASTSystemQuery` is not- * visited, so an unqualified name reaches each host's DDL worker unqualified- * and resolves against that server's default database (`default`) instead of- * `CLICKHOUSE_DB`.- */-export function qualifiedClickhouseTableName(- database: string,- table: string,-): string {- return `${quoteClickhouseIdentifier(database, "database")}.${quoteClickhouseIdentifier(table, "table")}`;-}+export {+ assertClickhouseIdentifier,+ qualifiedClickhouseTableName,+ quoteClickhouseIdentifier,+ quoteClickhouseString,+} from "@langfuse/shared/src/server/clickhouse/clickhouseIdentifiers";