🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(clickhouse): add alter_sync = 2 to clustered scores/dataset_run_items ALTERs (#16686)
Steffen Schmitz6aced4703bb3
diff --git a/.agents/skills/clickhouse-best-practices/SKILL.md b/.agents/skills/clickhouse-best-practices/SKILL.mdindex 1ae9e83325..87b573e219 100644--- a/.agents/skills/clickhouse-best-practices/SKILL.md+++ b/.agents/skills/clickhouse-best-practices/SKILL.md@@ -46,13 +46,21 @@ Comprehensive guidance for ClickHouse covering schema design, query optimizationrepository layer then reads baggage via `normalizeClickHouseQueryTags(...)`and writes it to `log_comment`. Prefer setting attribution at entry pointsrather than passing tags through every repository call.-- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with- more than one `ALTER` on the same table must end every metadata `ALTER`- (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,- and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)- with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs- against plain `MergeTree` and does not need (and should not duplicate)- these settings.+- In `packages/shared/clickhouse/migrations/clustered/**`, every metadata+ `ALTER` (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) must end with+ `SETTINGS alter_sync = 2`, and every mutation-creating `ALTER`+ (`MATERIALIZE …`, `UPDATE`, `DELETE`) with `SETTINGS mutations_sync = 2`.+ This applies to a file holding a single `ALTER` too — the race is across+ migration files, not within one. `alter_sync` defaults to `1`, so the+ statement returns as soon as the initiating replica has bumped the table's+ metadata version in Keeper; golang-migrate then opens the next file+ immediately, and its first `ALTER` on that table can land on a replica still+ on the previous version. ClickHouse refuses to queue it and aborts the whole+ run with `code 517 … metadata version on replica is N, while common metadata+ is N+1`. Note that `mutations_sync` does not substitute for `alter_sync`: it+ governs when mutations finish, not metadata propagation. The matching+ `unclustered/` file runs against plain `MergeTree` and does not need (and+ should not duplicate) these settings.- Never use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` /`EXCHANGE TABLES`) in ClickHouse migrations. The atomic replace requires`renameat2` filesystem support, which NFS-backed self-hosted deployments@@ -103,7 +111,7 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization- [ ] LowCardinality applied to appropriate string columns- [ ] Partition key cardinality bounded (100-1,000 values)- [ ] ReplacingMergeTree has version column if used-- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none+- [ ] Every metadata ALTER in a clustered migration ends with `SETTINGS alter_sync = 2` — including files with a single ALTER, since the next migration file is what breaks — and every `MATERIALIZE …` / `UPDATE` / `DELETE` with `SETTINGS mutations_sync = 2`; `mutations_sync` is not a substitute for `alter_sync`; unclustered mirror has none- [ ] No `CREATE OR REPLACE VIEW/TABLE` or `EXCHANGE TABLES` in migrations (breaks NFS/EFS self-hosting); plain views are redefined via `DROP VIEW IF EXISTS` + `CREATE VIEW` in the same file- [ ] Materialized views are never dropped and recreated while their source table takes inserts; SELECT changes go through `ALTER TABLE <mv> MODIFY QUERY` after the target-table `ALTER`sdiff --git a/.agents/skills/code-review/references/review-checklist.md b/.agents/skills/code-review/references/review-checklist.mdindex 31bf81451d..788a9eb8df 100644--- a/.agents/skills/code-review/references/review-checklist.md+++ b/.agents/skills/code-review/references/review-checklist.md@@ -10,6 +10,7 @@ This is the canonical shared review checklist for Langfuse.- E.g. `ReplacingMergeTree` is likely an error while `ReplicatedReplacingMergeTree` would be correct in most cases.- ClickHouse migrations in the `packages/shared/clickhouse/migrations/unclustered` directory must not include `ON CLUSTER` statements and must not use `Replicated` merge tree table types.- Migrations in `packages/shared/clickhouse/migrations/clustered` should match their counterparts in `packages/shared/clickhouse/migrations/unclustered` aside from the restrictions listed above.+- Every metadata `ALTER` (`ADD`/`DROP`/`MODIFY COLUMN`, `ADD`/`DROP INDEX`) in a `clustered` migration must end with `SETTINGS alter_sync = 2`, including files that contain only one `ALTER` — the race is across migration files, not within one. `alter_sync` defaults to `1`, so without it the statement returns once the initiating replica has bumped the table's metadata version, and the next migration file's first `ALTER` on that table can hit a replica still on the old version, aborting the run with `code 517 … Looks like this replica doesn't catchup with latest ALTER query updates`. `SETTINGS mutations_sync = 2` does not substitute for it: that setting governs when mutations finish, not metadata propagation. The `unclustered` counterpart runs against plain `MergeTree` and does not need (and should not duplicate) either setting.- Migrations must not use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` / `EXCHANGE TABLES`): the atomic replace requires `renameat2`, which fails on NFS-backed self-hosted deployments such as ClickHouse-on-EFS (GitHub issue #14906). Plain views are redefined with `DROP VIEW IF EXISTS <name>;` followed by `CREATE VIEW <name> AS ...` in the same migration file — the migration runner enables `x-multi-statement=true`, and every statement should stay idempotent (`IF EXISTS` / `IF NOT EXISTS`) so a dirty migration can be re-run.- Materialized views must never be dropped and recreated while their source table receives inserts — rows written between `DROP` and `CREATE` are silently and permanently missing from the target table. Redefine the SELECT with `ALTER TABLE <mv> MODIFY QUERY ...` instead, altering the target table(s) first when columns change.- For rationale and full patterns, see "Langfuse-Specific Rules" in the [`clickhouse-best-practices`](../../clickhouse-best-practices/SKILL.md) skill.diff --git a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sqlindex aedc54abed..0ea406281a 100644--- a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql+++ b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql@@ -1 +1 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String);+ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String) SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sqlindex 38a1061f54..a06e222c2f 100644--- a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql+++ b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql@@ -1 +1 @@-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3));+ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3)) SETTINGS alter_sync = 2;diff --git a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sqlindex 1e54324ca8..22d707f000 100644--- a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql+++ b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql@@ -1 +1 @@-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3));+ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3)) SETTINGS alter_sync = 2;