Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(clickhouse): add alter_sync = 2 to clustered scores/dataset_run_items ALTERs (#16686)

Steffen Schmitz6aced4703bb3
diff --git a/.agents/skills/clickhouse-best-practices/SKILL.md b/.agents/skills/clickhouse-best-practices/SKILL.md
index 1ae9e83325..87b573e219 100644
--- a/.agents/skills/clickhouse-best-practices/SKILL.md
+++ b/.agents/skills/clickhouse-best-practices/SKILL.md
@@ -46,13 +46,21 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization
repository layer then reads baggage via `normalizeClickHouseQueryTags(...)`
and writes it to `log_comment`. Prefer setting attribution at entry points
rather than passing tags through every repository call.
-- Any migration in `packages/shared/clickhouse/migrations/clustered/**` with
- more than one `ALTER` on the same table must end every metadata `ALTER`
- (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) with `SETTINGS alter_sync = 2`,
- and every mutation-creating `ALTER` (`MATERIALIZE …`, `UPDATE`, `DELETE`)
- with `SETTINGS mutations_sync = 2`. The matching `unclustered/` file runs
- against plain `MergeTree` and does not need (and should not duplicate)
- these settings.
+- In `packages/shared/clickhouse/migrations/clustered/**`, every metadata
+ `ALTER` (`ADD/DROP/MODIFY COLUMN`, `ADD/DROP INDEX`) must end with
+ `SETTINGS alter_sync = 2`, and every mutation-creating `ALTER`
+ (`MATERIALIZE …`, `UPDATE`, `DELETE`) with `SETTINGS mutations_sync = 2`.
+ This applies to a file holding a single `ALTER` too — the race is across
+ migration files, not within one. `alter_sync` defaults to `1`, so the
+ statement returns as soon as the initiating replica has bumped the table's
+ metadata version in Keeper; golang-migrate then opens the next file
+ immediately, and its first `ALTER` on that table can land on a replica still
+ on the previous version. ClickHouse refuses to queue it and aborts the whole
+ run with `code 517 … metadata version on replica is N, while common metadata
+ is N+1`. Note that `mutations_sync` does not substitute for `alter_sync`: it
+ governs when mutations finish, not metadata propagation. The matching
+ `unclustered/` file runs against plain `MergeTree` and does not need (and
+ should not duplicate) these settings.
- Never use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` /
`EXCHANGE TABLES`) in ClickHouse migrations. The atomic replace requires
`renameat2` filesystem support, which NFS-backed self-hosted deployments
@@ -103,7 +111,7 @@ Comprehensive guidance for ClickHouse covering schema design, query optimization
- [ ] LowCardinality applied to appropriate string columns
- [ ] Partition key cardinality bounded (100-1,000 values)
- [ ] ReplacingMergeTree has version column if used
-- [ ] Clustered migration files with multiple ALTERs on the same table use `SETTINGS alter_sync = 2` (metadata) and `SETTINGS mutations_sync = 2` (`MATERIALIZE …`, `UPDATE`, `DELETE`); unclustered mirror has none
+- [ ] Every metadata ALTER in a clustered migration ends with `SETTINGS alter_sync = 2` — including files with a single ALTER, since the next migration file is what breaks — and every `MATERIALIZE …` / `UPDATE` / `DELETE` with `SETTINGS mutations_sync = 2`; `mutations_sync` is not a substitute for `alter_sync`; unclustered mirror has none
- [ ] No `CREATE OR REPLACE VIEW/TABLE` or `EXCHANGE TABLES` in migrations (breaks NFS/EFS self-hosting); plain views are redefined via `DROP VIEW IF EXISTS` + `CREATE VIEW` in the same file
- [ ] Materialized views are never dropped and recreated while their source table takes inserts; SELECT changes go through `ALTER TABLE <mv> MODIFY QUERY` after the target-table `ALTER`s
diff --git a/.agents/skills/code-review/references/review-checklist.md b/.agents/skills/code-review/references/review-checklist.md
index 31bf81451d..788a9eb8df 100644
--- a/.agents/skills/code-review/references/review-checklist.md
+++ b/.agents/skills/code-review/references/review-checklist.md
@@ -10,6 +10,7 @@ This is the canonical shared review checklist for Langfuse.
- E.g. `ReplacingMergeTree` is likely an error while `ReplicatedReplacingMergeTree` would be correct in most cases.
- ClickHouse migrations in the `packages/shared/clickhouse/migrations/unclustered` directory must not include `ON CLUSTER` statements and must not use `Replicated` merge tree table types.
- Migrations in `packages/shared/clickhouse/migrations/clustered` should match their counterparts in `packages/shared/clickhouse/migrations/unclustered` aside from the restrictions listed above.
+- Every metadata `ALTER` (`ADD`/`DROP`/`MODIFY COLUMN`, `ADD`/`DROP INDEX`) in a `clustered` migration must end with `SETTINGS alter_sync = 2`, including files that contain only one `ALTER` — the race is across migration files, not within one. `alter_sync` defaults to `1`, so without it the statement returns once the initiating replica has bumped the table's metadata version, and the next migration file's first `ALTER` on that table can hit a replica still on the old version, aborting the run with `code 517 … Looks like this replica doesn't catchup with latest ALTER query updates`. `SETTINGS mutations_sync = 2` does not substitute for it: that setting governs when mutations finish, not metadata propagation. The `unclustered` counterpart runs against plain `MergeTree` and does not need (and should not duplicate) either setting.
- Migrations must not use `CREATE OR REPLACE VIEW` (nor `CREATE OR REPLACE TABLE` / `EXCHANGE TABLES`): the atomic replace requires `renameat2`, which fails on NFS-backed self-hosted deployments such as ClickHouse-on-EFS (GitHub issue #14906). Plain views are redefined with `DROP VIEW IF EXISTS <name>;` followed by `CREATE VIEW <name> AS ...` in the same migration file — the migration runner enables `x-multi-statement=true`, and every statement should stay idempotent (`IF EXISTS` / `IF NOT EXISTS`) so a dirty migration can be re-run.
- Materialized views must never be dropped and recreated while their source table receives inserts — rows written between `DROP` and `CREATE` are silently and permanently missing from the target table. Redefine the SELECT with `ALTER TABLE <mv> MODIFY QUERY ...` instead, altering the target table(s) first when columns change.
- For rationale and full patterns, see "Langfuse-Specific Rules" in the [`clickhouse-best-practices`](../../clickhouse-best-practices/SKILL.md) skill.
diff --git a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql
index aedc54abed..0ea406281a 100644
--- a/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql
+++ b/packages/shared/clickhouse/migrations/clustered/0030_add_eval_execution_trace_id_to_scores.up.sql
@@ -1 +1 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String);
+ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS execution_trace_id Nullable(String) SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
index 38a1061f54..a06e222c2f 100644
--- a/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
+++ b/packages/shared/clickhouse/migrations/clustered/0032_add_dataset_version_to_dataset_run_items_rmt.up.sql
@@ -1 +1 @@
-ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3));
+ALTER TABLE dataset_run_items_rmt ON CLUSTER default ADD COLUMN IF NOT EXISTS dataset_item_version Nullable(DateTime64(3)) SETTINGS alter_sync = 2;
diff --git a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql
index 1e54324ca8..22d707f000 100644
--- a/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql
+++ b/packages/shared/clickhouse/migrations/clustered/0034_add_long_string_value_to_scores.up.sql
@@ -1 +1 @@
-ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3));
+ALTER TABLE scores ON CLUSTER default ADD COLUMN IF NOT EXISTS long_string_value String CODEC(ZSTD(3)) SETTINGS alter_sync = 2;