Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

chore(ci): run query-AST golden SQL tests in tests-shared (#16745)

Niklas Semmler3c544ff60f77
diff --git a/.github/workflows/pipeline.yml b/.github/workflows/pipeline.yml
index 0517c13c67..d41d54df8d 100644
--- a/.github/workflows/pipeline.yml
+++ b/.github/workflows/pipeline.yml
@@ -455,7 +455,41 @@ jobs:
- name: generate prisma client
run: pnpm --filter=shared run db:generate
- name: run shared tests
+ # The clickhouse binary isn't installed yet, so the SQL-equivalence
+ # tests skip here (they run for real in the dedicated step below); every
+ # other shared test hard-fails as usual.
run: pnpm --filter @langfuse/shared run test
+ - name: Install ClickHouse binary for SQL-equivalence tests
+ # `clickhouse format` (from clickhouse-common-static) normalizes SQL for
+ # the equivalence snapshots. Its output is version-specific, so pin the
+ # exact version and regenerate snapshots on any bump; keep in sync with
+ # scripts/codex/cloud_services.sh. 26.4 is the v4-recommended version.
+ # continue-on-error + timeout-minutes: a missing binary OR a stalled apt
+ # mirror just skips the tests, never blocks (timeout-minutes bounds a
+ # hang, which continue-on-error alone would not).
+ continue-on-error: true
+ timeout-minutes: 5
+ env:
+ CLICKHOUSE_VERSION: 26.4.5.143
+ run: |
+ set -euo pipefail
+ sudo mkdir -p /etc/apt/keyrings
+ curl -fsSL https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key \
+ | gpg --dearmor \
+ | sudo tee /etc/apt/keyrings/clickhouse.gpg >/dev/null
+ echo "deb [signed-by=/etc/apt/keyrings/clickhouse.gpg] https://packages.clickhouse.com/deb stable main" \
+ | sudo tee /etc/apt/sources.list.d/clickhouse.list >/dev/null
+ sudo apt-get update
+ sudo apt-get install --allow-downgrades -y \
+ "clickhouse-common-static=${CLICKHOUSE_VERSION}"
+ clickhouse --version
+ - name: run SQL-equivalence tests (non-blocking)
+ # Non-blocking: a snapshot drift or an intentional SQL change emits a
+ # warning but must not fail the job (`|| echo` swallows the exit code).
+ # Make it a required check once the suite has proven stable.
+ run: |
+ pnpm --filter @langfuse/shared run test .golden.test \
+ || echo "::warning title=SQL-equivalence tests failed::Query-AST SQL-equivalence snapshots drifted or the tests failed (non-blocking). Regenerate with: pnpm --filter @langfuse/shared run test .golden.test -u"
test-docker-build:
name: test-docker-build (${{ matrix.component }})
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index a094261dd9..9860d76e2d 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -25,3 +25,19 @@ Regenerate baselines with `-u` after an intentional SQL change:
```
pnpm --filter @langfuse/shared run test src/server/query-ast -- -u
```
+
+## CI and the `clickhouse format` version
+
+These tests need the `clickhouse` binary (the `format` subcommand, shipped in
+`clickhouse-common-static`); without it they `describe.skip`. The `tests-shared`
+CI job installs it pinned to **26.4.5.143** — the ClickHouse version recommended
+for Langfuse v4, also pinned in `scripts/codex/cloud_services.sh`.
+
+`clickhouse format` output is version-sensitive (e.g. how `UNION ALL` branches
+are parenthesized changed between 25.x and 26.x), so the committed snapshots are
+coupled to that exact version. When bumping the CI pin, regenerate the snapshots
+against the new binary in the same PR, or the golden tests drift.
+
+The CI step is intentionally **non-blocking**: a drift surfaces as a warning
+annotation but never fails the pipeline (`|| echo "::warning::"`). Promote it to
+a required check once it has proven stable.