🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
chore(ci): run query-AST golden SQL tests in tests-shared (#16745)
Niklas Semmler3c544ff60f77
diff --git a/.github/workflows/pipeline.yml b/.github/workflows/pipeline.ymlindex 0517c13c67..d41d54df8d 100644--- a/.github/workflows/pipeline.yml+++ b/.github/workflows/pipeline.yml@@ -455,7 +455,41 @@ jobs:- name: generate prisma clientrun: pnpm --filter=shared run db:generate- name: run shared tests+ # The clickhouse binary isn't installed yet, so the SQL-equivalence+ # tests skip here (they run for real in the dedicated step below); every+ # other shared test hard-fails as usual.run: pnpm --filter @langfuse/shared run test+ - name: Install ClickHouse binary for SQL-equivalence tests+ # `clickhouse format` (from clickhouse-common-static) normalizes SQL for+ # the equivalence snapshots. Its output is version-specific, so pin the+ # exact version and regenerate snapshots on any bump; keep in sync with+ # scripts/codex/cloud_services.sh. 26.4 is the v4-recommended version.+ # continue-on-error + timeout-minutes: a missing binary OR a stalled apt+ # mirror just skips the tests, never blocks (timeout-minutes bounds a+ # hang, which continue-on-error alone would not).+ continue-on-error: true+ timeout-minutes: 5+ env:+ CLICKHOUSE_VERSION: 26.4.5.143+ run: |+ set -euo pipefail+ sudo mkdir -p /etc/apt/keyrings+ curl -fsSL https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key \+ | gpg --dearmor \+ | sudo tee /etc/apt/keyrings/clickhouse.gpg >/dev/null+ echo "deb [signed-by=/etc/apt/keyrings/clickhouse.gpg] https://packages.clickhouse.com/deb stable main" \+ | sudo tee /etc/apt/sources.list.d/clickhouse.list >/dev/null+ sudo apt-get update+ sudo apt-get install --allow-downgrades -y \+ "clickhouse-common-static=${CLICKHOUSE_VERSION}"+ clickhouse --version+ - name: run SQL-equivalence tests (non-blocking)+ # Non-blocking: a snapshot drift or an intentional SQL change emits a+ # warning but must not fail the job (`|| echo` swallows the exit code).+ # Make it a required check once the suite has proven stable.+ run: |+ pnpm --filter @langfuse/shared run test .golden.test \+ || echo "::warning title=SQL-equivalence tests failed::Query-AST SQL-equivalence snapshots drifted or the tests failed (non-blocking). Regenerate with: pnpm --filter @langfuse/shared run test .golden.test -u"test-docker-build:name: test-docker-build (${{ matrix.component }})diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex a094261dd9..9860d76e2d 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -25,3 +25,19 @@ Regenerate baselines with `-u` after an intentional SQL change:```pnpm --filter @langfuse/shared run test src/server/query-ast -- -u```++## CI and the `clickhouse format` version++These tests need the `clickhouse` binary (the `format` subcommand, shipped in+`clickhouse-common-static`); without it they `describe.skip`. The `tests-shared`+CI job installs it pinned to **26.4.5.143** — the ClickHouse version recommended+for Langfuse v4, also pinned in `scripts/codex/cloud_services.sh`.++`clickhouse format` output is version-sensitive (e.g. how `UNION ALL` branches+are parenthesized changed between 25.x and 26.x), so the committed snapshots are+coupled to that exact version. When bumping the CI pin, regenerate the snapshots+against the new binary in the same PR, or the golden tests drift.++The CI step is intentionally **non-blocking**: a drift surfaces as a warning+annotation but never fails the pipeline (`|| echo "::warning::"`). Promote it to+a required check once it has proven stable.