Skip to content

πŸͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

refactor(shared): flatten query-ast folder and colocate golden tests (#17596)

Niklas Semmler2239cee204c2
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.md
index 1e09792f5b..d58793d6c3 100644
--- a/packages/shared/AGENTS.md
+++ b/packages/shared/AGENTS.md
@@ -31,14 +31,17 @@
pairs and per-trace time windows when changing parameter chunking.
- Code evaluator dispatcher/error contract: `src/server/evals/codeEvalDispatcherTypes.ts`. Keep provider mappings, user-visible messages, and worker terminal-outcome classification aligned when adding an error code.
- Dashboard/monitor query feature (data model + server-only builder/executor): `src/features/query/*`
-- Query-builder AST (server half, WIP): `src/server/query-ast/*` β€” golden-SQL
- recording/diff harness that captures the current SQL at the
- `src/server/repositories/clickhouse.ts` exec seam and normalizes it via
- `clickhouse format` for snapshot comparison. Every migrated call site is
- proven against its baseline here. The Kysely ClickHouse dialect (ARRAY JOIN /
- LIMIT BY / metadata indexOf nodes, `ExecutionContext` tenancy injection,
- per-table dedup lowering, virtual views, catalog parity) lives under
- `src/server/query-ast/kysely/`.
+- Query-builder AST (server half, WIP): `src/server/query-ast/*` β€” the Kysely
+ ClickHouse dialect (ARRAY JOIN / LIMIT BY / metadata indexOf nodes,
+ `ExecutionContext` tenancy injection, per-table dedup lowering, virtual views,
+ catalog parity). Compile only through `compileClickhouseQuery` in
+ `src/server/query-ast/compile.ts`. SQL correctness is proven by a golden-SQL
+ harness (`src/server/repositories/goldenHarness.ts`, capturing at the
+ `src/server/repositories/clickhouse.ts` exec seam and normalizing via
+ `clickhouse format`); each migrated call site keeps its `*.golden.test.ts`
+ baseline next to the call site (e.g.
+ `src/server/repositories/environments.golden.test.ts`,
+ `src/server/queries/clickhouse-sql/event-filter-options.golden.test.ts`).
- Postgres schema: `prisma/schema.prisma`
- Prisma migrations: `prisma/migrations/*`
- Canonical ClickHouse migration templates (rendered for clustered and
diff --git a/packages/shared/src/server/query-ast/__snapshots__/event-filter-options.golden.test.ts.snap b/packages/shared/src/server/queries/clickhouse-sql/__snapshots__/event-filter-options.golden.test.ts.snap
similarity index 100%
rename from packages/shared/src/server/query-ast/__snapshots__/event-filter-options.golden.test.ts.snap
rename to packages/shared/src/server/queries/clickhouse-sql/__snapshots__/event-filter-options.golden.test.ts.snap
diff --git a/packages/shared/src/server/query-ast/event-filter-options.golden.test.ts b/packages/shared/src/server/queries/clickhouse-sql/event-filter-options.golden.test.ts
similarity index 91%
rename from packages/shared/src/server/query-ast/event-filter-options.golden.test.ts
rename to packages/shared/src/server/queries/clickhouse-sql/event-filter-options.golden.test.ts
index a8bb61b6a5..9bc51c9e9a 100644
--- a/packages/shared/src/server/query-ast/event-filter-options.golden.test.ts
+++ b/packages/shared/src/server/queries/clickhouse-sql/event-filter-options.golden.test.ts
@@ -28,26 +28,27 @@
// column, sortKey (the count/alpha/boolean rank), then value.
import { afterAll, beforeEach, describe, expect, it, vi } from "vitest";
-import { env } from "../../env";
-import type { EventFilterOptionColumn } from "../queries/clickhouse-sql/event-filter-options";
+import { env } from "../../../env";
+import type { EventFilterOptionColumn } from "./event-filter-options";
import {
capturedQueries,
clickhouseFormatAvailable,
normalizeCapturedQueries,
resetCaptures,
-} from "./goldenHarness";
+} from "../../repositories/goldenHarness";
// Record the exec seam instead of hitting ClickHouse. The factory is hoisted
// above imports, so it pulls the harness in via dynamic import; the captured
// store is a module singleton shared with the assertions below.
-vi.mock("../repositories/clickhouse", async (importOriginal) => {
+vi.mock("../../repositories/clickhouse", async (importOriginal) => {
const actual =
- await importOriginal<typeof import("../repositories/clickhouse")>();
- const { buildClickhouseMock } = await import("./goldenHarness.js");
+ await importOriginal<typeof import("../../repositories/clickhouse")>();
+ const { buildClickhouseMock } =
+ await import("../../repositories/goldenHarness.js");
return buildClickhouseMock(actual);
});
-import { getEventsExactFilterOptionsForColumns } from "../repositories/events";
+import { getEventsExactFilterOptionsForColumns } from "../../repositories/events";
const FIXED_PROJECT_ID = "golden-project";
// Fixed so the captured scope-window params are deterministic across runs.
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index 8a58949ded..f5145ef722 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -1,30 +1,45 @@
# query-ast (server)
-Server-only half of the query-builder AST module. Compiler,
-validation passes, physical registry, execution context, exec integration, and
-golden-check test utils live here, behind the `@langfuse/shared/src/server`
-export boundary. The client-safe half (node types, predicate AST, builder
-sugar, semantic field metadata, `FilterState` embedding) will land under
+Server-only half of the query-builder AST module: the Kysely-based ClickHouse
+compiler, validation passes, physical registry, execution context, and exec
+integration, behind the `@langfuse/shared/src/server` export boundary. The
+client-safe half (node types, predicate AST, builder sugar, semantic field
+metadata, `FilterState` embedding) will land under
`packages/shared/src/query-ast/`.
-## What's here today
-
-- `goldenHarness.ts` β€” captures SQL at the `repositories/clickhouse.ts` exec
- seam in test mode, then normalizes it (`clickhouse format` + positional param
- names) for snapshot comparison. No ClickHouse server needed.
-- `environments.golden.test.ts` β€” baselines `environments.getEnvironmentsForProject`
- (both write-mode branches Γ— the timestamp bound).
-- `kysely/` β€” compile-only ClickHouse dialect on Kysely 0.28. Real
- `OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf`
- subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;
- per-table dedup lowering from the registry; schema-typed selection; virtual
- views as WITH CTEs; catalog parity. Library-specific code lives only in
- this folder.
-- `kysely/schema.ts` β€” the physical table registry: one `defineTable`
- declaration per relation drives the Kysely row types (`ClickHouseDatabase`),
- the runtime column-type map (`COLUMN_DATA_TYPES`), the tenanted-table set
- (`TENANTED_TABLES`), and the per-table dedup specs (`DEDUP_SPECS`) β€”
- instead of hand-maintained tables that drift.
+This folder is a compile-only ClickHouse dialect on Kysely 0.28: real
+`OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf` subscripts; a
+mandatory tenancy injection pass keyed on `ExecutionContext`; per-table dedup
+lowering from the registry; schema-typed selection; virtual views as WITH CTEs;
+catalog parity.
+
+Two audiences: **[Using it](#using-it)** if you are calling the compiler from a
+repository, **[Developing the dialect](#developing-the-dialect)** if you are
+changing how it emits SQL.
+
+## Layout
+
+- `compile.ts` β€” the one supported entry point,
+ `compileClickhouseQuery(query, ctx)`, and the `ExecutionContext` re-export.
+- `tenancy.ts` β€” the `ExecutionContext` type and the tenancy injection pass.
+- `schema.ts` β€” the physical table registry: one `defineTable` declaration per
+ relation drives the Kysely row types (`ClickHouseDatabase`), the runtime
+ column-type map (`COLUMN_DATA_TYPES`), the tenanted-table set
+ (`TENANTED_TABLES`), and the per-table dedup specs (`DEDUP_SPECS`) β€” instead
+ of hand-maintained tables that drift.
+- `compiler.ts` / `transformer.ts` / `dialect.ts` / `nodes.ts` β€” the dialect.
+- `extensions.ts` β€” the `$call` helpers: `arrayJoin`, `mapKeys`, `mapValues`,
+ `limitBy`, `metadataValue`.
+- `catalog.ts` / `views.ts` / `dedup.ts` β€” catalog, virtual views as CTEs, and
+ per-table dedup lowering.
+- `catalog.golden.test.ts` β€” catalog SQL baselines.
+
+The golden-SQL harness lives in `../repositories/goldenHarness.ts` (it captures
+SQL at the `repositories/clickhouse.ts` exec seam in test mode, then normalizes
+it via `clickhouse format` + positional param names for snapshot comparison β€” no
+ClickHouse server needed). Each migrated call site keeps its SQL baseline next to
+that call site, e.g. `../repositories/environments.golden.test.ts` and
+`../queries/clickhouse-sql/event-filter-options.golden.test.ts`.
Regenerate baselines with `-u` after an intentional SQL change:
@@ -34,10 +49,11 @@ pnpm --filter @langfuse/shared run test src/server/query-ast -- -u
## CI and the `clickhouse format` version
-These tests need the `clickhouse` binary (the `format` subcommand, shipped in
-`clickhouse-common-static`); without it they `describe.skip`. The `tests-shared`
-CI job installs it pinned to **26.4.5.143** β€” the ClickHouse version recommended
-for Langfuse v4, also pinned in `scripts/codex/cloud_services.sh`.
+The `*.golden.test.ts` suites need the `clickhouse` binary (the `format`
+subcommand, shipped in `clickhouse-common-static`); without it they
+`describe.skip`. The `tests-shared` CI job installs it pinned to **26.4.5.143** β€”
+the ClickHouse version recommended for Langfuse v4, also pinned in
+`scripts/codex/cloud_services.sh`.
`clickhouse format` output is version-sensitive (e.g. how `UNION ALL` branches
are parenthesized changed between 25.x and 26.x), so the committed snapshots are
@@ -45,8 +61,9 @@ coupled to that exact version. When bumping the CI pin, regenerate the snapshots
against the new binary in the same PR, or the golden tests drift.
The CI step is intentionally **non-blocking**: a drift surfaces as a warning
-annotation but never fails the pipeline (`|| echo "::warning::"`). Promote it to
-a required check once it has proven stable.
+annotation but never fails the pipeline (`|| echo "::warning::"`). It selects
+suites by the `.golden.test` name, so a suite must carry the `.golden.test.ts`
+suffix to run there. Promote it to a required check once it has proven stable.
## Using it
@@ -56,7 +73,7 @@ a required check once it has proven stable.
is required β€” omitting it is a compile-time type error and an empty one throws.
- **Never filter `project_id` yourself.** The compile step injects
`project_id = {projectId}` into every tenanted relation, so call sites pass
- only `{ projectId }` (see `repositories/environments.ts`).
+ only `{ projectId }` (see `../repositories/environments.ts`).
- **Dedup is declared per table and must be an existing production idiom.**
`events_core` is `none` (immutable at read time β€” no LIMIT BY, no FINAL).
`limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` already
@@ -71,7 +88,7 @@ a required check once it has proven stable.
### Recipes
-**Write an ARRAY JOIN** (`arrayJoin` + `mapKeys`/`mapValues` from `./kysely/extensions`):
+**Write an ARRAY JOIN** (`arrayJoin` + `mapKeys`/`mapValues` from `./extensions`):
```ts
db.selectFrom("observations")
@@ -104,7 +121,133 @@ db.selectFrom("events_core as e")
// select metadata_values[indexof(e.metadata_names, {p:String})] as my_val …
```
-For how ARRAY JOIN / LIMIT BY / metadata nodes are implemented without forking
-Kysely, how tenancy injection works internally, the escape hatches, and the
-Kysely-internals upgrade hazard, see the developer guide in
-[`kysely/README.md`](kysely/README.md).
+## Developing the dialect
+
+Several patterns here are deliberately unusual. This is the guide for anyone
+**changing** how the builder emits SQL.
+
+### How ClickHouse clauses land (no fork)
+
+Kysely's `OperationNodeKind` union is closed, so a first-class node kind would
+need a fork. Instead each ClickHouse-only construct rides as an extra field or a
+special-cased node, and the compiler/transformer are overridden to emit and
+preserve it:
+
+| Clause | How it lands | Fork? |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
+| ARRAY JOIN | Plugin attaches `ArrayJoinNode` as an extra field on `SelectQueryNode`. `ClickHouseOperationNodeTransformer` preserves it. `ClickHouseQueryCompiler.visitSelectQuery` emits it after JOINs / before WHERE. | no |
+| LIMIT BY | Plugin attaches `LimitByNode` the same way. Compiler emits it after ORDER BY / before LIMIT. | no |
+| metadata `indexOf` | Helper builds an `ArrayIndexNode` whose index child is a `FunctionNode` (`indexOf`) over a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no |
+| Virtual view | Plugin rewrites `selectFrom(viewName)` into a WITH CTE. Outer types only expose the view's selected columns. | no |
+
+These are real node objects whose children are traced Kysely
+`FunctionNode`/`ColumnNode`/`ValueNode`/`IdentifierNode` values β€” not `RawNode`
+string splices. Any plugin using the default `OperationNodeTransformer` would
+drop them, so ours overrides `transformSelectQuery` to keep them. Upstream is
+not patched.
+
+### Kysely never runs β€” it only compiles
+
+`getClickhouseKysely()` returns a `Kysely` instance wired to a `DummyDriver`
+with SQLite adapter/introspector stand-ins (`dialect.ts`). Nothing ever
+connects or executes. The only supported output path is
+`compileClickhouseQuery(query, ctx)` β†’ `{ sql, params }`, which the repository
+layer hands to `queryClickhouse`. Do **not** call `.execute()` / `.compile()`
+directly: the compiler refuses any tree that did not go through the tenancy pass.
+
+### Tenancy injection β€” how the choke point works
+
+`compileClickhouseQuery(query, ctx)` is the only supported compile path, and it
+is where tenancy is enforced:
+
+1. A missing/empty `ExecutionContext` throws (`QueryCompileError`) β€” `ctx` is a
+ required parameter, so omitting it is also a compile-time type error.
+2. `TenancyInjectionPlugin` walks every FROM/JOIN and injects
+ `project_id = {projectId}` on each tenanted physical table, unless the tree
+ already carries a predicate that _proves_ that scope: the equality's value
+ must equal the context project, and β€” when more than one tenanted relation is
+ in scope β€” the column must be table-qualified. A qualified predicate covers a
+ relation only when the qualifier matches its alias (if aliased) or its table
+ name (if not) β€” so `scores AS traces` joined to `traces AS t` still scopes
+ both. It then identity-stamps the tree (`WeakSet`); a copied
+ `langfuseTenancy` property is not a valid stamp.
+3. `DedupLoweringPlugin` applies the table's declared read idiom
+ (`none` / `limitBy` / `final`). `events_core` is `none` β€” immutable at
+ read time, so the pass does not inject LIMIT BY or FINAL. `limitBy` is
+ the existing legacy `ORDER BY <version> DESC LIMIT 1 BY <key>`.
+ `final` is fail-closed until an emitter exists. The pass restamps the
+ rewritten root.
+4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is
+ present, so `qb.compile()` without the plugin also fails. Value binds take
+ their ClickHouse type from the compared column's registry entry when one is
+ in scope (`total_cost > 1` β†’ `{p:Float64}`).
+5. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a
+ `SELECT`/`FROM`/`JOIN` in SELECT/WHERE throw `UnscopedRelationError`. Kysely's
+ own keyword fragments (`asc`/`desc`) are not relations.
+
+So query bodies here never filter `project_id` by hand β€” it is redundant, and
+forgetting it is impossible.
+
+### Why `$call(helper())`, not a fluent `.arrayJoin(...)` method
+
+A real method would have to exist on _every_ builder instance, including the
+native Kysely builders that `.with((qb) => …)`, subquery, and `defineView`
+callbacks hand you. Getting that requires either forking Kysely's whole builder
+graph or globally mutating its prototype via an internal `kysely/dist/...`
+import (blocked here by Kysely's `exports` map under NodeNext). A curried helper
+is a plain function of a builder, so it works in any of those positions β€” which
+is exactly why ARRAY JOIN / LIMIT BY compose inside CTEs, subqueries, and views.
+`composition.test.ts` locks that property in.
+
+**`arrayJoin` widens the row type.** Each `{ alias: arrayExpr }` entry is added
+to the builder's output row, so an outer query over a CTE body can reference the
+produced column and a typo on the alias is a compile error. The element _value_
+type is `unknown`: Kysely's `Expression<T>` hides its type argument from
+inference. Precise value types would need a branded array-expression wrapper on
+`mapKeys` / `mapValues` / etc. β€” not done yet. `types.assert.ts` pins the
+widening behavior.
+
+**`arrayJoin` (clause) β‰  `arrayJoin()` (function).** ClickHouse has both. The
+helper here builds the ARRAY JOIN _clause_. The row-expanding SELECT _function_
+is just `eb.fn("arrayJoin", [...])`.
+
+### Escape hatches and their cost
+
+- `sql.ref("alias")` β€” the only way to reference a SELECT alias that is not a
+ schema column (ClickHouse allows `GROUP BY`/expression reuse of aliases;
+ Kysely's types do not model it). It is fully untyped β€” a typo reaches
+ ClickHouse unchecked. Use sparingly.
+- `eb.fn("ch_function", [...])` β€” arbitrary ClickHouse functions. The name is an
+ unchecked string and the return type defaults to `unknown`; no arity or
+ return checking.
+- New columns β€” add them to the table registry in `schema.ts` (one `defineTable`
+ entry per relation) as queries need them. That single declaration drives the
+ row types, the runtime column-type map, and the tenanted-table set.
+
+### The one Kysely-internals coupling (upgrade hazard)
+
+`compiler.ts` wraps Kysely's **private** `visitNode` / `nodeStack` to dispatch
+`ArrayIndexNode` (for `metadata[key]`), because it is not one of Kysely's closed
+`OperationNode` kinds. Kysely is pinned to **0.28.17** for this reason. Re-verify
+this hack on any Kysely bump; it is the single place that reaches past Kysely's
+documented surface. Everything else (plugins, dialect, transformer overrides)
+uses public or documented-protected API.
+
+### Types are asserted at compile time
+
+`types.assert.ts` holds `tsc`-only assertions (schema typing, view opacity,
+arrayJoin widening, limitBy preservation). It is never run; `schema.test.ts`
+anchors it so it stays in the build graph. `@ts-expect-error` lines there must
+stay live.
+
+### Verifying changes
+
+```
+CLICKHOUSE_BIN=clickhouse pnpm --filter @langfuse/shared run test src/server/query-ast
+```
+
+The `*.golden.test.ts` suites assert `compile(AST) ≑ referenceSQL` after
+`clickhouse format`, so they need a local `clickhouse` binary and otherwise
+`describe.skip`. `composition.test.ts` asserts on raw compiler output and runs
+everywhere. After an intentional SQL change, regenerate golden baselines with
+`-u` (see [Layout](#layout) above).
diff --git a/packages/shared/src/server/query-ast/kysely/binds.test.ts b/packages/shared/src/server/query-ast/binds.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/binds.test.ts
rename to packages/shared/src/server/query-ast/binds.test.ts
diff --git a/packages/shared/src/server/query-ast/kysely/catalog.golden.test.ts b/packages/shared/src/server/query-ast/catalog.golden.test.ts
similarity index 99%
rename from packages/shared/src/server/query-ast/kysely/catalog.golden.test.ts
rename to packages/shared/src/server/query-ast/catalog.golden.test.ts
index ecb3dfb0d3..fe3dc162fc 100644
--- a/packages/shared/src/server/query-ast/kysely/catalog.golden.test.ts
+++ b/packages/shared/src/server/query-ast/catalog.golden.test.ts
@@ -7,7 +7,7 @@ import {
formatSql,
normalizeParams,
substituteNamedParams,
-} from "../goldenHarness";
+} from "../repositories/goldenHarness";
import { CATALOG, CATALOG_PROJECT_ID } from "./catalog";
import { compileClickhouseQuery } from "./compile";
diff --git a/packages/shared/src/server/query-ast/kysely/catalog.ts b/packages/shared/src/server/query-ast/catalog.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/catalog.ts
rename to packages/shared/src/server/query-ast/catalog.ts
diff --git a/packages/shared/src/server/query-ast/kysely/compile.ts b/packages/shared/src/server/query-ast/compile.ts
similarity index 92%
rename from packages/shared/src/server/query-ast/kysely/compile.ts
rename to packages/shared/src/server/query-ast/compile.ts
index d60734c533..28a6bfbc67 100644
--- a/packages/shared/src/server/query-ast/kysely/compile.ts
+++ b/packages/shared/src/server/query-ast/compile.ts
@@ -4,12 +4,17 @@ import {
type RootOperationNode,
} from "kysely";
-import type { ExecutionContext } from "../executionContext";
import { ClickHouseQueryCompiler } from "./compiler";
import { DedupLoweringPlugin } from "./dedup";
-import { TenancyInjectionPlugin, requireExecutionContext } from "./tenancy";
+import {
+ type ExecutionContext,
+ TenancyInjectionPlugin,
+ requireExecutionContext,
+} from "./tenancy";
import { validateTypeCompatibility } from "./typecheck";
+export type { ExecutionContext } from "./tenancy";
+
export type CompiledClickhouseQuery = {
sql: string;
params: Record<string, unknown>;
diff --git a/packages/shared/src/server/query-ast/kysely/compiler.ts b/packages/shared/src/server/query-ast/compiler.ts
similarity index 99%
rename from packages/shared/src/server/query-ast/kysely/compiler.ts
rename to packages/shared/src/server/query-ast/compiler.ts
index 8bb26f3b3d..9156d82a3c 100644
--- a/packages/shared/src/server/query-ast/kysely/compiler.ts
+++ b/packages/shared/src/server/query-ast/compiler.ts
@@ -87,7 +87,7 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
// ArrayIndexNode is a custom node kind Kysely's dispatch cannot reach and
// Kysely has no public hook to register one, so we patch its private
// visitNode/nodeStack to route the node to visitArrayIndex. Kysely is
- // pinned to 0.28.17 for this; see kysely/README.md ("upgrade hazard").
+ // pinned to 0.28.17 for this; see README.md ("upgrade hazard").
const parentVisit = this.visitNode.bind(this);
// Anti-pattern: cast through `unknown` once to reach Kysely's private
// `visitNode` / `nodeStack` (not part of the public type). Deliberate, to
diff --git a/packages/shared/src/server/query-ast/kysely/composition.test.ts b/packages/shared/src/server/query-ast/composition.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/composition.test.ts
rename to packages/shared/src/server/query-ast/composition.test.ts
diff --git a/packages/shared/src/server/query-ast/kysely/dedup.test.ts b/packages/shared/src/server/query-ast/dedup.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/dedup.test.ts
rename to packages/shared/src/server/query-ast/dedup.test.ts
diff --git a/packages/shared/src/server/query-ast/kysely/dedup.ts b/packages/shared/src/server/query-ast/dedup.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/dedup.ts
rename to packages/shared/src/server/query-ast/dedup.ts
diff --git a/packages/shared/src/server/query-ast/kysely/dialect.ts b/packages/shared/src/server/query-ast/dialect.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/dialect.ts
rename to packages/shared/src/server/query-ast/dialect.ts
diff --git a/packages/shared/src/server/query-ast/kysely/errors.ts b/packages/shared/src/server/query-ast/errors.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/errors.ts
rename to packages/shared/src/server/query-ast/errors.ts
diff --git a/packages/shared/src/server/query-ast/executionContext.ts b/packages/shared/src/server/query-ast/executionContext.ts
deleted file mode 100644
index 7f00f76053..0000000000
--- a/packages/shared/src/server/query-ast/executionContext.ts
+++ /dev/null
@@ -1,8 +0,0 @@
-/**
- * Compile-time tenancy scope. Every ClickHouse query compiled through
- * {@link compileClickhouseQuery} must carry one of these; the tenancy
- * injection pass keys off `projectId`.
- */
-export type ExecutionContext = {
- projectId: string;
-};
diff --git a/packages/shared/src/server/query-ast/kysely/extensions.test.ts b/packages/shared/src/server/query-ast/extensions.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/extensions.test.ts
rename to packages/shared/src/server/query-ast/extensions.test.ts
diff --git a/packages/shared/src/server/query-ast/kysely/extensions.ts b/packages/shared/src/server/query-ast/extensions.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/extensions.ts
rename to packages/shared/src/server/query-ast/extensions.ts
diff --git a/packages/shared/src/server/query-ast/kysely/README.md b/packages/shared/src/server/query-ast/kysely/README.md
deleted file mode 100644
index daaa79ceac..0000000000
--- a/packages/shared/src/server/query-ast/kysely/README.md
+++ /dev/null
@@ -1,156 +0,0 @@
-# kysely/ β€” compile-only ClickHouse dialect: patterns & gotchas
-
-This folder adapts Kysely to emit ClickHouse SQL. Several patterns here are
-deliberately unusual β€” this is the guide for anyone **developing** the query
-builder. For how to **use** it from a call site, see [`../README.md`](../README.md).
-
-## How ClickHouse clauses land (no fork)
-
-Kysely's `OperationNodeKind` union is closed, so a first-class node kind would
-need a fork. Instead each ClickHouse-only construct rides as an extra field or a
-special-cased node, and the compiler/transformer are overridden to emit and
-preserve it:
-
-| Clause | How it lands | Fork? |
-| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
-| ARRAY JOIN | Plugin attaches `ArrayJoinNode` as an extra field on `SelectQueryNode`. `ClickHouseOperationNodeTransformer` preserves it. `ClickHouseQueryCompiler.visitSelectQuery` emits it after JOINs / before WHERE. | no |
-| LIMIT BY | Plugin attaches `LimitByNode` the same way. Compiler emits it after ORDER BY / before LIMIT. | no |
-| metadata `indexOf` | Helper builds an `ArrayIndexNode` whose index child is a `FunctionNode` (`indexOf`) over a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no |
-| Virtual view | Plugin rewrites `selectFrom(viewName)` into a WITH CTE. Outer types only expose the view's selected columns. | no |
-
-These are real node objects whose children are traced Kysely
-`FunctionNode`/`ColumnNode`/`ValueNode`/`IdentifierNode` values β€” not `RawNode`
-string splices. Any plugin using the default `OperationNodeTransformer` would
-drop them, so ours overrides `transformSelectQuery` to keep them. Upstream is
-not patched.
-
-## Kysely never runs β€” it only compiles
-
-`getClickhouseKysely()` returns a `Kysely` instance wired to a `DummyDriver`
-with SQLite adapter/introspector stand-ins (`dialect.ts`). Nothing ever
-connects or executes. The only supported output path is
-`compileClickhouseQuery(query, ctx)` β†’ `{ sql, params }`, which the repository
-layer hands to `queryClickhouse`. Do **not** call `.execute()` / `.compile()`
-directly: the compiler refuses any tree that did not go through the tenancy
-pass (see `../README.md`).
-
-## Tenancy injection β€” how the choke point works
-
-`compileClickhouseQuery(query, ctx)` is the only supported compile path, and it
-is where tenancy is enforced:
-
-1. A missing/empty `ExecutionContext` throws (`QueryCompileError`) β€” `ctx` is a
- required parameter, so omitting it is also a compile-time type error.
-2. `TenancyInjectionPlugin` walks every FROM/JOIN and injects
- `project_id = {projectId}` on each tenanted physical table, unless the tree
- already carries a predicate that _proves_ that scope: the equality's value
- must equal the context project, and β€” when more than one tenanted relation is
- in scope β€” the column must be table-qualified. A qualified predicate covers a
- relation only when the qualifier matches its alias (if aliased) or its table
- name (if not) β€” so `scores AS traces` joined to `traces AS t` still scopes
- both. It then identity-stamps the tree (`WeakSet`); a copied
- `langfuseTenancy` property is not a valid stamp.
-3. `DedupLoweringPlugin` applies the table's declared read idiom
- (`none` / `limitBy` / `final`). `events_core` is `none` β€” immutable at
- read time, so the pass does not inject LIMIT BY or FINAL. `limitBy` is
- the existing legacy `ORDER BY <version> DESC LIMIT 1 BY <key>`.
- `final` is fail-closed until an emitter exists. The pass restamps the
- rewritten root.
-4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is
- present, so `qb.compile()` without the plugin also fails. Value binds take
- their ClickHouse type from the compared column's registry entry when one is
- in scope (`total_cost > 1` β†’ `{p:Float64}`).
-5. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a
-`SELECT`/`FROM`/`JOIN`in SELECT/WHERE throw`UnscopedRelationError`. Kysely's
-own keyword fragments (`asc`/`desc`) are not relations.
-
-So query bodies here never filter `project_id` by hand β€” it is redundant, and
-forgetting it is impossible. Call sites like `repositories/environments.ts` pass
-only `{ projectId }`.
-
-## ClickHouse-only clauses use `$call(helper())`, not builder methods
-
-ARRAY JOIN and LIMIT BY are applied through curried helpers, invoked with
-Kysely's public `$call`:
-
-```ts
-db.selectFrom("observations")
- .select("environment")
- .$call(
- arrayJoin({
- cost_key: mapKeys("cost_details"),
- cost: mapValues("cost_details"),
- }),
- );
-
-db.selectFrom("events_core")
- .select(["span_id", "project_id"])
- .orderBy("event_ts", "desc")
- .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));
-```
-
-**Why `$call(...)` and not a fluent `.arrayJoin(...)` method:** a real method
-would have to exist on _every_ builder instance, including the native Kysely
-builders that `.with((qb) => …)`, subquery, and `defineView` callbacks hand you.
-Getting that requires either forking Kysely's whole builder graph or globally
-mutating its prototype via an internal `kysely/dist/...` import (blocked here by
-Kysely's `exports` map under NodeNext). A curried helper is a plain function of
-a builder, so it works in any of those positions β€” which is exactly why ARRAY
-JOIN / LIMIT BY compose inside CTEs, subqueries, and views. `composition.test.ts`
-locks that property in.
-
-**`arrayJoin` widens the row type.** Each `{ alias: arrayExpr }` entry is added
-to the builder's output row, so an outer query over a CTE body can reference the
-produced column and a typo on the alias is a compile error. The element _value_
-type is `unknown`: Kysely's `Expression<T>` hides its type argument from
-inference. Precise value types would need a branded array-expression wrapper on
-`mapKeys` / `mapValues` / etc. β€” not done yet. `types.assert.ts` pins the
-widening behavior.
-
-**`arrayJoin` (clause) β‰  `arrayJoin()` (function).** ClickHouse has both. The
-helper here builds the ARRAY JOIN _clause_. The row-expanding SELECT _function_
-is just `eb.fn("arrayJoin", [...])`.
-
-## Escape hatches and their cost
-
-- `sql.ref("alias")` β€” the only way to reference a SELECT alias that is not a
- schema column (ClickHouse allows `GROUP BY`/expression reuse of aliases;
- Kysely's types do not model it). It is fully untyped β€” a typo reaches
- ClickHouse unchecked. Use sparingly.
-- `eb.fn("ch_function", [...])` β€” arbitrary ClickHouse functions. The name is an
- unchecked string and the return type defaults to `unknown`; no arity or
- return checking.
-- New columns β€” add them to the table registry in `schema.ts` (one `defineTable`
- entry per relation) as queries need them. That single declaration drives the
- row types, the runtime column-type map, and the tenanted-table set.
-
-## The one Kysely-internals coupling (upgrade hazard)
-
-`compiler.ts` wraps Kysely's **private** `visitNode` / `nodeStack` to dispatch
-`ArrayIndexNode` (for `metadata[key]`), because it is not one of Kysely's closed
-`OperationNode` kinds. Kysely is pinned to **0.28.17** for this reason. Re-verify
-this hack on any Kysely bump; it is the single place that reaches past Kysely's
-documented surface. Everything else (plugins, dialect, transformer overrides)
-uses public or documented-protected API.
-
-## Types are asserted at compile time
-
-`types.assert.ts` holds `tsc`-only assertions (schema typing, view opacity,
-arrayJoin widening, limitBy preservation). It is never run; `schema.test.ts`
-anchors it so it stays in the build graph. `@ts-expect-error` lines there must
-stay live.
-
-## Verifying changes
-
-```
-CLICKHOUSE_BIN=clickhouse pnpm --filter @langfuse/shared run test src/server/query-ast
-```
-
-`catalog.golden.test.ts` and the other `*.golden.test.ts` suites assert
-`compile(AST) ≑ referenceSQL` after `clickhouse format`, so they need a local
-`clickhouse` binary and otherwise `describe.skip`. CI's non-blocking
-SQL-equivalence step installs the pinned binary and runs every `*.golden.test.ts`
-(it selects them by that name), so a suite must carry the `.golden.test.ts`
-suffix to run there. `composition.test.ts` asserts on raw compiler output and
-runs everywhere. After an intentional SQL change, regenerate golden baselines
-with `-u` (see `../README.md`).
diff --git a/packages/shared/src/server/query-ast/kysely/nodes.test.ts b/packages/shared/src/server/query-ast/nodes.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/nodes.test.ts
rename to packages/shared/src/server/query-ast/nodes.test.ts
diff --git a/packages/shared/src/server/query-ast/kysely/nodes.ts b/packages/shared/src/server/query-ast/nodes.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/nodes.ts
rename to packages/shared/src/server/query-ast/nodes.ts
diff --git a/packages/shared/src/server/query-ast/kysely/schema.test.ts b/packages/shared/src/server/query-ast/schema.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/schema.test.ts
rename to packages/shared/src/server/query-ast/schema.test.ts
diff --git a/packages/shared/src/server/query-ast/kysely/schema.ts b/packages/shared/src/server/query-ast/schema.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/schema.ts
rename to packages/shared/src/server/query-ast/schema.ts
diff --git a/packages/shared/src/server/query-ast/kysely/tenancy.ts b/packages/shared/src/server/query-ast/tenancy.ts
similarity index 97%
rename from packages/shared/src/server/query-ast/kysely/tenancy.ts
rename to packages/shared/src/server/query-ast/tenancy.ts
index b2f2c3636b..12789cc5ff 100644
--- a/packages/shared/src/server/query-ast/kysely/tenancy.ts
+++ b/packages/shared/src/server/query-ast/tenancy.ts
@@ -22,12 +22,20 @@ import {
type RootOperationNode,
} from "kysely";
-import type { ExecutionContext } from "../executionContext";
import { QueryCompileError, UnscopedRelationError } from "./errors";
import { type ClickHouseSelectQueryNode } from "./nodes";
import { TENANTED_TABLES } from "./schema";
import { ClickHouseOperationNodeTransformer } from "./transformer";
+/**
+ * Compile-time tenancy scope. Every ClickHouse query compiled through
+ * {@link compileClickhouseQuery} must carry one of these; the tenancy
+ * injection pass keys off `projectId`.
+ */
+export type ExecutionContext = {
+ projectId: string;
+};
+
const PROJECT_ID_COLUMN = "project_id";
/**
diff --git a/packages/shared/src/server/query-ast/kysely/transformer.ts b/packages/shared/src/server/query-ast/transformer.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/transformer.ts
rename to packages/shared/src/server/query-ast/transformer.ts
diff --git a/packages/shared/src/server/query-ast/kysely/typecheck.ts b/packages/shared/src/server/query-ast/typecheck.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/typecheck.ts
rename to packages/shared/src/server/query-ast/typecheck.ts
diff --git a/packages/shared/src/server/query-ast/kysely/types.assert.ts b/packages/shared/src/server/query-ast/types.assert.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/types.assert.ts
rename to packages/shared/src/server/query-ast/types.assert.ts
diff --git a/packages/shared/src/server/query-ast/kysely/views.ts b/packages/shared/src/server/query-ast/views.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/kysely/views.ts
rename to packages/shared/src/server/query-ast/views.ts
diff --git a/packages/shared/src/server/query-ast/__snapshots__/environments.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap
similarity index 100%
rename from packages/shared/src/server/query-ast/__snapshots__/environments.golden.test.ts.snap
rename to packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap
diff --git a/packages/shared/src/server/query-ast/environments.golden.test.ts b/packages/shared/src/server/repositories/environments.golden.test.ts
similarity index 91%
rename from packages/shared/src/server/query-ast/environments.golden.test.ts
rename to packages/shared/src/server/repositories/environments.golden.test.ts
index c8d9996f09..5ca535e693 100644
--- a/packages/shared/src/server/query-ast/environments.golden.test.ts
+++ b/packages/shared/src/server/repositories/environments.golden.test.ts
@@ -11,14 +11,13 @@ import {
// Record the exec seam instead of hitting ClickHouse. The factory is hoisted
// above imports, so it pulls the harness in via dynamic import; the captured
// store is a module singleton shared with the assertions below.
-vi.mock("../repositories/clickhouse", async (importOriginal) => {
- const actual =
- await importOriginal<typeof import("../repositories/clickhouse")>();
+vi.mock("./clickhouse", async (importOriginal) => {
+ const actual = await importOriginal<typeof import("./clickhouse")>();
const { buildClickhouseMock } = await import("./goldenHarness.js");
return buildClickhouseMock(actual);
});
-import { getEnvironmentsForProject } from "../repositories/environments";
+import { getEnvironmentsForProject } from "./environments";
const FIXED_PROJECT_ID = "golden-project";
// Fixed so the captured params are deterministic across runs.
diff --git a/packages/shared/src/server/repositories/environments.ts b/packages/shared/src/server/repositories/environments.ts
index ffbd67982d..4d860ee310 100644
--- a/packages/shared/src/server/repositories/environments.ts
+++ b/packages/shared/src/server/repositories/environments.ts
@@ -1,8 +1,10 @@
import { LISTABLE_SCORE_TYPES } from "../../domain/scores";
import { env } from "../../env";
-import type { ExecutionContext } from "../query-ast/executionContext";
-import { compileClickhouseQuery } from "../query-ast/kysely/compile";
-import { getClickhouseKysely } from "../query-ast/kysely/dialect";
+import {
+ compileClickhouseQuery,
+ type ExecutionContext,
+} from "../query-ast/compile";
+import { getClickhouseKysely } from "../query-ast/dialect";
import { queryClickhouse } from "./clickhouse";
export type EnvironmentFilterProps = {
diff --git a/packages/shared/src/server/query-ast/goldenHarness.test.ts b/packages/shared/src/server/repositories/goldenHarness.test.ts
similarity index 100%
rename from packages/shared/src/server/query-ast/goldenHarness.test.ts
rename to packages/shared/src/server/repositories/goldenHarness.test.ts
diff --git a/packages/shared/src/server/query-ast/goldenHarness.ts b/packages/shared/src/server/repositories/goldenHarness.ts
similarity index 99%
rename from packages/shared/src/server/query-ast/goldenHarness.ts
rename to packages/shared/src/server/repositories/goldenHarness.ts
index d761bbb6a3..066ad5fdae 100644
--- a/packages/shared/src/server/query-ast/goldenHarness.ts
+++ b/packages/shared/src/server/repositories/goldenHarness.ts
@@ -3,7 +3,7 @@ import { homedir } from "os";
import { join } from "path";
import { Readable } from "stream";
-import type * as ClickhouseModule from "../repositories/clickhouse";
+import type * as ClickhouseModule from "./clickhouse";
/**
* Golden-SQL recording/diff harness for the query-builder AST refactor.