πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
query-ast (server)
Server-only half of the query-builder AST module: the Kysely-based ClickHouse
compiler, validation passes, physical registry, execution context, and exec
integration, behind the @langfuse/shared/src/server export boundary. The
client-safe half (node types, predicate AST, builder sugar, semantic field
metadata, FilterState embedding) will land under
packages/shared/src/query-ast/.
This folder is a compile-only ClickHouse dialect on Kysely 0.28: real
OperationNodes for ARRAY JOIN, LIMIT BY, FINAL, and metadata map
subscripts; a mandatory tenancy injection pass keyed on one projectId;
per-table soft-delete filtering and dedup lowering from the registry;
schema-typed selection; virtual views as WITH CTEs; catalog parity.
Two audiences: Using it if you are calling the compiler from a repository, Developing the dialect if you are changing how it emits SQL.
Layout
compile.tsβ the one supported entry point,compileClickhouseQuery(query, projectId).ctes.tsβ the first pass: rejects CTE and FROM/JOIN names that could mean two relations, so later passes read every registry name as its table.tenancy.tsβ the tenancy injection pass.schema.tsβ the physical table registry: onedefineTabledeclaration per relation drives the Kysely row types (ClickHouseDatabase), the runtime column-type map (COLUMN_DATA_TYPES), the tenanted-table set (TENANTED_TABLES), the per-table dedup specs (DEDUP_SPECS), and the soft-delete flag columns (SOFT_DELETE_COLUMNS) β instead of hand-maintained tables that drift.compiler.ts/transformer.ts/dialect.ts/nodes.tsβ the dialect.extensions.tsβ the$callhelpers (arrayJoin,limitBy,useFinal,includeDeleted,softDeleteAfterLimit) and typed ClickHouse functions and aggregates (argMaxIf,countIf,sumMap,multiIf,metadataValue,literal, β¦) that takeeb.ref(...)expressions.views.ts/dedup.ts/softDelete.tsβ virtual views as CTEs, per-table dedup lowering, and soft-delete filtering.fixtures/catalog.tsβ test-only catalog of builder queries exercising the dialect's expressivity.catalog.golden.test.tsβ catalog SQL baselines.
The golden-SQL harness lives in ../repositories/goldenHarness.ts (it captures
SQL at the repositories/clickhouse.ts exec seam in test mode, then normalizes
it via clickhouse format + positional param names for snapshot comparison β no
ClickHouse server needed). Each migrated call site keeps its SQL baseline next to
that call site, e.g. ../repositories/environments.golden.test.ts,
../repositories/events.golden.test.ts,
../repositories/scores.golden.test.ts, and
../queries/clickhouse-sql/event-filter-options.golden.test.ts.
Regenerate baselines with -u after an intentional SQL change. Filter by the
.golden.test name so every relocated suite is picked up regardless of its
directory (the same selector CI uses), not by src/server/query-ast β two of
the three baselines now live outside this folder:
pnpm --filter @langfuse/shared run test .golden.test -- -u
CI and the clickhouse format version
The *.golden.test.ts suites need the clickhouse binary (the format
subcommand, shipped in clickhouse-common-static); without it they
describe.skip. The tests-shared CI job installs it pinned to 26.4.5.143 β
the ClickHouse version recommended for Langfuse v4, also pinned in
scripts/codex/cloud_services.sh.
clickhouse format output is version-sensitive (e.g. how UNION ALL branches
are parenthesized changed between 25.x and 26.x), so the committed snapshots are
coupled to that exact version. When bumping the CI pin, regenerate the snapshots
against the new binary in the same PR, or the golden tests drift.
The CI step is intentionally non-blocking: a drift surfaces as a warning
annotation but never fails the pipeline (|| echo "::warning::"). It selects
suites by the .golden.test name, so a suite must carry the .golden.test.ts
suffix to run there. Promote it to a required check once it has proven stable.
Using it
- Compile only through
compileClickhouseQuery(query, projectId). It is the one supported path from a builder to{ sql, params }; the repository layer hands that toqueryClickhouse.projectIdis required β omitting it is a compile-time type error and an empty one throws. There is no multi-project scope: a query reads one project. - Never filter
project_idyourself. The compile step injectsproject_id = {projectId}into every tenanted relation, so call sites pass onlyprojectId(see../repositories/environments.ts). - Read events through the logical
eventsrelation. Eacheventsrelation compiles toevents_coreunless a column reference that resolves to it reads a columnevents_coretruncates (input,output,metadata_values,input_length,output_length); then that relation becomesevents_full. Metadata=/startsWithpredicates onmetadataValuestay onevents_corewhenmetadataFilterIsEventsCoreSafeaccepts the value, as in the raw-SQL builders.*/e.*overevents, raw SQL text in a select that reads it, and an unqualified truncated column are compile errors whileeventsis in scope. Readevents_truncatedto get the truncated copy on purpose (input/output/metadata previews): it always compiles toevents_core, whatever columns it reads, and the pass does not reject*or raw SQL over it.events_core/events_fullare not part of the Kysely schema, and naming them anywhere β FROM/JOIN, alias, column qualifier, raw SQL β is a compile error; only the routing pass writes them. - Give each CTE (and view) its own plain name. A CTE name is a plain
identifier that is not a registry table (
traces,events_core, the logicalevents/events_truncated, β¦) and doesn't reuse the name of a sibling or an enclosing CTE (a CTE is not in scope inside its own body). Every FROM/JOIN name is an unqualified registry table or a CTE in scope; schema-qualified names (default.traces) are rejected, since the registry describes tables by name only. A CTE body sees only the CTEs declared before it, so recursive and cyclic CTEs are out. The passes then read every registry name as its table. Rename or reorder the CTEs when the compile rejects them; add a physical table to the registry to read it. - Identifiers, function and aggregate names must be plain identifiers
(
[A-Za-z_][A-Za-z0-9_]*). The compiler writes them unquoted, so anything else fails to compile. This covers columns, tables, aliases, CTE names,sql.refandsql.id;sql.ref("e.*")is rejected too, list the columns instead. A positive integer after a qualifier (x.1) is tuple element access and compiles as is.collateis not supported. - Never filter
is_deletedon a physical table yourself. Tables that declaresoftDeletein the registry (events_core,events_full) getis_deleted = 0on every FROM/JOIN. A latest-version read (ordered by version, thenLIMIT 1orLIMIT 1 BY) marks itself$call(softDeleteAfterLimit()), so a deletion (a newer row with the flag set) wins the pick and is dropped after it, instead of an older live version. A select that must see deleted rows opts out with$call(includeDeleted()). Reads without a version pick (lists, aggregates) filter each row on its own: like an updated row under dedupnone, a deleted row's older live version stays visible until a merge collapses the versions. - Dedup is declared per table and must be an existing production idiom.
events_coreandevents_fullarenone(immutable at read time β no LIMIT BY, no FINAL).limitByis the legacyORDER BY <version> DESC LIMIT 1 BY <key>already used on traces / observations / scores.$call(limitBy(...))remains for explicit non-version LIMIT BY. Auto-lowering does not declare scores as{ strategy: "final" }; callers opt in with$call(useFinal(["scores"])). - Value binds take their ClickHouse type from the compared column in the
table registry (
scores.value > 1emits{p:Float64}, not inferred{p:Int64}). Asql.refcolumn is typed the same way, and a column tuple (refTuple,tupleIn) per position. Same-value binds still intern to one placeholder. A value compared to a column the compiler can't type throwsQueryCompileError: a qualifier no relation declares, a table outside the registry (or a CTE that reads one), and an unqualified column that the joined relations type differently (a computed CTE column counts as its own type) or that a relation outside the registry may provide. Qualify the column with the alias of a registry relation, or add the table to the registry. A SELECT alias takes the type of the column it names, also through another alias; a computed alias, an ARRAY JOIN alias, or a column no relation has, binds by value. A*over a join types a shared column name from the leftmost relation, as ClickHouse does. - Bind dates as JS
Date, never as pre-formatted strings. The compiler sends aDateas the UTC wall-clock string ingestion writes (toClickhouseDateTime), so binds match stored rows whatever the server timezone. UsetoDate/toStartOfMinutefromextensions.tsfor date-truncated comparisons. - Inline code constants with
literal(...), bind everything else.eb("level", "=", literal("ERROR"))renderslevel = 'ERROR'; a value that comes from a caller is a plain bind,eb("level", "=", level).literalrejects backslashes and must never receive user input. For a constantINlist, wrap each item:eb("data_type", "in", TYPES.map((t) => literal(t)))rendersdata_type in ('A', 'B'). A list that mixesliteral(...)items with bound values is rejected. - Registry types are logical, not wire types.
DateTime64isDateandUInt64isnumberso binds type-check, but query results arrive as raw JSON (dates as strings). Type rows with a read schema and convert them; do not type results with the builder'sInferResult. - Build shared expressions from the calling query's expression builder.
A reusable events expression or select list is wrapped in
scopedToEvents((eb) => β¦)from../repositories/events-selects.ts, then passed as.select(traceAggregationSelects(TRACE_AGGREGATION_FIELDS))or called inside.where((eb) => β¦)with the query's owneb. Keyed select lists (traceAggregationSelects,sessionAggregationSelects,eventsRowFieldSelects) take the fields to select: pick only what the query reads, because one selectedinput,outputor metadata value routes a logicaleventsrelation, CTE bodies included, toevents_full. That only type-checks in a query that reads a full events row aseand nothing else: no join, no narrowere. Columns inside such a helper are writtene.β¦, so an alias of the same name in the calling query can't shadow them. A helper typedExpressionBuilder<{ e: β¦ }, "e">checks the alias and the columns its type names, but not joins or the rest of the row: fine for a fixed small column set (observationIoSelectsover{ e: IoColumns }), not a substitute forscopedToEvents. A module-level constant built from a detachedexpressionBuilder()loses the check entirely: helper results don't carry their table scope in their type, and Kysely'sselect()/where()accept any expression. - ClickHouse-only clauses use
$call(helper())β not fluent builder methods, so they compose inside CTEs, subqueries, and views. See the recipes below.
Recipes
Write an ARRAY JOIN (arrayJoin + mapKeys/mapValues from ./extensions):
db.selectFrom("observations")
.select("environment")
.$call(
arrayJoin((eb) => ({
cost_key: mapKeys(eb.ref("cost_details")),
cost: mapValues(eb.ref("cost_details")),
})),
);
// β¦ array join mapKeys(cost_details) as cost_key, mapValues(cost_details) as cost
Write a LIMIT BY (limitBy):
db.selectFrom("events")
.$call(softDeleteAfterLimit())
.select(["span_id", "project_id"])
.orderBy("event_ts", "desc")
.$call(
limitBy({
count: 1,
columns: (eb) => [eb.ref("span_id"), eb.ref("project_id")],
}),
);
// β¦ order by event_ts desc limit 1 by span_id, project_id, wrapped by the
// soft-delete pass so a deleted latest version is dropped
Write FINAL (useFinal β ReplacingMergeTree reads; never on the events relations):
db.selectFrom("scores as s")
.select("s.id")
.$call(useFinal(["scores"]));
// β¦ from scores as s final
Filter on a metadata value (metadataValue β lowers metadata[key] to a subscript of metadataMap, so the last duplicate key wins; like mapValue, a string key is bound and a literal(...) key is inlined; names / values must be one relation's metadata_names / metadata_values refs, anything else throws. Events reads use eventsMetadataValue(eb, key) from ../repositories/events-selects.ts):
const filterKey: string = input.metadataKey; // caller-supplied, so bound
db.selectFrom("events as e")
.select("e.span_id")
.where((eb) => eb(eventsMetadataValue(eb, filterKey), "=", "x"));
// select e.span_id from events_core as e where β¦ and mapfromarrays(arrayreverse(e.metadata_names), arrayreverse(e.metadata_values))[{p:String}] = {p:String}
Selecting metadataValue(...) reads the truncated metadata_values, so it
routes that relation to events_full.
Developing the dialect
Several patterns here are deliberately unusual. This is the guide for anyone changing how the builder emits SQL.
How ClickHouse clauses land (no fork)
Kysely's OperationNodeKind union is closed, so a first-class node kind would
need a fork. Instead each ClickHouse-only construct rides as an extra field or a
special-cased node, and the compiler/transformer are overridden to emit and
preserve it:
| Clause | How it lands | Fork? |
|---|---|---|
| ARRAY JOIN | Plugin attaches ArrayJoinNode as an extra field on SelectQueryNode. ClickHouseOperationNodeTransformer preserves it. ClickHouseQueryCompiler.visitSelectQuery emits it after JOINs / before WHERE. |
no |
| LIMIT BY | Plugin attaches LimitByNode the same way. Compiler emits it after ORDER BY / before LIMIT. |
no |
| FINAL | Plugin wraps a FROM/JOIN table expression in FinalTableNode. Compiler emits final after that expression (scores as s final). $call(useFinal(["scores"])). Tables declared dedup: none throw; so does FINAL on a limitBy table at compile (drop the useFinal). |
no |
| metadata subscript | Helper builds an ArrayIndexNode over the metadataMap FunctionNode, indexed by a string key expression (a bound value or a literal(...)). Transformer + compiler special-case the node. No plugin. |
no |
| Virtual view | Plugin rewrites selectFrom(viewName) into a WITH CTE. Outer types only expose the view's selected columns. |
no |
These are real node objects whose children are traced Kysely
FunctionNode/ColumnNode/ValueNode/IdentifierNode values β not RawNode
string splices. Any plugin using the default OperationNodeTransformer would
drop them, so ours overrides transformSelectQuery to keep them. Upstream is
not patched.
Kysely never runs β it only compiles
getClickhouseKysely() returns a Kysely instance wired to a DummyDriver
with SQLite adapter/introspector stand-ins (dialect.ts). Nothing ever
connects or executes. The only supported output path is
compileClickhouseQuery(query, projectId) β { sql, params }, which the
repository layer hands to queryClickhouse. Do not call .execute() / .compile()
directly: the compiler refuses any tree that did not go through the tenancy pass.
Tenancy injection β how the choke point works
compileClickhouseQuery(query, projectId) is the only supported compile path,
and it is where tenancy is enforced:
- A missing/empty
projectIdthrows (QueryCompileError) βprojectIdis a required parameter, so omitting it is also a compile-time type error.CteNamePluginthen rejects the CTE names the rules above forbid; it runs before every other pass. A FROM/JOIN name that resolves to no registry table or CTE in scope, and a schema-qualified name, throwUnscopedRelationError; naming errors throwQueryCompileError. SoftDeleteFilterPluginadds<flag> = 0for every FROM/JOIN relation whose table declaressoftDelete, except in selects marked withincludeDeleted(). A select markedsoftDeleteAfterLimit()is left unfiltered, gets the flag added to its outputs, and is wrapped in an outer select over those outputs that keeps rows withlatest.<flag> = 0. It runs before tenancy, so the WHERE readsproject_id = β¦ and is_deleted = β¦ and β¦.TenancyInjectionPluginwalks every FROM/JOIN and injectsproject_id = {projectId}on each tenanted physical table, unless the tree already carries a predicate that proves that scope: the equality's value must equalprojectId, and β when more than one tenanted relation is in scope β the column must be table-qualified. A qualified predicate covers a relation only when the qualifier matches its alias (if aliased) or its table name (if not) β soscores AS tracesjoined totraces AS tstill scopes both. It then identity-stamps the tree (WeakSet); a copiedlangfuseTenancyproperty is not a valid stamp.DedupLoweringPluginapplies the table's declared read idiom (none/limitBy/final).events_coreandevents_fullarenoneβ immutable at read time, so the pass does not inject LIMIT BY or FINAL.limitByis the existing legacyORDER BY <version> DESC LIMIT 1 BY <key>.finalremains fail-closed on the auto-lowering pass; callers opt in with$call(useFinal(["scores"])). FINAL on a table that declareslimitBythrows: the registry already picks the latest version, so drop theuseFinal. The pass restamps the rewritten root.ClickHouseQueryCompilerrefuses to emit SQL unless that identity stamp is present, soqb.compile()without the plugin also fails. Value binds take their ClickHouse type from the compared column's registry entry when one is in scope (scores.value > 1β{p:Float64}).- Raw-SQL table sources (
selectFrom(sql`...`)), raw fragments embedding aSELECT/FROM/JOINin SELECT/WHERE, and asql.table(...)reference inside raw SQL in any position throwUnscopedRelationError, as does any FROM/JOIN source that is neither a table reference nor a subquery (table functions such asremote(),clusterAllReplicas(),merge()). Kysely's own keyword fragments (asc/desc) are not relations.
Every pass and the compiler read a FROM/JOIN item through describeRelation
(relations.ts): it peels one FINAL and then one alias, so scores as s final
is the table scores under alias s. Any other nesting (an alias around
FINAL, FINAL around FINAL) is neither a table nor a subquery, and step 6
rejects it. A new relation shape is handled there, once.
So query bodies here never filter project_id by hand β it is redundant, and
forgetting it is impossible.
Why $call(helper()), not a fluent .arrayJoin(...) method
A real method would have to exist on every builder instance, including the
native Kysely builders that .with((qb) => β¦), subquery, and defineView
callbacks hand you. Getting that requires either forking Kysely's whole builder
graph or globally mutating its prototype via an internal kysely/dist/...
import (blocked here by Kysely's exports map under NodeNext). A curried helper
is a plain function of a builder, so it works in any of those positions β which
is exactly why ARRAY JOIN / LIMIT BY / FINAL compose inside CTEs, subqueries,
and views.
composition.test.ts locks that property in.
arrayJoin widens the row type. Each { alias: arrayExpr } entry is added
to the builder's output row, so an outer query over a CTE body can reference the
produced column and a typo on the alias is a compile error. The element value
type is unknown: Kysely's Expression<T> hides its type argument from
inference. Precise value types would need a branded array-expression wrapper on
mapKeys / mapValues / etc. β not done yet. types.assert.ts pins the
widening behavior.
arrayJoin (clause) β arrayJoin() (function). ClickHouse has both. The
helper here builds the ARRAY JOIN clause. The row-expanding SELECT function
is just eb.fn("arrayJoin", [...]).
Escape hatches and their cost
sql.ref("alias")β the only way to reference a SELECT alias that is not a schema column (ClickHouse allowsGROUP BY/expression reuse of aliases; Kysely's types do not model it). It is fully untyped β a typo reaches ClickHouse unchecked. Use sparingly.eb.fn("ch_function", [...])β arbitrary ClickHouse functions. The name is an unchecked string and the return type defaults tounknown; no arity or return checking.- New columns β add them to the table registry in
schema.ts(onedefineTableentry per relation) as queries need them. That single declaration drives the row types, the runtime column-type map, and the tenanted-table set.
The one Kysely-internals coupling (upgrade hazard)
compiler.ts wraps Kysely's private visitNode / nodeStack to dispatch
ArrayIndexNode (for metadata[key]) and FinalTableNode (for FINAL),
because they are not Kysely's closed OperationNode kinds. Kysely is pinned
to 0.28.17 for this reason. Re-verify this hack on any Kysely bump; it is
the single place that reaches past Kysely's documented surface. Everything
else (plugins, dialect, transformer overrides) uses public or
documented-protected API.
Types are asserted at compile time
types.assert.ts holds tsc-only assertions (schema typing, view opacity,
arrayJoin widening, limitBy / useFinal preservation). It is never run: tsc
checks it through the package's include, and knip treats *types.assert.ts
as an entry. @ts-expect-error lines there must stay live.
Verifying changes
The two positional filters cover the dialect unit tests under this folder plus
every .golden.test suite (two of which now live in ../repositories/ and
../queries/clickhouse-sql/):
CLICKHOUSE_BIN=clickhouse pnpm --filter @langfuse/shared run test src/server/query-ast .golden.test
The *.golden.test.ts suites assert compile(AST) β‘ referenceSQL after
clickhouse format, so they need a local clickhouse binary and otherwise
describe.skip. composition.test.ts asserts on raw compiler output and runs
everywhere. After an intentional SQL change, regenerate golden baselines with
-u (see Layout above).