🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(query-ast): bind values with the compared column's ClickHouse type (#17003)
Niklas Semmler29fa9ffbd8e0
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex ddc03f515f..8a58949ded 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -62,6 +62,9 @@ a required check once it has proven stable.`limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` alreadyused on traces / observations / scores. `$call(limitBy(...))` remains forexplicit non-version LIMIT BY.+- **Value binds take their ClickHouse type from the compared column** in the+ table registry (`total_cost > 1` emits `{p:Float64}`, not inferred+ `{p:Int64}`). Same-value binds still intern to one placeholder.- **ClickHouse-only clauses use `$call(helper())`** — not fluent buildermethods, so they compose inside CTEs, subqueries, and views. See the recipesbelow.@@ -73,7 +76,12 @@ a required check once it has proven stable.```tsdb.selectFrom("observations").select("environment")- .$call(arrayJoin({ cost_key: mapKeys("cost_details"), cost: mapValues("cost_details") }));+ .$call(+ arrayJoin({+ cost_key: mapKeys("cost_details"),+ cost: mapValues("cost_details"),+ }),+ );// … array join mapKeys(cost_details) as cost_key, mapValues(cost_details) as cost```diff --git a/packages/shared/src/server/query-ast/kysely/README.md b/packages/shared/src/server/query-ast/kysely/README.mdindex 0401a1cc7c..daaa79ceac 100644--- a/packages/shared/src/server/query-ast/kysely/README.md+++ b/packages/shared/src/server/query-ast/kysely/README.md@@ -57,10 +57,12 @@ is where tenancy is enforced:`final` is fail-closed until an emitter exists. The pass restamps therewritten root.4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is- present, so `qb.compile()` without the plugin also fails.+ present, so `qb.compile()` without the plugin also fails. Value binds take+ their ClickHouse type from the compared column's registry entry when one is+ in scope (`total_cost > 1` → `{p:Float64}`).5. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a- `SELECT`/`FROM`/`JOIN` in SELECT/WHERE throw `UnscopedRelationError`. Kysely's- own keyword fragments (`asc`/`desc`) are not relations.+`SELECT`/`FROM`/`JOIN`in SELECT/WHERE throw`UnscopedRelationError`. Kysely's+own keyword fragments (`asc`/`desc`) are not relations.So query bodies here never filter `project_id` by hand — it is redundant, andforgetting it is impossible. Call sites like `repositories/environments.ts` pass@@ -74,12 +76,17 @@ Kysely's public `$call`:```tsdb.selectFrom("observations").select("environment")- .$call(arrayJoin({ cost_key: mapKeys("cost_details"), cost: mapValues("cost_details") }))+ .$call(+ arrayJoin({+ cost_key: mapKeys("cost_details"),+ cost: mapValues("cost_details"),+ }),+ );db.selectFrom("events_core").select(["span_id", "project_id"]).orderBy("event_ts", "desc")- .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }))+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));```**Why `$call(...)` and not a fluent `.arrayJoin(...)` method:** a real methoddiff --git a/packages/shared/src/server/query-ast/kysely/binds.test.ts b/packages/shared/src/server/query-ast/kysely/binds.test.tsnew file mode 100644index 0000000000..9548a90b61--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/binds.test.ts@@ -0,0 +1,86 @@+import { describe, expect, it } from "vitest";++import { compileClickhouseQuery } from "./compile";+import { getClickhouseKysely } from "./dialect";++const ctx = { projectId: "proj-1" };++function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {+ return compileClickhouseQuery(builder, ctx);+}++describe("typed params from the column registry", () => {+ it("binds an integer compared to total_cost as Float64, not Int64", () => {+ const { sql, params } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("span_id")+ .where("total_cost", ">", 1),+ );+ expect(sql).toMatch(/total_cost > \{p\d+:Float64\}/);+ expect(sql).not.toMatch(/\{p\d+:Int64\}/);+ expect(Object.values(params)).toContain(1);+ });++ it("binds an IN list against a Float column as Array(Float64)", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("span_id")+ .where("total_cost", "in", [1, 2]),+ );+ expect(sql).toMatch(/total_cost in \(\{p\d+:Array\(Float64\)\}\)/);+ });++ it("wraps an IN list against an Array column as Array(Array(...))", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("span_id")+ .where("metadata_names", "in", [["a"], ["b"]]),+ );+ expect(sql).toMatch(+ /metadata_names in \(\{p\d+:Array\(Array\(String\)\)\}\)/,+ );+ expect(sql).not.toMatch(/metadata_names in \(\{p\d+:Array\(String\)\}\)/);+ });++ it("does not type a subquery LIMIT from an outer column comparison", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("trace_id")+ .where(+ "trace_id",+ "in",+ getClickhouseKysely().selectFrom("traces").select("id").limit(3),+ ),+ );+ expect(sql).toMatch(/limit \{p\d+:Int64\}/);+ expect(sql).not.toMatch(/limit \{p\d+:String\}/);+ });++ it("interns the same typed value to one placeholder across UNION branches", () => {+ const { sql, params } = compile(+ getClickhouseKysely()+ .selectFrom("traces")+ .select("environment")+ .where("id", "=", "same")+ .unionAll(+ getClickhouseKysely()+ .selectFrom("observations")+ .select("environment")+ .where("trace_id", "=", "same"),+ ),+ );+ expect(+ Object.entries(params).filter(([, value]) => value === "same"),+ ).toHaveLength(1);+ const sameName = Object.entries(params).find(+ ([, value]) => value === "same",+ )?.[0];+ expect(sameName).toBeDefined();+ const occurrences = sql.split(`{${sameName}:String}`).length - 1;+ expect(occurrences).toBe(2);+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/compiler.ts b/packages/shared/src/server/query-ast/kysely/compiler.tsindex 02fc6dbb79..8bb26f3b3d 100644--- a/packages/shared/src/server/query-ast/kysely/compiler.ts+++ b/packages/shared/src/server/query-ast/kysely/compiler.ts@@ -1,9 +1,12 @@import {+ BinaryOperationNode,+ ColumnNode,CreateTableNode,CreateViewNode,DefaultQueryCompiler,InsertQueryNode,ParensNode,+ ReferenceNode,SetOperationNode,ValueNode,type OperationNode,@@ -20,6 +23,7 @@ import {type ClickHouseSelectQueryNode,type LimitByNode,} from "./nodes";+import { COLUMN_BIND_TYPES } from "./schema";import { assertTenancyStamped } from "./tenancy";const ARRAY_JOIN_SQL: Record<ArrayJoinNode["variant"], string> = {@@ -71,6 +75,12 @@ function inferClickHouseType(value: unknown): string {export class ClickHouseQueryCompiler extends DefaultQueryCompiler {namedParameters: Record<string, unknown> = {};private intern = new Map<string, string>();+ /**+ * ClickHouse type taken from the compared column (registry bind type) while+ * visiting a binary operation. Falls back to JS-value inference when no+ * column is in scope.+ */+ private pendingBindType: string | undefined;constructor() {super();@@ -117,22 +127,37 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {}protected override appendValue(parameter: unknown): void {- this.append(this.bindValue(parameter));+ this.append(this.bindValue(parameter, this.pendingBindType));+ }++ /**+ * While visiting `column <op> value`, bind the value with the column's+ * registry type so an integer compared to a Float column still emits+ * `{p:Float64}` rather than inferred `{p:Int64}`.+ */+ protected override visitBinaryOperation(node: BinaryOperationNode): void {+ const columnType =+ bindTypeOfColumnOperand(node.leftOperand) ??+ bindTypeOfColumnOperand(node.rightOperand);+ const previous = this.pendingBindType;+ if (columnType) this.pendingBindType = columnType;+ super.visitBinaryOperation(node);+ this.pendingBindType = previous;}/*** ClickHouse `IN` takes a single `Array(T)` bind, not a parenthesized list of* scalar binds. So `col IN (1, 2, 3)` compiles to `col IN ({p:Array(Int64)})`* — one array parameter — which is also the shape the existing production SQL- * (e.g. the scores list query) already relies on.+ * (e.g. the scores list query) already relies on. `T` is the compared+ * column's registry type, including when that type is already `Array(...)`.*/protected override visitPrimitiveValueList(node: PrimitiveValueListNode,): void {+ const values = [...node.values];this.append("(");- this.append(- this.bindValue([...node.values], inferClickHouseType([...node.values])),- );+ this.append(this.bindValue(values, this.arrayBindType(values)));this.append(")");}@@ -142,13 +167,33 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {);if (values.every((v) => v !== undefined)) {this.append("(");- this.append(this.bindValue(values, inferClickHouseType(values)));+ this.append(this.bindValue(values, this.arrayBindType(values)));this.append(")");return;}super.visitValueList(node);}+ private arrayBindType(values: unknown[]): string {+ if (this.pendingBindType) {+ return `Array(${this.pendingBindType})`;+ }+ return inferClickHouseType(values);+ }++ protected override visitSelectQuery(node: SelectQueryNode): void {+ // A surrounding `column <op> (SELECT …)` must not type this select's+ // own LIMIT / OFFSET / literals with the outer column's bind type, so we+ // clear the pending bind type across the (forked) body and restore it after.+ const previousBindType = this.pendingBindType;+ this.pendingBindType = undefined;+ try {+ this.visitSelectQueryBody(node);+ } finally {+ this.pendingBindType = previousBindType;+ }+ }+/*** Wholesale copy of Kysely's `DefaultQueryCompiler.visitSelectQuery` (kept in* sync with the pinned 0.28.17) with two ClickHouse-only insertions:@@ -159,7 +204,7 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {* than call `super` because the parent emits clauses in a fixed order and* offers no hook to inject a clause between JOIN and WHERE.*/- protected override visitSelectQuery(node: SelectQueryNode): void {+ private visitSelectQueryBody(node: SelectQueryNode): void {const wrapInParens =this.parentNode !== undefined &&!ParensNode.is(this.parentNode) &&@@ -322,3 +367,11 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {return `{${name}:${inferred}}`;}}++function bindTypeOfColumnOperand(node: OperationNode): string | undefined {+ if (ColumnNode.is(node)) return COLUMN_BIND_TYPES[node.column.name];+ if (ReferenceNode.is(node) && ColumnNode.is(node.column)) {+ return COLUMN_BIND_TYPES[node.column.column.name];+ }+ return undefined;+}diff --git a/packages/shared/src/server/query-ast/kysely/schema.test.ts b/packages/shared/src/server/query-ast/kysely/schema.test.tsindex 4d502ed386..abe4fd564c 100644--- a/packages/shared/src/server/query-ast/kysely/schema.test.ts+++ b/packages/shared/src/server/query-ast/kysely/schema.test.ts@@ -1,6 +1,11 @@import { describe, expect, it } from "vitest";-import { COLUMN_DATA_TYPES, DEDUP_SPECS, TENANTED_TABLES } from "./schema";+import {+ COLUMN_BIND_TYPES,+ COLUMN_DATA_TYPES,+ DEDUP_SPECS,+ TENANTED_TABLES,+} from "./schema";import { schemaTypeAssertions, extensionTypeAssertions } from "./types.assert";describe("table registry derivation", () => {@@ -34,6 +39,13 @@ describe("table registry derivation", () => {expect(DEDUP_SPECS.scores).toBeUndefined();});+ it("derives ClickHouse bind types from the same column declarations", () => {+ expect(COLUMN_BIND_TYPES.total_cost).toBe("Float64");+ expect(COLUMN_BIND_TYPES.start_time).toBe("DateTime64(3)");+ expect(COLUMN_BIND_TYPES.project_id).toBe("String");+ expect(COLUMN_BIND_TYPES.metadata_names).toBe("Array(String)");+ });+// The assertions themselves are compile-time (`tsc` is the test); this only// anchors the file so its `@ts-expect-error` checks stay in the build graph.it("keeps the compile-time type assertions in the build graph", () => {diff --git a/packages/shared/src/server/query-ast/kysely/schema.ts b/packages/shared/src/server/query-ast/kysely/schema.tsindex a4548799a5..0ebd3fcaaa 100644--- a/packages/shared/src/server/query-ast/kysely/schema.ts+++ b/packages/shared/src/server/query-ast/kysely/schema.ts@@ -6,6 +6,8 @@* comparisons).* - `COLUMN_DATA_TYPES` — the coarse runtime type map the aggregate type-check* pass consults (`typecheck.ts`).+ * - `COLUMN_BIND_TYPES` — the ClickHouse bind types the compiler uses when a+ * value is compared to a known column (`compiler.ts`).* - `TENANTED_TABLES` — the relations the tenancy pass must scope* (`tenancy.ts`).* - `DEDUP_SPECS` — the per-table read idiom the lowering pass applies@@ -176,3 +178,26 @@ export const DEDUP_SPECS: Record<string, DedupSpec> = Object.fromEntries(spec.dedup ? [[name, spec.dedup]] : [],),);++/**+ * ClickHouse bind type for a JS value compared against this column. Wider than+ * the coarse {@link COLUMN_DATA_TYPES} category: `DateTime` columns bind as+ * `DateTime64(3)`, `Float` as `Float64`, so an integer literal compared to+ * `total_cost` still becomes `{p:Float64}`.+ */+const BIND_TYPE: Record<ChColumnType, string> = {+ String: "String",+ Float: "Float64",+ DateTime: "DateTime64(3)",+ "Array(String)": "Array(String)",+ "Map(String, Float)": "Map(String, Float64)",+};++export const COLUMN_BIND_TYPES: Record<string, string> = Object.fromEntries(+ Object.values(TABLE_REGISTRY).flatMap((table) =>+ Object.entries(table.columns).map(([name, chType]) => [+ name,+ BIND_TYPE[chType],+ ]),+ ),+);