Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(query-ast): bind values with the compared column's ClickHouse type (#17003)

Niklas Semmler29fa9ffbd8e0
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index ddc03f515f..8a58949ded 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -62,6 +62,9 @@ a required check once it has proven stable.
`limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` already
used on traces / observations / scores. `$call(limitBy(...))` remains for
explicit non-version LIMIT BY.
+- **Value binds take their ClickHouse type from the compared column** in the
+ table registry (`total_cost > 1` emits `{p:Float64}`, not inferred
+ `{p:Int64}`). Same-value binds still intern to one placeholder.
- **ClickHouse-only clauses use `$call(helper())`** — not fluent builder
methods, so they compose inside CTEs, subqueries, and views. See the recipes
below.
@@ -73,7 +76,12 @@ a required check once it has proven stable.
```ts
db.selectFrom("observations")
.select("environment")
- .$call(arrayJoin({ cost_key: mapKeys("cost_details"), cost: mapValues("cost_details") }));
+ .$call(
+ arrayJoin({
+ cost_key: mapKeys("cost_details"),
+ cost: mapValues("cost_details"),
+ }),
+ );
// … array join mapKeys(cost_details) as cost_key, mapValues(cost_details) as cost
```
diff --git a/packages/shared/src/server/query-ast/kysely/README.md b/packages/shared/src/server/query-ast/kysely/README.md
index 0401a1cc7c..daaa79ceac 100644
--- a/packages/shared/src/server/query-ast/kysely/README.md
+++ b/packages/shared/src/server/query-ast/kysely/README.md
@@ -57,10 +57,12 @@ is where tenancy is enforced:
`final` is fail-closed until an emitter exists. The pass restamps the
rewritten root.
4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is
- present, so `qb.compile()` without the plugin also fails.
+ present, so `qb.compile()` without the plugin also fails. Value binds take
+ their ClickHouse type from the compared column's registry entry when one is
+ in scope (`total_cost > 1` → `{p:Float64}`).
5. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a
- `SELECT`/`FROM`/`JOIN` in SELECT/WHERE throw `UnscopedRelationError`. Kysely's
- own keyword fragments (`asc`/`desc`) are not relations.
+`SELECT`/`FROM`/`JOIN`in SELECT/WHERE throw`UnscopedRelationError`. Kysely's
+own keyword fragments (`asc`/`desc`) are not relations.
So query bodies here never filter `project_id` by hand — it is redundant, and
forgetting it is impossible. Call sites like `repositories/environments.ts` pass
@@ -74,12 +76,17 @@ Kysely's public `$call`:
```ts
db.selectFrom("observations")
.select("environment")
- .$call(arrayJoin({ cost_key: mapKeys("cost_details"), cost: mapValues("cost_details") }))
+ .$call(
+ arrayJoin({
+ cost_key: mapKeys("cost_details"),
+ cost: mapValues("cost_details"),
+ }),
+ );
db.selectFrom("events_core")
.select(["span_id", "project_id"])
.orderBy("event_ts", "desc")
- .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }))
+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));
```
**Why `$call(...)` and not a fluent `.arrayJoin(...)` method:** a real method
diff --git a/packages/shared/src/server/query-ast/kysely/binds.test.ts b/packages/shared/src/server/query-ast/kysely/binds.test.ts
new file mode 100644
index 0000000000..9548a90b61
--- /dev/null
+++ b/packages/shared/src/server/query-ast/kysely/binds.test.ts
@@ -0,0 +1,86 @@
+import { describe, expect, it } from "vitest";
+
+import { compileClickhouseQuery } from "./compile";
+import { getClickhouseKysely } from "./dialect";
+
+const ctx = { projectId: "proj-1" };
+
+function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {
+ return compileClickhouseQuery(builder, ctx);
+}
+
+describe("typed params from the column registry", () => {
+ it("binds an integer compared to total_cost as Float64, not Int64", () => {
+ const { sql, params } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select("span_id")
+ .where("total_cost", ">", 1),
+ );
+ expect(sql).toMatch(/total_cost > \{p\d+:Float64\}/);
+ expect(sql).not.toMatch(/\{p\d+:Int64\}/);
+ expect(Object.values(params)).toContain(1);
+ });
+
+ it("binds an IN list against a Float column as Array(Float64)", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select("span_id")
+ .where("total_cost", "in", [1, 2]),
+ );
+ expect(sql).toMatch(/total_cost in \(\{p\d+:Array\(Float64\)\}\)/);
+ });
+
+ it("wraps an IN list against an Array column as Array(Array(...))", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select("span_id")
+ .where("metadata_names", "in", [["a"], ["b"]]),
+ );
+ expect(sql).toMatch(
+ /metadata_names in \(\{p\d+:Array\(Array\(String\)\)\}\)/,
+ );
+ expect(sql).not.toMatch(/metadata_names in \(\{p\d+:Array\(String\)\}\)/);
+ });
+
+ it("does not type a subquery LIMIT from an outer column comparison", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select("trace_id")
+ .where(
+ "trace_id",
+ "in",
+ getClickhouseKysely().selectFrom("traces").select("id").limit(3),
+ ),
+ );
+ expect(sql).toMatch(/limit \{p\d+:Int64\}/);
+ expect(sql).not.toMatch(/limit \{p\d+:String\}/);
+ });
+
+ it("interns the same typed value to one placeholder across UNION branches", () => {
+ const { sql, params } = compile(
+ getClickhouseKysely()
+ .selectFrom("traces")
+ .select("environment")
+ .where("id", "=", "same")
+ .unionAll(
+ getClickhouseKysely()
+ .selectFrom("observations")
+ .select("environment")
+ .where("trace_id", "=", "same"),
+ ),
+ );
+ expect(
+ Object.entries(params).filter(([, value]) => value === "same"),
+ ).toHaveLength(1);
+ const sameName = Object.entries(params).find(
+ ([, value]) => value === "same",
+ )?.[0];
+ expect(sameName).toBeDefined();
+ const occurrences = sql.split(`{${sameName}:String}`).length - 1;
+ expect(occurrences).toBe(2);
+ });
+});
diff --git a/packages/shared/src/server/query-ast/kysely/compiler.ts b/packages/shared/src/server/query-ast/kysely/compiler.ts
index 02fc6dbb79..8bb26f3b3d 100644
--- a/packages/shared/src/server/query-ast/kysely/compiler.ts
+++ b/packages/shared/src/server/query-ast/kysely/compiler.ts
@@ -1,9 +1,12 @@
import {
+ BinaryOperationNode,
+ ColumnNode,
CreateTableNode,
CreateViewNode,
DefaultQueryCompiler,
InsertQueryNode,
ParensNode,
+ ReferenceNode,
SetOperationNode,
ValueNode,
type OperationNode,
@@ -20,6 +23,7 @@ import {
type ClickHouseSelectQueryNode,
type LimitByNode,
} from "./nodes";
+import { COLUMN_BIND_TYPES } from "./schema";
import { assertTenancyStamped } from "./tenancy";
const ARRAY_JOIN_SQL: Record<ArrayJoinNode["variant"], string> = {
@@ -71,6 +75,12 @@ function inferClickHouseType(value: unknown): string {
export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
namedParameters: Record<string, unknown> = {};
private intern = new Map<string, string>();
+ /**
+ * ClickHouse type taken from the compared column (registry bind type) while
+ * visiting a binary operation. Falls back to JS-value inference when no
+ * column is in scope.
+ */
+ private pendingBindType: string | undefined;
constructor() {
super();
@@ -117,22 +127,37 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
}
protected override appendValue(parameter: unknown): void {
- this.append(this.bindValue(parameter));
+ this.append(this.bindValue(parameter, this.pendingBindType));
+ }
+
+ /**
+ * While visiting `column <op> value`, bind the value with the column's
+ * registry type so an integer compared to a Float column still emits
+ * `{p:Float64}` rather than inferred `{p:Int64}`.
+ */
+ protected override visitBinaryOperation(node: BinaryOperationNode): void {
+ const columnType =
+ bindTypeOfColumnOperand(node.leftOperand) ??
+ bindTypeOfColumnOperand(node.rightOperand);
+ const previous = this.pendingBindType;
+ if (columnType) this.pendingBindType = columnType;
+ super.visitBinaryOperation(node);
+ this.pendingBindType = previous;
}
/**
* ClickHouse `IN` takes a single `Array(T)` bind, not a parenthesized list of
* scalar binds. So `col IN (1, 2, 3)` compiles to `col IN ({p:Array(Int64)})`
* — one array parameter — which is also the shape the existing production SQL
- * (e.g. the scores list query) already relies on.
+ * (e.g. the scores list query) already relies on. `T` is the compared
+ * column's registry type, including when that type is already `Array(...)`.
*/
protected override visitPrimitiveValueList(
node: PrimitiveValueListNode,
): void {
+ const values = [...node.values];
this.append("(");
- this.append(
- this.bindValue([...node.values], inferClickHouseType([...node.values])),
- );
+ this.append(this.bindValue(values, this.arrayBindType(values)));
this.append(")");
}
@@ -142,13 +167,33 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
);
if (values.every((v) => v !== undefined)) {
this.append("(");
- this.append(this.bindValue(values, inferClickHouseType(values)));
+ this.append(this.bindValue(values, this.arrayBindType(values)));
this.append(")");
return;
}
super.visitValueList(node);
}
+ private arrayBindType(values: unknown[]): string {
+ if (this.pendingBindType) {
+ return `Array(${this.pendingBindType})`;
+ }
+ return inferClickHouseType(values);
+ }
+
+ protected override visitSelectQuery(node: SelectQueryNode): void {
+ // A surrounding `column <op> (SELECT …)` must not type this select's
+ // own LIMIT / OFFSET / literals with the outer column's bind type, so we
+ // clear the pending bind type across the (forked) body and restore it after.
+ const previousBindType = this.pendingBindType;
+ this.pendingBindType = undefined;
+ try {
+ this.visitSelectQueryBody(node);
+ } finally {
+ this.pendingBindType = previousBindType;
+ }
+ }
+
/**
* Wholesale copy of Kysely's `DefaultQueryCompiler.visitSelectQuery` (kept in
* sync with the pinned 0.28.17) with two ClickHouse-only insertions:
@@ -159,7 +204,7 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
* than call `super` because the parent emits clauses in a fixed order and
* offers no hook to inject a clause between JOIN and WHERE.
*/
- protected override visitSelectQuery(node: SelectQueryNode): void {
+ private visitSelectQueryBody(node: SelectQueryNode): void {
const wrapInParens =
this.parentNode !== undefined &&
!ParensNode.is(this.parentNode) &&
@@ -322,3 +367,11 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
return `{${name}:${inferred}}`;
}
}
+
+function bindTypeOfColumnOperand(node: OperationNode): string | undefined {
+ if (ColumnNode.is(node)) return COLUMN_BIND_TYPES[node.column.name];
+ if (ReferenceNode.is(node) && ColumnNode.is(node.column)) {
+ return COLUMN_BIND_TYPES[node.column.column.name];
+ }
+ return undefined;
+}
diff --git a/packages/shared/src/server/query-ast/kysely/schema.test.ts b/packages/shared/src/server/query-ast/kysely/schema.test.ts
index 4d502ed386..abe4fd564c 100644
--- a/packages/shared/src/server/query-ast/kysely/schema.test.ts
+++ b/packages/shared/src/server/query-ast/kysely/schema.test.ts
@@ -1,6 +1,11 @@
import { describe, expect, it } from "vitest";
-import { COLUMN_DATA_TYPES, DEDUP_SPECS, TENANTED_TABLES } from "./schema";
+import {
+ COLUMN_BIND_TYPES,
+ COLUMN_DATA_TYPES,
+ DEDUP_SPECS,
+ TENANTED_TABLES,
+} from "./schema";
import { schemaTypeAssertions, extensionTypeAssertions } from "./types.assert";
describe("table registry derivation", () => {
@@ -34,6 +39,13 @@ describe("table registry derivation", () => {
expect(DEDUP_SPECS.scores).toBeUndefined();
});
+ it("derives ClickHouse bind types from the same column declarations", () => {
+ expect(COLUMN_BIND_TYPES.total_cost).toBe("Float64");
+ expect(COLUMN_BIND_TYPES.start_time).toBe("DateTime64(3)");
+ expect(COLUMN_BIND_TYPES.project_id).toBe("String");
+ expect(COLUMN_BIND_TYPES.metadata_names).toBe("Array(String)");
+ });
+
// The assertions themselves are compile-time (`tsc` is the test); this only
// anchors the file so its `@ts-expect-error` checks stay in the build graph.
it("keeps the compile-time type assertions in the build graph", () => {
diff --git a/packages/shared/src/server/query-ast/kysely/schema.ts b/packages/shared/src/server/query-ast/kysely/schema.ts
index a4548799a5..0ebd3fcaaa 100644
--- a/packages/shared/src/server/query-ast/kysely/schema.ts
+++ b/packages/shared/src/server/query-ast/kysely/schema.ts
@@ -6,6 +6,8 @@
* comparisons).
* - `COLUMN_DATA_TYPES` — the coarse runtime type map the aggregate type-check
* pass consults (`typecheck.ts`).
+ * - `COLUMN_BIND_TYPES` — the ClickHouse bind types the compiler uses when a
+ * value is compared to a known column (`compiler.ts`).
* - `TENANTED_TABLES` — the relations the tenancy pass must scope
* (`tenancy.ts`).
* - `DEDUP_SPECS` — the per-table read idiom the lowering pass applies
@@ -176,3 +178,26 @@ export const DEDUP_SPECS: Record<string, DedupSpec> = Object.fromEntries(
spec.dedup ? [[name, spec.dedup]] : [],
),
);
+
+/**
+ * ClickHouse bind type for a JS value compared against this column. Wider than
+ * the coarse {@link COLUMN_DATA_TYPES} category: `DateTime` columns bind as
+ * `DateTime64(3)`, `Float` as `Float64`, so an integer literal compared to
+ * `total_cost` still becomes `{p:Float64}`.
+ */
+const BIND_TYPE: Record<ChColumnType, string> = {
+ String: "String",
+ Float: "Float64",
+ DateTime: "DateTime64(3)",
+ "Array(String)": "Array(String)",
+ "Map(String, Float)": "Map(String, Float64)",
+};
+
+export const COLUMN_BIND_TYPES: Record<string, string> = Object.fromEntries(
+ Object.values(TABLE_REGISTRY).flatMap((table) =>
+ Object.entries(table.columns).map(([name, chType]) => [
+ name,
+ BIND_TYPE[chType],
+ ]),
+ ),
+);