πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(query-ast): clean Kysely base for the ClickHouse query AST (#16687)
Niklas Semmler8521ece6c8d5
diff --git a/.codespellrc b/.codespellrcindex 9d7fda74e2..6fcafd986c 100644--- a/.codespellrc+++ b/.codespellrc@@ -1,4 +1,4 @@[codespell]skip = .git,*.pdf,*.svg,package-lock.json,*.prisma,pnpm-lock.yaml,./worker/src/__tests__/chatml/framework-traces-ignore-words-list = afterall,vertx,notIn,alue,allTime,rouge+ignore-words-list = afterall,vertx,notIn,alue,allTime,rouge,fromsdiff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.mdindex 13ed785621..d18d5e3ad5 100644--- a/packages/shared/AGENTS.md+++ b/packages/shared/AGENTS.md@@ -28,7 +28,10 @@recording/diff harness that captures the current SQL at the`src/server/repositories/clickhouse.ts` exec seam and normalizes it via`clickhouse format` for snapshot comparison. Every migrated call site is- proven against its baseline here.+ proven against its baseline here. The Kysely ClickHouse dialect (ARRAY JOIN /+ LIMIT BY / metadata indexOf nodes, `ExecutionContext` tenancy injection,+ typed selection, virtual views, catalog parity) lives under+ `src/server/query-ast/kysely/`.- Postgres schema: `prisma/schema.prisma`- Prisma migrations: `prisma/migrations/*`- ClickHouse migrations: `clickhouse/migrations/{clustered,unclustered}/*`diff --git a/packages/shared/package.json b/packages/shared/package.jsonindex d28ce20a24..4bccbbe9f8 100644--- a/packages/shared/package.json+++ b/packages/shared/package.json@@ -193,6 +193,7 @@"ioredis": "^5.10.1","ipaddr.js": "^2.2.0","jsonpath-plus": "10.3.0",+ "kysely": "0.28.17","langfuse": "3.38.4","lodash": "^4.18.1","lossless-json": "^4.1.1",diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex 9860d76e2d..e1e57352d0 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -9,16 +9,21 @@ sugar, semantic field metadata, `FilterState` embedding) will land under## What's here today-Only the golden-SQL harness β the thing every later migration is tested-against:-- `goldenHarness.ts` β captures SQL at the `repositories/clickhouse.ts` execseam in test mode, then normalizes it (`clickhouse format` + positional paramnames) for snapshot comparison. No ClickHouse server needed.-- `environments.golden.test.ts` β baselines the current SQL shapes of- `environments.getEnvironmentsForProject` (both write-mode branches Γ the- timestamp bound). This is the tier-0 exit target for the spike: once- `environments.ts` compiles from the AST, it must reproduce these snapshots.+- `environments.golden.test.ts` β baselines `environments.getEnvironmentsForProject`+ (both write-mode branches Γ the timestamp bound).+- `kysely/` β compile-only ClickHouse dialect on Kysely 0.28. Real+ `OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf`+ subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;+ schema-typed selection (TS + runtime validation); virtual views as WITH+ CTEs; catalog parity. Library-specific code lives only in this folder.+- `kysely/schema.ts` β the physical table registry: one `defineTable`+ declaration per relation drives all three downstream views β the Kysely row+ types (`ClickHouseDatabase`), the runtime column-type map the type-check pass+ consults (`COLUMN_DATA_TYPES`), and the set the tenancy pass scopes+ (`TENANTED_TABLES`) β instead of three hand-maintained tables that drift.Regenerate baselines with `-u` after an intentional SQL change:@@ -41,3 +46,51 @@ against the new binary in the same PR, or the golden tests drift.The CI step is intentionally **non-blocking**: a drift surfaces as a warningannotation but never fails the pipeline (`|| echo "::warning::"`). Promote it toa required check once it has proven stable.++## Using it++- **Compile only through `compileClickhouseQuery(query, ctx)`.** It is the one+ supported path from a builder to `{ sql, params }`; the repository layer hands+ that to `queryClickhouse`. `ctx` (an `ExecutionContext` carrying `projectId`)+ is required β omitting it is a compile-time type error and an empty one throws.+- **Never filter `project_id` yourself.** The compile step injects+ `project_id = {projectId}` into every tenanted relation, so call sites pass+ only `{ projectId }` (see `repositories/environments.ts`).+- **ClickHouse-only clauses use `$call(helper())`** β not fluent builder+ methods, so they compose inside CTEs, subqueries, and views. See the recipes+ below.++### Recipes++**Write an ARRAY JOIN** (`arrayJoin` + `mapKeys`/`mapValues` from `./kysely/extensions`):++```ts+db.selectFrom("observations")+ .select("environment")+ .$call(arrayJoin({ cost_key: mapKeys("cost_details"), cost: mapValues("cost_details") }));+// β¦ array join mapKeys(cost_details) as cost_key, mapValues(cost_details) as cost+```++**Write a LIMIT BY** (`limitBy`):++```ts+db.selectFrom("events_core")+ .select(["span_id", "project_id"])+ .orderBy("event_ts", "desc")+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));+// β¦ order by event_ts desc limit 1 by span_id, project_id+```++**Select a metadata value** (`metadataValue` β lowers `metadata[key]` to a bound `indexOf` subscript):++```ts+db.selectFrom("events_core as e")+ .select((eb) => [metadataValue("e", "my_key").as("my_val")])+ .where((eb) => eb(metadataValue("e", "my_key"), ">", 2));+// select metadata_values[indexof(e.metadata_names, {p:String})] as my_val β¦+```++For how ARRAY JOIN / LIMIT BY / metadata nodes are implemented without forking+Kysely, how tenancy injection works internally, the escape hatches, and the+Kysely-internals upgrade hazard, see the developer guide in+[`kysely/README.md`](kysely/README.md).diff --git a/packages/shared/src/server/query-ast/executionContext.ts b/packages/shared/src/server/query-ast/executionContext.tsnew file mode 100644index 0000000000..7f00f76053--- /dev/null+++ b/packages/shared/src/server/query-ast/executionContext.ts@@ -0,0 +1,8 @@+/**+ * Compile-time tenancy scope. Every ClickHouse query compiled through+ * {@link compileClickhouseQuery} must carry one of these; the tenancy+ * injection pass keys off `projectId`.+ */+export type ExecutionContext = {+ projectId: string;+};diff --git a/packages/shared/src/server/query-ast/goldenHarness.ts b/packages/shared/src/server/query-ast/goldenHarness.tsindex 62af4d9e08..2412481fa8 100644--- a/packages/shared/src/server/query-ast/goldenHarness.ts+++ b/packages/shared/src/server/query-ast/goldenHarness.ts@@ -1,4 +1,6 @@import { spawnSync } from "child_process";+import { homedir } from "os";+import { join } from "path";import { Readable } from "stream";import type * as ClickhouseModule from "../repositories/clickhouse";@@ -134,8 +136,31 @@ export type GoldenQuery = {};// The `clickhouse` multi-binary's `format` subcommand. Not `clickhouse-local`,-// which forces local mode and won't accept `format` as a subcommand.-const clickhouseBin = "clickhouse";+// which forces local mode and won't accept `format` as a subcommand. Prefer+// CLICKHOUSE_BIN, then PATH, then a user-local install β the PATH binary in+// some agent VMs is a docker wrapper that cannot run `format`.+function resolveClickhouseBin(): string {+ const candidates = [+ process.env.CLICKHOUSE_BIN,+ "clickhouse",+ join(homedir(), ".local/bin/clickhouse"),+ ].filter((bin): bin is string => Boolean(bin));++ for (const bin of candidates) {+ try {+ const res = spawnSync(bin, ["format"], {+ input: "SELECT 1",+ encoding: "utf8",+ });+ if (res.status === 0) return bin;+ } catch {+ // try the next candidate+ }+ }+ return "clickhouse";+}++const clickhouseBin = resolveClickhouseBin();let formatAvailability: boolean | undefined;@@ -164,7 +189,7 @@ export function clickhouseFormatAvailable(): boolean {* which `clickhouse-local` provides β no server, no state. Throws on a parse* error so malformed SQL fails the test instead of comparing raw.*/-function formatSql(sql: string): string {+export function formatSql(sql: string): string {const res = spawnSync(clickhouseBin, ["format"], {input: sql,encoding: "utf8",@@ -179,6 +204,78 @@ function formatSql(sql: string): string {return res.stdout.trimEnd();}+let localAvailability: boolean | undefined;++/** Whether `clickhouse local` can execute a query on this machine. */+export function clickhouseLocalAvailable(): boolean {+ if (localAvailability !== undefined) return localAvailability;+ try {+ const res = spawnSync(clickhouseBin, ["local", "--query", "SELECT 1"], {+ encoding: "utf8",+ });+ localAvailability = res.status === 0;+ } catch {+ localAvailability = false;+ }+ return localAvailability;+}++const NAMED_PARAM = /\{([A-Za-z_][A-Za-z0-9_]*)\s*:\s*([^}]+)\}/g;++function sqlLiteral(value: unknown, type: string): string {+ const t = type.trim();+ if (t.startsWith("Array")) {+ if (!Array.isArray(value)) {+ throw new Error(`Expected array bind for ${t}, got ${typeof value}`);+ }+ return `[${value.map((item) => sqlLiteral(item, "String")).join(", ")}]`;+ }+ if (t.startsWith("DateTime")) {+ const iso =+ value instanceof Date+ ? value.toISOString().replace("T", " ").replace("Z", "")+ : String(value);+ return `'${iso}'`;+ }+ if (typeof value === "number" || typeof value === "bigint") {+ return String(value);+ }+ if (typeof value === "boolean") return value ? "1" : "0";+ const escaped = String(value).replace(/\\/g, "\\\\").replace(/'/g, "\\'");+ return `'${escaped}'`;+}++/** Replace `{name:Type}` placeholders with ClickHouse literals for local exec. */+export function substituteNamedParams(+ sql: string,+ params: Record<string, unknown>,+): string {+ return sql.replace(NAMED_PARAM, (_match, name: string, type: string) => {+ if (!(name in params)) {+ throw new Error(`Missing bind '${name}' for local execution`);+ }+ return sqlLiteral(params[name], type);+ });+}++/**+ * Run SQL through `clickhouse local` (no server). Throws on a non-zero status+ * so an analyzer/parse error fails the test instead of comparing stdout.+ */+export function executeClickhouseLocal(sql: string): string {+ const res = spawnSync(clickhouseBin, ["local", "--query", sql], {+ encoding: "utf8",+ maxBuffer: 16 * 1024 * 1024,+ });+ if (res.error) throw res.error;+ if (res.status !== 0) {+ throw new Error(+ `clickhouse local failed (status ${res.status}): ${res.stderr?.trim()}\n--- SQL ---\n${sql}`,+ );+ }+ return (res.stdout ?? "").trimEnd();+}+const PARAM_PLACEHOLDER = /\{\s*([A-Za-z_][A-Za-z0-9_]*)\s*:/g;/**diff --git a/packages/shared/src/server/query-ast/kysely/README.md b/packages/shared/src/server/query-ast/kysely/README.mdnew file mode 100644index 0000000000..148a395afd--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/README.md@@ -0,0 +1,143 @@+# kysely/ β compile-only ClickHouse dialect: patterns & gotchas++This folder adapts Kysely to emit ClickHouse SQL. Several patterns here are+deliberately unusual β this is the guide for anyone **developing** the query+builder. For how to **use** it from a call site, see [`../README.md`](../README.md).++## How ClickHouse clauses land (no fork)++Kysely's `OperationNodeKind` union is closed, so a first-class node kind would+need a fork. Instead each ClickHouse-only construct rides as an extra field or a+special-cased node, and the compiler/transformer are overridden to emit and+preserve it:++| Clause | How it lands | Fork? |+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |+| ARRAY JOIN | Plugin attaches `ArrayJoinNode` as an extra field on `SelectQueryNode`. `ClickHouseOperationNodeTransformer` preserves it. `ClickHouseQueryCompiler.visitSelectQuery` emits it after JOINs / before WHERE. | no |+| LIMIT BY | Plugin attaches `LimitByNode` the same way. Compiler emits it after ORDER BY / before LIMIT. | no |+| metadata `indexOf` | Helper builds an `ArrayIndexNode` whose index child is a `FunctionNode` (`indexOf`) over a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no |+| Virtual view | Plugin rewrites `selectFrom(viewName)` into a WITH CTE. Outer types only expose the view's selected columns. | no |++These are real node objects whose children are traced Kysely+`FunctionNode`/`ColumnNode`/`ValueNode`/`IdentifierNode` values β not `RawNode`+string splices. Any plugin using the default `OperationNodeTransformer` would+drop them, so ours overrides `transformSelectQuery` to keep them. Upstream is+not patched.++## Kysely never runs β it only compiles++`getClickhouseKysely()` returns a `Kysely` instance wired to a `DummyDriver`+with SQLite adapter/introspector stand-ins (`dialect.ts`). Nothing ever+connects or executes. The only supported output path is+`compileClickhouseQuery(query, ctx)` β `{ sql, params }`, which the repository+layer hands to `queryClickhouse`. Do **not** call `.execute()` / `.compile()`+directly: the compiler refuses any tree that did not go through the tenancy+pass (see `../README.md`).++## Tenancy injection β how the choke point works++`compileClickhouseQuery(query, ctx)` is the only supported compile path, and it+is where tenancy is enforced:++1. A missing/empty `ExecutionContext` throws (`QueryCompileError`) β `ctx` is a+ required parameter, so omitting it is also a compile-time type error.+2. `TenancyInjectionPlugin` walks every FROM/JOIN and injects+ `project_id = {projectId}` on each tenanted physical table, unless the tree+ already carries a predicate that _proves_ that scope: the equality's value+ must equal the context project, and β when more than one tenanted relation is+ in scope β the column must be table-qualified. A qualified predicate covers a+ relation only when the qualifier matches its alias (if aliased) or its table+ name (if not) β so `scores AS traces` joined to `traces AS t` still scopes+ both. It then identity-stamps the tree (`WeakSet`); a copied+ `langfuseTenancy` property is not a valid stamp.+3. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is+ present, so `qb.compile()` without the plugin also fails.+4. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a+ `SELECT`/`FROM`/`JOIN` in SELECT/WHERE throw `UnscopedRelationError`. Kysely's+ own keyword fragments (`asc`/`desc`) are not relations.++So query bodies here never filter `project_id` by hand β it is redundant, and+forgetting it is impossible. Call sites like `repositories/environments.ts` pass+only `{ projectId }`.++## ClickHouse-only clauses use `$call(helper())`, not builder methods++ARRAY JOIN and LIMIT BY are applied through curried helpers, invoked with+Kysely's public `$call`:++```ts+db.selectFrom("observations")+ .select("environment")+ .$call(arrayJoin({ cost_key: mapKeys("cost_details"), cost: mapValues("cost_details") }))++db.selectFrom("events_core")+ .select(["span_id", "project_id"])+ .orderBy("event_ts", "desc")+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }))+```++**Why `$call(...)` and not a fluent `.arrayJoin(...)` method:** a real method+would have to exist on _every_ builder instance, including the native Kysely+builders that `.with((qb) => β¦)`, subquery, and `defineView` callbacks hand you.+Getting that requires either forking Kysely's whole builder graph or globally+mutating its prototype via an internal `kysely/dist/...` import (blocked here by+Kysely's `exports` map under NodeNext). A curried helper is a plain function of+a builder, so it works in any of those positions β which is exactly why ARRAY+JOIN / LIMIT BY compose inside CTEs, subqueries, and views. `composition.test.ts`+locks that property in.++**`arrayJoin` widens the row type.** Each `{ alias: arrayExpr }` entry is added+to the builder's output row, so an outer query over a CTE body can reference the+produced column and a typo on the alias is a compile error. The element _value_+type is `unknown`: Kysely's `Expression<T>` hides its type argument from+inference. Precise value types would need a branded array-expression wrapper on+`mapKeys` / `mapValues` / etc. β not done yet. `types.assert.ts` pins the+widening behavior.++**`arrayJoin` (clause) β `arrayJoin()` (function).** ClickHouse has both. The+helper here builds the ARRAY JOIN _clause_. The row-expanding SELECT _function_+is just `eb.fn("arrayJoin", [...])`.++## Escape hatches and their cost++- `sql.ref("alias")` β the only way to reference a SELECT alias that is not a+ schema column (ClickHouse allows `GROUP BY`/expression reuse of aliases;+ Kysely's types do not model it). It is fully untyped β a typo reaches+ ClickHouse unchecked. Use sparingly.+- `eb.fn("ch_function", [...])` β arbitrary ClickHouse functions. The name is an+ unchecked string and the return type defaults to `unknown`; no arity or+ return checking.+- New columns β add them to the table registry in `schema.ts` (one `defineTable`+ entry per relation) as queries need them. That single declaration drives the+ row types, the runtime column-type map, and the tenanted-table set.++## The one Kysely-internals coupling (upgrade hazard)++`compiler.ts` wraps Kysely's **private** `visitNode` / `nodeStack` to dispatch+`ArrayIndexNode` (for `metadata[key]`), because it is not one of Kysely's closed+`OperationNode` kinds. Kysely is pinned to **0.28.17** for this reason. Re-verify+this hack on any Kysely bump; it is the single place that reaches past Kysely's+documented surface. Everything else (plugins, dialect, transformer overrides)+uses public or documented-protected API.++## Types are asserted at compile time++`types.assert.ts` holds `tsc`-only assertions (schema typing, view opacity,+arrayJoin widening, limitBy preservation). It is never run; `schema.test.ts`+anchors it so it stays in the build graph. `@ts-expect-error` lines there must+stay live.++## Verifying changes++```+CLICKHOUSE_BIN=clickhouse pnpm --filter @langfuse/shared run test src/server/query-ast+```++`catalog.golden.test.ts` and the other `*.golden.test.ts` suites assert+`compile(AST) β‘ referenceSQL` after `clickhouse format`, so they need a local+`clickhouse` binary and otherwise `describe.skip`. CI's non-blocking+SQL-equivalence step installs the pinned binary and runs every `*.golden.test.ts`+(it selects them by that name), so a suite must carry the `.golden.test.ts`+suffix to run there. `composition.test.ts` asserts on raw compiler output and+runs everywhere. After an intentional SQL change, regenerate golden baselines+with `-u` (see `../README.md`).diff --git a/packages/shared/src/server/query-ast/kysely/catalog.golden.test.ts b/packages/shared/src/server/query-ast/kysely/catalog.golden.test.tsnew file mode 100644index 0000000000..cf83074d08--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/catalog.golden.test.ts@@ -0,0 +1,100 @@+import { describe, expect, it } from "vitest";++import {+ clickhouseFormatAvailable,+ clickhouseLocalAvailable,+ executeClickhouseLocal,+ formatSql,+ normalizeParams,+ substituteNamedParams,+} from "../goldenHarness";+import { CATALOG, CATALOG_PROJECT_ID } from "./catalog";+import { compileClickhouseQuery } from "./compile";++const describeWithClickhouse = clickhouseFormatAvailable()+ ? describe+ : describe.skip;++const describeWithClickhouseLocal = clickhouseLocalAvailable()+ ? describe+ : describe.skip;++if (!clickhouseFormatAvailable()) {+ console.warn(+ "[catalog] `clickhouse format` unavailable β skipping catalog parity tests.",+ );+}++if (!clickhouseLocalAvailable()) {+ console.warn(+ "[catalog] `clickhouse local` unavailable β skipping JOIN execution tests.",+ );+}++describeWithClickhouse("catalog parity", () => {+ for (const entry of CATALOG) {+ it(`${entry.id} (tier ${entry.tier}) compile(AST) β‘ referenceSQL`, () => {+ const compiled = compileClickhouseQuery(entry.build(), {+ projectId: CATALOG_PROJECT_ID,+ });++ const left = normalizeParams(formatSql(compiled.sql), compiled.params);+ const right = normalizeParams(+ formatSql(entry.referenceSql),+ compiled.params,+ );++ expect(left.sql).toBe(right.sql);+ });+ }+});++const JOIN_MEMORY_DDL = `+ CREATE TABLE observations (+ environment String,+ project_id String,+ start_time DateTime64(3),+ trace_id String,+ cost_details Map(String, Float64),+ usage_details Map(String, Float64)+ ) ENGINE = Memory;+ CREATE TABLE traces (+ environment String,+ project_id String,+ timestamp DateTime64(3),+ id String+ ) ENGINE = Memory;+`;++describeWithClickhouseLocal("catalog JOIN execution", () => {+ it("join_observations_traces is valid ClickHouse with table-qualified project_id on both sides", () => {+ const entry = CATALOG.find((e) => e.id === "join_observations_traces");+ expect(entry).toBeDefined();+ if (!entry) return;++ const compiled = compileClickhouseQuery(entry.build(), {+ projectId: CATALOG_PROJECT_ID,+ });+ const refs = compiled.sql.match(/[\w.]*project_id/gi) ?? [];+ expect(refs.length).toBeGreaterThanOrEqual(3);+ for (const ref of refs) {+ expect(ref).toMatch(/^(o|t)\.project_id$/i);+ }++ const executable = substituteNamedParams(compiled.sql, compiled.params);+ executeClickhouseLocal(`${JOIN_MEMORY_DDL}\n${executable}`);++ const explained = executeClickhouseLocal(+ `${JOIN_MEMORY_DDL}\nEXPLAIN QUERY TREE ${executable}`,+ );+ expect(explained).toMatch(/table_name: default\.observations/);+ expect(explained).toMatch(/table_name: default\.traces/);+ // Injected t.project_id and WHERE o.project_id resolve to different tables.+ expect(explained).toMatch(+ /column_name: project_id, result_type: String, source_id: 3/,+ );+ expect(explained).toMatch(+ /column_name: project_id, result_type: String, source_id: 5/,+ );+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/catalog.ts b/packages/shared/src/server/query-ast/kysely/catalog.tsnew file mode 100644index 0000000000..f9c9ad5ebd--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/catalog.ts@@ -0,0 +1,217 @@+import type { ClickhouseCompilable } from "./compile";+import { getClickhouseKysely } from "./dialect";+import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";++type CatalogTier = 0 | 1 | 2 | 3 | 4;++export type CatalogEntry = {+ id: string;+ tier: CatalogTier;+ /** Hand-written reference SQL the compiler must match after `clickhouse format`. */+ referenceSql: string;+ build: () => ClickhouseCompilable;+};++const PROJECT_ID = "golden-project";+const FROM_TS = new Date("2026-01-01T00:00:00.000Z");++function db() {+ return getClickhouseKysely();+}++/**+ * Catalog of reference query shapes, one entry per complexity tier.+ * Reference SQL is the intended ClickHouse shape, not a dump of Kysely's+ * first draft β `compile(AST) β‘ referenceSQL` after `clickhouse format`.+ *+ * No entry filters `project_id`: the reference SQL shows the `project_id`+ * predicate the mandatory tenancy pass injects, so these also prove injection.+ */+export const CATALOG: CatalogEntry[] = [+ {+ id: "environments_events_core",+ tier: 0,+ referenceSql: `+ SELECT DISTINCT environment+ FROM events_core+ WHERE project_id = {p1:String}+ `,+ build: () =>+ db().selectFrom("events_core").select("environment").distinct(),+ },+ {+ id: "environments_legacy_union",+ tier: 0,+ referenceSql: `+ SELECT DISTINCT environment+ FROM traces+ WHERE project_id = {p1:String}+ UNION ALL+ SELECT DISTINCT environment+ FROM observations+ WHERE project_id = {p1:String}+ `,+ build: () =>+ db()+ .selectFrom("traces")+ .select("environment")+ .distinct()+ .unionAll(+ db().selectFrom("observations").select("environment").distinct(),+ ),+ },+ {+ id: "environments_scores_in_array",+ tier: 0,+ referenceSql: `+ SELECT DISTINCT environment+ FROM scores+ WHERE (project_id = {p1:String}) AND (data_type IN ({p2:Array(String)}))+ `,+ build: () =>+ db()+ .selectFrom("scores")+ .select("environment")+ .distinct()+ .where("data_type", "in", [+ "NUMERIC",+ "BOOLEAN",+ "CATEGORICAL",+ "TEXT",+ ]),+ },+ {+ id: "events_core_time_bound",+ tier: 1,+ referenceSql: `+ SELECT DISTINCT environment+ FROM events_core+ WHERE (project_id = {p1:String}) AND (start_time >= {p2:DateTime64(3)})+ `,+ build: () =>+ db()+ .selectFrom("events_core")+ .select("environment")+ .distinct()+ .where("start_time", ">=", FROM_TS),+ },+ {+ id: "group_by_count",+ tier: 1,+ referenceSql: `+ SELECT environment, count(*) AS n+ FROM events_core+ WHERE project_id = {p1:String}+ GROUP BY environment+ `,+ build: () =>+ db()+ .selectFrom("events_core")+ .select((eb) => ["environment", eb.fn.countAll().as("n")])+ .groupBy("environment"),+ },+ {+ id: "join_observations_traces",+ tier: 2,+ referenceSql: `+ SELECT o.environment+ FROM observations AS o+ INNER JOIN traces AS t ON (o.trace_id = t.id) AND (o.project_id = t.project_id) AND (t.project_id = {p1:String})+ WHERE o.project_id = {p1:String}+ `,+ build: () =>+ db()+ .selectFrom("observations as o")+ .innerJoin("traces as t", (join) =>+ join+ .onRef("o.trace_id", "=", "t.id")+ .onRef("o.project_id", "=", "t.project_id"),+ )+ .select("o.environment"),+ },+ {+ id: "cte_trace_ids",+ tier: 2,+ referenceSql: `+ WITH traces_cte AS (+ SELECT id+ FROM traces+ WHERE project_id = {p1:String}+ )+ SELECT o.environment+ FROM observations AS o+ WHERE (o.project_id = {p1:String}) AND (o.trace_id IN (+ SELECT id+ FROM traces_cte+ ))+ `,+ build: () =>+ db()+ .with("traces_cte", (qb) => qb.selectFrom("traces").select("id"))+ .selectFrom("observations as o")+ .select("o.environment")+ .where("o.trace_id", "in", (eb) =>+ eb.selectFrom("traces_cte").select("id"),+ ),+ },+ {+ id: "array_join_cost_details",+ tier: 3,+ referenceSql: `+ SELECT environment+ FROM observations+ ARRAY JOIN mapKeys(cost_details) AS cost_key, mapValues(cost_details) AS cost+ WHERE project_id = {p1:String}+ `,+ build: () =>+ db()+ .selectFrom("observations")+ .select("environment")+ .$call(+ arrayJoin({+ cost_key: mapKeys("cost_details"),+ cost: mapValues("cost_details"),+ }),+ ),+ },+ {+ id: "limit_by_dedup",+ tier: 3,+ referenceSql: `+ SELECT span_id, project_id+ FROM events_core+ WHERE project_id = {p1:String}+ ORDER BY event_ts DESC+ LIMIT 1 BY span_id, project_id+ `,+ build: () =>+ db()+ .selectFrom("events_core")+ .select(["span_id", "project_id"])+ .orderBy("event_ts", "desc")+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] })),+ },+ {+ id: "window_rank",+ tier: 4,+ referenceSql: `+ SELECT span_id, rank() OVER (PARTITION BY trace_id ORDER BY start_time DESC) AS rk+ FROM events_core+ WHERE project_id = {p1:String}+ `,+ build: () =>+ db()+ .selectFrom("events_core")+ .select((eb) => [+ "span_id",+ eb.fn+ .agg("rank")+ .over((ob) =>+ ob.partitionBy("trace_id").orderBy("start_time", "desc"),+ )+ .as("rk"),+ ]),+ },+];++export const CATALOG_PROJECT_ID = PROJECT_ID;diff --git a/packages/shared/src/server/query-ast/kysely/compile.ts b/packages/shared/src/server/query-ast/kysely/compile.tsnew file mode 100644index 0000000000..88f5840fa4--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/compile.ts@@ -0,0 +1,51 @@+import {+ createQueryId,+ type KyselyPlugin,+ type RootOperationNode,+} from "kysely";++import type { ExecutionContext } from "../executionContext";+import { ClickHouseQueryCompiler } from "./compiler";+import { TenancyInjectionPlugin, requireExecutionContext } from "./tenancy";+import { validateTypeCompatibility } from "./typecheck";++export type CompiledClickhouseQuery = {+ sql: string;+ params: Record<string, unknown>;+};++/**+ * Anything Kysely will give us a SelectQueryNode for, including plugin-wrapped+ * builders. Kept structural so UNION / `$if` chains stay assignable.+ */+export type ClickhouseCompilable = {+ withPlugin(plugin: KyselyPlugin): ClickhouseCompilable;+ toOperationNode(): RootOperationNode;+};++/**+ * The choke point. Every query that should produce ClickHouse SQL goes+ * through here: the `ctx` parameter is required, so a caller cannot compile+ * without a tenancy scope β omitting it is a compile-time type error. Tenancy+ * is injected from that scope, and the dialect compiler refuses to emit SQL+ * unless the tree was identity-stamped by that pass (a copied property is not+ * enough). The runtime `requireExecutionContext` guard remains as+ * defense-in-depth for the one case the type cannot express (an empty+ * `projectId`) and for callers reaching this through an `any` boundary.+ */+export function compileClickhouseQuery(+ query: ClickhouseCompilable,+ ctx: ExecutionContext,+): CompiledClickhouseQuery {+ const scope = requireExecutionContext(ctx);+ const stamped = query+ .withPlugin(new TenancyInjectionPlugin(scope))+ .toOperationNode();+ const compiler = new ClickHouseQueryCompiler();+ validateTypeCompatibility(stamped);+ const compiled = compiler.compileQuery(stamped, createQueryId());+ return {+ sql: compiled.sql,+ params: { ...compiler.namedParameters },+ };+}diff --git a/packages/shared/src/server/query-ast/kysely/compiler.ts b/packages/shared/src/server/query-ast/kysely/compiler.tsnew file mode 100644index 0000000000..02fc6dbb79--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/compiler.ts@@ -0,0 +1,324 @@+import {+ CreateTableNode,+ CreateViewNode,+ DefaultQueryCompiler,+ InsertQueryNode,+ ParensNode,+ SetOperationNode,+ ValueNode,+ type OperationNode,+ type PrimitiveValueListNode,+ type RootOperationNode,+ type SelectQueryNode,+ type ValueListNode,+} from "kysely";+import type { QueryId } from "kysely";++import {+ ArrayJoinNode,+ ArrayIndexNode,+ type ClickHouseSelectQueryNode,+ type LimitByNode,+} from "./nodes";+import { assertTenancyStamped } from "./tenancy";++const ARRAY_JOIN_SQL: Record<ArrayJoinNode["variant"], string> = {+ default: "array join",+ left: "left array join",+ inner: "inner array join",+};++function canonicalParamValue(value: unknown): string {+ if (value instanceof Date) return `d:${value.toISOString()}`;+ if (Array.isArray(value)) {+ return `a:${JSON.stringify(value)}`;+ }+ return `${typeof value}:${String(value)}`;+}++function inferClickHouseType(value: unknown): string {+ if (value instanceof Date) return "DateTime64(3)";+ if (Array.isArray(value)) {+ const inner = value.length === 0 ? "String" : inferClickHouseType(value[0]);+ return `Array(${inner})`;+ }+ if (typeof value === "string") return "String";+ if (typeof value === "boolean") return "UInt8";+ if (typeof value === "number") {+ return Number.isInteger(value) ? "Int64" : "Float64";+ }+ if (typeof value === "bigint") return "Int64";+ return "String";+}++/**+ * ClickHouse SQL compiler. Identifier quoting is omitted so `clickhouse format`+ * matches the golden snapshots (unquoted identifiers). Parameters are named+ * `{pN:Type}` binds, interned by (type, value) so repeated values collapse to a+ * single placeholder, matching the named `{projectId:String}` style once the+ * golden harness's param normalizer runs.+ *+ * Example β a UNION whose two branches both filter `project_id = 'p'`:+ *+ * ... WHERE project_id = {p1:String} ... UNION ALL ... WHERE project_id = {p1:String}+ *+ * One `{p1:String}` bind is emitted and reused across both branches, not+ * separate `{p1:String}` / `{p2:String}` bound to the same value.+ *+ * ARRAY JOIN and LIMIT BY ride as extra fields on the select node as real+ * {@link ArrayJoinNode} / {@link LimitByNode} operation nodes.+ */+export class ClickHouseQueryCompiler extends DefaultQueryCompiler {+ namedParameters: Record<string, unknown> = {};+ private intern = new Map<string, string>();++ constructor() {+ super();+ // ArrayIndexNode is a custom node kind Kysely's dispatch cannot reach and+ // Kysely has no public hook to register one, so we patch its private+ // visitNode/nodeStack to route the node to visitArrayIndex. Kysely is+ // pinned to 0.28.17 for this; see kysely/README.md ("upgrade hazard").+ const parentVisit = this.visitNode.bind(this);+ // Anti-pattern: cast through `unknown` once to reach Kysely's private+ // `visitNode` / `nodeStack` (not part of the public type). Deliberate, to+ // avoid a fork.+ const self = this as unknown as {+ visitNode: (node: OperationNode) => void;+ nodeStack: OperationNode[];+ };+ self.visitNode = (node: OperationNode) => {+ if (ArrayIndexNode.is(node)) {+ self.nodeStack.push(node);+ this.visitArrayIndex(node);+ self.nodeStack.pop();+ return;+ }+ parentVisit(node);+ };+ }++ compileQuery(node: RootOperationNode, queryId: QueryId) {+ this.namedParameters = {};+ this.intern = new Map();+ assertTenancyStamped(node);+ return super.compileQuery(node, queryId);+ }++ // Emit identifiers unquoted. `clickhouse format` normalizes to unquoted+ // identifiers, so returning empty wrappers is what keeps raw compiler output+ // byte-comparable with the golden snapshots. (Column/table names in this+ // codebase are plain identifiers, so dropping the quotes is safe here.)+ protected override getLeftIdentifierWrapper(): string {+ return "";+ }++ protected override getRightIdentifierWrapper(): string {+ return "";+ }++ protected override appendValue(parameter: unknown): void {+ this.append(this.bindValue(parameter));+ }++ /**+ * ClickHouse `IN` takes a single `Array(T)` bind, not a parenthesized list of+ * scalar binds. So `col IN (1, 2, 3)` compiles to `col IN ({p:Array(Int64)})`+ * β one array parameter β which is also the shape the existing production SQL+ * (e.g. the scores list query) already relies on.+ */+ protected override visitPrimitiveValueList(+ node: PrimitiveValueListNode,+ ): void {+ this.append("(");+ this.append(+ this.bindValue([...node.values], inferClickHouseType([...node.values])),+ );+ this.append(")");+ }++ protected override visitValueList(node: ValueListNode): void {+ const values = node.values.map((v) =>+ ValueNode.is(v) ? v.value : undefined,+ );+ if (values.every((v) => v !== undefined)) {+ this.append("(");+ this.append(this.bindValue(values, inferClickHouseType(values)));+ this.append(")");+ return;+ }+ super.visitValueList(node);+ }++ /**+ * Wholesale copy of Kysely's `DefaultQueryCompiler.visitSelectQuery` (kept in+ * sync with the pinned 0.28.17) with two ClickHouse-only insertions:+ * - the ARRAY JOIN block, emitted after JOINs and before WHERE (line ~193)+ * - the LIMIT BY block, emitted before LIMIT (line ~225)+ * Both read extra fields off {@link ClickHouseSelectQueryNode}. Every other+ * clause mirrors the parent verbatim. We override the whole method rather+ * than call `super` because the parent emits clauses in a fixed order and+ * offers no hook to inject a clause between JOIN and WHERE.+ */+ protected override visitSelectQuery(node: SelectQueryNode): void {+ const wrapInParens =+ this.parentNode !== undefined &&+ !ParensNode.is(this.parentNode) &&+ !InsertQueryNode.is(this.parentNode) &&+ !CreateTableNode.is(this.parentNode) &&+ !CreateViewNode.is(this.parentNode) &&+ !SetOperationNode.is(this.parentNode);++ if (this.parentNode === undefined && node.explain) {+ this.visitNode(node.explain);+ this.append(" ");+ }++ if (wrapInParens) {+ this.append("(");+ }++ if (node.with) {+ this.visitNode(node.with);+ this.append(" ");+ }++ this.append("select");++ if (node.distinctOn) {+ this.append(" ");+ this.compileDistinctOn(node.distinctOn);+ }++ if (node.frontModifiers?.length) {+ this.append(" ");+ this.compileList(node.frontModifiers, " ");+ }++ if (node.top) {+ this.append(" ");+ this.visitNode(node.top);+ }++ if (node.selections) {+ this.append(" ");+ this.compileList(node.selections);+ }++ if (node.from) {+ this.append(" ");+ this.visitNode(node.from);+ }++ if (node.joins) {+ this.append(" ");+ this.compileList(node.joins, " ");+ }++ const chNode = node as ClickHouseSelectQueryNode;+ if (chNode.arrayJoins?.length) {+ for (const arrayJoin of chNode.arrayJoins) {+ this.append(" ");+ this.visitArrayJoin(arrayJoin);+ }+ }++ if (node.where) {+ this.append(" ");+ this.visitNode(node.where);+ }++ if (node.groupBy) {+ this.append(" ");+ this.visitNode(node.groupBy);+ }++ if (node.having) {+ this.append(" ");+ this.visitNode(node.having);+ }++ if (node.setOperations) {+ this.append(" ");+ this.compileList(node.setOperations, " ");+ }++ if (node.orderBy) {+ this.append(" ");+ this.visitNode(node.orderBy);+ }++ if (chNode.limitBy) {+ this.append(" ");+ this.visitLimitBy(chNode.limitBy);+ }++ if (node.limit) {+ this.append(" ");+ this.visitNode(node.limit);+ }++ if (node.offset) {+ this.append(" ");+ this.visitNode(node.offset);+ }++ if (node.fetch) {+ this.append(" ");+ this.visitNode(node.fetch);+ }++ if (node.endModifiers?.length) {+ this.append(" ");+ this.compileList(this.sortSelectModifiers([...node.endModifiers]), " ");+ }++ if (wrapInParens) {+ this.append(")");+ }+ }++ private visitArrayIndex(node: ArrayIndexNode): void {+ this.visitNode(node.array);+ this.append("[");+ this.visitNode(node.index);+ this.append("]");+ }++ private visitArrayJoin(node: ArrayJoinNode): void {+ this.append(ARRAY_JOIN_SQL[node.variant]);+ this.append(" ");+ node.items.forEach((item, index) => {+ if (index > 0) this.append(", ");+ this.visitNode(item.expression);+ this.append(" as ");+ this.visitNode(item.alias);+ });+ }++ private visitLimitBy(node: LimitByNode): void {+ this.append("limit ");+ this.visitNode(node.count);+ this.append(" by ");+ this.compileList(node.columns);+ }++ private bindValue(value: unknown, type?: string): string {+ // `type` is the caller-known ClickHouse type; when omitted we infer the+ // widest canonical type from the JS value (Int64 / Float64 / String /+ // DateTime64(3)). Inference only types the bind *placeholder* β ClickHouse+ // still coerces it to the compared column's type β so the widening cases we+ // emit are safe. Pass `type` explicitly where exact width or precision+ // matters (e.g. a Decimal column, where inferred Float64 could round).+ const inferred = type ?? inferClickHouseType(value);+ const key = `${inferred}:${canonicalParamValue(value)}`;+ const existing = this.intern.get(key);+ if (existing) {+ return `{${existing}:${inferred}}`;+ }+ this.addParameter(value);+ const name = `p${this.intern.size + 1}`;+ this.intern.set(key, name);+ this.namedParameters[name] = value;+ return `{${name}:${inferred}}`;+ }+}diff --git a/packages/shared/src/server/query-ast/kysely/composition.test.ts b/packages/shared/src/server/query-ast/kysely/composition.test.tsnew file mode 100644index 0000000000..5435aa0bf5--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/composition.test.ts@@ -0,0 +1,126 @@+import { describe, expect, it } from "vitest";++import { compileClickhouseQuery } from "./compile";+import { getClickhouseKysely } from "./dialect";+import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";+import { defineView, fromView } from "./views";++// Raw compiler output (no `clickhouse format` binary needed), so these run+// unconditionally in CI. The raw SQL is lowercase; match case-insensitively.+const ctx = { projectId: "p" };++const costArrayJoin = () =>+ arrayJoin({+ cost_key: mapKeys("cost_details"),+ cost: mapValues("cost_details"),+ });++const dedup = () => limitBy({ count: 1, columns: ["span_id", "project_id"] });++describe("ARRAY JOIN composition across nesting positions", () => {+ it("emits ARRAY JOIN at the top level (baseline)", () => {+ const db = getClickhouseKysely();+ const qb = db+ .selectFrom("observations")+ .select("environment")+ .$call(costArrayJoin());++ const { sql } = compileClickhouseQuery(qb, ctx);+ expect(sql).toMatch(/array join/i);+ expect(sql).toMatch(/mapkeys/i);+ expect(sql).toMatch(/mapvalues/i);+ });++ it("keeps ARRAY JOIN when applied inside a CTE body", () => {+ const db = getClickhouseKysely();+ const qb = db+ .with("aj", (qb) =>+ qb+ .selectFrom("observations")+ .select("environment")+ .$call(costArrayJoin()),+ )+ .selectFrom("aj")+ .select("environment");++ const { sql } = compileClickhouseQuery(qb, ctx);+ expect(sql).toMatch(/with aj as/i);+ // The ARRAY JOIN must live inside the CTE body and not leak into the outer+ // query. Split on the outer query (the `) select ... from aj` that closes+ // the CTE): the clause must appear before the split and be absent after it.+ const outerStart = sql.toLowerCase().search(/\)\s*select[\s\S]*from aj/i);+ expect(outerStart).toBeGreaterThan(-1);+ const cteBody = sql.slice(0, outerStart);+ const outerQuery = sql.slice(outerStart);+ expect(cteBody).toMatch(/array join/i);+ expect(outerQuery).not.toMatch(/array join/i);+ });++ it("keeps ARRAY JOIN when applied inside a subquery", () => {+ const db = getClickhouseKysely();+ const qb = db+ .selectFrom(+ db+ .selectFrom("observations")+ .select("environment")+ .$call(costArrayJoin())+ .as("sub"),+ )+ .select("environment");++ const { sql } = compileClickhouseQuery(qb, ctx);+ // Outer query has no ARRAY JOIN; presence proves the subquery clause held.+ expect(sql).toMatch(/array join/i);+ });++ it("keeps ARRAY JOIN when applied inside a virtual-view body", () => {+ const db = getClickhouseKysely();+ const view = defineView("aj_view")<{ environment: string }>(() =>+ db+ .selectFrom("observations")+ .select("environment")+ .$call(costArrayJoin()),+ );++ const qb = fromView(view).select("environment");++ const { sql } = compileClickhouseQuery(qb, ctx);+ // Outer query only selects from the view alias; presence proves the view+ // body's ARRAY JOIN composed into the emitted CTE.+ expect(sql).toMatch(/array join/i);+ expect(sql).toMatch(/with aj_view as/i);+ });+});++describe("LIMIT BY composition across nesting positions", () => {+ it("emits LIMIT BY at the top level (baseline)", () => {+ const db = getClickhouseKysely();+ const qb = db+ .selectFrom("events_core")+ .select(["span_id", "project_id"])+ .orderBy("event_ts", "desc")+ .$call(dedup());++ const { sql } = compileClickhouseQuery(qb, ctx);+ expect(sql).toMatch(/limit 1 by/i);+ });++ it("keeps LIMIT BY when applied inside a CTE body", () => {+ const db = getClickhouseKysely();+ const qb = db+ .with("lb", (qb) =>+ qb+ .selectFrom("events_core")+ .select(["span_id", "project_id"])+ .orderBy("event_ts", "desc")+ .$call(dedup()),+ )+ .selectFrom("lb")+ .select("span_id");++ const { sql } = compileClickhouseQuery(qb, ctx);+ // Outer query has no LIMIT BY; presence proves the CTE body's clause held.+ expect(sql).toMatch(/limit 1 by/i);+ expect(sql).toMatch(/with lb as/i);+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/dialect.ts b/packages/shared/src/server/query-ast/kysely/dialect.tsnew file mode 100644index 0000000000..8b85c5bebd--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/dialect.ts@@ -0,0 +1,42 @@+import {+ DummyDriver,+ Kysely,+ SqliteAdapter,+ SqliteIntrospector,+ type Dialect,+} from "kysely";++import { ClickHouseQueryCompiler } from "./compiler";+import type { ClickHouseDatabase } from "./schema";++/**+ * Compile-only ClickHouse dialect. Queries are never executed through Kysely;+ * the existing `queryClickhouse` seam remains the exec path. DummyDriver+ * throws if anyone tries to `.execute()`.+ */+export class ClickHouseCompileDialect implements Dialect {+ createDriver() {+ return new DummyDriver();+ }++ createQueryCompiler() {+ return new ClickHouseQueryCompiler();+ }++ createAdapter() {+ return new SqliteAdapter();+ }++ createIntrospector(db: Kysely<any>) {+ return new SqliteIntrospector(db);+ }+}++let compileDb: Kysely<ClickHouseDatabase> | undefined;++export function getClickhouseKysely(): Kysely<ClickHouseDatabase> {+ compileDb ??= new Kysely<ClickHouseDatabase>({+ dialect: new ClickHouseCompileDialect(),+ });+ return compileDb;+}diff --git a/packages/shared/src/server/query-ast/kysely/errors.ts b/packages/shared/src/server/query-ast/kysely/errors.tsnew file mode 100644index 0000000000..e7479512e7--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/errors.ts@@ -0,0 +1,20 @@+export class QueryCompileError extends Error {+ constructor(message: string) {+ super(message);+ this.name = "QueryCompileError";+ }+}++export class UnscopedRelationError extends QueryCompileError {+ constructor(message: string) {+ super(message);+ this.name = "UnscopedRelationError";+ }+}++export class TypeCompatibilityError extends QueryCompileError {+ constructor(message: string) {+ super(message);+ this.name = "TypeCompatibilityError";+ }+}diff --git a/packages/shared/src/server/query-ast/kysely/extensions.test.ts b/packages/shared/src/server/query-ast/kysely/extensions.test.tsnew file mode 100644index 0000000000..a96246165e--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/extensions.test.ts@@ -0,0 +1,148 @@+import { describe, expect, it } from "vitest";++import { compileClickhouseQuery } from "./compile";+import { getClickhouseKysely } from "./dialect";+import { TypeCompatibilityError } from "./errors";+import { metadataValue } from "./extensions";+import { ArrayIndexNode } from "./nodes";+import { TenancyInjectionPlugin } from "./tenancy";+import { defineView, fromView } from "./views";++const ctx = { projectId: "proj-1" };++describe("schema-typed selection (condition 7)", () => {+ it("accepts sum() over a numeric column and emits SQL", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select((eb) => [eb.fn.sum("total_cost").as("s")]);++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ expect(compiled.toLowerCase()).toContain("sum(total_cost)");+ expect(compiled.toLowerCase()).toContain("project_id");+ });++ it("rejects sum() over a String column at the runtime validation pass", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select((eb) => [eb.fn.sum("environment").as("s")]);++ expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ TypeCompatibilityError,+ );+ expect(() => compileClickhouseQuery(qb, ctx)).toThrow(/environment/);+ });+});++describe("metadata indexOf access (condition 7b)", () => {+ // These tests use a two-layer strategy. This first test inspects the AST+ // itself (serialized to JSON) to prove `metadata[key]` is *lowered* to real+ // operation nodes β an `ArrayIndexNode` subscript around an `indexOf(...)`+ // `FunctionNode` β and never a `RawNode` (a raw SQL string). The sibling+ // tests below assert the SQL the compiler then emits. Separating them+ // isolates "the builder produced the right tree" (composable, escaped,+ // parameter-bound) from "the compiler rendered that tree correctly".+ it("lowers metadata[key] to ArrayIndexNode + indexOf FunctionNode, not RawNode", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core as e")+ .select((_eb) => [metadataValue("e", "a").as("a")])+ .where((eb) => eb(metadataValue("e", "a"), ">", 2));++ const node = qb+ .withPlugin(new TenancyInjectionPlugin(ctx))+ .toOperationNode();+ const json = JSON.stringify(node);+ expect(json).toContain('"kind":"ArrayIndexNode"');+ expect(json).toContain('"func":"indexOf"');+ expect(json).not.toMatch(/"kind":"RawNode"/);++ // `.as("a")` wraps the expression in an AliasNode, so the first selection's+ // inner node must be that AliasNode rather than the bare ArrayIndexNode.+ const selections = (+ node as { selections?: { selection?: { kind: string } }[] }+ ).selections;+ expect(selections?.[0]?.selection?.kind).toBe("AliasNode");+ });++ it("emits subscript + bound key param in SELECT and WHERE", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core as e")+ .select((_eb) => [metadataValue("e", "a").as("a")])+ .where((eb) => eb(metadataValue("e", "a"), ">", 2));++ const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const lower = compiled.toLowerCase();+ expect(lower).toContain("metadata_values[indexof(e.metadata_names");+ expect(lower).toContain("as a");+ expect(Object.values(params)).toContain("a");+ expect(Object.values(params)).toContain(2);+ expect(compiled).not.toMatch(/metadata_names,\s*'a'/);+ });++ it("emits the same lowering in HAVING", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core as e")+ .select("environment")+ .groupBy("environment")+ .having((eb) => eb(metadataValue("e", "a"), ">", 2));++ const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ expect(compiled.toLowerCase()).toContain("having");+ expect(compiled.toLowerCase()).toContain(+ "metadata_values[indexof(e.metadata_names",+ );+ expect(Object.values(params)).toContain("a");+ });+});++describe("virtual view (condition 8)", () => {+ const environmentsView = defineView("environments_view")<{+ environment: string;+ }>(() =>+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("environment")+ .distinct(),+ );++ it("treats the view as a table and emits a WITH CTE", () => {+ const qb = fromView(environmentsView)+ .select("environment")+ .where("environment", "=", "production");++ const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const lower = compiled.toLowerCase();+ expect(lower).toContain("with environments_view as");+ expect(lower).toContain("from events_core");+ expect(lower).toContain("from environments_view");+ expect(lower).toContain("environment =");+ expect(Object.values(params)).toContain("proj-1");+ expect(Object.values(params)).toContain("production");+ });++ it("injects tenancy on the inner physical table, not the view alias", () => {+ const qb = fromView(environmentsView).select("environment");+ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const lower = compiled.toLowerCase();+ expect(lower).toMatch(+ /with environments_view as \([\s\S]*project_id[\s\S]*\)[\s\S]*from environments_view/,+ );+ const afterView = lower.split("from environments_view")[1] ?? "";+ expect(afterView).not.toContain("project_id");+ });+});++describe("ArrayIndexNode identity", () => {+ it("is a real node kind", () => {+ expect(+ ArrayIndexNode.is(+ ArrayIndexNode.create(+ {+ kind: "ColumnNode",+ column: { kind: "IdentifierNode", name: "x" },+ } as never,+ { kind: "ValueNode", value: 1 } as never,+ ),+ ),+ ).toBe(true);+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/extensions.ts b/packages/shared/src/server/query-ast/kysely/extensions.tsnew file mode 100644index 0000000000..dcb64d2c9a--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/extensions.ts@@ -0,0 +1,230 @@+/**+ * ClickHouse-specific query-builder extensions for the Kysely dialect.+ *+ * Kysely has no native representation for several ClickHouse constructs, so+ * this file supplies them in three forms:+ * - Plugins ({@link ArrayJoinPlugin}, {@link LimitByPlugin}) that attach a+ * custom operation node (`ArrayJoinNode` / `LimitByNode`) onto the select+ * node during query transformation; the ClickHouse compiler renders them.+ * - `$call` helpers ({@link arrayJoin}, {@link limitBy}) that apply those+ * plugins while extending the builder's output row type, so produced+ * aliases are type-checked and misspelled references are compile errors.+ * - Typed expression helpers ({@link mapKeys}, {@link mapValues},+ * {@link metadataValue}) that build ClickHouse map/array function calls and+ * `metadata[key]` subscripts as real, parameter-bound nodes β never raw SQL+ * strings β so they escape and compose like any other expression.+ */+import {+ ColumnNode,+ ExpressionWrapper,+ FunctionNode,+ IdentifierNode,+ ReferenceNode,+ TableNode,+ ValueNode,+ type Expression,+ type KyselyPlugin,+ type OperationNode,+ type PluginTransformQueryArgs,+ type PluginTransformResultArgs,+ type RootOperationNode,+ type SelectQueryBuilder,+} from "kysely";++import {+ ArrayIndexNode,+ ArrayJoinNode,+ LimitByNode,+ type ArrayJoinVariant,+ type ClickHouseSelectQueryNode,+} from "./nodes";+import type { ClickHouseDatabase } from "./schema";+import { ClickHouseOperationNodeTransformer } from "./transformer";++type OperationNodeSource = OperationNode | { toOperationNode(): OperationNode };++function toNode(source: OperationNodeSource): OperationNode {+ if (+ "toOperationNode" in source &&+ typeof source.toOperationNode === "function"+ ) {+ return source.toOperationNode();+ }+ return source as OperationNode;+}++type ArrayJoinItem = {+ expression: OperationNodeSource;+ as: string;+};++/** Plugin that attaches an {@link ArrayJoinNode} onto the select node. */+class ArrayJoinPlugin implements KyselyPlugin {+ constructor(+ private readonly items: ReadonlyArray<ArrayJoinItem>,+ private readonly variant: ArrayJoinVariant = "default",+ ) {}++ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ if (args.node.kind !== "SelectQueryNode") return args.node;+ const transformer = new ClickHouseOperationNodeTransformer();+ const node = transformer.transformNode(+ args.node,+ ) as ClickHouseSelectQueryNode;+ const arrayJoinNode = ArrayJoinNode.create(+ this.items.map((item) => ({+ expression: toNode(item.expression),+ alias: IdentifierNode.create(item.as),+ })),+ this.variant,+ );+ return {+ ...node,+ arrayJoins: [...(node.arrayJoins ?? []), arrayJoinNode],+ } as RootOperationNode;+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }+}++type LimitBySpec = {+ count: number;+ columns: ReadonlyArray<string>;+};++/** Plugin that attaches a {@link LimitByNode} onto the select node. */+class LimitByPlugin implements KyselyPlugin {+ constructor(private readonly spec: LimitBySpec) {}++ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ if (args.node.kind !== "SelectQueryNode") return args.node;+ const transformer = new ClickHouseOperationNodeTransformer();+ const node = transformer.transformNode(+ args.node,+ ) as ClickHouseSelectQueryNode;+ return {+ ...node,+ limitBy: LimitByNode.create(+ ValueNode.createImmediate(this.spec.count),+ this.spec.columns.map(columnRef),+ ),+ } as RootOperationNode;+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }+}++// Build a `table.column` reference straight from its parts. Prefer this+// whenever the caller already holds the table and column separately.+function qualifiedColumn(table: string, column: string): OperationNode {+ return ReferenceNode.create(+ ColumnNode.create(column),+ TableNode.create(table),+ );+}++// For callers whose column arrives as a single, possibly dotted string β e.g.+// the `limitBy` column list. Kysely only parses `"table.column"` at the+// expression-builder layer (`eb.ref`) and does not export its string-reference+// parser, and these plugins build raw OperationNodes with no ExpressionBuilder+// in scope, so we split the dotted name here rather than route through a string.+function columnRef(name: string): OperationNode {+ const parts = name.split(".");+ return parts.length === 2+ ? qualifiedColumn(parts[0], parts[1])+ : ColumnNode.create(name);+}++function arrayFunction<T>(fn: string, column: string): Expression<T[]> {+ return new ExpressionWrapper<+ ClickHouseDatabase,+ keyof ClickHouseDatabase,+ T[]+ >(FunctionNode.create(fn, [columnRef(column)]) as unknown as OperationNode);+}++/** `mapKeys(map)` as a typed array expression (element type defaults to string). */+export function mapKeys<K = string>(column: string): Expression<K[]> {+ return arrayFunction<K>("mapKeys", column);+}++/** `mapValues(map)` as a typed array expression (element type defaults to string). */+export function mapValues<V = string>(column: string): Expression<V[]> {+ return arrayFunction<V>("mapValues", column);+}++/**+ * Lower `metadata[key]` to a traced array-subscript + `indexOf` node.+ * The key is a bound `ValueNode`, not a SQL literal.+ */+export function metadataValue(+ tableAlias: string,+ key: string,+): ExpressionWrapper<ClickHouseDatabase, "events_core", string | number> {+ return new ExpressionWrapper(+ ArrayIndexNode.create(+ qualifiedColumn(tableAlias, "metadata_values"),+ FunctionNode.create("indexOf", [+ qualifiedColumn(tableAlias, "metadata_names"),+ ValueNode.create(key),+ ]),+ ) as unknown as OperationNode,+ );+}++// Extract the element type `T` from an array-typed `Expression<T[]>`. `arrayJoin`+// uses it to give each produced alias the array's *element* type in the+// builder's output row (ARRAY JOIN yields one row per element, so a+// `mapValues(...)` expression of `V[]` surfaces the alias as `V`, not `V[]`).+// Resolves to `never` for a non-array expression.+type ElementOf<E> =+ E extends Expression<infer A>+ ? A extends ReadonlyArray<infer T>+ ? T+ : never+ : never;++/**+ * ARRAY JOIN clause as a `$call` step: `qb.$call(arrayJoin({ alias: expr }))`.+ * Each entry's alias is added to the builder's output row type, so consumers of+ * the result (e.g. an outer query selecting from a CTE body) can reference the+ * produced columns and typos on the alias name are compile errors. The element+ * value type is opaque (Kysely's Expression hides its type arg), so aliases+ * surface as `unknown`. The alias must still be projected in a `select` to+ * appear in the emitted SQL. Distinct from the `arrayJoin()` SELECT function.+ */+export function arrayJoin<+ Items extends { [K in keyof Items]: Expression<ReadonlyArray<unknown>> },+>(items: Items, variant: ArrayJoinVariant = "default") {+ const list: ArrayJoinItem[] = Object.entries(items).map(+ ([as, expression]) => ({+ expression: expression as OperationNodeSource,+ as,+ }),+ );+ return <DB, TB extends keyof DB, O>(+ qb: SelectQueryBuilder<DB, TB, O>,+ ): SelectQueryBuilder<+ DB,+ TB,+ O & { [K in keyof Items]: ElementOf<Items[K]> }+ > =>+ qb.withPlugin(+ new ArrayJoinPlugin(list, variant),+ ) as unknown as SelectQueryBuilder<+ DB,+ TB,+ O & { [K in keyof Items]: ElementOf<Items[K]> }+ >;+}++/** LIMIT BY clause as a `$call` step: `qb.$call(limitBy({ count, columns }))`. */+export function limitBy(spec: LimitBySpec) {+ return <DB, TB extends keyof DB, O>(+ qb: SelectQueryBuilder<DB, TB, O>,+ ): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new LimitByPlugin(spec));+}diff --git a/packages/shared/src/server/query-ast/kysely/nodes.test.ts b/packages/shared/src/server/query-ast/kysely/nodes.test.tsnew file mode 100644index 0000000000..7b911e5350--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/nodes.test.ts@@ -0,0 +1,288 @@+import { createQueryId, sql, type KyselyPlugin } from "kysely";+import { describe, expect, it } from "vitest";++import { compileClickhouseQuery } from "./compile";+import { ClickHouseQueryCompiler } from "./compiler";+import { getClickhouseKysely } from "./dialect";+import { QueryCompileError, UnscopedRelationError } from "./errors";+import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";+import {+ ArrayJoinNode,+ LimitByNode,+ isClickHouseSelectQueryNode,+} from "./nodes";+import { TenancyInjectionPlugin } from "./tenancy";++const ctx = { projectId: "proj-1" };++describe("tenancy injection", () => {+ it("requires an ExecutionContext at compile time", () => {+ const qb = getClickhouseKysely()+ .selectFrom("traces")+ .select("environment")+ .where("project_id", "=", "proj-1");++ // The mandatory `ctx` parameter is the primary guard: omitting it is a+ // type error. The `@ts-expect-error` both documents and enforces that+ // contract, and the runtime throw is the defense-in-depth backstop for+ // callers that reach this through an `any` boundary.+ // @ts-expect-error - ctx is required+ expect(() => compileClickhouseQuery(qb)).toThrow(QueryCompileError);+ // @ts-expect-error - ctx is required+ expect(() => compileClickhouseQuery(qb)).toThrow(/no tenancy scope/);+ });++ it("refuses a context with an empty projectId", () => {+ const qb = getClickhouseKysely()+ .selectFrom("traces")+ .select("environment")+ .where("project_id", "=", "proj-1");++ // Type-valid but semantically empty: the type system cannot express+ // "non-empty string", so the runtime check earns its keep here.+ expect(() => compileClickhouseQuery(qb, { projectId: "" })).toThrow(+ /no tenancy scope/,+ );+ });++ it("refuses kysely.compile() when the tenancy pass did not run", () => {+ const qb = getClickhouseKysely()+ .selectFrom("traces")+ .select("environment")+ .where("project_id", "=", "proj-1");++ expect(() => qb.compile()).toThrow(QueryCompileError);+ });++ it("injects project_id when the builder omitted it", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select("environment");++ const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ expect(compiled.toLowerCase()).toContain("project_id");+ expect(Object.values(params)).toContain("proj-1");+ });++ it("does not duplicate project_id when the builder already scoped the table", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select("environment")+ .where("project_id", "=", "proj-1");++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const matches = compiled.match(/project_id/gi) ?? [];+ expect(matches).toHaveLength(1);+ });++ it("rejects a raw-SQL table source as an unscoped escape hatch", () => {+ const qb = getClickhouseKysely()+ .selectFrom(sql<{ environment: string }>`traces`.as("t"))+ .select("environment");++ expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ UnscopedRelationError,+ );+ });++ it("rejects aliased raw SQL that would introduce an unscoped relation", () => {+ const qb = getClickhouseKysely()+ .selectFrom(sql<{ environment: string }>`(SELECT * FROM traces)`.as("t"))+ .selectAll();++ expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ UnscopedRelationError,+ );+ });++ it("rejects raw SQL fragments in SELECT and WHERE, not only FROM/JOIN", () => {+ const selectRaw = getClickhouseKysely()+ .selectFrom("events_core")+ .select(+ sql<string>`(SELECT environment FROM traces LIMIT 1)`.as("stolen"),+ );++ const whereRaw = getClickhouseKysely()+ .selectFrom("events_core")+ .select("environment")+ .where(sql<boolean>`id IN (SELECT id FROM traces)`);++ expect(() => compileClickhouseQuery(selectRaw, ctx)).toThrow(+ UnscopedRelationError,+ );+ expect(() => compileClickhouseQuery(whereRaw, ctx)).toThrow(+ UnscopedRelationError,+ );+ });++ it("does not treat a copied langfuseTenancy property as a valid stamp", () => {+ const qb = getClickhouseKysely().selectFrom("traces").select("environment");+ const forged = {+ ...qb.toOperationNode(),+ langfuseTenancy: { projectId: "proj-1" },+ };+ const compiler = new ClickHouseQueryCompiler();+ expect(() => compiler.compileQuery(forged, createQueryId())).toThrow(+ QueryCompileError,+ );++ const fakeStamp: KyselyPlugin = {+ transformQuery(args) {+ return {+ ...args.node,+ langfuseTenancy: { projectId: "forged" },+ };+ },+ async transformResult(args) {+ return args.result;+ },+ };+ const stampedByProperty = getClickhouseKysely()+ .withPlugin(fakeStamp)+ .selectFrom("traces")+ .select("environment");+ expect(() => stampedByProperty.compile()).toThrow(QueryCompileError);+ });++ it("qualifies every project_id on a two-table JOIN", () => {+ const qb = getClickhouseKysely()+ .selectFrom("observations as o")+ .innerJoin("traces as t", (join) =>+ join+ .onRef("o.trace_id", "=", "t.id")+ .onRef("o.project_id", "=", "t.project_id"),+ )+ .select("o.environment")+ .where("o.project_id", "=", "proj-1");++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const refs = compiled.match(/[\w.]*project_id/gi) ?? [];+ expect(refs.length).toBeGreaterThanOrEqual(3);+ for (const ref of refs) {+ expect(ref).toMatch(/^(o|t)\.project_id$/i);+ }+ expect(compiled).toMatch(/o\.project_id/i);+ expect(compiled).toMatch(/t\.project_id/i);+ });++ it("injects the context project even when the builder scoped a different project", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select("environment")+ .where("project_id", "=", "other-project");++ const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ // The builder's `other-project` predicate does not scope the request's+ // project, so the pass must still inject `project_id = proj-1`.+ expect(Object.values(params)).toContain("proj-1");+ const matches = compiled.match(/project_id/gi) ?? [];+ expect(matches.length).toBeGreaterThanOrEqual(2);+ });++ it("does not let one unqualified project_id cover multiple tenanted JOIN relations", () => {+ const qb = getClickhouseKysely()+ .selectFrom("observations as o")+ .innerJoin("traces as t", (join) => join.onRef("o.trace_id", "=", "t.id"))+ .select("o.environment")+ // Unqualified, and ambiguous across two tenanted tables β must not be+ // treated as scoping either relation.+ .where("project_id", "=", "proj-1");++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ expect(compiled).toMatch(/o\.project_id/i);+ expect(compiled).toMatch(/t\.project_id/i);+ });++ it("scopes an aliased relation whose alias collides with another table's physical name", () => {+ // `scores AS traces` aliases one tenanted table to the *physical name* of+ // another tenanted table (`traces`). A qualified `traces.project_id`+ // predicate scopes the alias only; the physically-`traces` relation+ // (aliased `t`) must still get its own injected predicate. Regression for+ // an alias/table-name collision that previously left `t` unscoped.+ const qb = getClickhouseKysely()+ .selectFrom("scores as traces")+ .innerJoin("traces as t", (join) => join.onRef("traces.id", "=", "t.id"))+ .select("t.environment")+ .where("traces.project_id", "=", "proj-1");++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ expect(compiled).toMatch(/t\.project_id/i);+ const matches = compiled.match(/project_id/gi) ?? [];+ expect(matches.length).toBeGreaterThanOrEqual(2);+ });+});++describe("ARRAY JOIN and LIMIT BY nodes", () => {+ it("attaches a real ArrayJoinNode, not a RawNode", () => {+ const qb = getClickhouseKysely()+ .selectFrom("observations")+ .select("environment")+ .where("project_id", "=", "proj-1")+ .$call(+ arrayJoin({+ cost_key: mapKeys("cost_details"),+ cost: mapValues("cost_details"),+ }),+ );++ const node = qb+ .withPlugin(new TenancyInjectionPlugin(ctx))+ .toOperationNode();+ expect(isClickHouseSelectQueryNode(node)).toBe(true);+ if (!isClickHouseSelectQueryNode(node)) return;+ expect(node.arrayJoins).toHaveLength(1);+ const arrayJoinNode = node.arrayJoins![0];+ expect(ArrayJoinNode.is(arrayJoinNode)).toBe(true);+ expect(arrayJoinNode.items).toHaveLength(2);+ expect(arrayJoinNode.items[0].expression.kind).toBe("FunctionNode");+ expect(arrayJoinNode.items[0].alias.kind).toBe("IdentifierNode");+ expect(JSON.stringify(node)).not.toMatch(/"kind":"RawNode"/);+ });++ it("emits ARRAY JOIN from the node, not a string splice", () => {+ const qb = getClickhouseKysely()+ .selectFrom("observations")+ .select("environment")+ .where("project_id", "=", "proj-1")+ .$call(arrayJoin({ cost_key: mapKeys("cost_details") }));++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ expect(compiled.toLowerCase()).toContain("array join");+ expect(compiled.toLowerCase()).toContain(+ "mapkeys(cost_details) as cost_key",+ );+ });++ it("attaches a real LimitByNode", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select("span_id")+ .where("project_id", "=", "proj-1")+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));++ const node = qb+ .withPlugin(new TenancyInjectionPlugin(ctx))+ .toOperationNode();+ expect(isClickHouseSelectQueryNode(node)).toBe(true);+ if (!isClickHouseSelectQueryNode(node)) return;+ expect(node.limitBy).toBeDefined();+ expect(LimitByNode.is(node.limitBy!)).toBe(true);+ expect(node.limitBy!.columns).toHaveLength(2);+ expect(node.limitBy!.count.kind).toBe("ValueNode");+ });++ it("emits LIMIT BY before a plain LIMIT", () => {+ const qb = getClickhouseKysely()+ .selectFrom("events_core")+ .select("span_id")+ .where("project_id", "=", "proj-1")+ .limit(50)+ .$call(limitBy({ count: 1, columns: ["span_id"] }));++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const lower = compiled.toLowerCase();+ expect(lower).toContain("limit ");+ expect(lower).toContain(" by ");+ expect(lower.indexOf(" by ")).toBeLessThan(lower.lastIndexOf("limit"));+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/nodes.ts b/packages/shared/src/server/query-ast/kysely/nodes.tsnew file mode 100644index 0000000000..8b08f21423--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/nodes.ts@@ -0,0 +1,114 @@+import {+ type IdentifierNode,+ type OperationNode,+ type SelectQueryNode,+} from "kysely";++/**+ * ClickHouse `ARRAY JOIN` / `LEFT ARRAY JOIN` / `INNER ARRAY JOIN`.+ *+ * Not a Kysely `OperationNode` kind β Kysely's `kind` union is closed, so a+ * real custom kind would collapse to `never`. These nodes live as extra+ * fields on {@link ClickHouseSelectQueryNode}. Children (`expression`,+ * `alias`) are traced Kysely nodes, not raw SQL.+ */+export type ArrayJoinVariant = "default" | "left" | "inner";++type ArrayJoinItemNode = {+ readonly expression: OperationNode;+ readonly alias: IdentifierNode;+};++export type ArrayJoinNode = {+ readonly kind: "ArrayJoinNode";+ readonly variant: ArrayJoinVariant;+ readonly items: ReadonlyArray<ArrayJoinItemNode>;+};++export const ArrayJoinNode = {+ is(node: { kind: string }): node is ArrayJoinNode {+ return node.kind === "ArrayJoinNode";+ },+ create(+ items: ReadonlyArray<ArrayJoinItemNode>,+ variant: ArrayJoinVariant = "default",+ ): ArrayJoinNode {+ return Object.freeze({+ kind: "ArrayJoinNode",+ variant,+ items: Object.freeze([...items]),+ });+ },+};++/**+ * ClickHouse `LIMIT n BY col1, col2, ...`.+ *+ * Distinct from Kysely's `LimitNode` (`LIMIT n`). ClickHouse places LIMIT BY+ * after ORDER BY and before a plain LIMIT.+ */+export type LimitByNode = {+ readonly kind: "LimitByNode";+ readonly count: OperationNode;+ readonly columns: ReadonlyArray<OperationNode>;+};++export const LimitByNode = {+ is(node: { kind: string }): node is LimitByNode {+ return node.kind === "LimitByNode";+ },+ create(+ count: OperationNode,+ columns: ReadonlyArray<OperationNode>,+ ): LimitByNode {+ return Object.freeze({+ kind: "LimitByNode",+ count,+ columns: Object.freeze([...columns]),+ });+ },+};++/**+ * ClickHouse array subscript `arr[index]`. Children are traced nodes+ * (typically a column ref and an `indexOf(...)` FunctionNode), not raw SQL.+ *+ * Same closed-`kind` constraint as ARRAY JOIN: not a Kysely visitor-map+ * kind. The dialect compiler and transformer special-case this object.+ */+export type ArrayIndexNode = {+ readonly kind: "ArrayIndexNode";+ readonly array: OperationNode;+ readonly index: OperationNode;+};++export const ArrayIndexNode = {+ is(node: { kind: string }): node is ArrayIndexNode {+ return node.kind === "ArrayIndexNode";+ },+ create(array: OperationNode, index: OperationNode): ArrayIndexNode {+ return Object.freeze({+ kind: "ArrayIndexNode",+ array,+ index,+ });+ },+};++/**+ * Select query extended with ClickHouse-only clauses.+ * Extra fields are ignored by Kysely's default compiler/transformer; our+ * dialect compiler and {@link ClickHouseOperationNodeTransformer} are the+ * ones that read them. Tenancy is stamped by identity (WeakSet), not a+ * copyable property on this node.+ */+export type ClickHouseSelectQueryNode = SelectQueryNode & {+ readonly arrayJoins?: ReadonlyArray<ArrayJoinNode>;+ readonly limitBy?: LimitByNode;+};++export function isClickHouseSelectQueryNode(+ node: OperationNode,+): node is ClickHouseSelectQueryNode {+ return node.kind === "SelectQueryNode";+}diff --git a/packages/shared/src/server/query-ast/kysely/schema.test.ts b/packages/shared/src/server/query-ast/kysely/schema.test.tsnew file mode 100644index 0000000000..815486231c--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/schema.test.ts@@ -0,0 +1,36 @@+import { describe, expect, it } from "vitest";++import { COLUMN_DATA_TYPES, TENANTED_TABLES } from "./schema";+import { schemaTypeAssertions, extensionTypeAssertions } from "./types.assert";++describe("table registry derivation", () => {+ it("derives the tenanted table set from the registry", () => {+ // Only relations in the registry are scoped. `events_full` is just as+ // project-gated as `events_core` and would be `tenant: true` if modeled β+ // it is absent only because the builder targets `events_core` (the MV+ // projection) and never selects `events_full`. Since it is not in the row+ // type and raw sources are rejected, it is unreachable, so it needs no+ // entry. Add one (tenant defaults true) the day a query selects it.+ expect([...TENANTED_TABLES].sort()).toEqual([+ "events_core",+ "observations",+ "scores",+ "traces",+ ]);+ });++ it("derives coarse runtime column types for the type-check pass", () => {+ expect(COLUMN_DATA_TYPES.total_cost).toBe("number");+ expect(COLUMN_DATA_TYPES.environment).toBe("string");+ expect(COLUMN_DATA_TYPES.timestamp).toBe("date");+ expect(COLUMN_DATA_TYPES.metadata_names).toBe("array");+ expect(COLUMN_DATA_TYPES.cost_details).toBe("map");+ });++ // The assertions themselves are compile-time (`tsc` is the test); this only+ // anchors the file so its `@ts-expect-error` checks stay in the build graph.+ it("keeps the compile-time type assertions in the build graph", () => {+ expect(typeof schemaTypeAssertions).toBe("function");+ expect(typeof extensionTypeAssertions).toBe("function");+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/schema.ts b/packages/shared/src/server/query-ast/kysely/schema.tsnew file mode 100644index 0000000000..0fd812f599--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/schema.ts@@ -0,0 +1,140 @@+/**+ * Single source of truth for the physical ClickHouse relations the Kysely+ * compiler targets. One table declaration drives all three downstream views:+ *+ * - `ClickHouseDatabase` β the Kysely row types (column autocomplete and typed+ * comparisons).+ * - `COLUMN_DATA_TYPES` β the coarse runtime type map the aggregate type-check+ * pass consults (`typecheck.ts`).+ * - `TENANTED_TABLES` β the relations the tenancy pass must scope+ * (`tenancy.ts`).+ *+ * Deriving all three from one declaration keeps them from drifting apart.+ *+ * Column sets cover the relations the compiler targets, not a full schema+ * dump. Physical tuning metadata (partition / sort / dedup keys) is not+ * modeled here; dedup lowering will extend this same declaration.+ */++/**+ * ClickHouse column types the registry understands. Each maps to a TS row type+ * (for Kysely) and a coarse runtime category (for the type-check pass).+ */+export type ChColumnType =+ | "String"+ | "Float"+ | "DateTime"+ | "Array(String)"+ | "Map(String, Float)";++type TsTypeOfColumn<T extends ChColumnType> = T extends "String"+ ? string+ : T extends "Float"+ ? number+ : T extends "DateTime"+ ? Date+ : T extends "Array(String)"+ ? string[]+ : T extends "Map(String, Float)"+ ? Record<string, number>+ : never;++/** Coarse runtime category the aggregate type-check pass reasons about. */+export type ColumnDataType = "string" | "number" | "date" | "array" | "map";++const RUNTIME_TYPE: Record<ChColumnType, ColumnDataType> = {+ String: "string",+ Float: "number",+ DateTime: "date",+ "Array(String)": "array",+ "Map(String, Float)": "map",+};++function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {+ columns: Cols;+ /**+ * Whether the tenancy pass must inject a `project_id` scope. Defaults to+ * `true`: every relation modeled here is project-gated, and defaulting on is+ * fail-closed β a table added without this flag is still scoped. Set `false`+ * only for a genuinely global relation.+ */+ tenant?: boolean;+}) {+ return { columns: spec.columns, tenant: spec.tenant ?? true };+}++const TABLE_REGISTRY = {+ traces: defineTable({+ columns: {+ environment: "String",+ project_id: "String",+ timestamp: "DateTime",+ id: "String",+ },+ }),+ observations: defineTable({+ columns: {+ environment: "String",+ project_id: "String",+ start_time: "DateTime",+ trace_id: "String",+ cost_details: "Map(String, Float)",+ usage_details: "Map(String, Float)",+ },+ }),+ events_core: defineTable({+ columns: {+ environment: "String",+ project_id: "String",+ start_time: "DateTime",+ span_id: "String",+ trace_id: "String",+ event_ts: "DateTime",+ type: "String",+ total_cost: "Float",+ metadata_names: "Array(String)",+ metadata_values: "Array(String)",+ },+ }),+ scores: defineTable({+ columns: {+ environment: "String",+ project_id: "String",+ timestamp: "DateTime",+ data_type: "String",+ },+ }),+} as const;++type RowOf<Cols extends Record<string, ChColumnType>> = {+ [C in keyof Cols]: TsTypeOfColumn<Cols[C]>;+};++/** Derived Kysely row types. */+export type ClickHouseDatabase = {+ [T in keyof typeof TABLE_REGISTRY]: RowOf<+ (typeof TABLE_REGISTRY)[T]["columns"]+ >;+};++/**+ * Derived coarse runtime column-type map (column name -> category), consumed by+ * the aggregate type-check pass. Column names are consistent across tables in+ * this schema, so a flat map is unambiguous.+ */+export const COLUMN_DATA_TYPES: Record<string, ColumnDataType> =+ Object.fromEntries(+ Object.values(TABLE_REGISTRY).flatMap((table) =>+ Object.entries(table.columns).map(([name, chType]) => [+ name,+ RUNTIME_TYPE[chType],+ ]),+ ),+ );++/** Derived set of relations the tenancy pass must scope. */+export const TENANTED_TABLES = new Set<string>(+ Object.entries(TABLE_REGISTRY)+ .filter(([, spec]) => spec.tenant)+ .map(([name]) => name),+);diff --git a/packages/shared/src/server/query-ast/kysely/tenancy.ts b/packages/shared/src/server/query-ast/kysely/tenancy.tsnew file mode 100644index 0000000000..60dbd7fb83--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/tenancy.ts@@ -0,0 +1,345 @@+import {+ AliasNode,+ AndNode,+ BinaryOperationNode,+ ColumnNode,+ IdentifierNode,+ JoinNode,+ OperatorNode,+ OrNode,+ ParensNode,+ RawNode,+ ReferenceNode,+ SelectQueryNode,+ TableNode,+ ValueNode,+ WhereNode,+ type KyselyPlugin,+ type OperationNode,+ type PluginTransformQueryArgs,+ type PluginTransformResultArgs,+ type QueryId,+ type RootOperationNode,+} from "kysely";++import type { ExecutionContext } from "../executionContext";+import { QueryCompileError, UnscopedRelationError } from "./errors";+import { type ClickHouseSelectQueryNode } from "./nodes";+import { TENANTED_TABLES } from "./schema";+import { ClickHouseOperationNodeTransformer } from "./transformer";++const PROJECT_ID_COLUMN = "project_id";++/**+ * Identity-based stamp. A copied `langfuseTenancy` property on a cloned node+ * is not enough β only trees that actually went through+ * {@link TenancyInjectionPlugin} are in this set.+ */+const TENANCY_STAMPED = new WeakSet<object>();++type Relation =+ | { kind: "table"; tableName: string; alias?: string }+ | { kind: "subquery"; node: SelectQueryNode }+ | { kind: "raw" }+ | { kind: "other" };++/**+ * Mandatory tenancy injection. Attaches `project_id = {projectId}` to every+ * tenanted physical relation that does not already have that predicate, then+ * stamps the tree so {@link ClickHouseQueryCompiler} will compile it.+ *+ * Any `RawNode` whose SQL fragments introduce a relation (`SELECT` / `FROM` /+ * `JOIN`) is rejected β not only FROM/JOIN table sources. Kysely's own+ * keyword fragments (`asc` / `desc`) are not relations and are allowed.+ */+export class TenancyInjectionPlugin implements KyselyPlugin {+ constructor(private readonly ctx: ExecutionContext) {+ if (!ctx?.projectId) {+ throw new QueryCompileError(+ "ExecutionContext.projectId is required for tenancy injection",+ );+ }+ }++ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ const transformer = new TenancyInjectionTransformer(this.ctx);+ const injected = transformer.transformNode(args.node) as RootOperationNode;+ rejectUnscopedRawSql(injected);+ return stampTenancy(injected);+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }+}++class TenancyInjectionTransformer extends ClickHouseOperationNodeTransformer {+ constructor(private readonly ctx: ExecutionContext) {+ super();+ }++ protected override transformSelectQuery(+ node: SelectQueryNode,+ queryId?: QueryId,+ ): SelectQueryNode {+ const withChildren = super.transformSelectQuery(+ node,+ queryId,+ ) as ClickHouseSelectQueryNode;+ return injectSelect(withChildren, this.ctx, collectCteNames(withChildren));+ }+}++function collectCteNames(node: SelectQueryNode): Set<string> {+ const names = new Set<string>();+ for (const expr of node.with?.expressions ?? []) {+ names.add(tableNameOf(expr.name.table));+ }+ return names;+}++function injectSelect(+ node: ClickHouseSelectQueryNode,+ ctx: ExecutionContext,+ cteNames: Set<string>,+): ClickHouseSelectQueryNode {+ const fromRelations = (node.from?.froms ?? []).map(describeRelation);+ const joinRelations = (node.joins ?? []).map((join) => ({+ join,+ relation: describeRelation(join.table),+ }));++ for (const relation of fromRelations) {+ assertRelationAllowed(relation);+ }+ for (const { relation } of joinRelations) {+ assertRelationAllowed(relation);+ }++ const isTenantedTable = (+ r: Relation,+ ): r is Extract<Relation, { kind: "table" }> =>+ r.kind === "table" &&+ TENANTED_TABLES.has(r.tableName) &&+ !cteNames.has(r.tableName);++ const tenantedFrom = fromRelations.filter(isTenantedTable);+ const tenantedJoinCount = joinRelations.filter(({ relation }) =>+ isTenantedTable(relation),+ ).length;++ // When more than one tenanted relation is in scope, an unqualified+ // `project_id = β¦` predicate is ambiguous β it cannot be proven to scope a+ // *specific* relation β so a table-qualified reference is required before a+ // relation counts as already covered. With a single tenanted relation an+ // unqualified reference is unambiguous and accepted.+ const requireQualified = tenantedFrom.length + tenantedJoinCount > 1;++ let where = node.where;+ for (const table of tenantedFrom) {+ if (+ !predicateCovers(where?.where, table, ctx.projectId, requireQualified)+ ) {+ const predicate = projectIdPredicate(table, ctx.projectId);+ // Prepend: the tenancy scope is the leading WHERE predicate, so its bound+ // value keeps a stable (first) parameter position regardless of the other+ // predicates a caller wrote.+ where = where+ ? WhereNode.create(AndNode.create(predicate, where.where))+ : WhereNode.create(predicate);+ }+ }++ const joins = joinRelations.map(({ join, relation }) => {+ if (!isTenantedTable(relation)) {+ return join;+ }+ const onExpr = join.on?.on;+ if (+ predicateCovers(onExpr, relation, ctx.projectId, requireQualified) ||+ predicateCovers(where?.where, relation, ctx.projectId, requireQualified)+ ) {+ return join;+ }+ const predicate = projectIdPredicate(relation, ctx.projectId);+ if (!join.on) {+ return JoinNode.createWithOn(join.joinType, join.table, predicate);+ }+ return JoinNode.cloneWithOn(join, predicate);+ });++ return {+ ...node,+ ...(where ? { where } : {}),+ ...(joins.length ? { joins } : {}),+ };+}++function assertRelationAllowed(relation: Relation): void {+ if (relation.kind === "raw") {+ throw new UnscopedRelationError(+ "Raw SQL table sources are rejected: they can introduce an unscoped relation that the tenancy pass cannot prove. Use a traced table reference instead.",+ );+ }+}++function describeRelation(node: OperationNode): Relation {+ if (RawNode.is(node)) {+ return { kind: "raw" };+ }+ if (AliasNode.is(node)) {+ const inner = describeRelation(node.node);+ if (inner.kind === "table") {+ const alias = identifierName(node.alias);+ return { kind: "table", tableName: inner.tableName, alias };+ }+ return inner;+ }+ if (TableNode.is(node)) {+ return { kind: "table", tableName: tableNameOf(node) };+ }+ if (SelectQueryNode.is(node)) {+ return { kind: "subquery", node };+ }+ return { kind: "other" };+}++function tableNameOf(node: TableNode): string {+ return node.table.identifier.name;+}++function identifierName(node: OperationNode): string | undefined {+ if (IdentifierNode.is(node)) return node.name;+ return undefined;+}++function projectIdPredicate(+ table: Extract<Relation, { kind: "table" }>,+ projectId: string,+): OperationNode {+ const column = ColumnNode.create(PROJECT_ID_COLUMN);+ const left = table.alias+ ? ReferenceNode.create(column, TableNode.create(table.alias))+ : column;+ return BinaryOperationNode.create(+ left,+ OperatorNode.create("="),+ ValueNode.create(projectId),+ );+}++/**+ * True only when `expr` provably constrains `table` to `projectId`. Both halves+ * matter: the left operand must be `table`'s `project_id` column (qualified when+ * {@link requireQualified}, see {@link injectSelect}) *and* the right operand+ * must be the literal `projectId` from the {@link ExecutionContext}. A predicate+ * such as `project_id = <someOtherProject>` or `o.project_id = t.project_id`+ * does not scope the relation to the request's project, so it is not covered and+ * the pass injects the correct predicate.+ */+function predicateCovers(+ expr: OperationNode | undefined,+ table: Extract<Relation, { kind: "table" }>,+ projectId: string,+ requireQualified: boolean,+): boolean {+ if (!expr) return false;+ if (AndNode.is(expr)) {+ return (+ predicateCovers(expr.left, table, projectId, requireQualified) ||+ predicateCovers(expr.right, table, projectId, requireQualified)+ );+ }+ if (OrNode.is(expr)) {+ return (+ predicateCovers(expr.left, table, projectId, requireQualified) &&+ predicateCovers(expr.right, table, projectId, requireQualified)+ );+ }+ if (ParensNode.is(expr)) {+ return predicateCovers(expr.node, table, projectId, requireQualified);+ }+ if (!BinaryOperationNode.is(expr)) return false;+ if (!OperatorNode.is(expr.operator) || expr.operator.operator !== "=") {+ return false;+ }+ return (+ isProjectIdColumn(expr.leftOperand, table, requireQualified) &&+ isProjectIdValue(expr.rightOperand, projectId)+ );+}++function isProjectIdColumn(+ node: OperationNode,+ table: Extract<Relation, { kind: "table" }>,+ requireQualified: boolean,+): boolean {+ if (ColumnNode.is(node)) {+ // Bare column with no table qualifier: accept only when unambiguous.+ return !requireQualified && node.column.name === PROJECT_ID_COLUMN;+ }+ if (ReferenceNode.is(node) && ColumnNode.is(node.column)) {+ if (node.column.column.name !== PROJECT_ID_COLUMN) return false;+ if (!node.table) return !requireQualified;+ const referenced = tableNameOf(node.table);+ // Once a relation is aliased, its physical table name is no longer a valid+ // SQL qualifier for it β only the alias is. Accepting the physical name+ // here would let one relation's predicate "cover" a *different* relation+ // whose physical name collides with this alias (e.g. `scores AS traces`+ // joined to `traces AS t`), leaving the second relation unscoped. Mirror+ // projectIdPredicate: alias when present, else the table name.+ return table.alias+ ? referenced === table.alias+ : referenced === table.tableName;+ }+ return false;+}++function isProjectIdValue(node: OperationNode, projectId: string): boolean {+ return ValueNode.is(node) && node.value === projectId;+}++export function requireExecutionContext(+ ctx: ExecutionContext | undefined,+): ExecutionContext {+ if (!ctx?.projectId) {+ throw new QueryCompileError(+ "ExecutionContext is required: a query with no tenancy scope cannot compile.",+ );+ }+ return ctx;+}++function stampTenancy<T extends object>(node: T): T {+ TENANCY_STAMPED.add(node);+ return node;+}++export function assertTenancyStamped(node: RootOperationNode): void {+ if (!TENANCY_STAMPED.has(node)) {+ throw new QueryCompileError(+ "Refusing to compile: the tenancy injection pass was not applied. Compile through compileClickhouseQuery() with an ExecutionContext.",+ );+ }+}++const RAW_RELATION_SQL = /\b(?:from|join|select)\b/i;++function rejectUnscopedRawSql(node: unknown): void {+ if (!node || typeof node !== "object") return;+ if (Array.isArray(node)) {+ for (const item of node) rejectUnscopedRawSql(item);+ return;+ }+ if (RawNode.is(node as OperationNode)) {+ const fragments = (node as RawNode).sqlFragments;+ if (fragments.some((fragment) => RAW_RELATION_SQL.test(fragment))) {+ throw new UnscopedRelationError(+ "Raw SQL fragments that introduce a relation (SELECT/FROM/JOIN) are rejected in any position: they can bypass tenancy injection. Use traced table and column references instead.",+ );+ }+ }+ for (const value of Object.values(node)) {+ rejectUnscopedRawSql(value);+ }+}diff --git a/packages/shared/src/server/query-ast/kysely/transformer.ts b/packages/shared/src/server/query-ast/kysely/transformer.tsnew file mode 100644index 0000000000..732b6ee430--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/transformer.ts@@ -0,0 +1,84 @@+import {+ OperationNodeTransformer,+ type OperationNode,+ type QueryId,+ type SelectQueryNode,+} from "kysely";++import {+ ArrayJoinNode,+ ArrayIndexNode,+ LimitByNode,+ type ClickHouseSelectQueryNode,+} from "./nodes";++/**+ * Transformer that knows the ClickHouse extra fields on SelectQueryNode.+ *+ * Kysely's default {@link OperationNodeTransformer.transformSelectQuery}+ * reconstructs a SelectQueryNode from known slots and drops anything else β+ * so without this override, ARRAY JOIN / LIMIT BY / the tenancy stamp would+ * vanish the moment any plugin walked the tree.+ */+export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer {+ protected override transformNodeImpl<T extends OperationNode>(+ node: T,+ queryId?: QueryId,+ ): T {+ if (ArrayIndexNode.is(node)) {+ // Anti-pattern: cast through `unknown`. ArrayIndexNode is a custom kind+ // outside Kysely's closed OperationNode union, so `transformArrayIndex`'s+ // return type cannot be proven assignable to the generic `T` the base+ // method is parameterized over. The `.is` guard above establishes at+ // runtime that `T` is ArrayIndexNode here, so the cast is sound.+ return this.transformArrayIndex(node) as unknown as T;+ }+ return super.transformNodeImpl(node, queryId);+ }++ protected transformArrayIndex(node: ArrayIndexNode): ArrayIndexNode {+ return ArrayIndexNode.create(+ this.transformNode(node.array),+ this.transformNode(node.index),+ );+ }++ protected override transformSelectQuery(+ node: SelectQueryNode,+ queryId?: QueryId,+ ): SelectQueryNode {+ const transformed = super.transformSelectQuery(+ node,+ queryId,+ ) as ClickHouseSelectQueryNode;+ const extra = node as ClickHouseSelectQueryNode;+ return {+ ...transformed,+ ...(extra.arrayJoins+ ? {+ arrayJoins: extra.arrayJoins.map((n) => this.transformArrayJoin(n)),+ }+ : {}),+ ...(extra.limitBy+ ? { limitBy: this.transformLimitBy(extra.limitBy) }+ : {}),+ } as ClickHouseSelectQueryNode;+ }++ protected transformArrayJoin(node: ArrayJoinNode): ArrayJoinNode {+ return ArrayJoinNode.create(+ node.items.map((item) => ({+ expression: this.transformNode(item.expression),+ alias: this.transformNode(item.alias) as typeof item.alias,+ })),+ node.variant,+ );+ }++ protected transformLimitBy(node: LimitByNode): LimitByNode {+ return LimitByNode.create(+ this.transformNode(node.count),+ node.columns.map((column) => this.transformNode(column)),+ );+ }+}diff --git a/packages/shared/src/server/query-ast/kysely/typecheck.ts b/packages/shared/src/server/query-ast/kysely/typecheck.tsnew file mode 100644index 0000000000..2a4034ba7b--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/typecheck.ts@@ -0,0 +1,61 @@+import {+ AggregateFunctionNode,+ ColumnNode,+ ReferenceNode,+ type OperationNode,+} from "kysely";++import { TypeCompatibilityError } from "./errors";+import { COLUMN_DATA_TYPES } from "./schema";++// Kysely already type-checks comparisons against the schema; `sum()` / `avg()`+// do not constrain the argument column, so this pass catches those. Column data+// types come from the table registry in `schema.ts`.++const NUMERIC_AGGREGATES = new Set(["sum", "avg"]);++/**+ * Runtime validation pass: numeric aggregates over non-numeric columns.+ * Walks every node object (including ClickHouse extra fields).+ */+export function validateTypeCompatibility(root: OperationNode): void {+ walk(root, (node) => {+ if (!AggregateFunctionNode.is(node)) return;+ if (!NUMERIC_AGGREGATES.has(node.func.toLowerCase())) return;+ for (const arg of node.aggregated) {+ const column = columnNameOf(arg);+ if (!column) continue;+ const dataType = COLUMN_DATA_TYPES[column];+ if (dataType && dataType !== "number") {+ throw new TypeCompatibilityError(+ `Aggregate ${node.func}() is not compatible with ${dataType} column "${column}"`,+ );+ }+ }+ });+}++function columnNameOf(node: OperationNode): string | undefined {+ if (ColumnNode.is(node)) return node.column.name;+ if (ReferenceNode.is(node) && ColumnNode.is(node.column)) {+ return node.column.column.name;+ }+ return undefined;+}++function walk(value: unknown, visit: (node: OperationNode) => void): void {+ if (!value || typeof value !== "object") return;+ if (+ "kind" in value &&+ typeof (value as { kind: unknown }).kind === "string"+ ) {+ visit(value as OperationNode);+ }+ for (const child of Object.values(value)) {+ if (Array.isArray(child)) {+ for (const item of child) walk(item, visit);+ } else {+ walk(child, visit);+ }+ }+}diff --git a/packages/shared/src/server/query-ast/kysely/types.assert.ts b/packages/shared/src/server/query-ast/kysely/types.assert.tsnew file mode 100644index 0000000000..4083d53e50--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/types.assert.ts@@ -0,0 +1,88 @@+import { type InferResult } from "kysely";++import { getClickhouseKysely } from "./dialect";+import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";+import { defineView, fromView } from "./views";++/**+ * Compile-time assertions that the typed schema and virtual views enforce their+ * guarantees at the type layer. Never called at runtime; `tsc` is the test, and+ * every `@ts-expect-error` must stay a real (unused-error-free) error.+ *+ * Which layer catches what:+ * - DateTime vs Int comparison: TS types (Kysely OperandValueExpression)+ * - unknown column name: TS types+ * - view exposing only declared columns: TS types+ * - sum() over a String column: NOT caught by TS (Kysely's sum() argument+ * is any ReferenceExpression) β caught by the runtime validation pass+ * - column-name auto-complete: yes, from ClickHouseDatabase schema keys+ */+export function schemaTypeAssertions(): void {+ const db = getClickhouseKysely();++ db.selectFrom("events_core")+ .select("environment")+ // @ts-expect-error DateTime column compared to Int+ .where("start_time", "=", 123);++ db.selectFrom("events_core")+ .select("environment")+ .where("start_time", ">=", new Date("2026-01-01T00:00:00.000Z"));++ db.selectFrom("events_core").select((eb) => eb.fn.sum("total_cost").as("s"));++ // Intentionally valid at the TS layer β Kysely does not constrain sum()+ // to numeric columns. The runtime pass in typecheck.ts rejects this.+ db.selectFrom("events_core").select((eb) => eb.fn.sum("environment").as("s"));++ db.selectFrom("events_core")+ // @ts-expect-error unknown column is not in the schema+ .select("not_a_column");++ const view = defineView("environments_view")<{ environment: string }>(() =>+ db.selectFrom("events_core").select("environment").distinct(),+ );+ fromView(view).select("environment");+ fromView(view)+ // @ts-expect-error view is a black box: inner columns are not exposed+ .select("span_id");+}++/**+ * Compile-time assertions that the `$call` extension helpers infer correctly:+ * `arrayJoin` widens the output row type with each declared alias, and `limitBy`+ * preserves it. `tsc` is the test.+ */+export function extensionTypeAssertions(): void {+ const db = getClickhouseKysely();++ const _widened = db+ .selectFrom("observations")+ .select("environment")+ .$call(+ arrayJoin({+ cost_key: mapKeys("cost_details"),+ cost: mapValues("cost_details"),+ }),+ );+ type Widened = InferResult<typeof _widened>[number];+ // Each arrayJoin alias is added to the output row, so it is referenceable+ // downstream (e.g. an outer query over a CTE body) and a typo on the alias+ // name is a compile error (indexing an absent key fails). The element value+ // type is opaque β Kysely's Expression hides its type arg from inference β+ // so it surfaces as unknown.+ type _Key = Widened["cost_key"];+ type _Cost = Widened["cost"];+ // @ts-expect-error an alias the arrayJoin did not declare is absent+ type _Missing = Widened["not_joined"];++ // limitBy preserves the row type: it declares no new columns.+ const _limited = db+ .selectFrom("events_core")+ .select("span_id")+ .$call(limitBy({ count: 1, columns: ["span_id"] }));+ type Limited = InferResult<typeof _limited>[number];+ type _Span = Limited["span_id"];+ // @ts-expect-error limitBy adds no columns+ type _NoExtra = Limited["cost_key"];+}diff --git a/packages/shared/src/server/query-ast/kysely/views.ts b/packages/shared/src/server/query-ast/kysely/views.tsnew file mode 100644index 0000000000..103b80dc39--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/views.ts@@ -0,0 +1,110 @@+import {+ AliasNode,+ CommonTableExpressionNameNode,+ CommonTableExpressionNode,+ Kysely,+ TableNode,+ WithNode,+ type KyselyPlugin,+ type OperationNode,+ type PluginTransformQueryArgs,+ type PluginTransformResultArgs,+ type RootOperationNode,+ type SelectQueryBuilder,+ type SelectQueryNode,+} from "kysely";++import type { ClickhouseCompilable } from "./compile";+import { ClickHouseCompileDialect } from "./dialect";+import { ClickHouseOperationNodeTransformer } from "./transformer";++export type ClickHouseView<N extends string, O> = {+ name: N;+ build: () => ClickhouseCompilable;+ /** Phantom: the view's exposed row type. */+ readonly $row?: O;+};++/**+ * Name a compilable query as a black-box relation. The outer query can only+ * see `O` (the view's selected columns), never the inner physical schema.+ * Implemented as a plugin that rewrites `selectFrom(viewName)` into a WITH CTE;+ * tenancy injection still walks the CTE body.+ */+export function defineView<N extends string>(name: N) {+ return <O>(build: () => ClickhouseCompilable): ClickHouseView<N, O> => ({+ name,+ build,+ });+}++class VirtualViewPlugin implements KyselyPlugin {+ private readonly views: Map<string, ClickHouseView<string, unknown>>;++ constructor(+ views: ClickHouseView<string, unknown> | ClickHouseView<string, unknown>[],+ ) {+ const list = Array.isArray(views) ? views : [views];+ this.views = new Map(list.map((view) => [view.name, view]));+ }++ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ if (args.node.kind !== "SelectQueryNode") return args.node;+ const transformer = new ClickHouseOperationNodeTransformer();+ const node = transformer.transformNode(args.node) as SelectQueryNode;++ const froms = node.from?.froms ?? [];+ const viewNames = froms+ .map(tableNameOfFrom)+ .filter((name): name is string => Boolean(name && this.views.has(name)));++ if (viewNames.length === 0) return node;++ let withNode = node.with;+ for (const name of viewNames) {+ const def = this.views.get(name)!;+ const inner = def.build().toOperationNode();+ const cte = CommonTableExpressionNode.create(+ CommonTableExpressionNameNode.create(name),+ inner,+ );+ withNode = withNode+ ? WithNode.cloneWithExpression(withNode, cte)+ : WithNode.create(cte);+ }++ return { ...node, with: withNode };+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }+}++function tableNameOfFrom(node: OperationNode): string | undefined {+ if (TableNode.is(node)) {+ return node.table.identifier.name;+ }+ if (AliasNode.is(node)) {+ return tableNameOfFrom(node.node);+ }+ return undefined;+}++/**+ * Open a query against a named view as if it were a table. Only the view's+ * exposed columns are in the type.+ */+export function fromView<N extends string, O>(+ view: ClickHouseView<N, O>,+): SelectQueryBuilder<Record<N, O>, N, Record<string, never>> {+ type DB = Record<N, O>;+ const db = new Kysely<DB>({ dialect: new ClickHouseCompileDialect() });+ return db+ .withPlugin(new VirtualViewPlugin(view as ClickHouseView<string, unknown>))+ .selectFrom(view.name) as SelectQueryBuilder<+ Record<N, O>,+ N,+ Record<string, never>+ >;+}diff --git a/packages/shared/src/server/repositories/environments.ts b/packages/shared/src/server/repositories/environments.tsindex 42d80140b9..ffbd67982d 100644--- a/packages/shared/src/server/repositories/environments.ts+++ b/packages/shared/src/server/repositories/environments.ts@@ -1,5 +1,8 @@import { LISTABLE_SCORE_TYPES } from "../../domain/scores";import { env } from "../../env";+import type { ExecutionContext } from "../query-ast/executionContext";+import { compileClickhouseQuery } from "../query-ast/kysely/compile";+import { getClickhouseKysely } from "../query-ast/kysely/dialect";import { queryClickhouse } from "./clickhouse";export type EnvironmentFilterProps = {@@ -11,54 +14,69 @@ export const getEnvironmentsForProject = async (props: EnvironmentFilterProps,): Promise<{ environment: string }[]> => {const { projectId, fromTimestamp } = props;+ const writeMode = env.LANGFUSE_MIGRATION_V4_WRITE_MODE;+ const db = getClickhouseKysely();+ // Tenancy is applied at compile time: compileClickhouseQuery injects+ // `project_id = {projectId}` into every tenanted relation and refuses to+ // compile an unscoped query, so the query bodies below carry no project_id+ // filter of their own.+ const ctx: ExecutionContext = { projectId };- // In dual and events_only write modes all tracing data lands in the events- // tables: a single events_core scan covers traces and observations and is- // the only populated source under events_only. Scores keep their own table- // in every write mode. The events read may be routed to a dedicated- // ClickHouse service (CLICKHOUSE_EVENTS_READ_ONLY_URL), so it cannot share- // a query with the scores read.- const tracingEnvironmentsPromise =- env.LANGFUSE_MIGRATION_V4_WRITE_MODE === "legacy"- ? queryClickhouse<{ environment: string }>({- query: `- (- SELECT distinct environment- FROM traces- WHERE project_id = {projectId: String}- ${fromTimestamp ? "AND timestamp >= {fromTimestamp: DateTime64(3)}" : ""}- ) UNION ALL (- SELECT distinct environment- FROM observations- WHERE project_id = {projectId: String}- ${fromTimestamp ? "AND start_time >= {fromTimestamp: DateTime64(3)}" : ""}- )- `,- params: { projectId, fromTimestamp },- tags: { projectId, route: "environments.getEnvironmentsForProject" },- preferredClickhouseService: "ReadOnly",- })- : queryClickhouse<{ environment: string }>({- query: `- SELECT distinct environment- FROM events_core- WHERE project_id = {projectId: String}- ${fromTimestamp ? "AND start_time >= {fromTimestamp: DateTime64(3)}" : ""}- `,- params: { projectId, fromTimestamp },- tags: { projectId, route: "environments.getEnvironmentsForProject" },- preferredClickhouseService: "EventsReadOnly",- });+ // In dual/events_only write modes all tracing data lands in events_core (the+ // only populated source under events_only), so one scan covers traces and+ // observations. Legacy still reads the two separate tables and unions them.+ const tracingQuery =+ writeMode === "legacy"+ ? db+ .selectFrom("traces")+ .select("environment")+ .distinct()+ .$if(fromTimestamp != null, (qb) =>+ qb.where("timestamp", ">=", fromTimestamp!),+ )+ .unionAll(+ db+ .selectFrom("observations")+ .select("environment")+ .distinct()+ .$if(fromTimestamp != null, (qb) =>+ qb.where("start_time", ">=", fromTimestamp!),+ ),+ )+ : db+ .selectFrom("events_core")+ .select("environment")+ .distinct()+ .$if(fromTimestamp != null, (qb) =>+ qb.where("start_time", ">=", fromTimestamp!),+ );++ const scoresQuery = db+ .selectFrom("scores")+ .select("environment")+ .distinct()+ .where("data_type", "in", [...LISTABLE_SCORE_TYPES])+ .$if(fromTimestamp != null, (qb) =>+ qb.where("timestamp", ">=", fromTimestamp!),+ );++ const tracing = compileClickhouseQuery(tracingQuery, ctx);+ const scores = compileClickhouseQuery(scoresQuery, ctx);++ // The events read may route to a dedicated ClickHouse service+ // (CLICKHOUSE_EVENTS_READ_ONLY_URL) while scores always read the primary, so+ // the two cannot share one query.+ const tracingEnvironmentsPromise = queryClickhouse<{ environment: string }>({+ query: tracing.sql,+ params: tracing.params,+ tags: { projectId, route: "environments.getEnvironmentsForProject" },+ preferredClickhouseService:+ writeMode === "legacy" ? "ReadOnly" : "EventsReadOnly",+ });const scoreEnvironmentsPromise = queryClickhouse<{ environment: string }>({- query: `- SELECT distinct environment- FROM scores- WHERE project_id = {projectId: String}- AND data_type IN ({dataTypes: Array(String)})- ${fromTimestamp ? "AND timestamp >= {fromTimestamp: DateTime64(3)}" : ""}- `,- params: { projectId, fromTimestamp, dataTypes: LISTABLE_SCORE_TYPES },+ query: scores.sql,+ params: scores.params,tags: { projectId, route: "environments.getEnvironmentsForProject" },preferredClickhouseService: "ReadOnly",});@@ -67,7 +85,8 @@ export const getEnvironmentsForProject = async (await Promise.all([tracingEnvironmentsPromise, scoreEnvironmentsPromise])).flat();- // Always add default environment to list+ // "default" always exists as a selectable environment even when no rows have+ // been written under it, so it is not guaranteed to appear in the scan.results.push({ environment: "default" });return Array.from(new Set(results.map((e) => e.environment))).map(diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 579771d605..5837a3dd22 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -380,6 +380,9 @@ importers:jsonpath-plus:specifier: 10.3.0version: 10.3.0+ kysely:+ specifier: 0.28.17+ version: 0.28.17langfuse:specifier: 3.38.4version: 3.38.4@@ -8659,6 +8662,10 @@ packages:resolution: {integrity: sha512-Xq9nH7KlWZmXAtodXDDRE7vs6DU1gTU8zYDHDiWLSip45Egwq3plLHzPn27NgvzL2r1LMPC1vdqh98sQxtqj4A==}+ resolution: {integrity: sha512-nbD8lB9EB3wNdMhOCdx5Li8DxnLbvKByylRLcJ1h+4SkrowVeECAyZlyiKMThF7xFdRz0jSQ2MoJr+wXux2y0Q==}+ engines: {node: '>=20.0.0'}+resolution: {integrity: sha512-zBKVmQN/1oT5VWZUBYlWzvokIlkC/6mnpgr/2atMyTeAm+jR3ia7w2iJMjlrF5/oG8ukO1s8+LDRCzJpF1QeEA==}engines: {node: '>=18'}@@ -19584,6 +19591,8 @@ snapshots:+ [email protected]: {}+dependencies:mustache: 4.2.0diff --git a/turbo.json b/turbo.jsonindex 433fed3eb7..ce122ceaa7 100644--- a/turbo.json+++ b/turbo.json@@ -5,7 +5,8 @@"globalEnv": ["NEXT_PUBLIC_LANGFUSE_BLOB_EXPORT_CUTOFF","NEXT_PUBLIC_LANGFUSE_BLOB_EXPORTER_CUTOFF",- "NEXT_PUBLIC_LANGFUSE_ANALYTICS_EXPORTER_CUTOFF"+ "NEXT_PUBLIC_LANGFUSE_ANALYTICS_EXPORTER_CUTOFF",+ "CLICKHOUSE_BIN"],"envMode": "loose","tasks": {