πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
refactor(query-ast): take a bare projectId in compileClickhouseQuery (#18817)
Niklas Semmler46debfeb6560
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.mdindex b3a5bd1dca..5cf2159109 100644--- a/packages/shared/AGENTS.md+++ b/packages/shared/AGENTS.md@@ -48,7 +48,7 @@- Dashboard/monitor query feature (data model + server-only builder/executor): `src/features/query/*`- Query-builder AST (server half, WIP): `src/server/query-ast/*` β the KyselyClickHouse dialect (ARRAY JOIN / LIMIT BY / FINAL / metadata subscript nodes,- `ExecutionContext` tenancy injection, per-table soft-delete filtering and+ single-project tenancy injection, per-table soft-delete filtering anddedup lowering, virtual views, catalog parity). Compile only through`compileClickhouseQuery` in `src/server/query-ast/compile.ts`. SQLcorrectness is proven by a golden-SQL harnessdiff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex ec2c321366..a2185e07de 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -9,7 +9,7 @@ metadata, `FilterState` embedding) will land underThis folder is a compile-only ClickHouse dialect on Kysely 0.28: real`OperationNode`s for ARRAY JOIN, LIMIT BY, FINAL, and metadata map-subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;+subscripts; a mandatory tenancy injection pass keyed on one `projectId`;per-table soft-delete filtering and dedup lowering from the registry;schema-typed selection; virtual views as WITH CTEs; catalog parity.@@ -20,8 +20,8 @@ changing how it emits SQL.## Layout- `compile.ts` β the one supported entry point,- `compileClickhouseQuery(query, ctx)`, and the `ExecutionContext` re-export.-- `tenancy.ts` β the `ExecutionContext` type and the tenancy injection pass.+ `compileClickhouseQuery(query, projectId)`.+- `tenancy.ts` β the tenancy injection pass.- `schema.ts` β the physical table registry: one `defineTable` declaration perrelation drives the Kysely row types (`ClickHouseDatabase`), the runtimecolumn-type map (`COLUMN_DATA_TYPES`), the tenanted-table set@@ -75,15 +75,14 @@ suffix to run there. Promote it to a required check once it has proven stable.## Using it-- **Compile only through `compileClickhouseQuery(query, ctx)`.** It is the one- supported path from a builder to `{ sql, params }`; the repository layer hands- that to `queryClickhouse`. `ctx` (an `ExecutionContext` carrying exactly one- `projectId`) is required β omitting it is a compile-time type error and- an empty one throws. There is no multi-project scope: a query reads one- project.+- **Compile only through `compileClickhouseQuery(query, projectId)`.** It is+ the one supported path from a builder to `{ sql, params }`; the repository+ layer hands that to `queryClickhouse`. `projectId` is required β omitting it+ is a compile-time type error and an empty one throws. There is no+ multi-project scope: a query reads one project.- **Never filter `project_id` yourself.** The compile step injects`project_id = {projectId}` into every tenanted relation, so call sites pass- only `{ projectId }` (see `../repositories/environments.ts`).+ only `projectId` (see `../repositories/environments.ts`).- **Never filter `is_deleted` on a physical table yourself.** Tables thatdeclare `softDelete` in the registry (`events_core`, `events_full`) get`is_deleted = 0` on every FROM/JOIN. A select that must see deleted rows@@ -183,16 +182,16 @@ not patched.`getClickhouseKysely()` returns a `Kysely` instance wired to a `DummyDriver`with SQLite adapter/introspector stand-ins (`dialect.ts`). Nothing everconnects or executes. The only supported output path is-`compileClickhouseQuery(query, ctx)` β `{ sql, params }`, which the repository-layer hands to `queryClickhouse`. Do **not** call `.execute()` / `.compile()`+`compileClickhouseQuery(query, projectId)` β `{ sql, params }`, which the+repository layer hands to `queryClickhouse`. Do **not** call `.execute()` / `.compile()`directly: the compiler refuses any tree that did not go through the tenancy pass.### Tenancy injection β how the choke point works-`compileClickhouseQuery(query, ctx)` is the only supported compile path, and it-is where tenancy is enforced:+`compileClickhouseQuery(query, projectId)` is the only supported compile path,+and it is where tenancy is enforced:-1. A missing/empty `ExecutionContext` throws (`QueryCompileError`) β `ctx` is a+1. A missing/empty `projectId` throws (`QueryCompileError`) β `projectId` is arequired parameter, so omitting it is also a compile-time type error.2. `SoftDeleteFilterPlugin` adds `<flag> = 0` for every FROM/JOIN relationwhose table declares `softDelete`, except in selects marked with@@ -201,7 +200,7 @@ is where tenancy is enforced:3. `TenancyInjectionPlugin` walks every FROM/JOIN and injects`project_id = {projectId}` on each tenanted physical table, unless the treealready carries a predicate that _proves_ that scope: the equality's value- must equal the context project, and β when more than one tenanted relation is+ must equal `projectId`, and β when more than one tenanted relation isin scope β the column must be table-qualified. A qualified predicate covers arelation only when the qualifier matches its alias (if aliased) or its tablename (if not) β so `scores AS traces` joined to `traces AS t` still scopesdiff --git a/packages/shared/src/server/query-ast/binds.test.ts b/packages/shared/src/server/query-ast/binds.test.tsindex b552184eed..4e62bb23ed 100644--- a/packages/shared/src/server/query-ast/binds.test.ts+++ b/packages/shared/src/server/query-ast/binds.test.ts@@ -6,11 +6,11 @@ import { compileClickhouseQuery } from "./compile";import { getClickhouseKysely } from "./dialect";import { tupleIn, useFinal } from "./extensions";-const ctx = { projectId: "proj-1" };+const projectId = "proj-1";const START = new Date("2026-01-01T00:00:00.000Z");function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {- return compileClickhouseQuery(builder, ctx);+ return compileClickhouseQuery(builder, projectId);}describe("typed params from the column registry", () => {diff --git a/packages/shared/src/server/query-ast/catalog.golden.test.ts b/packages/shared/src/server/query-ast/catalog.golden.test.tsindex c8b766f569..5af8999ae0 100644--- a/packages/shared/src/server/query-ast/catalog.golden.test.ts+++ b/packages/shared/src/server/query-ast/catalog.golden.test.ts@@ -34,9 +34,10 @@ if (!clickhouseLocalAvailable()) {describeWithClickhouse("catalog parity", () => {for (const entry of CATALOG) {it(`${entry.id} (tier ${entry.tier}) compile(AST) β‘ referenceSQL`, () => {- const compiled = compileClickhouseQuery(entry.build(), {- projectId: CATALOG_PROJECT_ID,- });+ const compiled = compileClickhouseQuery(+ entry.build(),+ CATALOG_PROJECT_ID,+ );const left = normalizeParams(formatSql(compiled.sql), compiled.params);const right = normalizeParams(@@ -72,9 +73,7 @@ describeWithClickhouseLocal("catalog JOIN execution", () => {expect(entry).toBeDefined();if (!entry) return;- const compiled = compileClickhouseQuery(entry.build(), {- projectId: CATALOG_PROJECT_ID,- });+ const compiled = compileClickhouseQuery(entry.build(), CATALOG_PROJECT_ID);const refs = compiled.sql.match(/[\w.]*project_id/gi) ?? [];expect(refs.length).toBeGreaterThanOrEqual(3);for (const ref of refs) {@@ -154,9 +153,10 @@ describeWithClickhouseLocal("catalog tier 5-7 execution", () => {expect(entry).toBeDefined();if (!entry) return;- const compiled = compileClickhouseQuery(entry.build(), {- projectId: CATALOG_PROJECT_ID,- });+ const compiled = compileClickhouseQuery(+ entry.build(),+ CATALOG_PROJECT_ID,+ );const executable = substituteNamedParams(compiled.sql, compiled.params);executeClickhouseLocal(`${CATALOG_MEMORY_DDL}\n${executable}`);});@@ -169,9 +169,7 @@ describeWithClickhouseLocal("catalog metadata semantics", () => {expect(entry).toBeDefined();if (!entry) return;- const compiled = compileClickhouseQuery(entry.build(), {- projectId: CATALOG_PROJECT_ID,- });+ const compiled = compileClickhouseQuery(entry.build(), CATALOG_PROJECT_ID);const executable = substituteNamedParams(compiled.sql, compiled.params);const output = executeClickhouseLocal(`${CATALOG_MEMORY_DDL}INSERT INTO events_core (project_id, metadata_names, metadata_values)diff --git a/packages/shared/src/server/query-ast/compile.ts b/packages/shared/src/server/query-ast/compile.tsindex 6f8cb12ce4..1e75dc5c7e 100644--- a/packages/shared/src/server/query-ast/compile.ts+++ b/packages/shared/src/server/query-ast/compile.ts@@ -9,15 +9,12 @@ import { ClickHouseQueryCompiler } from "./compiler";import { DedupLoweringPlugin } from "./dedup";import { SoftDeleteFilterPlugin } from "./softDelete";import {- type ExecutionContext,TenancyInjectionPlugin,assertRelationFree,- requireExecutionContext,+ requireProjectId,} from "./tenancy";import { validateTypeCompatibility } from "./typecheck";-export type { ExecutionContext } from "./tenancy";-export type CompiledClickhouseQuery = {sql: string;params: Record<string, unknown>;@@ -34,21 +31,21 @@ export type ClickhouseCompilable = {/*** The choke point. Every query that should produce ClickHouse SQL goes- * through here: the `ctx` parameter is required, so a caller cannot compile- * without a tenancy scope β omitting it is a compile-time type error.- * Soft-deleted rows are filtered out, tenancy is injected from that scope,- * per-table dedup is applied from the registry, and the dialect compiler- * refuses to emit SQL unless the tree was identity-stamped by the tenancy and- * dedup passes (a copied property is not enough). The runtime- * `requireExecutionContext` guard remains as defense-in-depth for the- * one case the type cannot express (an empty `projectId`) and for callers- * reaching this through an `any` boundary.+ * through here: the `projectId` parameter is required, so a caller cannot+ * compile without a tenancy scope β omitting it is a compile-time type error.+ * The query reads exactly that one project. Soft-deleted rows are filtered+ * out, tenancy is injected from that project, per-table dedup is applied from+ * the registry, and the dialect compiler refuses to emit SQL unless the tree+ * was identity-stamped by the tenancy and dedup passes (a copied property is+ * not enough). The runtime `requireProjectId` guard remains as+ * defense-in-depth for the one case the type cannot express (an empty+ * `projectId`) and for callers reaching this through an `any` boundary.*/export function compileClickhouseQuery(query: ClickhouseCompilable,- ctx: ExecutionContext,+ projectId: string,): CompiledClickhouseQuery {- const scope = requireExecutionContext(ctx);+ const scope = requireProjectId(projectId);const stamped = query.withPlugin(new SoftDeleteFilterPlugin()).withPlugin(new TenancyInjectionPlugin(scope))diff --git a/packages/shared/src/server/query-ast/composition.test.ts b/packages/shared/src/server/query-ast/composition.test.tsindex 95f1b5a56c..30a915f051 100644--- a/packages/shared/src/server/query-ast/composition.test.ts+++ b/packages/shared/src/server/query-ast/composition.test.ts@@ -7,7 +7,7 @@ import { defineView, fromView } from "./views";// Raw compiler output (no `clickhouse format` binary needed), so these run// unconditionally in CI. The raw SQL is lowercase; match case-insensitively.-const ctx = { projectId: "p" };+const projectId = "p";const costArrayJoin = () =>arrayJoin({@@ -25,7 +25,7 @@ describe("ARRAY JOIN composition across nesting positions", () => {.select("environment").$call(costArrayJoin());- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);expect(sql).toMatch(/array join/i);expect(sql).toMatch(/mapkeys/i);expect(sql).toMatch(/mapvalues/i);@@ -43,7 +43,7 @@ describe("ARRAY JOIN composition across nesting positions", () => {.selectFrom("aj").select("environment");- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);expect(sql).toMatch(/with aj as/i);// The ARRAY JOIN must live inside the CTE body and not leak into the outer// query. Split on the outer query (the `) select ... from aj` that closes@@ -68,7 +68,7 @@ describe("ARRAY JOIN composition across nesting positions", () => {).select("environment");- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);// Outer query has no ARRAY JOIN; presence proves the subquery clause held.expect(sql).toMatch(/array join/i);});@@ -84,7 +84,7 @@ describe("ARRAY JOIN composition across nesting positions", () => {const qb = fromView(view).select("environment");- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);// Outer query only selects from the view alias; presence proves the view// body's ARRAY JOIN composed into the emitted CTE.expect(sql).toMatch(/array join/i);@@ -101,7 +101,7 @@ describe("LIMIT BY composition across nesting positions", () => {.orderBy("event_ts", "desc").$call(dedup());- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);expect(sql).toMatch(/limit 1 by/i);});@@ -118,7 +118,7 @@ describe("LIMIT BY composition across nesting positions", () => {.selectFrom("lb").select("span_id");- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);// Outer query has no LIMIT BY; presence proves the CTE body's clause held.expect(sql).toMatch(/limit 1 by/i);expect(sql).toMatch(/with lb as/i);@@ -133,7 +133,7 @@ describe("FINAL composition across nesting positions", () => {.select("s.id").$call(useFinal(["scores"]));- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);expect(sql).toMatch(/scores as s final/i);});@@ -149,7 +149,7 @@ describe("FINAL composition across nesting positions", () => {.selectFrom("scored").select("id");- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);const outerStart = sql.toLowerCase().search(/\)\s*select[\s\S]*from scored/i);@@ -172,7 +172,7 @@ describe("FINAL composition across nesting positions", () => {.$call(useFinal(["scores"])).select("s.id");- const { sql } = compileClickhouseQuery(qb, ctx);+ const { sql } = compileClickhouseQuery(qb, projectId);expect(sql).toMatch(/scores as s final/i);expect(sql).not.toMatch(/events_core as e final/i);});diff --git a/packages/shared/src/server/query-ast/dedup.test.ts b/packages/shared/src/server/query-ast/dedup.test.tsindex 52993dd254..4ecd6aa343 100644--- a/packages/shared/src/server/query-ast/dedup.test.ts+++ b/packages/shared/src/server/query-ast/dedup.test.ts@@ -5,10 +5,10 @@ import { getClickhouseKysely } from "./dialect";import { limitBy } from "./extensions";import { DEDUP_SPECS } from "./schema";-const ctx = { projectId: "proj-1" };+const projectId = "proj-1";function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {- return compileClickhouseQuery(builder, ctx);+ return compileClickhouseQuery(builder, projectId);}describe("per-table dedup lowering", () => {diff --git a/packages/shared/src/server/query-ast/extensions.test.ts b/packages/shared/src/server/query-ast/extensions.test.tsindex a0fd7c1845..24bf8d3608 100644--- a/packages/shared/src/server/query-ast/extensions.test.ts+++ b/packages/shared/src/server/query-ast/extensions.test.ts@@ -21,7 +21,7 @@ import type { ClickHouseDatabase } from "./schema";import { TenancyInjectionPlugin } from "./tenancy";import { defineView, fromView } from "./views";-const ctx = { projectId: "proj-1" };+const projectId = "proj-1";describe("schema-typed selection (condition 7)", () => {it("accepts sum() over a numeric column and emits SQL", () => {@@ -29,7 +29,7 @@ describe("schema-typed selection (condition 7)", () => {.selectFrom("events_core").select((eb) => [eb.fn.sum("total_cost").as("s")]);- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled.toLowerCase()).toContain("sum(total_cost)");expect(compiled.toLowerCase()).toContain("project_id");});@@ -39,10 +39,10 @@ describe("schema-typed selection (condition 7)", () => {.selectFrom("events_core").select((eb) => [eb.fn.sum("environment").as("s")]);- expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ expect(() => compileClickhouseQuery(qb, projectId)).toThrow(TypeCompatibilityError,);- expect(() => compileClickhouseQuery(qb, ctx)).toThrow(/environment/);+ expect(() => compileClickhouseQuery(qb, projectId)).toThrow(/environment/);});});@@ -61,7 +61,7 @@ describe("metadata subscript access (condition 7b)", () => {.where((eb) => eb(metadataValue("e", "a"), "=", "x"));const node = qb- .withPlugin(new TenancyInjectionPlugin(ctx))+ .withPlugin(new TenancyInjectionPlugin(projectId)).toOperationNode();const json = JSON.stringify(node);expect(json).toContain('"kind":"ArrayIndexNode"');@@ -82,7 +82,7 @@ describe("metadata subscript access (condition 7b)", () => {.select((_eb) => [metadataValue("e", "a").as("a")]).where((eb) => eb(metadataValue("e", "a"), "=", "x"));- const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled, params } = compileClickhouseQuery(qb, projectId);const lower = compiled.toLowerCase();expect(lower).toContain("mapfromarrays(arrayreverse(e.metadata_names), arrayreverse(e.metadata_values))[",@@ -100,7 +100,7 @@ describe("metadata subscript access (condition 7b)", () => {.groupBy("environment").having((eb) => eb(metadataValue("e", "a"), "=", "x"));- const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled, params } = compileClickhouseQuery(qb, projectId);expect(compiled.toLowerCase()).toContain("having");expect(compiled.toLowerCase()).toContain("mapfromarrays(arrayreverse(e.metadata_names), arrayreverse(e.metadata_values))[",@@ -124,7 +124,7 @@ describe("virtual view (condition 8)", () => {.select("environment").where("environment", "=", "production");- const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled, params } = compileClickhouseQuery(qb, projectId);const lower = compiled.toLowerCase();expect(lower).toContain("with environments_view as");expect(lower).toContain("from events_core");@@ -136,7 +136,7 @@ describe("virtual view (condition 8)", () => {it("injects tenancy on the inner physical table, not the view alias", () => {const qb = fromView(environmentsView).select("environment");- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);const lower = compiled.toLowerCase();expect(lower).toMatch(/with environments_view as \([\s\S]*project_id[\s\S]*\)[\s\S]*from environments_view/,@@ -168,7 +168,7 @@ describe("leftUTF8", () => {.selectFrom("events_core as e").select((eb) => [leftUTF8(eb.ref("e.input"), 100).as("input")]);- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled).toContain("leftUTF8(e.input, 100)");});@@ -200,7 +200,7 @@ describe("arrayMap", () => {).as("sibling"),]);- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled).toContain("arrayMap(v -> arrayMap(v1 -> concat(v, v1), e.tags), e.metadata_names)",);diff --git a/packages/shared/src/server/query-ast/nodes.test.ts b/packages/shared/src/server/query-ast/nodes.test.tsindex 0c6a364b3f..8098e49796 100644--- a/packages/shared/src/server/query-ast/nodes.test.ts+++ b/packages/shared/src/server/query-ast/nodes.test.ts@@ -14,26 +14,26 @@ import {} from "./nodes";import { TenancyInjectionPlugin } from "./tenancy";-const ctx = { projectId: "proj-1" };+const projectId = "proj-1";describe("tenancy injection", () => {- it("requires an ExecutionContext at compile time", () => {+ it("requires a projectId at compile time", () => {const qb = getClickhouseKysely().selectFrom("traces").select("environment").where("project_id", "=", "proj-1");- // The mandatory `ctx` parameter is the primary guard: omitting it is a- // type error. The `@ts-expect-error` both documents and enforces that+ // The mandatory `projectId` parameter is the primary guard: omitting it is+ // a type error. The `@ts-expect-error` both documents and enforces that// contract, and the runtime throw is the defense-in-depth backstop for// callers that reach this through an `any` boundary.- // @ts-expect-error - ctx is required+ // @ts-expect-error - projectId is requiredexpect(() => compileClickhouseQuery(qb)).toThrow(QueryCompileError);- // @ts-expect-error - ctx is required+ // @ts-expect-error - projectId is requiredexpect(() => compileClickhouseQuery(qb)).toThrow(/no tenancy scope/);});- it("refuses a context with an empty projectId", () => {+ it("refuses an empty projectId", () => {const qb = getClickhouseKysely().selectFrom("traces").select("environment")@@ -41,19 +41,21 @@ describe("tenancy injection", () => {// Type-valid but semantically empty: the type system cannot express// "non-empty string", so the runtime check earns its keep here.- expect(() => compileClickhouseQuery(qb, { projectId: "" })).toThrow(- /no tenancy scope/,- );+ expect(() => compileClickhouseQuery(qb, "")).toThrow(/no tenancy scope/);});- it("has no multi-project scope", () => {+ it("refuses a scope that is not a single projectId string", () => {const qb = getClickhouseKysely().selectFrom("events_core").select("environment");expect(() =>// @ts-expect-error - a query is scoped to exactly one project- compileClickhouseQuery(qb, { projectIds: ["proj-a", "proj-b"] }),+ compileClickhouseQuery(qb, ["proj-a", "proj-b"]),+ ).toThrow(/no tenancy scope/);+ expect(() =>+ // @ts-expect-error - the scope is the projectId itself, not a context+ compileClickhouseQuery(qb, { projectId: "proj-a" }),).toThrow(/no tenancy scope/);});@@ -71,7 +73,7 @@ describe("tenancy injection", () => {.selectFrom("events_core").select("environment");- const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled, params } = compileClickhouseQuery(qb, projectId);expect(compiled.toLowerCase()).toContain("project_id");expect(Object.values(params)).toContain("proj-1");});@@ -82,7 +84,7 @@ describe("tenancy injection", () => {.select("environment").where("project_id", "=", "proj-1");- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);const matches = compiled.match(/project_id/gi) ?? [];expect(matches).toHaveLength(1);});@@ -92,7 +94,7 @@ describe("tenancy injection", () => {.selectFrom(sql<{ environment: string }>`traces`.as("t")).select("environment");- expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ expect(() => compileClickhouseQuery(qb, projectId)).toThrow(UnscopedRelationError,);});@@ -102,7 +104,7 @@ describe("tenancy injection", () => {.selectFrom(sql<{ environment: string }>`(SELECT * FROM traces)`.as("t")).selectAll();- expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ expect(() => compileClickhouseQuery(qb, projectId)).toThrow(UnscopedRelationError,);});@@ -120,7 +122,7 @@ describe("tenancy injection", () => {).select("t.environment");- expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ expect(() => compileClickhouseQuery(qb, projectId)).toThrow(UnscopedRelationError,);},@@ -140,7 +142,7 @@ describe("tenancy injection", () => {).select("t.id");- expect(() => compileClickhouseQuery(qb, ctx)).toThrow(+ expect(() => compileClickhouseQuery(qb, projectId)).toThrow(UnscopedRelationError,);},@@ -158,10 +160,10 @@ describe("tenancy injection", () => {.select("environment").where(sql<boolean>`id IN (SELECT id FROM traces)`);- expect(() => compileClickhouseQuery(selectRaw, ctx)).toThrow(+ expect(() => compileClickhouseQuery(selectRaw, projectId)).toThrow(UnscopedRelationError,);- expect(() => compileClickhouseQuery(whereRaw, ctx)).toThrow(+ expect(() => compileClickhouseQuery(whereRaw, projectId)).toThrow(UnscopedRelationError,);});@@ -206,7 +208,7 @@ describe("tenancy injection", () => {.select("o.environment").where("o.project_id", "=", "proj-1");- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);const refs = compiled.match(/[\w.]*project_id/gi) ?? [];expect(refs.length).toBeGreaterThanOrEqual(3);for (const ref of refs) {@@ -222,7 +224,7 @@ describe("tenancy injection", () => {.select("environment").where("project_id", "=", "other-project");- const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled, params } = compileClickhouseQuery(qb, projectId);// The builder's `other-project` predicate does not scope the request's// project, so the pass must still inject `project_id = proj-1`.expect(Object.values(params)).toContain("proj-1");@@ -239,7 +241,7 @@ describe("tenancy injection", () => {// treated as scoping either relation..where("project_id", "=", "proj-1");- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled).toMatch(/o\.project_id/i);expect(compiled).toMatch(/t\.project_id/i);});@@ -256,7 +258,7 @@ describe("tenancy injection", () => {.select("t.environment").where("traces.project_id", "=", "proj-1");- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled).toMatch(/t\.project_id/i);const matches = compiled.match(/project_id/gi) ?? [];expect(matches.length).toBeGreaterThanOrEqual(2);@@ -277,7 +279,7 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {);const node = qb- .withPlugin(new TenancyInjectionPlugin(ctx))+ .withPlugin(new TenancyInjectionPlugin(projectId)).toOperationNode();expect(isClickHouseSelectQueryNode(node)).toBe(true);if (!isClickHouseSelectQueryNode(node)) return;@@ -297,7 +299,7 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {.where("project_id", "=", "proj-1").$call(arrayJoin({ cost_key: mapKeys("cost_details") }));- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled.toLowerCase()).toContain("array join");expect(compiled.toLowerCase()).toContain("mapkeys(cost_details) as cost_key",@@ -312,7 +314,7 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {.$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));const node = qb- .withPlugin(new TenancyInjectionPlugin(ctx))+ .withPlugin(new TenancyInjectionPlugin(projectId)).toOperationNode();expect(isClickHouseSelectQueryNode(node)).toBe(true);if (!isClickHouseSelectQueryNode(node)) return;@@ -330,7 +332,7 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {.limit(50).$call(limitBy({ count: 1, columns: ["span_id"] }));- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);const lower = compiled.toLowerCase();expect(lower).toContain("limit ");expect(lower).toContain(" by ");@@ -344,7 +346,7 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {.$call(useFinal(["scores"]));const node = qb- .withPlugin(new TenancyInjectionPlugin(ctx))+ .withPlugin(new TenancyInjectionPlugin(projectId)).toOperationNode();expect(node.kind).toBe("SelectQueryNode");const from = (node as { from?: { froms?: Array<{ kind: string }> } }).from@@ -359,7 +361,7 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {.select("s.id").$call(useFinal(["scores"]));- const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ const { sql: compiled } = compileClickhouseQuery(qb, projectId);expect(compiled.toLowerCase()).toMatch(/from scores as s final/);expect(compiled.toLowerCase()).not.toMatch(/from scores as s,/);});diff --git a/packages/shared/src/server/query-ast/softDelete.test.ts b/packages/shared/src/server/query-ast/softDelete.test.tsindex fffa723449..77b9eb05b8 100644--- a/packages/shared/src/server/query-ast/softDelete.test.ts+++ b/packages/shared/src/server/query-ast/softDelete.test.ts@@ -15,10 +15,10 @@ import { SOFT_DELETE_COLUMNS } from "./schema";import { SoftDeleteFilterPlugin } from "./softDelete";// Raw compiler output, so these run without a `clickhouse` binary.-const ctx = { projectId: "p" };+const projectId = "p";function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {- return compileClickhouseQuery(builder, ctx).sql;+ return compileClickhouseQuery(builder, projectId).sql;}describe("soft-delete filtering", () => {diff --git a/packages/shared/src/server/query-ast/tenancy.ts b/packages/shared/src/server/query-ast/tenancy.tsindex 10cd8834e7..efe295f20c 100644--- a/packages/shared/src/server/query-ast/tenancy.ts+++ b/packages/shared/src/server/query-ast/tenancy.ts@@ -31,16 +31,6 @@ import {import { TENANTED_TABLES } from "./schema";import { ClickHouseOperationNodeTransformer } from "./transformer";-/**- * Compile-time tenancy scope. Every ClickHouse query compiled through- * {@link compileClickhouseQuery} must carry one of these; the tenancy- * injection pass keys off `projectId`. A query is always scoped to exactly one- * project.- */-export type ExecutionContext = {- projectId: string;-};-const PROJECT_ID_COLUMN = "project_id";/**@@ -68,12 +58,12 @@ type Relation =* not relations and are allowed.*/export class TenancyInjectionPlugin implements KyselyPlugin {- constructor(private readonly ctx: ExecutionContext) {- requireExecutionContext(ctx);+ constructor(private readonly projectId: string) {+ requireProjectId(projectId);}transformQuery(args: PluginTransformQueryArgs): RootOperationNode {- const transformer = new TenancyInjectionTransformer(this.ctx);+ const transformer = new TenancyInjectionTransformer(this.projectId);const injected = transformer.transformNode(args.node) as RootOperationNode;rejectUnscopedRawSql(injected);return stampTenancy(injected);@@ -85,7 +75,7 @@ export class TenancyInjectionPlugin implements KyselyPlugin {}class TenancyInjectionTransformer extends ClickHouseOperationNodeTransformer {- constructor(private readonly ctx: ExecutionContext) {+ constructor(private readonly projectId: string) {super();}@@ -97,7 +87,11 @@ class TenancyInjectionTransformer extends ClickHouseOperationNodeTransformer {node,queryId,) as ClickHouseSelectQueryNode;- return injectSelect(withChildren, this.ctx, collectCteNames(withChildren));+ return injectSelect(+ withChildren,+ this.projectId,+ collectCteNames(withChildren),+ );}}@@ -111,7 +105,7 @@ function collectCteNames(node: SelectQueryNode): Set<string> {function injectSelect(node: ClickHouseSelectQueryNode,- ctx: ExecutionContext,+ projectId: string,cteNames: Set<string>,): ClickHouseSelectQueryNode {const fromRelations = (node.from?.froms ?? []).map(describeRelation);@@ -148,8 +142,8 @@ function injectSelect(let where = node.where;for (const table of tenantedFrom) {- if (!predicateCovers(where?.where, table, ctx, requireQualified)) {- const predicate = projectIdPredicate(table, ctx);+ if (!predicateCovers(where?.where, table, projectId, requireQualified)) {+ const predicate = projectIdPredicate(table, projectId);// Prepend: the tenancy scope is the leading WHERE predicate, so its bound// value keeps a stable (first) parameter position regardless of the other// predicates a caller wrote.@@ -165,12 +159,12 @@ function injectSelect(}const onExpr = join.on?.on;if (- predicateCovers(onExpr, relation, ctx, requireQualified) ||- predicateCovers(where?.where, relation, ctx, requireQualified)+ predicateCovers(onExpr, relation, projectId, requireQualified) ||+ predicateCovers(where?.where, relation, projectId, requireQualified)) {return join;}- const predicate = projectIdPredicate(relation, ctx);+ const predicate = projectIdPredicate(relation, projectId);if (!join.on) {return JoinNode.createWithOn(join.joinType, join.table, predicate);}@@ -241,12 +235,12 @@ function projectIdColumn(function projectIdPredicate(table: Extract<Relation, { kind: "table" }>,- ctx: ExecutionContext,+ projectId: string,): OperationNode {return BinaryOperationNode.create(projectIdColumn(table),OperatorNode.create("="),- ValueNode.create(ctx.projectId),+ ValueNode.create(projectId),);}@@ -254,7 +248,7 @@ function projectIdPredicate(* True only when `expr` provably constrains `table` to `projectId`. Both halves* matter: the left operand must be `table`'s `project_id` column (qualified when* {@link requireQualified}, see {@link injectSelect}) *and* the right operand- * must be the literal `projectId` from the {@link ExecutionContext}. A predicate+ * must be the literal `projectId` the query is compiled for. A predicate* such as `project_id = <someOtherProject>` or `o.project_id = t.project_id`* does not scope the relation to the request's project, so it is not covered and* the pass injects the correct predicate.@@ -262,24 +256,24 @@ function projectIdPredicate(function predicateCovers(expr: OperationNode | undefined,table: Extract<Relation, { kind: "table" }>,- ctx: ExecutionContext,+ projectId: string,requireQualified: boolean,): boolean {if (!expr) return false;if (AndNode.is(expr)) {return (- predicateCovers(expr.left, table, ctx, requireQualified) ||- predicateCovers(expr.right, table, ctx, requireQualified)+ predicateCovers(expr.left, table, projectId, requireQualified) ||+ predicateCovers(expr.right, table, projectId, requireQualified));}if (OrNode.is(expr)) {return (- predicateCovers(expr.left, table, ctx, requireQualified) &&- predicateCovers(expr.right, table, ctx, requireQualified)+ predicateCovers(expr.left, table, projectId, requireQualified) &&+ predicateCovers(expr.right, table, projectId, requireQualified));}if (ParensNode.is(expr)) {- return predicateCovers(expr.node, table, ctx, requireQualified);+ return predicateCovers(expr.node, table, projectId, requireQualified);}if (!BinaryOperationNode.is(expr) || !OperatorNode.is(expr.operator)) {return false;@@ -289,7 +283,7 @@ function predicateCovers(}return (expr.operator.operator === "=" &&- isProjectIdValue(expr.rightOperand, ctx.projectId)+ isProjectIdValue(expr.rightOperand, projectId));}@@ -323,15 +317,13 @@ function isProjectIdValue(node: OperationNode, projectId: string): boolean {return ValueNode.is(node) && node.value === projectId;}-export function requireExecutionContext(- ctx: ExecutionContext | undefined,-): ExecutionContext {- if (!ctx?.projectId) {+export function requireProjectId(projectId: string | undefined): string {+ if (typeof projectId !== "string" || !projectId) {throw new QueryCompileError(- "ExecutionContext is required: a query with no tenancy scope cannot compile.",+ "projectId is required: a query with no tenancy scope cannot compile.",);}- return ctx;+ return projectId;}function stampTenancy<T extends object>(node: T): T {@@ -352,7 +344,7 @@ export function stampCompiledTree<T extends object>(node: T): T {export function assertTenancyStamped(node: RootOperationNode): void {if (!TENANCY_STAMPED.has(node)) {throw new QueryCompileError(- "Refusing to compile: the tenancy injection pass was not applied. Compile through compileClickhouseQuery() with an ExecutionContext.",+ "Refusing to compile: the tenancy injection pass was not applied. Compile through compileClickhouseQuery() with a projectId.",);}}diff --git a/packages/shared/src/server/repositories/environments.ts b/packages/shared/src/server/repositories/environments.tsindex 6d547ac3c1..26728d073f 100644--- a/packages/shared/src/server/repositories/environments.ts+++ b/packages/shared/src/server/repositories/environments.ts@@ -1,9 +1,6 @@import { LISTABLE_SCORE_TYPES } from "../../domain/scores";import { env } from "../../env";-import {- compileClickhouseQuery,- type ExecutionContext,-} from "../query-ast/compile";+import { compileClickhouseQuery } from "../query-ast/compile";import { getClickhouseKysely } from "../query-ast/dialect";import { includeDeleted } from "../query-ast/extensions";import { queryClickhouse } from "./clickhouse";@@ -19,11 +16,9 @@ export const getEnvironmentsForProject = async (const { projectId, fromTimestamp } = props;const writeMode = env.LANGFUSE_MIGRATION_V4_WRITE_MODE;const db = getClickhouseKysely();- // Tenancy is applied at compile time: compileClickhouseQuery injects- // `project_id = {projectId}` into every tenanted relation and refuses to- // compile an unscoped query, so the query bodies below carry no project_id- // filter of their own.- const ctx: ExecutionContext = { projectId };+ // The query bodies below carry no project_id filter of their own:+ // compileClickhouseQuery injects `project_id = {projectId}` into every+ // tenanted relation.// In dual/events_only write modes all tracing data lands in events_core (the// only populated source under events_only), so one scan covers traces and@@ -64,8 +59,8 @@ export const getEnvironmentsForProject = async (qb.where("timestamp", ">=", fromTimestamp!),);- const tracing = compileClickhouseQuery(tracingQuery, ctx);- const scores = compileClickhouseQuery(scoresQuery, ctx);+ const tracing = compileClickhouseQuery(tracingQuery, projectId);+ const scores = compileClickhouseQuery(scoresQuery, projectId);// The events read may route to a dedicated ClickHouse service// (CLICKHOUSE_EVENTS_READ_ONLY_URL) while scores always read the primary, sodiff --git a/packages/shared/src/server/repositories/events.ts b/packages/shared/src/server/repositories/events.tsindex 44aa3e4e5f..2a7100c71f 100644--- a/packages/shared/src/server/repositories/events.ts+++ b/packages/shared/src/server/repositories/events.ts@@ -29,10 +29,7 @@ import {} from "../clickhouse/client";import { recordDistribution } from "../instrumentation";import { logger } from "../logger";-import {- compileClickhouseQuery,- type ExecutionContext,-} from "../query-ast/compile";+import { compileClickhouseQuery } from "../query-ast/compile";import { getClickhouseKysely } from "../query-ast/dialect";import { promptGenerationEvents } from "./prompt-events";import {@@ -680,7 +677,7 @@ export const getObservationMetricsForPromptsFromEvents = async (.where("e.is_deleted", "=", 0).groupBy(["e.prompt_id", "e.prompt_version"]).orderBy("e.prompt_version", "desc");- const { sql: query, params } = compileClickhouseQuery(builder, { projectId });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);const rows = await queryClickhouse<{count: string;prompt_id: string;@@ -731,7 +728,7 @@ export const getObservationsWithPromptNameFromEvents = async (qb.where("start_time", "<=", toTimestamp!),).groupBy("prompt_name");- const { sql: query, params } = compileClickhouseQuery(builder, { projectId });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);const rows = await queryClickhouse<{ count: string; prompt_name: string }>({query,params,@@ -1128,9 +1125,7 @@ async function getObservationByIdFromEventsTableInternal({.orderBy("event_ts", "desc").limit(1);- const { sql: query, params } = compileClickhouseQuery(builder, {- projectId,- });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);return await queryClickhouse<EventsObservationRecordReadType>({query,@@ -1240,9 +1235,7 @@ export const getTraceByIdFromEventsTable = async ({.orderBy("t.timestamp", "desc").limit(1);- const { sql: query, params } = compileClickhouseQuery(builder, {- projectId,- });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);const records = await queryClickhouse<TraceRecordReadType>({query,@@ -1383,7 +1376,6 @@ function compileHasAnyFromEventsTable(opts: {kind: HasAnyEventsKind;}) {const db = getClickhouseKysely();- const ctx: ExecutionContext = { projectId: opts.projectId };const query = db.selectFrom("events_core")@@ -1396,7 +1388,7 @@ function compileHasAnyFromEventsTable(opts: {).limit(1);- return compileClickhouseQuery(query, ctx);+ return compileClickhouseQuery(query, opts.projectId);}/**@@ -2647,9 +2639,10 @@ export async function getAgentGraphDataFromEventsTable(params: {.where("e.start_time", ">=", minStartTime).where("e.start_time", "<=", maxStartTime);- const { sql: query, params: queryParams } = compileClickhouseQuery(builder, {+ const { sql: query, params: queryParams } = compileClickhouseQuery(+ builder,projectId,- });+ );return queryClickhouse({query,@@ -2842,9 +2835,10 @@ export const getObservationsBatchIOFromEventsTable = async <).where("e.start_time", ">=", minTimestamp).where("e.start_time", "<=", maxTimestamp);- const { sql: query, params } = compileClickhouseQuery(builder, {- projectId: opts.projectId,- });+ const { sql: query, params } = compileClickhouseQuery(+ builder,+ opts.projectId,+ );const results = await queryClickhouse<{id: string;@@ -2929,7 +2923,6 @@ function compileObservationIOFieldRead<A extends string>(select: (field: AliasableExpression<string>) => AliasedExpression<unknown, A>,) {const db = getClickhouseKysely();- const ctx: ExecutionContext = { projectId: opts.projectId };const query = db.selectFrom("events_full as e")@@ -2953,7 +2946,7 @@ function compileObservationIOFieldRead<A extends string>(.orderBy("e.event_ts", "desc").limit(1);- return compileClickhouseQuery(query, ctx);+ return compileClickhouseQuery(query, opts.projectId);}/**@@ -3050,7 +3043,6 @@ export const getObservationFullIOForSessionFromEventsTable = async (opts: {const maxTimestamp = new Date(opts.startTime.getTime() + 1000);const db = getClickhouseKysely();- const ctx: ExecutionContext = { projectId: opts.projectId };const builder = db.selectFrom("events_full as e").$call(includeDeleted())@@ -3072,7 +3064,10 @@ export const getObservationFullIOForSessionFromEventsTable = async (opts: {.where("e.start_time", "<=", maxTimestamp).orderBy("e.event_ts", "desc").limit(1);- const { sql: query, params } = compileClickhouseQuery(builder, ctx);+ const { sql: query, params } = compileClickhouseQuery(+ builder,+ opts.projectId,+ );const rows = await queryClickhouse<{id: string;@@ -3124,9 +3119,10 @@ export const getObservationsTraceIdsFromEventsTable = async (opts: {.select(["e.trace_id as trace_id", "e.span_id as span_id"]).where("e.span_id", "in", observationIds);- const { sql: query, params: queryParams } = compileClickhouseQuery(builder, {+ const { sql: query, params: queryParams } = compileClickhouseQuery(+ builder,projectId,- });+ );const result = await queryClickhouse<{trace_id: string;@@ -3620,9 +3616,10 @@ export const getTraceMetadataByIdsFromEvents = async (props: {.where("e.trace_id", "in", props.traceIds).$call(limitBy({ count: 1, columns: ["e.trace_id"] }));- const { sql: query, params } = compileClickhouseQuery(builder, {- projectId: props.projectId,- });+ const { sql: query, params } = compileClickhouseQuery(+ builder,+ props.projectId,+ );return queryClickhouse<{id: string;@@ -3892,7 +3889,7 @@ export const getSessionMetricsFromEvents = async <.where("session_id", "!=", "").groupBy("session_id").limit(sessionIds.length);- const { sql: query, params } = compileClickhouseQuery(builder, { projectId });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);const rows = await queryClickhouse<Pick<SessionEventsMetricsRow, F>>({query,@@ -3956,9 +3953,10 @@ export async function getLatestSdkVersionInfoFromEvents(params: {"e.telemetry_sdk_language",]).orderBy("last_seen", "desc");- const { sql: query, params: queryParams } = compileClickhouseQuery(builder, {+ const { sql: query, params: queryParams } = compileClickhouseQuery(+ builder,projectId,- });+ );const result = await queryClickhouse<{ingestion_sdk_name: string;ingestion_sdk_version: string;diff --git a/packages/shared/src/server/repositories/experiments.ts b/packages/shared/src/server/repositories/experiments.tsindex c2eb153933..253710c89c 100644--- a/packages/shared/src/server/repositories/experiments.ts+++ b/packages/shared/src/server/repositories/experiments.ts@@ -375,7 +375,7 @@ export const getExperimentMetricsFromEvents = async (props: {.where("e.experiment_id", "in", experimentIds).where("e.experiment_id", "!=", "").groupBy(["e.project_id", "e.experiment_id"]);- const { sql: query, params } = compileClickhouseQuery(builder, { projectId });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);const res = await queryClickhouse<ExperimentMetricsReturnType>({query,diff --git a/packages/shared/src/server/repositories/scores.ts b/packages/shared/src/server/repositories/scores.tsindex 9f5f73c0e0..7f9bd80fab 100644--- a/packages/shared/src/server/repositories/scores.ts+++ b/packages/shared/src/server/repositories/scores.ts@@ -489,7 +489,7 @@ export const getScoresForExperimentItems = async (const compiled = compileClickhouseQuery(scoresForExperimentItemsQuery(experimentIds),- { projectId },+ projectId,);const rows = await queryClickhouse<@@ -2333,7 +2333,7 @@ export const buildAggregatedScoresForPromptsFromEventsQuery = (fetchScoreRelation,timeWindow,),- { projectId },+ projectId,);return { query: compiled.sql, params: compiled.params };};diff --git a/packages/shared/src/server/services/sessions-ui-table-events-service.ts b/packages/shared/src/server/services/sessions-ui-table-events-service.tsindex d07393986a..a1f165a7c9 100644--- a/packages/shared/src/server/services/sessions-ui-table-events-service.ts+++ b/packages/shared/src/server/services/sessions-ui-table-events-service.ts@@ -56,7 +56,7 @@ export const getSessionTracesFromEvents = async (props: {.where("e.session_id", "=", sessionId).groupBy(["trace_id", "project_id"]).orderBy(sql.ref("timestamp"), "asc");- const { sql: query, params } = compileClickhouseQuery(builder, { projectId });+ const { sql: query, params } = compileClickhouseQuery(builder, projectId);const rows = await queryClickhouse<{id: string;