πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(scores): compile experiment-item and prompt-event reads through the query AST (#17553)
Niklas Semmlerdd4ca52fcfad
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.mdindex 67b7213ca4..1abd74207a 100644--- a/packages/shared/AGENTS.md+++ b/packages/shared/AGENTS.md@@ -41,7 +41,7 @@- Code evaluator dispatcher/error contract: `src/server/evals/codeEvalDispatcherTypes.ts`. Keep provider mappings, user-visible messages, and worker terminal-outcome classification aligned when adding an error code.- Dashboard/monitor query feature (data model + server-only builder/executor): `src/features/query/*`- Query-builder AST (server half, WIP): `src/server/query-ast/*` β the Kysely- ClickHouse dialect (ARRAY JOIN / LIMIT BY / metadata indexOf nodes,+ ClickHouse dialect (ARRAY JOIN / LIMIT BY / FINAL / metadata indexOf nodes,`ExecutionContext` tenancy injection, per-table dedup lowering, virtual views,catalog parity). Compile only through `compileClickhouseQuery` in`src/server/query-ast/compile.ts`. SQL correctness is proven by a golden-SQL@@ -50,6 +50,7 @@`clickhouse format`); each migrated call site keeps its `*.golden.test.ts`baseline next to the call site (e.g.`src/server/repositories/environments.golden.test.ts`,+ `src/server/repositories/scores.golden.test.ts`,`src/server/queries/clickhouse-sql/event-filter-options.golden.test.ts`).- Postgres schema: `prisma/schema.prisma`- Prisma migrations: `prisma/migrations/*`diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex 3949c0ab22..ccb9862838 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -8,10 +8,10 @@ metadata, `FilterState` embedding) will land under`packages/shared/src/query-ast/`.This folder is a compile-only ClickHouse dialect on Kysely 0.28: real-`OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf` subscripts; a-mandatory tenancy injection pass keyed on `ExecutionContext`; per-table dedup-lowering from the registry; schema-typed selection; virtual views as WITH CTEs;-catalog parity.+`OperationNode`s for ARRAY JOIN, LIMIT BY, FINAL, and metadata `indexOf`+subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;+per-table dedup lowering from the registry; schema-typed selection; virtual+views as WITH CTEs; catalog parity.Two audiences: **[Using it](#using-it)** if you are calling the compiler from arepository, **[Developing the dialect](#developing-the-dialect)** if you are@@ -29,16 +29,18 @@ changing how it emits SQL.of hand-maintained tables that drift.- `compiler.ts` / `transformer.ts` / `dialect.ts` / `nodes.ts` β the dialect.- `extensions.ts` β the `$call` helpers: `arrayJoin`, `mapKeys`, `mapValues`,- `limitBy`, `metadataValue`.-- `catalog.ts` / `views.ts` / `dedup.ts` β catalog, virtual views as CTEs, and- per-table dedup lowering.+ `limitBy`, `useFinal`, `metadataValue`.+- `views.ts` / `dedup.ts` β virtual views as CTEs and per-table dedup lowering.+- `fixtures/catalog.ts` β test-only catalog of builder queries exercising the+ dialect's expressivity.- `catalog.golden.test.ts` β catalog SQL baselines.The golden-SQL harness lives in `../repositories/goldenHarness.ts` (it capturesSQL at the `repositories/clickhouse.ts` exec seam in test mode, then normalizesit via `clickhouse format` + positional param names for snapshot comparison β noClickHouse server needed). Each migrated call site keeps its SQL baseline next to-that call site, e.g. `../repositories/environments.golden.test.ts` and+that call site, e.g. `../repositories/environments.golden.test.ts`,+`../repositories/scores.golden.test.ts`, and`../queries/clickhouse-sql/event-filter-options.golden.test.ts`.Regenerate baselines with `-u` after an intentional SQL change. Filter by the@@ -81,7 +83,8 @@ suffix to run there. Promote it to a required check once it has proven stable.`events_core` is `none` (immutable at read time β no LIMIT BY, no FINAL).`limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` alreadyused on traces / observations / scores. `$call(limitBy(...))` remains for- explicit non-version LIMIT BY.+ explicit non-version LIMIT BY. Auto-lowering does not declare scores as+ `{ strategy: "final" }`; callers opt in with `$call(useFinal(["scores"]))`.- **Value binds take their ClickHouse type from the compared column** in thetable registry (`total_cost > 1` emits `{p:Float64}`, not inferred`{p:Int64}`). Same-value binds still intern to one placeholder.@@ -115,6 +118,15 @@ db.selectFrom("events_core")// β¦ order by event_ts desc limit 1 by span_id, project_id```+**Write FINAL** (`useFinal` β ReplacingMergeTree reads; never on `events_core`):++```ts+db.selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"]));+// β¦ from scores as s final+```+**Select a metadata value** (`metadataValue` β lowers `metadata[key]` to a bound `indexOf` subscript):```ts@@ -140,6 +152,7 @@ preserve it:| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- || ARRAY JOIN | Plugin attaches `ArrayJoinNode` as an extra field on `SelectQueryNode`. `ClickHouseOperationNodeTransformer` preserves it. `ClickHouseQueryCompiler.visitSelectQuery` emits it after JOINs / before WHERE. | no || LIMIT BY | Plugin attaches `LimitByNode` the same way. Compiler emits it after ORDER BY / before LIMIT. | no |+| FINAL | Plugin wraps a FROM/JOIN table expression in `FinalTableNode`. Compiler emits `final` after that expression (`scores as s final`). `$call(useFinal(["scores"]))`. Tables declared `dedup: none` throw. | no || metadata `indexOf` | Helper builds an `ArrayIndexNode` whose index child is a `FunctionNode` (`indexOf`) over a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no || Virtual view | Plugin rewrites `selectFrom(viewName)` into a WITH CTE. Outer types only expose the view's selected columns. | no |@@ -178,8 +191,8 @@ is where tenancy is enforced:(`none` / `limitBy` / `final`). `events_core` is `none` β immutable atread time, so the pass does not inject LIMIT BY or FINAL. `limitBy` isthe existing legacy `ORDER BY <version> DESC LIMIT 1 BY <key>`.- `final` is fail-closed until an emitter exists. The pass restamps the- rewritten root.+ `final` remains fail-closed on the auto-lowering pass; callers opt in+ with `$call(useFinal(["scores"]))`. The pass restamps the rewritten root.4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp ispresent, so `qb.compile()` without the plugin also fails. Value binds taketheir ClickHouse type from the compared column's registry entry when one is@@ -199,7 +212,8 @@ callbacks hand you. Getting that requires either forking Kysely's whole buildergraph or globally mutating its prototype via an internal `kysely/dist/...`import (blocked here by Kysely's `exports` map under NodeNext). A curried helperis a plain function of a builder, so it works in any of those positions β which-is exactly why ARRAY JOIN / LIMIT BY compose inside CTEs, subqueries, and views.+is exactly why ARRAY JOIN / LIMIT BY / FINAL compose inside CTEs, subqueries,+and views.`composition.test.ts` locks that property in.**`arrayJoin` widens the row type.** Each `{ alias: arrayExpr }` entry is added@@ -230,16 +244,17 @@ is just `eb.fn("arrayJoin", [...])`.### The one Kysely-internals coupling (upgrade hazard)`compiler.ts` wraps Kysely's **private** `visitNode` / `nodeStack` to dispatch-`ArrayIndexNode` (for `metadata[key]`), because it is not one of Kysely's closed-`OperationNode` kinds. Kysely is pinned to **0.28.17** for this reason. Re-verify-this hack on any Kysely bump; it is the single place that reaches past Kysely's-documented surface. Everything else (plugins, dialect, transformer overrides)-uses public or documented-protected API.+`ArrayIndexNode` (for `metadata[key]`) and `FinalTableNode` (for `FINAL`),+because they are not Kysely's closed `OperationNode` kinds. Kysely is pinned+to **0.28.17** for this reason. Re-verify this hack on any Kysely bump; it is+the single place that reaches past Kysely's documented surface. Everything+else (plugins, dialect, transformer overrides) uses public or+documented-protected API.### Types are asserted at compile time`types.assert.ts` holds `tsc`-only assertions (schema typing, view opacity,-arrayJoin widening, limitBy preservation). It is never run; `schema.test.ts`+arrayJoin widening, limitBy / useFinal preservation). It is never run; `schema.test.ts`anchors it so it stays in the build graph. `@ts-expect-error` lines there muststay live.diff --git a/packages/shared/src/server/query-ast/catalog.golden.test.ts b/packages/shared/src/server/query-ast/catalog.golden.test.tsindex fe3dc162fc..8af77f40f8 100644--- a/packages/shared/src/server/query-ast/catalog.golden.test.ts+++ b/packages/shared/src/server/query-ast/catalog.golden.test.ts@@ -8,7 +8,7 @@ import {normalizeParams,substituteNamedParams,} from "../repositories/goldenHarness";-import { CATALOG, CATALOG_PROJECT_ID } from "./catalog";+import { CATALOG, CATALOG_PROJECT_ID } from "./fixtures/catalog";import { compileClickhouseQuery } from "./compile";const describeWithClickhouse = clickhouseFormatAvailable()diff --git a/packages/shared/src/server/query-ast/compiler.ts b/packages/shared/src/server/query-ast/compiler.tsindex 9156d82a3c..9f868df245 100644--- a/packages/shared/src/server/query-ast/compiler.ts+++ b/packages/shared/src/server/query-ast/compiler.ts@@ -20,6 +20,7 @@ import type { QueryId } from "kysely";import {ArrayJoinNode,ArrayIndexNode,+ FinalTableNode,type ClickHouseSelectQueryNode,type LimitByNode,} from "./nodes";@@ -103,6 +104,12 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {self.nodeStack.pop();return;}+ if (FinalTableNode.is(node)) {+ self.nodeStack.push(node);+ this.visitFinalTable(node);+ self.nodeStack.pop();+ return;+ }parentVisit(node);};}@@ -347,6 +354,11 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {this.compileList(node.columns);}+ private visitFinalTable(node: FinalTableNode): void {+ this.visitNode(node.table);+ this.append(" final");+ }+private bindValue(value: unknown, type?: string): string {// `type` is the caller-known ClickHouse type; when omitted we infer the// widest canonical type from the JS value (Int64 / Float64 / String /diff --git a/packages/shared/src/server/query-ast/composition.test.ts b/packages/shared/src/server/query-ast/composition.test.tsindex 5435aa0bf5..95f1b5a56c 100644--- a/packages/shared/src/server/query-ast/composition.test.ts+++ b/packages/shared/src/server/query-ast/composition.test.ts@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";import { compileClickhouseQuery } from "./compile";import { getClickhouseKysely } from "./dialect";-import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";+import { arrayJoin, limitBy, mapKeys, mapValues, useFinal } from "./extensions";import { defineView, fromView } from "./views";// Raw compiler output (no `clickhouse format` binary needed), so these run@@ -124,3 +124,56 @@ describe("LIMIT BY composition across nesting positions", () => {expect(sql).toMatch(/with lb as/i);});});++describe("FINAL composition across nesting positions", () => {+ it("emits FINAL at the top level (baseline)", () => {+ const db = getClickhouseKysely();+ const qb = db+ .selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"]));++ const { sql } = compileClickhouseQuery(qb, ctx);+ expect(sql).toMatch(/scores as s final/i);+ });++ it("keeps FINAL when applied inside a CTE body", () => {+ const db = getClickhouseKysely();+ const qb = db+ .with("scored", (inner) =>+ inner+ .selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"])),+ )+ .selectFrom("scored")+ .select("id");++ const { sql } = compileClickhouseQuery(qb, ctx);+ const outerStart = sql+ .toLowerCase()+ .search(/\)\s*select[\s\S]*from scored/i);+ expect(outerStart).toBeGreaterThan(-1);+ const cteBody = sql.slice(0, outerStart);+ const outerQuery = sql.slice(outerStart);+ expect(cteBody).toMatch(/scores as s final/i);+ expect(outerQuery).not.toMatch(/final/i);+ });++ it("keeps FINAL on a JOIN table and not on events_core", () => {+ const db = getClickhouseKysely();+ const qb = db+ .selectFrom("events_core as e")+ .innerJoin("scores as s", (join) =>+ join+ .onRef("e.trace_id", "=", "s.trace_id")+ .onRef("e.project_id", "=", "s.project_id"),+ )+ .$call(useFinal(["scores"]))+ .select("s.id");++ const { sql } = compileClickhouseQuery(qb, ctx);+ expect(sql).toMatch(/scores as s final/i);+ expect(sql).not.toMatch(/events_core as e final/i);+ });+});diff --git a/packages/shared/src/server/query-ast/dedup.ts b/packages/shared/src/server/query-ast/dedup.tsindex b088ca61b3..9a86e6195c 100644--- a/packages/shared/src/server/query-ast/dedup.ts+++ b/packages/shared/src/server/query-ast/dedup.ts@@ -42,7 +42,12 @@ import {} from "kysely";import { QueryCompileError } from "./errors";-import { LimitByNode, type ClickHouseSelectQueryNode } from "./nodes";+import {+ FinalTableNode,+ LimitByNode,+ unwrapFinalTable,+ type ClickHouseSelectQueryNode,+} from "./nodes";import { DEDUP_SPECS, type DedupSpec } from "./schema";import { stampCompiledTree } from "./tenancy";import { ClickHouseOperationNodeTransformer } from "./transformer";@@ -133,6 +138,9 @@ function singlePhysicalFrom(}function describeFrom(node: OperationNode): PhysicalFrom | undefined {+ if (FinalTableNode.is(node)) {+ return describeFrom(unwrapFinalTable(node));+ }if (AliasNode.is(node)) {const inner = describeFrom(node.node);if (!inner) return undefined;diff --git a/packages/shared/src/server/query-ast/extensions.ts b/packages/shared/src/server/query-ast/extensions.tsindex dcb64d2c9a..1279aafe37 100644--- a/packages/shared/src/server/query-ast/extensions.ts+++ b/packages/shared/src/server/query-ast/extensions.ts@@ -6,19 +6,23 @@* - Plugins ({@link ArrayJoinPlugin}, {@link LimitByPlugin}) that attach a* custom operation node (`ArrayJoinNode` / `LimitByNode`) onto the select* node during query transformation; the ClickHouse compiler renders them.- * - `$call` helpers ({@link arrayJoin}, {@link limitBy}) that apply those- * plugins while extending the builder's output row type, so produced- * aliases are type-checked and misspelled references are compile errors.+ * - `$call` helpers ({@link arrayJoin}, {@link limitBy}, {@link useFinal})+ * that apply those plugins while extending the builder's output row type,+ * so produced aliases are type-checked and misspelled references are+ * compile errors.* - Typed expression helpers ({@link mapKeys}, {@link mapValues},* {@link metadataValue}) that build ClickHouse map/array function calls and* `metadata[key]` subscripts as real, parameter-bound nodes β never raw SQL* strings β so they escape and compose like any other expression.*/import {+ AliasNode,ColumnNode,ExpressionWrapper,+ FromNode,FunctionNode,IdentifierNode,+ JoinNode,ReferenceNode,TableNode,ValueNode,@@ -34,11 +38,14 @@ import {import {ArrayIndexNode,ArrayJoinNode,+ FinalTableNode,LimitByNode,+ unwrapFinalTable,type ArrayJoinVariant,type ClickHouseSelectQueryNode,} from "./nodes";-import type { ClickHouseDatabase } from "./schema";+import { QueryCompileError } from "./errors";+import { DEDUP_SPECS, type ClickHouseDatabase } from "./schema";import { ClickHouseOperationNodeTransformer } from "./transformer";type OperationNodeSource = OperationNode | { toOperationNode(): OperationNode };@@ -228,3 +235,78 @@ export function limitBy(spec: LimitBySpec) {qb: SelectQueryBuilder<DB, TB, O>,): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new LimitByPlugin(spec));}++function physicalTableName(node: OperationNode): string | undefined {+ if (FinalTableNode.is(node)) return physicalTableName(unwrapFinalTable(node));+ if (AliasNode.is(node)) return physicalTableName(node.node);+ if (TableNode.is(node)) return node.table.identifier.name;+ return undefined;+}++function wrapFinal(+ node: OperationNode,+ tables: ReadonlySet<string>,+): OperationNode {+ if (FinalTableNode.is(node)) return node;+ const name = physicalTableName(node);+ return name && tables.has(name)+ ? (FinalTableNode.create(node) as unknown as OperationNode)+ : node;+}++/**+ * Plugin that marks named physical tables for `FINAL`. Walks this select's+ * FROM/JOIN table expressions only β apply it on the builder that owns the+ * scores (or other ReplacingMergeTree) relation, including a CTE body.+ */+class FinalPlugin implements KyselyPlugin {+ constructor(private readonly tables: ReadonlySet<string>) {}++ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ if (args.node.kind !== "SelectQueryNode") return args.node;+ const transformer = new ClickHouseOperationNodeTransformer();+ const node = transformer.transformNode(+ args.node,+ ) as ClickHouseSelectQueryNode;+ const froms = (node.from?.froms ?? []).map((from) =>+ wrapFinal(from, this.tables),+ );+ const joins = (node.joins ?? []).map((join) => {+ const table = wrapFinal(join.table, this.tables);+ if (table === join.table) return join;+ return join.on+ ? JoinNode.createWithOn(join.joinType, table, join.on.on)+ : JoinNode.create(join.joinType, table);+ });+ return {+ ...node,+ // cloneWithFroms appends; replacing the list avoids+ // `FROM scores AS s, scores AS s FINAL`.+ ...(node.from ? { from: FromNode.create(froms) } : {}),+ ...(joins.length ? { joins } : {}),+ } as RootOperationNode;+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }+}++/**+ * `FROM <table> FINAL` / `JOIN <table> AS alias FINAL` as a `$call` step:+ * `qb.$call(useFinal(["scores"]))`. Tables declared `dedup: none` (immutable+ * at read time, including `events_core`) cannot take FINAL.+ */+export function useFinal(tables: ReadonlyArray<keyof ClickHouseDatabase>) {+ for (const table of tables) {+ if (DEDUP_SPECS[table]?.strategy === "none") {+ throw new QueryCompileError(+ `Cannot apply FINAL to ${table}: declared immutable (dedup strategy none).`,+ );+ }+ }+ const names = new Set<string>(tables);+ return <DB, TB extends keyof DB, O>(+ qb: SelectQueryBuilder<DB, TB, O>,+ ): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new FinalPlugin(names));+}diff --git a/packages/shared/src/server/query-ast/catalog.ts b/packages/shared/src/server/query-ast/fixtures/catalog.tssimilarity index 86%rename from packages/shared/src/server/query-ast/catalog.tsrename to packages/shared/src/server/query-ast/fixtures/catalog.tsindex 95df98fa28..12c2415201 100644--- a/packages/shared/src/server/query-ast/catalog.ts+++ b/packages/shared/src/server/query-ast/fixtures/catalog.ts@@ -1,14 +1,15 @@import type { SqlBool } from "kysely";-import type { ClickhouseCompilable } from "./compile";-import { getClickhouseKysely } from "./dialect";+import type { ClickhouseCompilable } from "../compile";+import { getClickhouseKysely } from "../dialect";import {arrayJoin,limitBy,mapKeys,mapValues,metadataValue,-} from "./extensions";+ useFinal,+} from "../extensions";type CatalogTier = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7;@@ -189,6 +190,57 @@ export const CATALOG: CatalogEntry[] = [eb.selectFrom("traces_cte").select("id"),),},+ {+ // Explicit FINAL on a ReplacingMergeTree read. Auto-lowering does not+ // declare scores as `{ strategy: "final" }` (that would rewrite every+ // scores scan and has no JOIN emitter); callers opt in with useFinal.+ id: "scores_final",+ tier: 2,+ referenceSql: `+ SELECT s.id+ FROM scores AS s FINAL+ WHERE s.project_id = {p1:String}+ `,+ build: () =>+ db()+ .selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"])),+ },+ {+ // Tuple IN-subquery: `(trace_id, observation_id) IN (SELECT β¦)`.+ id: "tuple_in_subquery",+ tier: 2,+ referenceSql: `+ WITH prompt_events AS (+ SELECT trace_id, span_id+ FROM events_core+ WHERE project_id = {p1:String}+ )+ SELECT s.id+ FROM scores AS s+ WHERE (s.project_id = {p1:String}) AND ((s.trace_id, s.observation_id) IN (+ SELECT trace_id, span_id+ FROM prompt_events+ ))+ `,+ build: () =>+ db()+ .with("prompt_events", (qb) =>+ qb.selectFrom("events_core").select(["trace_id", "span_id"]),+ )+ .selectFrom("scores as s")+ .select("s.id")+ .where(({ eb, refTuple, selectFrom }) =>+ eb(+ refTuple("s.trace_id", "s.observation_id"),+ "in",+ selectFrom("prompt_events")+ .select(["trace_id", "span_id"])+ .$asTuple("trace_id", "span_id"),+ ),+ ),+ },{id: "array_join_cost_details",tier: 3,diff --git a/packages/shared/src/server/query-ast/nodes.test.ts b/packages/shared/src/server/query-ast/nodes.test.tsindex 7b911e5350..48d7ab3d5f 100644--- a/packages/shared/src/server/query-ast/nodes.test.ts+++ b/packages/shared/src/server/query-ast/nodes.test.ts@@ -5,9 +5,10 @@ import { compileClickhouseQuery } from "./compile";import { ClickHouseQueryCompiler } from "./compiler";import { getClickhouseKysely } from "./dialect";import { QueryCompileError, UnscopedRelationError } from "./errors";-import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";+import { arrayJoin, limitBy, mapKeys, mapValues, useFinal } from "./extensions";import {ArrayJoinNode,+ FinalTableNode,LimitByNode,isClickHouseSelectQueryNode,} from "./nodes";@@ -285,4 +286,36 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {expect(lower).toContain(" by ");expect(lower.indexOf(" by ")).toBeLessThan(lower.lastIndexOf("limit"));});++ it("wraps the scores FROM table in a FinalTableNode", () => {+ const qb = getClickhouseKysely()+ .selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"]));++ const node = qb+ .withPlugin(new TenancyInjectionPlugin(ctx))+ .toOperationNode();+ expect(node.kind).toBe("SelectQueryNode");+ const from = (node as { from?: { froms?: Array<{ kind: string }> } }).from+ ?.froms?.[0];+ expect(from).toBeDefined();+ expect(FinalTableNode.is(from!)).toBe(true);+ });++ it("emits FINAL after the scores table expression", () => {+ const qb = getClickhouseKysely()+ .selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"]));++ const { sql: compiled } = compileClickhouseQuery(qb, ctx);+ expect(compiled.toLowerCase()).toMatch(/from scores as s final/);+ expect(compiled.toLowerCase()).not.toMatch(/from scores as s,/);+ });++ it("rejects FINAL on events_core", () => {+ expect(() => useFinal(["events_core"])).toThrow(QueryCompileError);+ expect(() => useFinal(["events_core"])).toThrow(/immutable/);+ });});diff --git a/packages/shared/src/server/query-ast/nodes.ts b/packages/shared/src/server/query-ast/nodes.tsindex 8b08f21423..f5ef78b6ae 100644--- a/packages/shared/src/server/query-ast/nodes.ts+++ b/packages/shared/src/server/query-ast/nodes.ts@@ -112,3 +112,38 @@ export function isClickHouseSelectQueryNode(): node is ClickHouseSelectQueryNode {return node.kind === "SelectQueryNode";}++/**+ * ClickHouse `FROM <table> FINAL` / `JOIN <table> AS alias FINAL`.+ *+ * Wraps a table expression (bare `TableNode` or `AliasNode`) so the compiler+ * emits `final` after that expression. Same closed-`kind` constraint as+ * ARRAY JOIN: not a Kysely visitor-map kind. The dialect compiler and+ * transformer special-case this object.+ */+export type FinalTableNode = {+ readonly kind: "FinalTableNode";+ readonly table: OperationNode;+};++export const FinalTableNode = {+ is(node: { kind: string }): node is FinalTableNode {+ return node.kind === "FinalTableNode";+ },+ create(table: OperationNode): FinalTableNode {+ return Object.freeze({+ kind: "FinalTableNode",+ table,+ });+ },+};++/**+ * Read the wrapped table expression. `FinalTableNode.kind` is outside+ * Kysely's closed `OperationNodeKind`, so a narrowed+ * `OperationNode & FinalTableNode` is `never` and `.table` is unreadable+ * without this cast.+ */+export function unwrapFinalTable(node: { kind: string }): OperationNode {+ return (node as unknown as FinalTableNode).table;+}diff --git a/packages/shared/src/server/query-ast/schema.test.ts b/packages/shared/src/server/query-ast/schema.test.tsindex abe4fd564c..43c6be5231 100644--- a/packages/shared/src/server/query-ast/schema.test.ts+++ b/packages/shared/src/server/query-ast/schema.test.ts@@ -30,6 +30,8 @@ describe("table registry derivation", () => {expect(COLUMN_DATA_TYPES.timestamp).toBe("date");expect(COLUMN_DATA_TYPES.metadata_names).toBe("array");expect(COLUMN_DATA_TYPES.cost_details).toBe("map");+ expect(COLUMN_DATA_TYPES.is_deleted).toBe("number");+ expect(COLUMN_DATA_TYPES.metadata).toBe("map");});it("declares events_core as immutable and leaves legacy tables undeclared", () => {@@ -44,6 +46,8 @@ describe("table registry derivation", () => {expect(COLUMN_BIND_TYPES.start_time).toBe("DateTime64(3)");expect(COLUMN_BIND_TYPES.project_id).toBe("String");expect(COLUMN_BIND_TYPES.metadata_names).toBe("Array(String)");+ expect(COLUMN_BIND_TYPES.is_deleted).toBe("UInt8");+ expect(COLUMN_BIND_TYPES.metadata).toBe("Map(String, String)");});// The assertions themselves are compile-time (`tsc` is the test); this onlydiff --git a/packages/shared/src/server/query-ast/schema.ts b/packages/shared/src/server/query-ast/schema.tsindex 0ebd3fcaaa..ad4cc6585f 100644--- a/packages/shared/src/server/query-ast/schema.ts+++ b/packages/shared/src/server/query-ast/schema.ts@@ -28,20 +28,26 @@ export type ChColumnType =| "String"| "Float"| "DateTime"+ | "UInt8"| "Array(String)"- | "Map(String, Float)";+ | "Map(String, Float)"+ | "Map(String, String)";type TsTypeOfColumn<T extends ChColumnType> = T extends "String"? string: T extends "Float"? number- : T extends "DateTime"- ? Date- : T extends "Array(String)"- ? string[]- : T extends "Map(String, Float)"- ? Record<string, number>- : never;+ : T extends "UInt8"+ ? number+ : T extends "DateTime"+ ? Date+ : T extends "Array(String)"+ ? string[]+ : T extends "Map(String, Float)"+ ? Record<string, number>+ : T extends "Map(String, String)"+ ? Record<string, string>+ : never;/** Coarse runtime category the aggregate type-check pass reasons about. */export type ColumnDataType = "string" | "number" | "date" | "array" | "map";@@ -49,9 +55,11 @@ export type ColumnDataType = "string" | "number" | "date" | "array" | "map";const RUNTIME_TYPE: Record<ChColumnType, ColumnDataType> = {String: "string",Float: "number",+ UInt8: "number",DateTime: "date","Array(String)": "array","Map(String, Float)": "map",+ "Map(String, String)": "map",};/**@@ -123,6 +131,9 @@ const TABLE_REGISTRY = {event_ts: "DateTime",type: "String",total_cost: "Float",+ experiment_id: "String",+ prompt_id: "String",+ is_deleted: "UInt8",metadata_names: "Array(String)",metadata_values: "Array(String)",},@@ -131,10 +142,29 @@ const TABLE_REGISTRY = {}),scores: defineTable({columns: {+ id: "String",environment: "String",project_id: "String",timestamp: "DateTime",+ event_ts: "DateTime",+ created_at: "DateTime",+ updated_at: "DateTime",data_type: "String",+ trace_id: "String",+ session_id: "String",+ observation_id: "String",+ dataset_run_id: "String",+ name: "String",+ value: "Float",+ source: "String",+ comment: "String",+ author_user_id: "String",+ config_id: "String",+ string_value: "String",+ queue_id: "String",+ execution_trace_id: "String",+ is_deleted: "UInt8",+ metadata: "Map(String, String)",},}),} as const;@@ -188,9 +218,11 @@ export const DEDUP_SPECS: Record<string, DedupSpec> = Object.fromEntries(const BIND_TYPE: Record<ChColumnType, string> = {String: "String",Float: "Float64",+ UInt8: "UInt8",DateTime: "DateTime64(3)","Array(String)": "Array(String)","Map(String, Float)": "Map(String, Float64)",+ "Map(String, String)": "Map(String, String)",};export const COLUMN_BIND_TYPES: Record<string, string> = Object.fromEntries(diff --git a/packages/shared/src/server/query-ast/tenancy.ts b/packages/shared/src/server/query-ast/tenancy.tsindex 12789cc5ff..599c572332 100644--- a/packages/shared/src/server/query-ast/tenancy.ts+++ b/packages/shared/src/server/query-ast/tenancy.ts@@ -23,7 +23,11 @@ import {} from "kysely";import { QueryCompileError, UnscopedRelationError } from "./errors";-import { type ClickHouseSelectQueryNode } from "./nodes";+import {+ FinalTableNode,+ unwrapFinalTable,+ type ClickHouseSelectQueryNode,+} from "./nodes";import { TENANTED_TABLES } from "./schema";import { ClickHouseOperationNodeTransformer } from "./transformer";@@ -192,6 +196,9 @@ function assertRelationAllowed(relation: Relation): void {}function describeRelation(node: OperationNode): Relation {+ if (FinalTableNode.is(node)) {+ return describeRelation(unwrapFinalTable(node));+ }if (RawNode.is(node)) {return { kind: "raw" };}diff --git a/packages/shared/src/server/query-ast/transformer.ts b/packages/shared/src/server/query-ast/transformer.tsindex 732b6ee430..f307743a54 100644--- a/packages/shared/src/server/query-ast/transformer.ts+++ b/packages/shared/src/server/query-ast/transformer.ts@@ -8,6 +8,7 @@ import {import {ArrayJoinNode,ArrayIndexNode,+ FinalTableNode,LimitByNode,type ClickHouseSelectQueryNode,} from "./nodes";@@ -33,6 +34,9 @@ export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer// runtime that `T` is ArrayIndexNode here, so the cast is sound.return this.transformArrayIndex(node) as unknown as T;}+ if (FinalTableNode.is(node)) {+ return this.transformFinalTable(node) as unknown as T;+ }return super.transformNodeImpl(node, queryId);}@@ -43,6 +47,10 @@ export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer);}+ protected transformFinalTable(node: FinalTableNode): FinalTableNode {+ return FinalTableNode.create(this.transformNode(node.table));+ }+protected override transformSelectQuery(node: SelectQueryNode,queryId?: QueryId,diff --git a/packages/shared/src/server/query-ast/types.assert.ts b/packages/shared/src/server/query-ast/types.assert.tsindex 4083d53e50..8fb383a3d5 100644--- a/packages/shared/src/server/query-ast/types.assert.ts+++ b/packages/shared/src/server/query-ast/types.assert.ts@@ -1,7 +1,7 @@import { type InferResult } from "kysely";import { getClickhouseKysely } from "./dialect";-import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";+import { arrayJoin, limitBy, mapKeys, mapValues, useFinal } from "./extensions";import { defineView, fromView } from "./views";/**@@ -51,7 +51,7 @@ export function schemaTypeAssertions(): void {/*** Compile-time assertions that the `$call` extension helpers infer correctly:* `arrayJoin` widens the output row type with each declared alias, and `limitBy`- * preserves it. `tsc` is the test.+ * / `useFinal` preserve it. `tsc` is the test.*/export function extensionTypeAssertions(): void {const db = getClickhouseKysely();@@ -85,4 +85,14 @@ export function extensionTypeAssertions(): void {type _Span = Limited["span_id"];// @ts-expect-error limitBy adds no columnstype _NoExtra = Limited["cost_key"];++ // useFinal preserves the row type: it declares no new columns.+ const _final = db+ .selectFrom("scores as s")+ .select("s.id")+ .$call(useFinal(["scores"]));+ type FinalRow = InferResult<typeof _final>[number];+ type _Id = FinalRow["id"];+ // @ts-expect-error useFinal adds no columns+ type _NoFinalExtra = FinalRow["cost_key"];}diff --git a/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snapnew file mode 100644index 0000000000..ae07dd35d8--- /dev/null+++ b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap@@ -0,0 +1,337 @@+// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html++exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=observation time=from+to 1`] = `+{+ "params": {+ "param1": "golden-project",+ "param2": "GENERATION",+ "param3": [+ "prompt-a",+ "prompt-b",+ ],+ "param4": 2026-01-01T00:00:00.000Z,+ "param5": 2026-01-31T23:59:59.000Z,+ "param6": 0,+ "param7": 0,+ "param8": [+ "NUMERIC",+ "BOOLEAN",+ "CATEGORICAL",+ "TEXT",+ ],+ },+ "sql": "WITH+ prompt_events AS+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(3)}) AND (e.start_time <= {param5:DateTime64(3)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ )+SELECT+ e.prompt_id AS prompt_id,+ s.id AS id,+ s.name AS name,+ s.string_value AS string_value,+ s.value AS value,+ s.source AS source,+ s.data_type AS data_type,+ s.comment AS comment,+ s.timestamp AS timestamp,+ length(mapKeys(s.metadata)) > {param6:Int64} AS has_metadata+FROM scores AS s+FINAL+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param7:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param8:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (+ SELECT+ trace_id,+ span_id+ FROM prompt_events+ WHERE is_deleted = {param7:UInt8}+))+GROUP BY+ e.prompt_id,+ s.id,+ s.name,+ s.string_value,+ s.value,+ s.source,+ s.data_type,+ s.comment,+ s.timestamp,+ s.metadata",+}+`;++exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=observation time=unset 1`] = `+{+ "params": {+ "param1": "golden-project",+ "param2": "GENERATION",+ "param3": [+ "prompt-a",+ "prompt-b",+ ],+ "param4": 0,+ "param5": 0,+ "param6": [+ "NUMERIC",+ "BOOLEAN",+ "CATEGORICAL",+ "TEXT",+ ],+ },+ "sql": "WITH+ prompt_events AS+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ )+SELECT+ e.prompt_id AS prompt_id,+ s.id AS id,+ s.name AS name,+ s.string_value AS string_value,+ s.value AS value,+ s.source AS source,+ s.data_type AS data_type,+ s.comment AS comment,+ s.timestamp AS timestamp,+ length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata+FROM scores AS s+FINAL+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (+ SELECT+ trace_id,+ span_id+ FROM prompt_events+ WHERE is_deleted = {param5:UInt8}+))+GROUP BY+ e.prompt_id,+ s.id,+ s.name,+ s.string_value,+ s.value,+ s.source,+ s.data_type,+ s.comment,+ s.timestamp,+ s.metadata",+}+`;++exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace time=from 1`] = `+{+ "params": {+ "param1": "golden-project",+ "param2": "GENERATION",+ "param3": [+ "prompt-a",+ "prompt-b",+ ],+ "param4": 2026-01-01T00:00:00.000Z,+ "param5": 0,+ "param6": 0,+ "param7": [+ "NUMERIC",+ "BOOLEAN",+ "CATEGORICAL",+ "TEXT",+ ],+ },+ "sql": "WITH+ prompt_events AS+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(3)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ )+SELECT+ e.prompt_id AS prompt_id,+ s.id AS id,+ s.name AS name,+ s.string_value AS string_value,+ s.value AS value,+ s.source AS source,+ s.data_type AS data_type,+ s.comment AS comment,+ s.timestamp AS timestamp,+ length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadata+FROM scores AS s+FINAL+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param6:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param7:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (+ SELECT trace_id+ FROM prompt_events+ WHERE is_deleted = {param6:UInt8}+))+GROUP BY+ e.prompt_id,+ s.id,+ s.name,+ s.string_value,+ s.value,+ s.source,+ s.data_type,+ s.comment,+ s.timestamp,+ s.metadata",+}+`;++exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace time=unset 1`] = `+{+ "params": {+ "param1": "golden-project",+ "param2": "GENERATION",+ "param3": [+ "prompt-a",+ "prompt-b",+ ],+ "param4": 0,+ "param5": 0,+ "param6": [+ "NUMERIC",+ "BOOLEAN",+ "CATEGORICAL",+ "TEXT",+ ],+ },+ "sql": "WITH+ prompt_events AS+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ )+SELECT+ e.prompt_id AS prompt_id,+ s.id AS id,+ s.name AS name,+ s.string_value AS string_value,+ s.value AS value,+ s.source AS source,+ s.data_type AS data_type,+ s.comment AS comment,+ s.timestamp AS timestamp,+ length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata+FROM scores AS s+FINAL+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (+ SELECT trace_id+ FROM prompt_events+ WHERE is_deleted = {param5:UInt8}+))+GROUP BY+ e.prompt_id,+ s.id,+ s.name,+ s.string_value,+ s.value,+ s.source,+ s.data_type,+ s.comment,+ s.timestamp,+ s.metadata",+}+`;++exports[`golden: scores.getScoresForExperimentItems > experimentIds set 1`] = `+{+ "params": {+ "param1": 0,+ "param2": "golden-project",+ "param3": [+ "exp-a",+ "exp-b",+ ],+ "param4": "",+ "param5": [+ "NUMERIC",+ "BOOLEAN",+ "CATEGORICAL",+ ],+ },+ "sql": "SELECT+ s.id AS id,+ s.timestamp AS timestamp,+ s.project_id AS project_id,+ s.environment AS environment,+ s.trace_id AS trace_id,+ s.session_id AS session_id,+ s.observation_id AS observation_id,+ s.dataset_run_id AS dataset_run_id,+ s.name AS name,+ s.value AS value,+ s.source AS source,+ s.comment AS comment,+ s.author_user_id AS author_user_id,+ s.config_id AS config_id,+ s.data_type AS data_type,+ s.string_value AS string_value,+ s.queue_id AS queue_id,+ s.execution_trace_id AS execution_trace_id,+ s.created_at AS created_at,+ s.updated_at AS updated_at,+ s.event_ts AS event_ts,+ s.is_deleted AS is_deleted,+ length(mapKeys(s.metadata)) > {param1:Int64} AS has_metadata,+ e.experiment_id AS experiment_id+FROM+(+ SELECT+ e.project_id,+ e.experiment_id,+ e.trace_id+ FROM events_core AS e+ WHERE (e.project_id = {param2:String}) AND (e.experiment_id IN ({param3:Array(String)})) AND (e.experiment_id != {param4:String})+) AS e+INNER JOIN scores AS s+FINAL ON (e.trace_id = s.trace_id) AND (e.project_id = s.project_id) AND (s.project_id = {param2:String})+WHERE s.data_type IN ({param5:Array(String)})+ORDER BY s.event_ts DESC+LIMIT 1 BY+ s.id,+ s.project_id,+ e.experiment_id",+}+`;diff --git a/packages/shared/src/server/repositories/scores.golden.test.ts b/packages/shared/src/server/repositories/scores.golden.test.tsnew file mode 100644index 0000000000..961a1ce248--- /dev/null+++ b/packages/shared/src/server/repositories/scores.golden.test.ts@@ -0,0 +1,117 @@+import { beforeEach, describe, expect, it, vi } from "vitest";++const mockQueryClickhouse = vi.hoisted(() => vi.fn().mockResolvedValue([]));++vi.mock("./clickhouse", () => ({+ queryClickhouse: mockQueryClickhouse,+ queryClickhouseStream: vi.fn(),+ queryClickhouseExecRaw: vi.fn(),+ commandClickhouse: vi.fn(),+ upsertClickhouse: vi.fn(),+ parseClickhouseUTCDateTimeFormat: vi.fn(),+ clickhouseCompliantRandomCharacters: vi.fn(() => "x"),+ BLOB_EXPORT_PARQUET_CLICKHOUSE_SETTINGS: {},+}));++// Break the query-options -> server-barrel import cycle that otherwise fails+// module initialization when the graph is entered from this repository file.+vi.mock("../queries/clickhouse-sql/query-options", () => ({+ shouldSkipObservationsFinal: vi.fn().mockResolvedValue(false),+}));++import {+ clickhouseFormatAvailable,+ formatSql,+ normalizeParams,+} from "./goldenHarness";+import {+ getAggregatedScoresForPromptsFromEvents,+ getScoresForExperimentItems,+} from "../index";++const FIXED_PROJECT_ID = "golden-project";+const FIXED_EXPERIMENT_IDS = ["exp-a", "exp-b"];+const FIXED_PROMPT_IDS = ["prompt-a", "prompt-b"];+const FIXED_FROM_TIMESTAMP = new Date("2026-01-01T00:00:00.000Z");+const FIXED_TO_TIMESTAMP = new Date("2026-01-31T23:59:59.000Z");++const describeWithClickhouse = clickhouseFormatAvailable()+ ? describe+ : describe.skip;++if (!clickhouseFormatAvailable()) {+ console.warn(+ "[golden-harness] `clickhouse format` unavailable β skipping golden SQL tests. Install clickhouse-local to run them.",+ );+}++function snapshotCapturedQuery() {+ expect(mockQueryClickhouse).toHaveBeenCalledTimes(1);+ const { query, params } = mockQueryClickhouse.mock.calls[0][0] as {+ query: string;+ params: Record<string, unknown>;+ };+ return normalizeParams(formatSql(query), params);+}++describeWithClickhouse("golden: scores.getScoresForExperimentItems", () => {+ beforeEach(() => {+ vi.clearAllMocks();+ mockQueryClickhouse.mockResolvedValue([]);+ });++ it("experimentIds set", async () => {+ await getScoresForExperimentItems(FIXED_PROJECT_ID, FIXED_EXPERIMENT_IDS);++ expect(snapshotCapturedQuery()).toMatchSnapshot();+ });+});++describeWithClickhouse(+ "golden: scores.getAggregatedScoresForPromptsFromEvents",+ () => {+ beforeEach(() => {+ vi.clearAllMocks();+ mockQueryClickhouse.mockResolvedValue([]);+ });++ const variants: Array<{+ relation: "observation" | "trace";+ fromTimestamp?: Date;+ toTimestamp?: Date;+ }> = [+ { relation: "observation" },+ { relation: "trace" },+ {+ relation: "observation",+ fromTimestamp: FIXED_FROM_TIMESTAMP,+ toTimestamp: FIXED_TO_TIMESTAMP,+ },+ { relation: "trace", fromTimestamp: FIXED_FROM_TIMESTAMP },+ ];++ for (const variant of variants) {+ const bounds = [+ variant.fromTimestamp ? "from" : null,+ variant.toTimestamp ? "to" : null,+ ]+ .filter(Boolean)+ .join("+");+ const name = `relation=${variant.relation} time=${bounds || "unset"}`;++ it(name, async () => {+ await getAggregatedScoresForPromptsFromEvents(+ FIXED_PROJECT_ID,+ FIXED_PROMPT_IDS,+ variant.relation,+ {+ fromTimestamp: variant.fromTimestamp,+ toTimestamp: variant.toTimestamp,+ },+ );++ expect(snapshotCapturedQuery()).toMatchSnapshot();+ });+ }+ },+);diff --git a/packages/shared/src/server/repositories/scores.ts b/packages/shared/src/server/repositories/scores.tsindex f7774b1957..d1d17ebe51 100644--- a/packages/shared/src/server/repositories/scores.ts+++ b/packages/shared/src/server/repositories/scores.ts@@ -35,7 +35,6 @@ import {orderByToClickhouseSql,StringOptionsFilter,DateTimeFilter,- CTEQueryBuilder,NumberFilter,scoreOnlyFiltersAreSeekEligible,} from "../queries";@@ -73,9 +72,10 @@ import { scoresColumnsTableUiColumnDefinitions } from "../tableMappings/mapScoreimport {eventsTraceMetadata,eventsExperimentTraceIds,- eventsExperiments,- promptEventsForMetrics,} from "../queries/clickhouse-sql/query-fragments";+import { compileClickhouseQuery } from "../query-ast/compile";+import { getClickhouseKysely } from "../query-ast/dialect";+import { limitBy, mapKeys, useFinal } from "../query-ast/extensions";import { scoresTableCols } from "../../tableDefinitions/scoresTable";import {findUiColumnMapping,@@ -408,6 +408,57 @@ export const getTraceScoresForDatasetRuns = async (}));};+const experimentItemEventsQuery = (experimentIds: string[]) =>+ getClickhouseKysely()+ .selectFrom("events_core as e")+ .select(["e.project_id", "e.experiment_id", "e.trace_id"])+ .where("e.experiment_id", "in", experimentIds)+ .where("e.experiment_id", "!=", "");++const scoresForExperimentItemsQuery = (experimentIds: string[]) =>+ getClickhouseKysely()+ .selectFrom(experimentItemEventsQuery(experimentIds).as("e"))+ .innerJoin("scores as s", (join) =>+ join+ .onRef("e.trace_id", "=", "s.trace_id")+ .onRef("e.project_id", "=", "s.project_id"),+ )+ .$call(useFinal(["scores"]))+ .select((eb) => [+ "s.id as id",+ "s.timestamp as timestamp",+ "s.project_id as project_id",+ "s.environment as environment",+ "s.trace_id as trace_id",+ "s.session_id as session_id",+ "s.observation_id as observation_id",+ "s.dataset_run_id as dataset_run_id",+ "s.name as name",+ "s.value as value",+ "s.source as source",+ "s.comment as comment",+ "s.author_user_id as author_user_id",+ "s.config_id as config_id",+ "s.data_type as data_type",+ "s.string_value as string_value",+ "s.queue_id as queue_id",+ "s.execution_trace_id as execution_trace_id",+ "s.created_at as created_at",+ "s.updated_at as updated_at",+ "s.event_ts as event_ts",+ "s.is_deleted as is_deleted",+ eb(eb.fn("length", [mapKeys("s.metadata")]), ">", 0).as("has_metadata"),+ "e.experiment_id as experiment_id",+ ])+ .where("s.data_type", "in", [...AGGREGATABLE_SCORE_TYPES])+ .orderBy("s.event_ts", "desc")+ .$call(+ limitBy({+ count: 1,+ columns: ["s.id", "s.project_id", "e.experiment_id"],+ }),+ );+export const getScoresForExperimentItems = async (projectId: string,experimentIds: string[],@@ -421,48 +472,10 @@ export const getScoresForExperimentItems = async (> => {if (experimentIds.length === 0) return [];- // Build events subquery using the query builder- const eventsSubquery = eventsExperiments({- projectId,- experimentIds,- })- .selectRaw("e.project_id", "e.experiment_id", "e.trace_id")- .buildWithParams();-- const query = `- SELECT- s.id as id,- s.timestamp as timestamp,- s.project_id as project_id,- s.environment as environment,- s.trace_id as trace_id,- s.session_id as session_id,- s.observation_id as observation_id,- s.dataset_run_id as dataset_run_id,- s.name as name,- s.value as value,- s.source as source,- s.comment as comment,- s.author_user_id as author_user_id,- s.config_id as config_id,- s.data_type as data_type,- s.string_value as string_value,- s.queue_id as queue_id,- s.execution_trace_id as execution_trace_id,- s.created_at as created_at,- s.updated_at as updated_at,- s.event_ts as event_ts,- s.is_deleted as is_deleted,- length(mapKeys(s.metadata)) > 0 AS has_metadata,- e.experiment_id as experiment_id- FROM (${eventsSubquery.query}) e- JOIN scores s FINAL ON e.trace_id = s.trace_id- AND e.project_id = s.project_id- WHERE s.project_id = {projectId: String}- AND s.data_type IN ({dataTypes: Array(String)})- ORDER BY s.event_ts DESC- LIMIT 1 BY s.id, s.project_id, e.experiment_id- `;+ const compiled = compileClickhouseQuery(+ scoresForExperimentItemsQuery(experimentIds),+ { projectId },+ );const rows = await queryClickhouse<Omit<ScoreRecordReadType, "metadata"> & {@@ -470,13 +483,9 @@ export const getScoresForExperimentItems = async (experiment_id: string;}>({- query,- params: {- projectId,- ...eventsSubquery.params,- dataTypes: AGGREGATABLE_SCORE_TYPES,- },- tags: { projectId },+ query: compiled.sql,+ params: compiled.params,+ tags: { projectId, route: "scores.getScoresForExperimentItems" },preferredClickhouseService: "EventsReadOnly",});@@ -2197,92 +2206,90 @@ export const getAggregatedScoresForPrompts = async (}));};-export const buildAggregatedScoresForPromptsFromEventsQuery = (- projectId: string,+const promptEventsForScoresQuery = (promptIds: string[],- fetchScoreRelation: "observation" | "trace",- {- fromTimestamp,- toTimestamp,- }: { fromTimestamp?: Date; toTimestamp?: Date } = {},-) => {- const promptEvents = promptEventsForMetrics({- projectId,- promptIds,- ...(fromTimestamp- ? { fromTimestamp: convertDateToClickhouseDateTime(fromTimestamp) }- : {}),- ...(toTimestamp- ? { toTimestamp: convertDateToClickhouseDateTime(toTimestamp) }- : {}),- });-- const scoreRows = {- query: `- SELECT- e.prompt_id AS prompt_id,- s.id AS id,- s.name AS name,- s.string_value AS string_value,- s.value AS value,- s.source AS source,- s.data_type AS data_type,- s.comment AS comment,- s.timestamp AS timestamp,- s.metadata AS metadata- FROM scores s FINAL- INNER JOIN prompt_events e- ON s.project_id = e.project_id- AND s.trace_id = e.trace_id- ${fetchScoreRelation === "observation" ? "AND s.observation_id = e.span_id" : ""}- WHERE s.project_id = {scoreProjectId: String}- AND e.is_deleted = 0- AND s.name IS NOT NULL- AND s.data_type IN ({dataTypes: Array(String)})- ${- fetchScoreRelation === "trace"- ? `AND s.observation_id IS NULL- AND s.trace_id IN (SELECT trace_id FROM prompt_events WHERE is_deleted = 0)`- : `AND s.observation_id IS NOT NULL- AND (s.trace_id, s.observation_id) IN (SELECT trace_id, span_id FROM prompt_events WHERE is_deleted = 0)`- }- `,- params: {- scoreProjectId: projectId,- dataTypes: LISTABLE_SCORE_TYPES,- },- schema: [- "prompt_id",- "id",- "name",- "string_value",- "value",- "source",- "data_type",- "comment",- "timestamp",- "metadata",- ],- };+ timeWindow: { fromTimestamp?: Date; toTimestamp?: Date } = {},+) =>+ getClickhouseKysely()+ .selectFrom("events_core as e")+ .select([+ "e.project_id as project_id",+ "e.prompt_id as prompt_id",+ "e.trace_id as trace_id",+ "e.span_id as span_id",+ "e.is_deleted as is_deleted",+ ])+ .where("e.type", "=", "GENERATION")+ .where("e.prompt_id", "in", promptIds)+ .$if(timeWindow.fromTimestamp !== undefined, (qb) =>+ qb.where("e.start_time", ">=", timeWindow.fromTimestamp!),+ )+ .$if(timeWindow.toTimestamp !== undefined, (qb) =>+ qb.where("e.start_time", "<=", timeWindow.toTimestamp!),+ )+ .orderBy("e.event_ts", "desc")+ .$call(limitBy({ count: 1, columns: ["e.span_id", "e.project_id"] }));- return new CTEQueryBuilder()- .withCTE("prompt_events", promptEvents)- .withCTE("score_rows", scoreRows)- .from("score_rows", "s")- .select(- "s.prompt_id AS prompt_id",- "s.id AS id",- "s.name AS name",- "s.string_value AS string_value",- "s.value AS value",- "s.source AS source",- "s.data_type AS data_type",- "s.comment AS comment",- "s.timestamp AS timestamp",- "length(mapKeys(s.metadata)) > 0 AS has_metadata",+const aggregatedScoresForPromptsFromEventsQuery = (+ promptIds: string[],+ fetchScoreRelation: "observation" | "trace",+ timeWindow: { fromTimestamp?: Date; toTimestamp?: Date } = {},+) =>+ getClickhouseKysely()+ .with("prompt_events", () =>+ promptEventsForScoresQuery(promptIds, timeWindow),+ )+ .selectFrom("scores as s")+ .$call(useFinal(["scores"]))+ .innerJoin("prompt_events as e", (join) => {+ const base = join+ .onRef("s.project_id", "=", "e.project_id")+ .onRef("s.trace_id", "=", "e.trace_id");+ return fetchScoreRelation === "observation"+ ? base.onRef("s.observation_id", "=", "e.span_id")+ : base;+ })+ .select((eb) => [+ "e.prompt_id as prompt_id",+ "s.id as id",+ "s.name as name",+ "s.string_value as string_value",+ "s.value as value",+ "s.source as source",+ "s.data_type as data_type",+ "s.comment as comment",+ "s.timestamp as timestamp",+ eb(eb.fn("length", [mapKeys("s.metadata")]), ">", 0).as("has_metadata"),+ ])+ .where("e.is_deleted", "=", 0)+ .where("s.name", "is not", null)+ .where("s.data_type", "in", [...LISTABLE_SCORE_TYPES])+ .$if(fetchScoreRelation === "trace", (qb) =>+ qb+ .where("s.observation_id", "is", null)+ .where("s.trace_id", "in", (eb) =>+ eb+ .selectFrom("prompt_events")+ .select("trace_id")+ .where("is_deleted", "=", 0),+ ),)- .groupBy(- "s.prompt_id",+ .$if(fetchScoreRelation === "observation", (qb) =>+ qb+ .where("s.observation_id", "is not", null)+ .where(({ eb, refTuple, selectFrom }) =>+ eb(+ refTuple("s.trace_id", "s.observation_id"),+ "in",+ selectFrom("prompt_events")+ .select(["trace_id", "span_id"])+ .where("is_deleted", "=", 0)+ .$asTuple("trace_id", "span_id"),+ ),+ ),+ )+ .groupBy([+ "e.prompt_id","s.id","s.name","s.string_value",@@ -2292,8 +2299,23 @@ export const buildAggregatedScoresForPromptsFromEventsQuery = ("s.comment","s.timestamp","s.metadata",- )- .buildWithParams();+ ]);++export const buildAggregatedScoresForPromptsFromEventsQuery = (+ projectId: string,+ promptIds: string[],+ fetchScoreRelation: "observation" | "trace",+ timeWindow: { fromTimestamp?: Date; toTimestamp?: Date } = {},+) => {+ const compiled = compileClickhouseQuery(+ aggregatedScoresForPromptsFromEventsQuery(+ promptIds,+ fetchScoreRelation,+ timeWindow,+ ),+ { projectId },+ );+ return { query: compiled.sql, params: compiled.params };};export const getAggregatedScoresForPromptsFromEvents = async (@@ -2317,7 +2339,10 @@ export const getAggregatedScoresForPromptsFromEvents = async (>({query,params,- tags: { projectId },+ tags: {+ projectId,+ route: "scores.getAggregatedScoresForPromptsFromEvents",+ },preferredClickhouseService: "EventsReadOnly",});diff --git a/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts b/web/src/__tests__/server/api/tables/prompts-ui.servertest.tsindex 6f9f2cb1fe..5bb062f3f3 100644--- a/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts+++ b/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts@@ -587,9 +587,9 @@ describe("UI Prompts Table", () => {"observation",);- expect(query).toContain("INNER JOIN prompt_events");- expect(query).toContain(- "(s.trace_id, s.observation_id) IN (SELECT trace_id, span_id FROM prompt_events",+ expect(query).toMatch(/inner join prompt_events/i);+ expect(query).toMatch(+ /\(s\.trace_id, s\.observation_id\) in \(select trace_id, span_id from prompt_events/i,);});