Skip to content

πŸͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(scores): compile experiment-item and prompt-event reads through the query AST (#17553)

Niklas Semmlerdd4ca52fcfad
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.md
index 67b7213ca4..1abd74207a 100644
--- a/packages/shared/AGENTS.md
+++ b/packages/shared/AGENTS.md
@@ -41,7 +41,7 @@
- Code evaluator dispatcher/error contract: `src/server/evals/codeEvalDispatcherTypes.ts`. Keep provider mappings, user-visible messages, and worker terminal-outcome classification aligned when adding an error code.
- Dashboard/monitor query feature (data model + server-only builder/executor): `src/features/query/*`
- Query-builder AST (server half, WIP): `src/server/query-ast/*` β€” the Kysely
- ClickHouse dialect (ARRAY JOIN / LIMIT BY / metadata indexOf nodes,
+ ClickHouse dialect (ARRAY JOIN / LIMIT BY / FINAL / metadata indexOf nodes,
`ExecutionContext` tenancy injection, per-table dedup lowering, virtual views,
catalog parity). Compile only through `compileClickhouseQuery` in
`src/server/query-ast/compile.ts`. SQL correctness is proven by a golden-SQL
@@ -50,6 +50,7 @@
`clickhouse format`); each migrated call site keeps its `*.golden.test.ts`
baseline next to the call site (e.g.
`src/server/repositories/environments.golden.test.ts`,
+ `src/server/repositories/scores.golden.test.ts`,
`src/server/queries/clickhouse-sql/event-filter-options.golden.test.ts`).
- Postgres schema: `prisma/schema.prisma`
- Prisma migrations: `prisma/migrations/*`
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index 3949c0ab22..ccb9862838 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -8,10 +8,10 @@ metadata, `FilterState` embedding) will land under
`packages/shared/src/query-ast/`.
This folder is a compile-only ClickHouse dialect on Kysely 0.28: real
-`OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf` subscripts; a
-mandatory tenancy injection pass keyed on `ExecutionContext`; per-table dedup
-lowering from the registry; schema-typed selection; virtual views as WITH CTEs;
-catalog parity.
+`OperationNode`s for ARRAY JOIN, LIMIT BY, FINAL, and metadata `indexOf`
+subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;
+per-table dedup lowering from the registry; schema-typed selection; virtual
+views as WITH CTEs; catalog parity.
Two audiences: **[Using it](#using-it)** if you are calling the compiler from a
repository, **[Developing the dialect](#developing-the-dialect)** if you are
@@ -29,16 +29,18 @@ changing how it emits SQL.
of hand-maintained tables that drift.
- `compiler.ts` / `transformer.ts` / `dialect.ts` / `nodes.ts` β€” the dialect.
- `extensions.ts` β€” the `$call` helpers: `arrayJoin`, `mapKeys`, `mapValues`,
- `limitBy`, `metadataValue`.
-- `catalog.ts` / `views.ts` / `dedup.ts` β€” catalog, virtual views as CTEs, and
- per-table dedup lowering.
+ `limitBy`, `useFinal`, `metadataValue`.
+- `views.ts` / `dedup.ts` β€” virtual views as CTEs and per-table dedup lowering.
+- `fixtures/catalog.ts` β€” test-only catalog of builder queries exercising the
+ dialect's expressivity.
- `catalog.golden.test.ts` β€” catalog SQL baselines.
The golden-SQL harness lives in `../repositories/goldenHarness.ts` (it captures
SQL at the `repositories/clickhouse.ts` exec seam in test mode, then normalizes
it via `clickhouse format` + positional param names for snapshot comparison β€” no
ClickHouse server needed). Each migrated call site keeps its SQL baseline next to
-that call site, e.g. `../repositories/environments.golden.test.ts` and
+that call site, e.g. `../repositories/environments.golden.test.ts`,
+`../repositories/scores.golden.test.ts`, and
`../queries/clickhouse-sql/event-filter-options.golden.test.ts`.
Regenerate baselines with `-u` after an intentional SQL change. Filter by the
@@ -81,7 +83,8 @@ suffix to run there. Promote it to a required check once it has proven stable.
`events_core` is `none` (immutable at read time β€” no LIMIT BY, no FINAL).
`limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` already
used on traces / observations / scores. `$call(limitBy(...))` remains for
- explicit non-version LIMIT BY.
+ explicit non-version LIMIT BY. Auto-lowering does not declare scores as
+ `{ strategy: "final" }`; callers opt in with `$call(useFinal(["scores"]))`.
- **Value binds take their ClickHouse type from the compared column** in the
table registry (`total_cost > 1` emits `{p:Float64}`, not inferred
`{p:Int64}`). Same-value binds still intern to one placeholder.
@@ -115,6 +118,15 @@ db.selectFrom("events_core")
// … order by event_ts desc limit 1 by span_id, project_id
```
+**Write FINAL** (`useFinal` β€” ReplacingMergeTree reads; never on `events_core`):
+
+```ts
+db.selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"]));
+// … from scores as s final
+```
+
**Select a metadata value** (`metadataValue` β€” lowers `metadata[key]` to a bound `indexOf` subscript):
```ts
@@ -140,6 +152,7 @@ preserve it:
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
| ARRAY JOIN | Plugin attaches `ArrayJoinNode` as an extra field on `SelectQueryNode`. `ClickHouseOperationNodeTransformer` preserves it. `ClickHouseQueryCompiler.visitSelectQuery` emits it after JOINs / before WHERE. | no |
| LIMIT BY | Plugin attaches `LimitByNode` the same way. Compiler emits it after ORDER BY / before LIMIT. | no |
+| FINAL | Plugin wraps a FROM/JOIN table expression in `FinalTableNode`. Compiler emits `final` after that expression (`scores as s final`). `$call(useFinal(["scores"]))`. Tables declared `dedup: none` throw. | no |
| metadata `indexOf` | Helper builds an `ArrayIndexNode` whose index child is a `FunctionNode` (`indexOf`) over a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no |
| Virtual view | Plugin rewrites `selectFrom(viewName)` into a WITH CTE. Outer types only expose the view's selected columns. | no |
@@ -178,8 +191,8 @@ is where tenancy is enforced:
(`none` / `limitBy` / `final`). `events_core` is `none` β€” immutable at
read time, so the pass does not inject LIMIT BY or FINAL. `limitBy` is
the existing legacy `ORDER BY <version> DESC LIMIT 1 BY <key>`.
- `final` is fail-closed until an emitter exists. The pass restamps the
- rewritten root.
+ `final` remains fail-closed on the auto-lowering pass; callers opt in
+ with `$call(useFinal(["scores"]))`. The pass restamps the rewritten root.
4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is
present, so `qb.compile()` without the plugin also fails. Value binds take
their ClickHouse type from the compared column's registry entry when one is
@@ -199,7 +212,8 @@ callbacks hand you. Getting that requires either forking Kysely's whole builder
graph or globally mutating its prototype via an internal `kysely/dist/...`
import (blocked here by Kysely's `exports` map under NodeNext). A curried helper
is a plain function of a builder, so it works in any of those positions β€” which
-is exactly why ARRAY JOIN / LIMIT BY compose inside CTEs, subqueries, and views.
+is exactly why ARRAY JOIN / LIMIT BY / FINAL compose inside CTEs, subqueries,
+and views.
`composition.test.ts` locks that property in.
**`arrayJoin` widens the row type.** Each `{ alias: arrayExpr }` entry is added
@@ -230,16 +244,17 @@ is just `eb.fn("arrayJoin", [...])`.
### The one Kysely-internals coupling (upgrade hazard)
`compiler.ts` wraps Kysely's **private** `visitNode` / `nodeStack` to dispatch
-`ArrayIndexNode` (for `metadata[key]`), because it is not one of Kysely's closed
-`OperationNode` kinds. Kysely is pinned to **0.28.17** for this reason. Re-verify
-this hack on any Kysely bump; it is the single place that reaches past Kysely's
-documented surface. Everything else (plugins, dialect, transformer overrides)
-uses public or documented-protected API.
+`ArrayIndexNode` (for `metadata[key]`) and `FinalTableNode` (for `FINAL`),
+because they are not Kysely's closed `OperationNode` kinds. Kysely is pinned
+to **0.28.17** for this reason. Re-verify this hack on any Kysely bump; it is
+the single place that reaches past Kysely's documented surface. Everything
+else (plugins, dialect, transformer overrides) uses public or
+documented-protected API.
### Types are asserted at compile time
`types.assert.ts` holds `tsc`-only assertions (schema typing, view opacity,
-arrayJoin widening, limitBy preservation). It is never run; `schema.test.ts`
+arrayJoin widening, limitBy / useFinal preservation). It is never run; `schema.test.ts`
anchors it so it stays in the build graph. `@ts-expect-error` lines there must
stay live.
diff --git a/packages/shared/src/server/query-ast/catalog.golden.test.ts b/packages/shared/src/server/query-ast/catalog.golden.test.ts
index fe3dc162fc..8af77f40f8 100644
--- a/packages/shared/src/server/query-ast/catalog.golden.test.ts
+++ b/packages/shared/src/server/query-ast/catalog.golden.test.ts
@@ -8,7 +8,7 @@ import {
normalizeParams,
substituteNamedParams,
} from "../repositories/goldenHarness";
-import { CATALOG, CATALOG_PROJECT_ID } from "./catalog";
+import { CATALOG, CATALOG_PROJECT_ID } from "./fixtures/catalog";
import { compileClickhouseQuery } from "./compile";
const describeWithClickhouse = clickhouseFormatAvailable()
diff --git a/packages/shared/src/server/query-ast/compiler.ts b/packages/shared/src/server/query-ast/compiler.ts
index 9156d82a3c..9f868df245 100644
--- a/packages/shared/src/server/query-ast/compiler.ts
+++ b/packages/shared/src/server/query-ast/compiler.ts
@@ -20,6 +20,7 @@ import type { QueryId } from "kysely";
import {
ArrayJoinNode,
ArrayIndexNode,
+ FinalTableNode,
type ClickHouseSelectQueryNode,
type LimitByNode,
} from "./nodes";
@@ -103,6 +104,12 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
self.nodeStack.pop();
return;
}
+ if (FinalTableNode.is(node)) {
+ self.nodeStack.push(node);
+ this.visitFinalTable(node);
+ self.nodeStack.pop();
+ return;
+ }
parentVisit(node);
};
}
@@ -347,6 +354,11 @@ export class ClickHouseQueryCompiler extends DefaultQueryCompiler {
this.compileList(node.columns);
}
+ private visitFinalTable(node: FinalTableNode): void {
+ this.visitNode(node.table);
+ this.append(" final");
+ }
+
private bindValue(value: unknown, type?: string): string {
// `type` is the caller-known ClickHouse type; when omitted we infer the
// widest canonical type from the JS value (Int64 / Float64 / String /
diff --git a/packages/shared/src/server/query-ast/composition.test.ts b/packages/shared/src/server/query-ast/composition.test.ts
index 5435aa0bf5..95f1b5a56c 100644
--- a/packages/shared/src/server/query-ast/composition.test.ts
+++ b/packages/shared/src/server/query-ast/composition.test.ts
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
import { compileClickhouseQuery } from "./compile";
import { getClickhouseKysely } from "./dialect";
-import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";
+import { arrayJoin, limitBy, mapKeys, mapValues, useFinal } from "./extensions";
import { defineView, fromView } from "./views";
// Raw compiler output (no `clickhouse format` binary needed), so these run
@@ -124,3 +124,56 @@ describe("LIMIT BY composition across nesting positions", () => {
expect(sql).toMatch(/with lb as/i);
});
});
+
+describe("FINAL composition across nesting positions", () => {
+ it("emits FINAL at the top level (baseline)", () => {
+ const db = getClickhouseKysely();
+ const qb = db
+ .selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"]));
+
+ const { sql } = compileClickhouseQuery(qb, ctx);
+ expect(sql).toMatch(/scores as s final/i);
+ });
+
+ it("keeps FINAL when applied inside a CTE body", () => {
+ const db = getClickhouseKysely();
+ const qb = db
+ .with("scored", (inner) =>
+ inner
+ .selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"])),
+ )
+ .selectFrom("scored")
+ .select("id");
+
+ const { sql } = compileClickhouseQuery(qb, ctx);
+ const outerStart = sql
+ .toLowerCase()
+ .search(/\)\s*select[\s\S]*from scored/i);
+ expect(outerStart).toBeGreaterThan(-1);
+ const cteBody = sql.slice(0, outerStart);
+ const outerQuery = sql.slice(outerStart);
+ expect(cteBody).toMatch(/scores as s final/i);
+ expect(outerQuery).not.toMatch(/final/i);
+ });
+
+ it("keeps FINAL on a JOIN table and not on events_core", () => {
+ const db = getClickhouseKysely();
+ const qb = db
+ .selectFrom("events_core as e")
+ .innerJoin("scores as s", (join) =>
+ join
+ .onRef("e.trace_id", "=", "s.trace_id")
+ .onRef("e.project_id", "=", "s.project_id"),
+ )
+ .$call(useFinal(["scores"]))
+ .select("s.id");
+
+ const { sql } = compileClickhouseQuery(qb, ctx);
+ expect(sql).toMatch(/scores as s final/i);
+ expect(sql).not.toMatch(/events_core as e final/i);
+ });
+});
diff --git a/packages/shared/src/server/query-ast/dedup.ts b/packages/shared/src/server/query-ast/dedup.ts
index b088ca61b3..9a86e6195c 100644
--- a/packages/shared/src/server/query-ast/dedup.ts
+++ b/packages/shared/src/server/query-ast/dedup.ts
@@ -42,7 +42,12 @@ import {
} from "kysely";
import { QueryCompileError } from "./errors";
-import { LimitByNode, type ClickHouseSelectQueryNode } from "./nodes";
+import {
+ FinalTableNode,
+ LimitByNode,
+ unwrapFinalTable,
+ type ClickHouseSelectQueryNode,
+} from "./nodes";
import { DEDUP_SPECS, type DedupSpec } from "./schema";
import { stampCompiledTree } from "./tenancy";
import { ClickHouseOperationNodeTransformer } from "./transformer";
@@ -133,6 +138,9 @@ function singlePhysicalFrom(
}
function describeFrom(node: OperationNode): PhysicalFrom | undefined {
+ if (FinalTableNode.is(node)) {
+ return describeFrom(unwrapFinalTable(node));
+ }
if (AliasNode.is(node)) {
const inner = describeFrom(node.node);
if (!inner) return undefined;
diff --git a/packages/shared/src/server/query-ast/extensions.ts b/packages/shared/src/server/query-ast/extensions.ts
index dcb64d2c9a..1279aafe37 100644
--- a/packages/shared/src/server/query-ast/extensions.ts
+++ b/packages/shared/src/server/query-ast/extensions.ts
@@ -6,19 +6,23 @@
* - Plugins ({@link ArrayJoinPlugin}, {@link LimitByPlugin}) that attach a
* custom operation node (`ArrayJoinNode` / `LimitByNode`) onto the select
* node during query transformation; the ClickHouse compiler renders them.
- * - `$call` helpers ({@link arrayJoin}, {@link limitBy}) that apply those
- * plugins while extending the builder's output row type, so produced
- * aliases are type-checked and misspelled references are compile errors.
+ * - `$call` helpers ({@link arrayJoin}, {@link limitBy}, {@link useFinal})
+ * that apply those plugins while extending the builder's output row type,
+ * so produced aliases are type-checked and misspelled references are
+ * compile errors.
* - Typed expression helpers ({@link mapKeys}, {@link mapValues},
* {@link metadataValue}) that build ClickHouse map/array function calls and
* `metadata[key]` subscripts as real, parameter-bound nodes β€” never raw SQL
* strings β€” so they escape and compose like any other expression.
*/
import {
+ AliasNode,
ColumnNode,
ExpressionWrapper,
+ FromNode,
FunctionNode,
IdentifierNode,
+ JoinNode,
ReferenceNode,
TableNode,
ValueNode,
@@ -34,11 +38,14 @@ import {
import {
ArrayIndexNode,
ArrayJoinNode,
+ FinalTableNode,
LimitByNode,
+ unwrapFinalTable,
type ArrayJoinVariant,
type ClickHouseSelectQueryNode,
} from "./nodes";
-import type { ClickHouseDatabase } from "./schema";
+import { QueryCompileError } from "./errors";
+import { DEDUP_SPECS, type ClickHouseDatabase } from "./schema";
import { ClickHouseOperationNodeTransformer } from "./transformer";
type OperationNodeSource = OperationNode | { toOperationNode(): OperationNode };
@@ -228,3 +235,78 @@ export function limitBy(spec: LimitBySpec) {
qb: SelectQueryBuilder<DB, TB, O>,
): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new LimitByPlugin(spec));
}
+
+function physicalTableName(node: OperationNode): string | undefined {
+ if (FinalTableNode.is(node)) return physicalTableName(unwrapFinalTable(node));
+ if (AliasNode.is(node)) return physicalTableName(node.node);
+ if (TableNode.is(node)) return node.table.identifier.name;
+ return undefined;
+}
+
+function wrapFinal(
+ node: OperationNode,
+ tables: ReadonlySet<string>,
+): OperationNode {
+ if (FinalTableNode.is(node)) return node;
+ const name = physicalTableName(node);
+ return name && tables.has(name)
+ ? (FinalTableNode.create(node) as unknown as OperationNode)
+ : node;
+}
+
+/**
+ * Plugin that marks named physical tables for `FINAL`. Walks this select's
+ * FROM/JOIN table expressions only β€” apply it on the builder that owns the
+ * scores (or other ReplacingMergeTree) relation, including a CTE body.
+ */
+class FinalPlugin implements KyselyPlugin {
+ constructor(private readonly tables: ReadonlySet<string>) {}
+
+ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {
+ if (args.node.kind !== "SelectQueryNode") return args.node;
+ const transformer = new ClickHouseOperationNodeTransformer();
+ const node = transformer.transformNode(
+ args.node,
+ ) as ClickHouseSelectQueryNode;
+ const froms = (node.from?.froms ?? []).map((from) =>
+ wrapFinal(from, this.tables),
+ );
+ const joins = (node.joins ?? []).map((join) => {
+ const table = wrapFinal(join.table, this.tables);
+ if (table === join.table) return join;
+ return join.on
+ ? JoinNode.createWithOn(join.joinType, table, join.on.on)
+ : JoinNode.create(join.joinType, table);
+ });
+ return {
+ ...node,
+ // cloneWithFroms appends; replacing the list avoids
+ // `FROM scores AS s, scores AS s FINAL`.
+ ...(node.from ? { from: FromNode.create(froms) } : {}),
+ ...(joins.length ? { joins } : {}),
+ } as RootOperationNode;
+ }
+
+ async transformResult(args: PluginTransformResultArgs) {
+ return args.result;
+ }
+}
+
+/**
+ * `FROM <table> FINAL` / `JOIN <table> AS alias FINAL` as a `$call` step:
+ * `qb.$call(useFinal(["scores"]))`. Tables declared `dedup: none` (immutable
+ * at read time, including `events_core`) cannot take FINAL.
+ */
+export function useFinal(tables: ReadonlyArray<keyof ClickHouseDatabase>) {
+ for (const table of tables) {
+ if (DEDUP_SPECS[table]?.strategy === "none") {
+ throw new QueryCompileError(
+ `Cannot apply FINAL to ${table}: declared immutable (dedup strategy none).`,
+ );
+ }
+ }
+ const names = new Set<string>(tables);
+ return <DB, TB extends keyof DB, O>(
+ qb: SelectQueryBuilder<DB, TB, O>,
+ ): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new FinalPlugin(names));
+}
diff --git a/packages/shared/src/server/query-ast/catalog.ts b/packages/shared/src/server/query-ast/fixtures/catalog.ts
similarity index 86%
rename from packages/shared/src/server/query-ast/catalog.ts
rename to packages/shared/src/server/query-ast/fixtures/catalog.ts
index 95df98fa28..12c2415201 100644
--- a/packages/shared/src/server/query-ast/catalog.ts
+++ b/packages/shared/src/server/query-ast/fixtures/catalog.ts
@@ -1,14 +1,15 @@
import type { SqlBool } from "kysely";
-import type { ClickhouseCompilable } from "./compile";
-import { getClickhouseKysely } from "./dialect";
+import type { ClickhouseCompilable } from "../compile";
+import { getClickhouseKysely } from "../dialect";
import {
arrayJoin,
limitBy,
mapKeys,
mapValues,
metadataValue,
-} from "./extensions";
+ useFinal,
+} from "../extensions";
type CatalogTier = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7;
@@ -189,6 +190,57 @@ export const CATALOG: CatalogEntry[] = [
eb.selectFrom("traces_cte").select("id"),
),
},
+ {
+ // Explicit FINAL on a ReplacingMergeTree read. Auto-lowering does not
+ // declare scores as `{ strategy: "final" }` (that would rewrite every
+ // scores scan and has no JOIN emitter); callers opt in with useFinal.
+ id: "scores_final",
+ tier: 2,
+ referenceSql: `
+ SELECT s.id
+ FROM scores AS s FINAL
+ WHERE s.project_id = {p1:String}
+ `,
+ build: () =>
+ db()
+ .selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"])),
+ },
+ {
+ // Tuple IN-subquery: `(trace_id, observation_id) IN (SELECT …)`.
+ id: "tuple_in_subquery",
+ tier: 2,
+ referenceSql: `
+ WITH prompt_events AS (
+ SELECT trace_id, span_id
+ FROM events_core
+ WHERE project_id = {p1:String}
+ )
+ SELECT s.id
+ FROM scores AS s
+ WHERE (s.project_id = {p1:String}) AND ((s.trace_id, s.observation_id) IN (
+ SELECT trace_id, span_id
+ FROM prompt_events
+ ))
+ `,
+ build: () =>
+ db()
+ .with("prompt_events", (qb) =>
+ qb.selectFrom("events_core").select(["trace_id", "span_id"]),
+ )
+ .selectFrom("scores as s")
+ .select("s.id")
+ .where(({ eb, refTuple, selectFrom }) =>
+ eb(
+ refTuple("s.trace_id", "s.observation_id"),
+ "in",
+ selectFrom("prompt_events")
+ .select(["trace_id", "span_id"])
+ .$asTuple("trace_id", "span_id"),
+ ),
+ ),
+ },
{
id: "array_join_cost_details",
tier: 3,
diff --git a/packages/shared/src/server/query-ast/nodes.test.ts b/packages/shared/src/server/query-ast/nodes.test.ts
index 7b911e5350..48d7ab3d5f 100644
--- a/packages/shared/src/server/query-ast/nodes.test.ts
+++ b/packages/shared/src/server/query-ast/nodes.test.ts
@@ -5,9 +5,10 @@ import { compileClickhouseQuery } from "./compile";
import { ClickHouseQueryCompiler } from "./compiler";
import { getClickhouseKysely } from "./dialect";
import { QueryCompileError, UnscopedRelationError } from "./errors";
-import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";
+import { arrayJoin, limitBy, mapKeys, mapValues, useFinal } from "./extensions";
import {
ArrayJoinNode,
+ FinalTableNode,
LimitByNode,
isClickHouseSelectQueryNode,
} from "./nodes";
@@ -285,4 +286,36 @@ describe("ARRAY JOIN and LIMIT BY nodes", () => {
expect(lower).toContain(" by ");
expect(lower.indexOf(" by ")).toBeLessThan(lower.lastIndexOf("limit"));
});
+
+ it("wraps the scores FROM table in a FinalTableNode", () => {
+ const qb = getClickhouseKysely()
+ .selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"]));
+
+ const node = qb
+ .withPlugin(new TenancyInjectionPlugin(ctx))
+ .toOperationNode();
+ expect(node.kind).toBe("SelectQueryNode");
+ const from = (node as { from?: { froms?: Array<{ kind: string }> } }).from
+ ?.froms?.[0];
+ expect(from).toBeDefined();
+ expect(FinalTableNode.is(from!)).toBe(true);
+ });
+
+ it("emits FINAL after the scores table expression", () => {
+ const qb = getClickhouseKysely()
+ .selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"]));
+
+ const { sql: compiled } = compileClickhouseQuery(qb, ctx);
+ expect(compiled.toLowerCase()).toMatch(/from scores as s final/);
+ expect(compiled.toLowerCase()).not.toMatch(/from scores as s,/);
+ });
+
+ it("rejects FINAL on events_core", () => {
+ expect(() => useFinal(["events_core"])).toThrow(QueryCompileError);
+ expect(() => useFinal(["events_core"])).toThrow(/immutable/);
+ });
});
diff --git a/packages/shared/src/server/query-ast/nodes.ts b/packages/shared/src/server/query-ast/nodes.ts
index 8b08f21423..f5ef78b6ae 100644
--- a/packages/shared/src/server/query-ast/nodes.ts
+++ b/packages/shared/src/server/query-ast/nodes.ts
@@ -112,3 +112,38 @@ export function isClickHouseSelectQueryNode(
): node is ClickHouseSelectQueryNode {
return node.kind === "SelectQueryNode";
}
+
+/**
+ * ClickHouse `FROM <table> FINAL` / `JOIN <table> AS alias FINAL`.
+ *
+ * Wraps a table expression (bare `TableNode` or `AliasNode`) so the compiler
+ * emits `final` after that expression. Same closed-`kind` constraint as
+ * ARRAY JOIN: not a Kysely visitor-map kind. The dialect compiler and
+ * transformer special-case this object.
+ */
+export type FinalTableNode = {
+ readonly kind: "FinalTableNode";
+ readonly table: OperationNode;
+};
+
+export const FinalTableNode = {
+ is(node: { kind: string }): node is FinalTableNode {
+ return node.kind === "FinalTableNode";
+ },
+ create(table: OperationNode): FinalTableNode {
+ return Object.freeze({
+ kind: "FinalTableNode",
+ table,
+ });
+ },
+};
+
+/**
+ * Read the wrapped table expression. `FinalTableNode.kind` is outside
+ * Kysely's closed `OperationNodeKind`, so a narrowed
+ * `OperationNode & FinalTableNode` is `never` and `.table` is unreadable
+ * without this cast.
+ */
+export function unwrapFinalTable(node: { kind: string }): OperationNode {
+ return (node as unknown as FinalTableNode).table;
+}
diff --git a/packages/shared/src/server/query-ast/schema.test.ts b/packages/shared/src/server/query-ast/schema.test.ts
index abe4fd564c..43c6be5231 100644
--- a/packages/shared/src/server/query-ast/schema.test.ts
+++ b/packages/shared/src/server/query-ast/schema.test.ts
@@ -30,6 +30,8 @@ describe("table registry derivation", () => {
expect(COLUMN_DATA_TYPES.timestamp).toBe("date");
expect(COLUMN_DATA_TYPES.metadata_names).toBe("array");
expect(COLUMN_DATA_TYPES.cost_details).toBe("map");
+ expect(COLUMN_DATA_TYPES.is_deleted).toBe("number");
+ expect(COLUMN_DATA_TYPES.metadata).toBe("map");
});
it("declares events_core as immutable and leaves legacy tables undeclared", () => {
@@ -44,6 +46,8 @@ describe("table registry derivation", () => {
expect(COLUMN_BIND_TYPES.start_time).toBe("DateTime64(3)");
expect(COLUMN_BIND_TYPES.project_id).toBe("String");
expect(COLUMN_BIND_TYPES.metadata_names).toBe("Array(String)");
+ expect(COLUMN_BIND_TYPES.is_deleted).toBe("UInt8");
+ expect(COLUMN_BIND_TYPES.metadata).toBe("Map(String, String)");
});
// The assertions themselves are compile-time (`tsc` is the test); this only
diff --git a/packages/shared/src/server/query-ast/schema.ts b/packages/shared/src/server/query-ast/schema.ts
index 0ebd3fcaaa..ad4cc6585f 100644
--- a/packages/shared/src/server/query-ast/schema.ts
+++ b/packages/shared/src/server/query-ast/schema.ts
@@ -28,20 +28,26 @@ export type ChColumnType =
| "String"
| "Float"
| "DateTime"
+ | "UInt8"
| "Array(String)"
- | "Map(String, Float)";
+ | "Map(String, Float)"
+ | "Map(String, String)";
type TsTypeOfColumn<T extends ChColumnType> = T extends "String"
? string
: T extends "Float"
? number
- : T extends "DateTime"
- ? Date
- : T extends "Array(String)"
- ? string[]
- : T extends "Map(String, Float)"
- ? Record<string, number>
- : never;
+ : T extends "UInt8"
+ ? number
+ : T extends "DateTime"
+ ? Date
+ : T extends "Array(String)"
+ ? string[]
+ : T extends "Map(String, Float)"
+ ? Record<string, number>
+ : T extends "Map(String, String)"
+ ? Record<string, string>
+ : never;
/** Coarse runtime category the aggregate type-check pass reasons about. */
export type ColumnDataType = "string" | "number" | "date" | "array" | "map";
@@ -49,9 +55,11 @@ export type ColumnDataType = "string" | "number" | "date" | "array" | "map";
const RUNTIME_TYPE: Record<ChColumnType, ColumnDataType> = {
String: "string",
Float: "number",
+ UInt8: "number",
DateTime: "date",
"Array(String)": "array",
"Map(String, Float)": "map",
+ "Map(String, String)": "map",
};
/**
@@ -123,6 +131,9 @@ const TABLE_REGISTRY = {
event_ts: "DateTime",
type: "String",
total_cost: "Float",
+ experiment_id: "String",
+ prompt_id: "String",
+ is_deleted: "UInt8",
metadata_names: "Array(String)",
metadata_values: "Array(String)",
},
@@ -131,10 +142,29 @@ const TABLE_REGISTRY = {
}),
scores: defineTable({
columns: {
+ id: "String",
environment: "String",
project_id: "String",
timestamp: "DateTime",
+ event_ts: "DateTime",
+ created_at: "DateTime",
+ updated_at: "DateTime",
data_type: "String",
+ trace_id: "String",
+ session_id: "String",
+ observation_id: "String",
+ dataset_run_id: "String",
+ name: "String",
+ value: "Float",
+ source: "String",
+ comment: "String",
+ author_user_id: "String",
+ config_id: "String",
+ string_value: "String",
+ queue_id: "String",
+ execution_trace_id: "String",
+ is_deleted: "UInt8",
+ metadata: "Map(String, String)",
},
}),
} as const;
@@ -188,9 +218,11 @@ export const DEDUP_SPECS: Record<string, DedupSpec> = Object.fromEntries(
const BIND_TYPE: Record<ChColumnType, string> = {
String: "String",
Float: "Float64",
+ UInt8: "UInt8",
DateTime: "DateTime64(3)",
"Array(String)": "Array(String)",
"Map(String, Float)": "Map(String, Float64)",
+ "Map(String, String)": "Map(String, String)",
};
export const COLUMN_BIND_TYPES: Record<string, string> = Object.fromEntries(
diff --git a/packages/shared/src/server/query-ast/tenancy.ts b/packages/shared/src/server/query-ast/tenancy.ts
index 12789cc5ff..599c572332 100644
--- a/packages/shared/src/server/query-ast/tenancy.ts
+++ b/packages/shared/src/server/query-ast/tenancy.ts
@@ -23,7 +23,11 @@ import {
} from "kysely";
import { QueryCompileError, UnscopedRelationError } from "./errors";
-import { type ClickHouseSelectQueryNode } from "./nodes";
+import {
+ FinalTableNode,
+ unwrapFinalTable,
+ type ClickHouseSelectQueryNode,
+} from "./nodes";
import { TENANTED_TABLES } from "./schema";
import { ClickHouseOperationNodeTransformer } from "./transformer";
@@ -192,6 +196,9 @@ function assertRelationAllowed(relation: Relation): void {
}
function describeRelation(node: OperationNode): Relation {
+ if (FinalTableNode.is(node)) {
+ return describeRelation(unwrapFinalTable(node));
+ }
if (RawNode.is(node)) {
return { kind: "raw" };
}
diff --git a/packages/shared/src/server/query-ast/transformer.ts b/packages/shared/src/server/query-ast/transformer.ts
index 732b6ee430..f307743a54 100644
--- a/packages/shared/src/server/query-ast/transformer.ts
+++ b/packages/shared/src/server/query-ast/transformer.ts
@@ -8,6 +8,7 @@ import {
import {
ArrayJoinNode,
ArrayIndexNode,
+ FinalTableNode,
LimitByNode,
type ClickHouseSelectQueryNode,
} from "./nodes";
@@ -33,6 +34,9 @@ export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer
// runtime that `T` is ArrayIndexNode here, so the cast is sound.
return this.transformArrayIndex(node) as unknown as T;
}
+ if (FinalTableNode.is(node)) {
+ return this.transformFinalTable(node) as unknown as T;
+ }
return super.transformNodeImpl(node, queryId);
}
@@ -43,6 +47,10 @@ export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer
);
}
+ protected transformFinalTable(node: FinalTableNode): FinalTableNode {
+ return FinalTableNode.create(this.transformNode(node.table));
+ }
+
protected override transformSelectQuery(
node: SelectQueryNode,
queryId?: QueryId,
diff --git a/packages/shared/src/server/query-ast/types.assert.ts b/packages/shared/src/server/query-ast/types.assert.ts
index 4083d53e50..8fb383a3d5 100644
--- a/packages/shared/src/server/query-ast/types.assert.ts
+++ b/packages/shared/src/server/query-ast/types.assert.ts
@@ -1,7 +1,7 @@
import { type InferResult } from "kysely";
import { getClickhouseKysely } from "./dialect";
-import { arrayJoin, limitBy, mapKeys, mapValues } from "./extensions";
+import { arrayJoin, limitBy, mapKeys, mapValues, useFinal } from "./extensions";
import { defineView, fromView } from "./views";
/**
@@ -51,7 +51,7 @@ export function schemaTypeAssertions(): void {
/**
* Compile-time assertions that the `$call` extension helpers infer correctly:
* `arrayJoin` widens the output row type with each declared alias, and `limitBy`
- * preserves it. `tsc` is the test.
+ * / `useFinal` preserve it. `tsc` is the test.
*/
export function extensionTypeAssertions(): void {
const db = getClickhouseKysely();
@@ -85,4 +85,14 @@ export function extensionTypeAssertions(): void {
type _Span = Limited["span_id"];
// @ts-expect-error limitBy adds no columns
type _NoExtra = Limited["cost_key"];
+
+ // useFinal preserves the row type: it declares no new columns.
+ const _final = db
+ .selectFrom("scores as s")
+ .select("s.id")
+ .$call(useFinal(["scores"]));
+ type FinalRow = InferResult<typeof _final>[number];
+ type _Id = FinalRow["id"];
+ // @ts-expect-error useFinal adds no columns
+ type _NoFinalExtra = FinalRow["cost_key"];
}
diff --git a/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap
new file mode 100644
index 0000000000..ae07dd35d8
--- /dev/null
+++ b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap
@@ -0,0 +1,337 @@
+// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html
+
+exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=observation time=from+to 1`] = `
+{
+ "params": {
+ "param1": "golden-project",
+ "param2": "GENERATION",
+ "param3": [
+ "prompt-a",
+ "prompt-b",
+ ],
+ "param4": 2026-01-01T00:00:00.000Z,
+ "param5": 2026-01-31T23:59:59.000Z,
+ "param6": 0,
+ "param7": 0,
+ "param8": [
+ "NUMERIC",
+ "BOOLEAN",
+ "CATEGORICAL",
+ "TEXT",
+ ],
+ },
+ "sql": "WITH
+ prompt_events AS
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(3)}) AND (e.start_time <= {param5:DateTime64(3)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ )
+SELECT
+ e.prompt_id AS prompt_id,
+ s.id AS id,
+ s.name AS name,
+ s.string_value AS string_value,
+ s.value AS value,
+ s.source AS source,
+ s.data_type AS data_type,
+ s.comment AS comment,
+ s.timestamp AS timestamp,
+ length(mapKeys(s.metadata)) > {param6:Int64} AS has_metadata
+FROM scores AS s
+FINAL
+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)
+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param7:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param8:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (
+ SELECT
+ trace_id,
+ span_id
+ FROM prompt_events
+ WHERE is_deleted = {param7:UInt8}
+))
+GROUP BY
+ e.prompt_id,
+ s.id,
+ s.name,
+ s.string_value,
+ s.value,
+ s.source,
+ s.data_type,
+ s.comment,
+ s.timestamp,
+ s.metadata",
+}
+`;
+
+exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=observation time=unset 1`] = `
+{
+ "params": {
+ "param1": "golden-project",
+ "param2": "GENERATION",
+ "param3": [
+ "prompt-a",
+ "prompt-b",
+ ],
+ "param4": 0,
+ "param5": 0,
+ "param6": [
+ "NUMERIC",
+ "BOOLEAN",
+ "CATEGORICAL",
+ "TEXT",
+ ],
+ },
+ "sql": "WITH
+ prompt_events AS
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ )
+SELECT
+ e.prompt_id AS prompt_id,
+ s.id AS id,
+ s.name AS name,
+ s.string_value AS string_value,
+ s.value AS value,
+ s.source AS source,
+ s.data_type AS data_type,
+ s.comment AS comment,
+ s.timestamp AS timestamp,
+ length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata
+FROM scores AS s
+FINAL
+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)
+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (
+ SELECT
+ trace_id,
+ span_id
+ FROM prompt_events
+ WHERE is_deleted = {param5:UInt8}
+))
+GROUP BY
+ e.prompt_id,
+ s.id,
+ s.name,
+ s.string_value,
+ s.value,
+ s.source,
+ s.data_type,
+ s.comment,
+ s.timestamp,
+ s.metadata",
+}
+`;
+
+exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace time=from 1`] = `
+{
+ "params": {
+ "param1": "golden-project",
+ "param2": "GENERATION",
+ "param3": [
+ "prompt-a",
+ "prompt-b",
+ ],
+ "param4": 2026-01-01T00:00:00.000Z,
+ "param5": 0,
+ "param6": 0,
+ "param7": [
+ "NUMERIC",
+ "BOOLEAN",
+ "CATEGORICAL",
+ "TEXT",
+ ],
+ },
+ "sql": "WITH
+ prompt_events AS
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(3)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ )
+SELECT
+ e.prompt_id AS prompt_id,
+ s.id AS id,
+ s.name AS name,
+ s.string_value AS string_value,
+ s.value AS value,
+ s.source AS source,
+ s.data_type AS data_type,
+ s.comment AS comment,
+ s.timestamp AS timestamp,
+ length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadata
+FROM scores AS s
+FINAL
+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)
+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param6:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param7:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (
+ SELECT trace_id
+ FROM prompt_events
+ WHERE is_deleted = {param6:UInt8}
+))
+GROUP BY
+ e.prompt_id,
+ s.id,
+ s.name,
+ s.string_value,
+ s.value,
+ s.source,
+ s.data_type,
+ s.comment,
+ s.timestamp,
+ s.metadata",
+}
+`;
+
+exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace time=unset 1`] = `
+{
+ "params": {
+ "param1": "golden-project",
+ "param2": "GENERATION",
+ "param3": [
+ "prompt-a",
+ "prompt-b",
+ ],
+ "param4": 0,
+ "param5": 0,
+ "param6": [
+ "NUMERIC",
+ "BOOLEAN",
+ "CATEGORICAL",
+ "TEXT",
+ ],
+ },
+ "sql": "WITH
+ prompt_events AS
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ )
+SELECT
+ e.prompt_id AS prompt_id,
+ s.id AS id,
+ s.name AS name,
+ s.string_value AS string_value,
+ s.value AS value,
+ s.source AS source,
+ s.data_type AS data_type,
+ s.comment AS comment,
+ s.timestamp AS timestamp,
+ length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata
+FROM scores AS s
+FINAL
+INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)
+WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (
+ SELECT trace_id
+ FROM prompt_events
+ WHERE is_deleted = {param5:UInt8}
+))
+GROUP BY
+ e.prompt_id,
+ s.id,
+ s.name,
+ s.string_value,
+ s.value,
+ s.source,
+ s.data_type,
+ s.comment,
+ s.timestamp,
+ s.metadata",
+}
+`;
+
+exports[`golden: scores.getScoresForExperimentItems > experimentIds set 1`] = `
+{
+ "params": {
+ "param1": 0,
+ "param2": "golden-project",
+ "param3": [
+ "exp-a",
+ "exp-b",
+ ],
+ "param4": "",
+ "param5": [
+ "NUMERIC",
+ "BOOLEAN",
+ "CATEGORICAL",
+ ],
+ },
+ "sql": "SELECT
+ s.id AS id,
+ s.timestamp AS timestamp,
+ s.project_id AS project_id,
+ s.environment AS environment,
+ s.trace_id AS trace_id,
+ s.session_id AS session_id,
+ s.observation_id AS observation_id,
+ s.dataset_run_id AS dataset_run_id,
+ s.name AS name,
+ s.value AS value,
+ s.source AS source,
+ s.comment AS comment,
+ s.author_user_id AS author_user_id,
+ s.config_id AS config_id,
+ s.data_type AS data_type,
+ s.string_value AS string_value,
+ s.queue_id AS queue_id,
+ s.execution_trace_id AS execution_trace_id,
+ s.created_at AS created_at,
+ s.updated_at AS updated_at,
+ s.event_ts AS event_ts,
+ s.is_deleted AS is_deleted,
+ length(mapKeys(s.metadata)) > {param1:Int64} AS has_metadata,
+ e.experiment_id AS experiment_id
+FROM
+(
+ SELECT
+ e.project_id,
+ e.experiment_id,
+ e.trace_id
+ FROM events_core AS e
+ WHERE (e.project_id = {param2:String}) AND (e.experiment_id IN ({param3:Array(String)})) AND (e.experiment_id != {param4:String})
+) AS e
+INNER JOIN scores AS s
+FINAL ON (e.trace_id = s.trace_id) AND (e.project_id = s.project_id) AND (s.project_id = {param2:String})
+WHERE s.data_type IN ({param5:Array(String)})
+ORDER BY s.event_ts DESC
+LIMIT 1 BY
+ s.id,
+ s.project_id,
+ e.experiment_id",
+}
+`;
diff --git a/packages/shared/src/server/repositories/scores.golden.test.ts b/packages/shared/src/server/repositories/scores.golden.test.ts
new file mode 100644
index 0000000000..961a1ce248
--- /dev/null
+++ b/packages/shared/src/server/repositories/scores.golden.test.ts
@@ -0,0 +1,117 @@
+import { beforeEach, describe, expect, it, vi } from "vitest";
+
+const mockQueryClickhouse = vi.hoisted(() => vi.fn().mockResolvedValue([]));
+
+vi.mock("./clickhouse", () => ({
+ queryClickhouse: mockQueryClickhouse,
+ queryClickhouseStream: vi.fn(),
+ queryClickhouseExecRaw: vi.fn(),
+ commandClickhouse: vi.fn(),
+ upsertClickhouse: vi.fn(),
+ parseClickhouseUTCDateTimeFormat: vi.fn(),
+ clickhouseCompliantRandomCharacters: vi.fn(() => "x"),
+ BLOB_EXPORT_PARQUET_CLICKHOUSE_SETTINGS: {},
+}));
+
+// Break the query-options -> server-barrel import cycle that otherwise fails
+// module initialization when the graph is entered from this repository file.
+vi.mock("../queries/clickhouse-sql/query-options", () => ({
+ shouldSkipObservationsFinal: vi.fn().mockResolvedValue(false),
+}));
+
+import {
+ clickhouseFormatAvailable,
+ formatSql,
+ normalizeParams,
+} from "./goldenHarness";
+import {
+ getAggregatedScoresForPromptsFromEvents,
+ getScoresForExperimentItems,
+} from "../index";
+
+const FIXED_PROJECT_ID = "golden-project";
+const FIXED_EXPERIMENT_IDS = ["exp-a", "exp-b"];
+const FIXED_PROMPT_IDS = ["prompt-a", "prompt-b"];
+const FIXED_FROM_TIMESTAMP = new Date("2026-01-01T00:00:00.000Z");
+const FIXED_TO_TIMESTAMP = new Date("2026-01-31T23:59:59.000Z");
+
+const describeWithClickhouse = clickhouseFormatAvailable()
+ ? describe
+ : describe.skip;
+
+if (!clickhouseFormatAvailable()) {
+ console.warn(
+ "[golden-harness] `clickhouse format` unavailable β€” skipping golden SQL tests. Install clickhouse-local to run them.",
+ );
+}
+
+function snapshotCapturedQuery() {
+ expect(mockQueryClickhouse).toHaveBeenCalledTimes(1);
+ const { query, params } = mockQueryClickhouse.mock.calls[0][0] as {
+ query: string;
+ params: Record<string, unknown>;
+ };
+ return normalizeParams(formatSql(query), params);
+}
+
+describeWithClickhouse("golden: scores.getScoresForExperimentItems", () => {
+ beforeEach(() => {
+ vi.clearAllMocks();
+ mockQueryClickhouse.mockResolvedValue([]);
+ });
+
+ it("experimentIds set", async () => {
+ await getScoresForExperimentItems(FIXED_PROJECT_ID, FIXED_EXPERIMENT_IDS);
+
+ expect(snapshotCapturedQuery()).toMatchSnapshot();
+ });
+});
+
+describeWithClickhouse(
+ "golden: scores.getAggregatedScoresForPromptsFromEvents",
+ () => {
+ beforeEach(() => {
+ vi.clearAllMocks();
+ mockQueryClickhouse.mockResolvedValue([]);
+ });
+
+ const variants: Array<{
+ relation: "observation" | "trace";
+ fromTimestamp?: Date;
+ toTimestamp?: Date;
+ }> = [
+ { relation: "observation" },
+ { relation: "trace" },
+ {
+ relation: "observation",
+ fromTimestamp: FIXED_FROM_TIMESTAMP,
+ toTimestamp: FIXED_TO_TIMESTAMP,
+ },
+ { relation: "trace", fromTimestamp: FIXED_FROM_TIMESTAMP },
+ ];
+
+ for (const variant of variants) {
+ const bounds = [
+ variant.fromTimestamp ? "from" : null,
+ variant.toTimestamp ? "to" : null,
+ ]
+ .filter(Boolean)
+ .join("+");
+ const name = `relation=${variant.relation} time=${bounds || "unset"}`;
+
+ it(name, async () => {
+ await getAggregatedScoresForPromptsFromEvents(
+ FIXED_PROJECT_ID,
+ FIXED_PROMPT_IDS,
+ variant.relation,
+ {
+ fromTimestamp: variant.fromTimestamp,
+ toTimestamp: variant.toTimestamp,
+ },
+ );
+
+ expect(snapshotCapturedQuery()).toMatchSnapshot();
+ });
+ }
+ },
+);
diff --git a/packages/shared/src/server/repositories/scores.ts b/packages/shared/src/server/repositories/scores.ts
index f7774b1957..d1d17ebe51 100644
--- a/packages/shared/src/server/repositories/scores.ts
+++ b/packages/shared/src/server/repositories/scores.ts
@@ -35,7 +35,6 @@ import {
orderByToClickhouseSql,
StringOptionsFilter,
DateTimeFilter,
- CTEQueryBuilder,
NumberFilter,
scoreOnlyFiltersAreSeekEligible,
} from "../queries";
@@ -73,9 +72,10 @@ import { scoresColumnsTableUiColumnDefinitions } from "../tableMappings/mapScore
import {
eventsTraceMetadata,
eventsExperimentTraceIds,
- eventsExperiments,
- promptEventsForMetrics,
} from "../queries/clickhouse-sql/query-fragments";
+import { compileClickhouseQuery } from "../query-ast/compile";
+import { getClickhouseKysely } from "../query-ast/dialect";
+import { limitBy, mapKeys, useFinal } from "../query-ast/extensions";
import { scoresTableCols } from "../../tableDefinitions/scoresTable";
import {
findUiColumnMapping,
@@ -408,6 +408,57 @@ export const getTraceScoresForDatasetRuns = async (
}));
};
+const experimentItemEventsQuery = (experimentIds: string[]) =>
+ getClickhouseKysely()
+ .selectFrom("events_core as e")
+ .select(["e.project_id", "e.experiment_id", "e.trace_id"])
+ .where("e.experiment_id", "in", experimentIds)
+ .where("e.experiment_id", "!=", "");
+
+const scoresForExperimentItemsQuery = (experimentIds: string[]) =>
+ getClickhouseKysely()
+ .selectFrom(experimentItemEventsQuery(experimentIds).as("e"))
+ .innerJoin("scores as s", (join) =>
+ join
+ .onRef("e.trace_id", "=", "s.trace_id")
+ .onRef("e.project_id", "=", "s.project_id"),
+ )
+ .$call(useFinal(["scores"]))
+ .select((eb) => [
+ "s.id as id",
+ "s.timestamp as timestamp",
+ "s.project_id as project_id",
+ "s.environment as environment",
+ "s.trace_id as trace_id",
+ "s.session_id as session_id",
+ "s.observation_id as observation_id",
+ "s.dataset_run_id as dataset_run_id",
+ "s.name as name",
+ "s.value as value",
+ "s.source as source",
+ "s.comment as comment",
+ "s.author_user_id as author_user_id",
+ "s.config_id as config_id",
+ "s.data_type as data_type",
+ "s.string_value as string_value",
+ "s.queue_id as queue_id",
+ "s.execution_trace_id as execution_trace_id",
+ "s.created_at as created_at",
+ "s.updated_at as updated_at",
+ "s.event_ts as event_ts",
+ "s.is_deleted as is_deleted",
+ eb(eb.fn("length", [mapKeys("s.metadata")]), ">", 0).as("has_metadata"),
+ "e.experiment_id as experiment_id",
+ ])
+ .where("s.data_type", "in", [...AGGREGATABLE_SCORE_TYPES])
+ .orderBy("s.event_ts", "desc")
+ .$call(
+ limitBy({
+ count: 1,
+ columns: ["s.id", "s.project_id", "e.experiment_id"],
+ }),
+ );
+
export const getScoresForExperimentItems = async (
projectId: string,
experimentIds: string[],
@@ -421,48 +472,10 @@ export const getScoresForExperimentItems = async (
> => {
if (experimentIds.length === 0) return [];
- // Build events subquery using the query builder
- const eventsSubquery = eventsExperiments({
- projectId,
- experimentIds,
- })
- .selectRaw("e.project_id", "e.experiment_id", "e.trace_id")
- .buildWithParams();
-
- const query = `
- SELECT
- s.id as id,
- s.timestamp as timestamp,
- s.project_id as project_id,
- s.environment as environment,
- s.trace_id as trace_id,
- s.session_id as session_id,
- s.observation_id as observation_id,
- s.dataset_run_id as dataset_run_id,
- s.name as name,
- s.value as value,
- s.source as source,
- s.comment as comment,
- s.author_user_id as author_user_id,
- s.config_id as config_id,
- s.data_type as data_type,
- s.string_value as string_value,
- s.queue_id as queue_id,
- s.execution_trace_id as execution_trace_id,
- s.created_at as created_at,
- s.updated_at as updated_at,
- s.event_ts as event_ts,
- s.is_deleted as is_deleted,
- length(mapKeys(s.metadata)) > 0 AS has_metadata,
- e.experiment_id as experiment_id
- FROM (${eventsSubquery.query}) e
- JOIN scores s FINAL ON e.trace_id = s.trace_id
- AND e.project_id = s.project_id
- WHERE s.project_id = {projectId: String}
- AND s.data_type IN ({dataTypes: Array(String)})
- ORDER BY s.event_ts DESC
- LIMIT 1 BY s.id, s.project_id, e.experiment_id
- `;
+ const compiled = compileClickhouseQuery(
+ scoresForExperimentItemsQuery(experimentIds),
+ { projectId },
+ );
const rows = await queryClickhouse<
Omit<ScoreRecordReadType, "metadata"> & {
@@ -470,13 +483,9 @@ export const getScoresForExperimentItems = async (
experiment_id: string;
}
>({
- query,
- params: {
- projectId,
- ...eventsSubquery.params,
- dataTypes: AGGREGATABLE_SCORE_TYPES,
- },
- tags: { projectId },
+ query: compiled.sql,
+ params: compiled.params,
+ tags: { projectId, route: "scores.getScoresForExperimentItems" },
preferredClickhouseService: "EventsReadOnly",
});
@@ -2197,92 +2206,90 @@ export const getAggregatedScoresForPrompts = async (
}));
};
-export const buildAggregatedScoresForPromptsFromEventsQuery = (
- projectId: string,
+const promptEventsForScoresQuery = (
promptIds: string[],
- fetchScoreRelation: "observation" | "trace",
- {
- fromTimestamp,
- toTimestamp,
- }: { fromTimestamp?: Date; toTimestamp?: Date } = {},
-) => {
- const promptEvents = promptEventsForMetrics({
- projectId,
- promptIds,
- ...(fromTimestamp
- ? { fromTimestamp: convertDateToClickhouseDateTime(fromTimestamp) }
- : {}),
- ...(toTimestamp
- ? { toTimestamp: convertDateToClickhouseDateTime(toTimestamp) }
- : {}),
- });
-
- const scoreRows = {
- query: `
- SELECT
- e.prompt_id AS prompt_id,
- s.id AS id,
- s.name AS name,
- s.string_value AS string_value,
- s.value AS value,
- s.source AS source,
- s.data_type AS data_type,
- s.comment AS comment,
- s.timestamp AS timestamp,
- s.metadata AS metadata
- FROM scores s FINAL
- INNER JOIN prompt_events e
- ON s.project_id = e.project_id
- AND s.trace_id = e.trace_id
- ${fetchScoreRelation === "observation" ? "AND s.observation_id = e.span_id" : ""}
- WHERE s.project_id = {scoreProjectId: String}
- AND e.is_deleted = 0
- AND s.name IS NOT NULL
- AND s.data_type IN ({dataTypes: Array(String)})
- ${
- fetchScoreRelation === "trace"
- ? `AND s.observation_id IS NULL
- AND s.trace_id IN (SELECT trace_id FROM prompt_events WHERE is_deleted = 0)`
- : `AND s.observation_id IS NOT NULL
- AND (s.trace_id, s.observation_id) IN (SELECT trace_id, span_id FROM prompt_events WHERE is_deleted = 0)`
- }
- `,
- params: {
- scoreProjectId: projectId,
- dataTypes: LISTABLE_SCORE_TYPES,
- },
- schema: [
- "prompt_id",
- "id",
- "name",
- "string_value",
- "value",
- "source",
- "data_type",
- "comment",
- "timestamp",
- "metadata",
- ],
- };
+ timeWindow: { fromTimestamp?: Date; toTimestamp?: Date } = {},
+) =>
+ getClickhouseKysely()
+ .selectFrom("events_core as e")
+ .select([
+ "e.project_id as project_id",
+ "e.prompt_id as prompt_id",
+ "e.trace_id as trace_id",
+ "e.span_id as span_id",
+ "e.is_deleted as is_deleted",
+ ])
+ .where("e.type", "=", "GENERATION")
+ .where("e.prompt_id", "in", promptIds)
+ .$if(timeWindow.fromTimestamp !== undefined, (qb) =>
+ qb.where("e.start_time", ">=", timeWindow.fromTimestamp!),
+ )
+ .$if(timeWindow.toTimestamp !== undefined, (qb) =>
+ qb.where("e.start_time", "<=", timeWindow.toTimestamp!),
+ )
+ .orderBy("e.event_ts", "desc")
+ .$call(limitBy({ count: 1, columns: ["e.span_id", "e.project_id"] }));
- return new CTEQueryBuilder()
- .withCTE("prompt_events", promptEvents)
- .withCTE("score_rows", scoreRows)
- .from("score_rows", "s")
- .select(
- "s.prompt_id AS prompt_id",
- "s.id AS id",
- "s.name AS name",
- "s.string_value AS string_value",
- "s.value AS value",
- "s.source AS source",
- "s.data_type AS data_type",
- "s.comment AS comment",
- "s.timestamp AS timestamp",
- "length(mapKeys(s.metadata)) > 0 AS has_metadata",
+const aggregatedScoresForPromptsFromEventsQuery = (
+ promptIds: string[],
+ fetchScoreRelation: "observation" | "trace",
+ timeWindow: { fromTimestamp?: Date; toTimestamp?: Date } = {},
+) =>
+ getClickhouseKysely()
+ .with("prompt_events", () =>
+ promptEventsForScoresQuery(promptIds, timeWindow),
+ )
+ .selectFrom("scores as s")
+ .$call(useFinal(["scores"]))
+ .innerJoin("prompt_events as e", (join) => {
+ const base = join
+ .onRef("s.project_id", "=", "e.project_id")
+ .onRef("s.trace_id", "=", "e.trace_id");
+ return fetchScoreRelation === "observation"
+ ? base.onRef("s.observation_id", "=", "e.span_id")
+ : base;
+ })
+ .select((eb) => [
+ "e.prompt_id as prompt_id",
+ "s.id as id",
+ "s.name as name",
+ "s.string_value as string_value",
+ "s.value as value",
+ "s.source as source",
+ "s.data_type as data_type",
+ "s.comment as comment",
+ "s.timestamp as timestamp",
+ eb(eb.fn("length", [mapKeys("s.metadata")]), ">", 0).as("has_metadata"),
+ ])
+ .where("e.is_deleted", "=", 0)
+ .where("s.name", "is not", null)
+ .where("s.data_type", "in", [...LISTABLE_SCORE_TYPES])
+ .$if(fetchScoreRelation === "trace", (qb) =>
+ qb
+ .where("s.observation_id", "is", null)
+ .where("s.trace_id", "in", (eb) =>
+ eb
+ .selectFrom("prompt_events")
+ .select("trace_id")
+ .where("is_deleted", "=", 0),
+ ),
)
- .groupBy(
- "s.prompt_id",
+ .$if(fetchScoreRelation === "observation", (qb) =>
+ qb
+ .where("s.observation_id", "is not", null)
+ .where(({ eb, refTuple, selectFrom }) =>
+ eb(
+ refTuple("s.trace_id", "s.observation_id"),
+ "in",
+ selectFrom("prompt_events")
+ .select(["trace_id", "span_id"])
+ .where("is_deleted", "=", 0)
+ .$asTuple("trace_id", "span_id"),
+ ),
+ ),
+ )
+ .groupBy([
+ "e.prompt_id",
"s.id",
"s.name",
"s.string_value",
@@ -2292,8 +2299,23 @@ export const buildAggregatedScoresForPromptsFromEventsQuery = (
"s.comment",
"s.timestamp",
"s.metadata",
- )
- .buildWithParams();
+ ]);
+
+export const buildAggregatedScoresForPromptsFromEventsQuery = (
+ projectId: string,
+ promptIds: string[],
+ fetchScoreRelation: "observation" | "trace",
+ timeWindow: { fromTimestamp?: Date; toTimestamp?: Date } = {},
+) => {
+ const compiled = compileClickhouseQuery(
+ aggregatedScoresForPromptsFromEventsQuery(
+ promptIds,
+ fetchScoreRelation,
+ timeWindow,
+ ),
+ { projectId },
+ );
+ return { query: compiled.sql, params: compiled.params };
};
export const getAggregatedScoresForPromptsFromEvents = async (
@@ -2317,7 +2339,10 @@ export const getAggregatedScoresForPromptsFromEvents = async (
>({
query,
params,
- tags: { projectId },
+ tags: {
+ projectId,
+ route: "scores.getAggregatedScoresForPromptsFromEvents",
+ },
preferredClickhouseService: "EventsReadOnly",
});
diff --git a/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts b/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts
index 6f9f2cb1fe..5bb062f3f3 100644
--- a/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts
+++ b/web/src/__tests__/server/api/tables/prompts-ui.servertest.ts
@@ -587,9 +587,9 @@ describe("UI Prompts Table", () => {
"observation",
);
- expect(query).toContain("INNER JOIN prompt_events");
- expect(query).toContain(
- "(s.trace_id, s.observation_id) IN (SELECT trace_id, span_id FROM prompt_events",
+ expect(query).toMatch(/inner join prompt_events/i);
+ expect(query).toMatch(
+ /\(s\.trace_id, s\.observation_id\) in \(select trace_id, span_id from prompt_events/i,
);
});