🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(query-ast): hide soft-deleted rows from events reads (#18869)
Niklas Semmler1896859401ef
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex 7898c8521d..4f1d7b5114 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -30,9 +30,9 @@ changing how it emits SQL.hand-maintained tables that drift.- `compiler.ts` / `transformer.ts` / `dialect.ts` / `nodes.ts` — the dialect.- `extensions.ts` — the `$call` helpers (`arrayJoin`, `limitBy`, `useFinal`,- `includeDeleted`) and typed ClickHouse functions and aggregates (`argMaxIf`,- `countIf`, `sumMap`, `multiIf`, `metadataValue`, `literal`, …) that take- `eb.ref(...)` expressions.+ `includeDeleted`, `softDeleteAfterLimit`) and typed ClickHouse functions and+ aggregates (`argMaxIf`, `countIf`, `sumMap`, `multiIf`, `metadataValue`,+ `literal`, …) that take `eb.ref(...)` expressions.- `views.ts` / `dedup.ts` / `softDelete.ts` — virtual views as CTEs, per-tablededup lowering, and soft-delete filtering.- `fixtures/catalog.ts` — test-only catalog of builder queries exercising the@@ -87,9 +87,15 @@ suffix to run there. Promote it to a required check once it has proven stable.only `projectId` (see `../repositories/environments.ts`).- **Never filter `is_deleted` on a physical table yourself.** Tables thatdeclare `softDelete` in the registry (`events_core`, `events_full`) get- `is_deleted = 0` on every FROM/JOIN. A select that must see deleted rows- opts out with `$call(includeDeleted())`, e.g. a latest-version read that- drops tombstones after its `LIMIT 1 BY`.+ `is_deleted = 0` on every FROM/JOIN. A latest-version read (ordered by+ version, then `LIMIT 1` or `LIMIT 1 BY`) marks itself+ `$call(softDeleteAfterLimit())`, so a deletion (a newer row with the flag+ set) wins the pick and is dropped after it, instead of an older live+ version. A select that must see deleted rows opts out with+ `$call(includeDeleted())`. Reads without a version pick (lists,+ aggregates) filter each row on its own: like an updated row under dedup+ `none`, a deleted row's older live version stays visible until a merge+ collapses the versions.- **Dedup is declared per table and must be an existing production idiom.**`events_core` and `events_full` are `none` (immutable at read time — noLIMIT BY, no FINAL).@@ -137,10 +143,12 @@ db.selectFrom("observations")```tsdb.selectFrom("events_core")+ .$call(softDeleteAfterLimit()).select(["span_id", "project_id"]).orderBy("event_ts", "desc").$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));-// … order by event_ts desc limit 1 by span_id, project_id+// … order by event_ts desc limit 1 by span_id, project_id, wrapped by the+// soft-delete pass so a deleted latest version is dropped```**Write FINAL** (`useFinal` — ReplacingMergeTree reads; never on `events_core`):@@ -206,7 +214,10 @@ and it is where tenancy is enforced:required parameter, so omitting it is also a compile-time type error.2. `SoftDeleteFilterPlugin` adds `<flag> = 0` for every FROM/JOIN relationwhose table declares `softDelete`, except in selects marked with- `includeDeleted()`. It runs before tenancy, so the WHERE reads+ `includeDeleted()`. A select marked `softDeleteAfterLimit()` is left+ unfiltered, gets the flag added to its outputs, and is wrapped in an outer+ select over those outputs that keeps rows with `latest.<flag> = 0`. It+ runs before tenancy, so the WHERE reads`project_id = … and is_deleted = … and …`.3. `TenancyInjectionPlugin` walks every FROM/JOIN and injects`project_id = {projectId}` on each tenanted physical table, unless the treediff --git a/packages/shared/src/server/query-ast/extensions.ts b/packages/shared/src/server/query-ast/extensions.tsindex 1a12ae18fa..3948c25df2 100644--- a/packages/shared/src/server/query-ast/extensions.ts+++ b/packages/shared/src/server/query-ast/extensions.ts@@ -782,18 +782,6 @@ export function useFinal(tables: ReadonlyArray<keyof ClickHouseDatabase>) {): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new FinalPlugin(names));}-/** Plugin that marks the select so the soft-delete pass leaves it unfiltered. */-class IncludeDeletedPlugin implements KyselyPlugin {- transformQuery(args: PluginTransformQueryArgs): RootOperationNode {- if (args.node.kind !== "SelectQueryNode") return args.node;- return { ...args.node, includeDeleted: true } as RootOperationNode;- }-- async transformResult(args: PluginTransformResultArgs) {- return args.result;- }-}-/*** Read soft-deleted rows too: `qb.$call(includeDeleted())`. Applies to this* select's FROM/JOIN relations only; nested selects are still filtered. A@@ -802,5 +790,39 @@ class IncludeDeletedPlugin implements KyselyPlugin {export function includeDeleted() {return <DB, TB extends keyof DB, O>(qb: SelectQueryBuilder<DB, TB, O>,- ): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new IncludeDeletedPlugin());+ ): SelectQueryBuilder<DB, TB, O> =>+ qb.withPlugin(new SoftDeleteMarkerPlugin("includeDeleted"));+}++/**+ * Drop soft-deleted rows after this select's ORDER BY / LIMIT / LIMIT BY+ * instead of before: `qb.$call(softDeleteAfterLimit())`. For a latest-version+ * pick, where a deletion is a newer version with the flag set; filtering+ * first would return the older live version. Compiles to+ * `select latest.<output>, … from (<this select, flag added>) as latest where+ * latest.<flag> = 0`. The select reads exactly one soft-delete table, has a+ * LIMIT or LIMIT BY, has no GROUP BY or set operation, names every output (no+ * `*`, unaliased expressions), and does not select the flag itself.+ */+export function softDeleteAfterLimit() {+ return <DB, TB extends keyof DB, O>(+ qb: SelectQueryBuilder<DB, TB, O>,+ ): SelectQueryBuilder<DB, TB, O> =>+ qb.withPlugin(new SoftDeleteMarkerPlugin("softDeleteAfterLimit"));+}++/** Sets a soft-delete marker on the select; the soft-delete pass reads it. */+class SoftDeleteMarkerPlugin implements KyselyPlugin {+ constructor(+ private readonly marker: "includeDeleted" | "softDeleteAfterLimit",+ ) {}++ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ if (args.node.kind !== "SelectQueryNode") return args.node;+ return { ...args.node, [this.marker]: true } as RootOperationNode;+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }}diff --git a/packages/shared/src/server/query-ast/nodes.ts b/packages/shared/src/server/query-ast/nodes.tsindex 74ea1ab4a9..0356d89f12 100644--- a/packages/shared/src/server/query-ast/nodes.ts+++ b/packages/shared/src/server/query-ast/nodes.ts@@ -107,6 +107,11 @@ export type ClickHouseSelectQueryNode = SelectQueryNode & {readonly limitBy?: LimitByNode;/** Set by `includeDeleted()`: the soft-delete pass skips this select. */readonly includeDeleted?: true;+ /**+ * Set by `softDeleteAfterLimit()`: the soft-delete pass filters this+ * select's output instead of its relation.+ */+ readonly softDeleteAfterLimit?: true;};export function isClickHouseSelectQueryNode(diff --git a/packages/shared/src/server/query-ast/schema.ts b/packages/shared/src/server/query-ast/schema.tsindex 274f64cf6f..48d466b762 100644--- a/packages/shared/src/server/query-ast/schema.ts+++ b/packages/shared/src/server/query-ast/schema.ts@@ -136,7 +136,7 @@ function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {/*** Column that marks a soft-deleted row. The soft-delete pass injects* `<column> = 0` on every read of this table unless the select opts out- * with `includeDeleted()`.+ * with `includeDeleted()`, or after its LIMIT with `softDeleteAfterLimit()`.*/softDelete?: keyof Cols & string;}) {diff --git a/packages/shared/src/server/query-ast/softDelete.test.ts b/packages/shared/src/server/query-ast/softDelete.test.tsindex 4446cca18f..6363b538a2 100644--- a/packages/shared/src/server/query-ast/softDelete.test.ts+++ b/packages/shared/src/server/query-ast/softDelete.test.ts@@ -9,7 +9,7 @@ import { describe, expect, it } from "vitest";import { compileClickhouseQuery } from "./compile";import { getClickhouseKysely } from "./dialect";import { QueryCompileError } from "./errors";-import { includeDeleted, limitBy } from "./extensions";+import { includeDeleted, limitBy, softDeleteAfterLimit } from "./extensions";import { FinalTableNode } from "./nodes";import { SOFT_DELETE_COLUMNS } from "./schema";import { SoftDeleteFilterPlugin } from "./softDelete";@@ -309,4 +309,153 @@ describe("soft-delete filtering", () => {),).toThrow(QueryCompileError);});++ it("softDeleteAfterLimit() filters after the pick, projecting outputs by name", () => {+ expect(+ compile(+ getClickhouseKysely()+ .selectFrom("events_full as e")+ .$call(softDeleteAfterLimit())+ .select((eb) => [+ "e.span_id as id",+ "e.event_ts",+ "input",+ eb.fn<number>("length", ["e.output"]).as("len"),+ ])+ .where("e.span_id", "=", "o")+ .orderBy("e.event_ts", "desc")+ .limit(1),+ ),+ ).toMatchInlineSnapshot(+ `"select latest.id, latest.event_ts, latest.input, latest.len from (select e.span_id as id, e.event_ts, input, length(e.output) as len, e.is_deleted as is_deleted from events_full as e where e.project_id = {p1:String} and e.span_id = {p2:String} order by e.event_ts desc limit {p3:Int64}) as latest where latest.is_deleted = {p4:UInt8}"`,+ );+ });++ it("softDeleteAfterLimit() wraps a CTE body and keeps its LIMIT BY inside", () => {+ expect(+ compile(+ getClickhouseKysely()+ .with("latest_events", () =>+ getClickhouseKysely()+ .selectFrom("events_core as e")+ .$call(softDeleteAfterLimit())+ .select(["e.span_id as span_id", "e.trace_id as trace_id"])+ .orderBy("e.event_ts", "desc")+ .$call(limitBy({ count: 1, columns: ["e.span_id"] })),+ )+ .selectFrom("latest_events as e")+ .select("e.trace_id"),+ ),+ ).toMatchInlineSnapshot(+ `"with latest_events as (select latest.span_id, latest.trace_id from (select e.span_id as span_id, e.trace_id as trace_id, e.is_deleted as is_deleted from events_core as e where e.project_id = {p1:String} order by e.event_ts desc limit 1 by e.span_id) as latest where latest.is_deleted = {p2:UInt8}) select e.trace_id from latest_events as e"`,+ );+ });++ it("softDeleteAfterLimit() qualifies the flag by table name on an unaliased FROM", () => {+ expect(+ compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .$call(softDeleteAfterLimit())+ .select("span_id")+ .limit(1),+ ),+ ).toMatchInlineSnapshot(+ `"select latest.span_id from (select span_id, events_core.is_deleted as is_deleted from events_core where project_id = {p1:String} limit {p2:Int64}) as latest where latest.is_deleted = {p3:UInt8}"`,+ );+ });++ it("softDeleteAfterLimit() leaves nested selects filtered", () => {+ expect(+ compile(+ getClickhouseKysely()+ .selectFrom("events_core as e")+ .$call(softDeleteAfterLimit())+ .select("e.span_id")+ .where("e.trace_id", "in", (eb) =>+ eb.selectFrom("events_core").select("trace_id"),+ )+ .limit(1),+ ),+ ).toContain("events_core.is_deleted = {p2:UInt8}");+ });++ it("rejects softDeleteAfterLimit() on selects it cannot wrap", () => {+ const marked = () =>+ getClickhouseKysely()+ .selectFrom("events_core as e")+ .$call(softDeleteAfterLimit())+ .limit(1);+ const rejects = (+ builder: Parameters<typeof compileClickhouseQuery>[0],+ message: RegExp,+ ) => expect(() => compile(builder)).toThrow(message);++ rejects(marked(), /without outputs/);+ rejects(+ getClickhouseKysely()+ .selectFrom("events_core as e")+ .$call(softDeleteAfterLimit())+ .select("e.span_id"),+ /needs a LIMIT or LIMIT BY/,+ );+ rejects(+ marked().select("e.trace_id").groupBy("e.trace_id"),+ /cannot wrap a GROUP BY or set operation/,+ );+ rejects(+ marked()+ .select("e.span_id")+ .unionAll(+ getClickhouseKysely().selectFrom("events_core").select("span_id"),+ ),+ /cannot wrap a GROUP BY or set operation/,+ );+ rejects(marked().selectAll(), /every output named/);+ rejects(marked().selectAll("e"), /every output named/);+ rejects(marked().select("e.is_deleted"), /adds `is_deleted` itself/);+ rejects(+ marked().select((eb) => [eb.lit(0).as("is_deleted")]),+ /adds `is_deleted` itself/,+ );+ rejects(+ getClickhouseKysely()+ .selectFrom("scores")+ .$call(softDeleteAfterLimit())+ .select("id"),+ /reads 0/,+ );+ rejects(+ marked()+ .innerJoin("events_full as f", (join) =>+ join.onRef("f.span_id", "=", "e.span_id"),+ )+ .select("e.span_id"),+ /reads 2/,+ );+ rejects(+ getClickhouseKysely()+ .selectFrom("scores as s")+ .leftJoin("events_core as e", (join) =>+ join.onRef("e.trace_id", "=", "s.trace_id"),+ )+ .$call(softDeleteAfterLimit())+ .select("s.id")+ .limit(1),+ /cannot wrap a LeftJoin/,+ );+ rejects(+ marked()+ .rightJoin("scores as s", (join) =>+ join.onRef("s.trace_id", "=", "e.trace_id"),+ )+ .select("s.id"),+ /cannot wrap a RightJoin/,+ );+ rejects(marked().distinct().select("e.span_id"), /cannot wrap a DISTINCT/);+ rejects(+ marked().$call(includeDeleted()).select("e.span_id"),+ /cannot mark the same select/,+ );+ });});diff --git a/packages/shared/src/server/query-ast/softDelete.ts b/packages/shared/src/server/query-ast/softDelete.tsindex ad24332d93..e89ae881b7 100644--- a/packages/shared/src/server/query-ast/softDelete.ts+++ b/packages/shared/src/server/query-ast/softDelete.ts@@ -6,7 +6,8 @@* for a JOIN with one, prepended to the WHERE otherwise. A RIGHT or FULL JOIN* in such a select fails to compile. A select that opts out with* `$call(includeDeleted())` is left unchanged; nested selects decide for- * themselves.+ * themselves. A select marked `$call(softDeleteAfterLimit())` keeps its rows+ * and is wrapped in an outer select that filters on its output instead.** Runs before tenancy, so the tenancy pass prepends `project_id` ahead of the* flag: `where project_id = … and is_deleted = … and <caller predicates>`.@@ -20,6 +21,8 @@ import {JoinNode,OperatorNode,ReferenceNode,+ SelectionNode,+ SelectQueryNode,TableNode,ValueNode,WhereNode,@@ -41,6 +44,8 @@ import { SOFT_DELETE_COLUMNS } from "./schema";import { CteScopedTransformer } from "./transformer";const OUTER_PRESERVING_JOINS = new Set<JoinType>(["RightJoin", "FullJoin"]);+const AFTER_LIMIT_ALIAS = "latest";+const AFTER_LIMIT_JOINS = new Set<JoinType>(["InnerJoin", "CrossJoin"]);export class SoftDeleteFilterPlugin implements KyselyPlugin {transformQuery(args: PluginTransformQueryArgs): RootOperationNode {@@ -78,6 +83,12 @@ function filterSelect(const readsSoftDelete =froms.some(Boolean) || joins.some(({ relation }) => relation);+ if (node.softDeleteAfterLimit) {+ return filterAfterLimit(node, [+ ...froms,+ ...joins.map(({ relation }) => relation),+ ]);+ }if (node.includeDeleted) {if (!readsSoftDelete) {throw new QueryCompileError(@@ -126,6 +137,142 @@ function filterSelect(};}+/**+ * `select latest.<output>, … from (<node + flag>) as latest where+ * latest.<flag> = 0`: the inner select keeps its ORDER BY / LIMIT, so a newer+ * deleted version wins the pick and the outer filter then drops it.+ */+function filterAfterLimit(+ node: ClickHouseSelectQueryNode,+ relations: ReadonlyArray<SoftDeleteRelation | undefined>,+): ClickHouseSelectQueryNode {+ const { relation, outputs } = checkAfterLimitShape(node, relations);+ return wrapAfterLimit(node, relation, outputs);+}++/**+ * Throws on shapes the wrap would change; returns the soft-delete relation and+ * the output names to wrap.+ */+function checkAfterLimitShape(+ node: ClickHouseSelectQueryNode,+ relations: ReadonlyArray<SoftDeleteRelation | undefined>,+): { relation: SoftDeleteRelation; outputs: string[] } {+ if (node.includeDeleted) {+ throw new QueryCompileError(+ "includeDeleted() and softDeleteAfterLimit() cannot mark the same select.",+ );+ }+ // The soft-delete tables this select reads directly (FROM and JOINs).+ const read = relations.filter((relation) => relation !== undefined);+ if (read.length !== 1) {+ throw new QueryCompileError(+ `softDeleteAfterLimit() needs a select that reads exactly one soft-delete table; it reads ${read.length}.`,+ );+ }+ const relation = read[0];+ if (!node.limit && !node.limitBy) {+ throw new QueryCompileError(+ "softDeleteAfterLimit() needs a LIMIT or LIMIT BY to filter after.",+ );+ }+ if (node.groupBy || node.setOperations) {+ throw new QueryCompileError(+ "softDeleteAfterLimit() cannot wrap a GROUP BY or set operation: the added flag would not match its shape.",+ );+ }+ // The added flag would join the DISTINCT key and change which rows survive.+ const distinct =+ node.distinctOn ||+ node.frontModifiers?.some(({ modifier }) => modifier === "Distinct");+ if (distinct) {+ throw new QueryCompileError(+ "softDeleteAfterLimit() cannot wrap a DISTINCT: the added flag would change its key.",+ );+ }+ // The outer filter would drop rows an outer join preserves; an ON filter keeps them.+ const outerJoin = node.joins?.find(+ ({ joinType }) => !AFTER_LIMIT_JOINS.has(joinType),+ );+ if (outerJoin) {+ throw new QueryCompileError(+ `softDeleteAfterLimit() cannot wrap a ${outerJoin.joinType}: the outer filter would drop its preserved rows.`,+ );+ }+ // Names of the outputs the outer select lists again; `*` is rejected below.+ const outputs = (node.selections ?? []).map((selection) => {+ const name = outputName(selection.selection);+ if (name === undefined) {+ throw new QueryCompileError(+ "softDeleteAfterLimit() needs every output named: select columns or alias expressions, not `*`.",+ );+ }+ return name;+ });+ if (outputs.length === 0) {+ throw new QueryCompileError(+ "softDeleteAfterLimit() was applied to a select without outputs.",+ );+ }+ if (outputs.includes(relation.flag)) {+ throw new QueryCompileError(+ `softDeleteAfterLimit() adds \`${relation.flag}\` itself; do not select it.`,+ );+ }+ return { relation, outputs };+}++function wrapAfterLimit(+ node: ClickHouseSelectQueryNode,+ relation: SoftDeleteRelation,+ outputs: ReadonlyArray<string>,+): ClickHouseSelectQueryNode {+ // Adds `<alias>.<flag> AS <flag>` as an output for the outer filter.+ const inner = SelectQueryNode.cloneWithSelections(node, [+ SelectionNode.create(+ AliasNode.create(+ qualifiedFlag(relation),+ IdentifierNode.create(relation.flag),+ ),+ ),+ ]);+ // `select latest.<output>, … from (<inner>) as latest`, without the flag.+ const outer = SelectQueryNode.cloneWithSelections(+ SelectQueryNode.createFrom([+ AliasNode.create(inner, IdentifierNode.create(AFTER_LIMIT_ALIAS)),+ ]),+ outputs.map((name) =>+ SelectionNode.create(+ ReferenceNode.create(+ ColumnNode.create(name),+ TableNode.create(AFTER_LIMIT_ALIAS),+ ),+ ),+ ),+ );+ // `where latest.<flag> = 0` drops the pick if it is a deleted version.+ return {+ ...outer,+ where: WhereNode.create(+ notDeleted({+ tableName: AFTER_LIMIT_ALIAS,+ alias: AFTER_LIMIT_ALIAS,+ flag: relation.flag,+ }),+ ),+ };+}++/** Name ClickHouse gives a select output: `e.span_id` → `span_id`, `x as y` → `y`. */+function outputName(node: OperationNode): string | undefined {+ if (AliasNode.is(node)) {+ return IdentifierNode.is(node.alias) ? node.alias.name : undefined;+ }+ if (ReferenceNode.is(node)) return outputName(node.column);+ if (ColumnNode.is(node)) return node.column.name;+ return undefined;+}+function describeRelation(node: OperationNode): SoftDeleteRelation | undefined {if (FinalTableNode.is(node)) {return describeRelation(unwrapFinalTable(node));@@ -147,20 +294,27 @@ function describeRelation(node: OperationNode): SoftDeleteRelation | undefined {* Always qualified (by alias, else table name): a bare flag column would bind* to a same-named SELECT alias, e.g. `0 AS is_deleted`.*/-function notDeleted(relation: SoftDeleteRelation): OperationNode {+function notDeleted(relation: FlagColumn): OperationNode {return BinaryOperationNode.create(- ReferenceNode.create(- ColumnNode.create(relation.flag),- TableNode.create(relation.alias ?? relation.tableName),- ),+ qualifiedFlag(relation),OperatorNode.create("="),ValueNode.create(0),);}+function qualifiedFlag(relation: FlagColumn): ReferenceNode {+ return ReferenceNode.create(+ ColumnNode.create(relation.flag),+ TableNode.create(relation.alias ?? relation.tableName),+ );+}+type SoftDeleteRelation = {tableName: string;alias?: string;flag: string;node: TableNode;};++/** No `node`: {@link filterAfterLimit} filters a derived table, not a TableNode. */+type FlagColumn = Pick<SoftDeleteRelation, "tableName" | "alias" | "flag">;diff --git a/packages/shared/src/server/query-ast/transformer.ts b/packages/shared/src/server/query-ast/transformer.tsindex 2b745427cd..4508812d64 100644--- a/packages/shared/src/server/query-ast/transformer.ts+++ b/packages/shared/src/server/query-ast/transformer.ts@@ -24,8 +24,8 @@ import {** Kysely's default {@link OperationNodeTransformer.transformSelectQuery}* reconstructs a SelectQueryNode from known slots and drops anything else —- * so without this override, ARRAY JOIN / LIMIT BY / the `includeDeleted`- * opt-out would vanish the moment any plugin walked the tree.+ * so without this override, ARRAY JOIN / LIMIT BY / the soft-delete+ * markers would vanish the moment any plugin walked the tree.*/export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer {protected override transformNodeImpl<T extends OperationNode>(@@ -77,6 +77,7 @@ export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer? { limitBy: this.transformLimitBy(extra.limitBy) }: {}),...(extra.includeDeleted ? { includeDeleted: true } : {}),+ ...(extra.softDeleteAfterLimit ? { softDeleteAfterLimit: true } : {}),} as ClickHouseSelectQueryNode;}diff --git a/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snapindex 596cf1763b..55eb40634a 100644--- a/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap+++ b/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap@@ -6,12 +6,13 @@ exports[`golden: environments.getEnvironmentsForProject > writeMode=events_only"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": "2026-01-01 00:00:00.000",+ "param2": 0,+ "param3": "2026-01-01 00:00:00.000",},"route": "environments.getEnvironmentsForProject","sql": "SELECT DISTINCT environmentFROM events_core-WHERE (project_id = {param1:String}) AND (start_time >= {param2:DateTime64(6)})",+WHERE (project_id = {param1:String}) AND (events_core.is_deleted = {param2:UInt8}) AND (start_time >= {param3:DateTime64(6)})",},{"fn": "queryClickhouse",@@ -39,11 +40,12 @@ exports[`golden: environments.getEnvironmentsForProject > writeMode=events_only"fn": "queryClickhouse","params": {"param1": "golden-project",+ "param2": 0,},"route": "environments.getEnvironmentsForProject","sql": "SELECT DISTINCT environmentFROM events_core-WHERE project_id = {param1:String}",+WHERE (project_id = {param1:String}) AND (events_core.is_deleted = {param2:UInt8})",},{"fn": "queryClickhouse",diff --git a/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snapindex 7cb7132074..fd15c54088 100644--- a/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap+++ b/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap@@ -12,17 +12,28 @@ exports[`golden: events IO payload reads family > getObservationFullIOForSession"param5": "2025-12-31 23:59:59.000","param6": "2026-01-01 00:00:01.000","param7": 1,+ "param8": 0,},"route": "unknown","sql": "SELECT- e.span_id AS id,- e.input AS input,- e.output AS output,- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.session_id = {param2:String}) AND (xxHash32(e.trace_id) = xxHash32({param3:String})) AND (e.trace_id = {param3:String}) AND (e.span_id = {param4:String}) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param7:Int64}",+ latest.id,+ latest.input,+ latest.output,+ latest.metadata+FROM+(+ SELECT+ e.span_id AS id,+ e.input AS input,+ e.output AS output,+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.session_id = {param2:String}) AND (xxHash32(e.trace_id) = xxHash32({param3:String})) AND (e.trace_id = {param3:String}) AND (e.span_id = {param4:String}) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param7:Int64}+) AS latest+WHERE latest.is_deleted = {param8:UInt8}",},]`;@@ -38,13 +49,21 @@ exports[`golden: events IO payload reads family > getObservationIOFieldByteLengt"param4": "2025-12-31 23:59:59.000","param5": "2026-01-01 00:00:01.000","param6": 1,+ "param7": 0,},"route": "observation-io-stream-length",- "sql": "SELECT length(e.input) AS len-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param6:Int64}",+ "sql": "SELECT latest.len+FROM+(+ SELECT+ length(e.input) AS len,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param6:Int64}+) AS latest+WHERE latest.is_deleted = {param7:UInt8}",},]`;@@ -60,13 +79,21 @@ exports[`golden: events IO payload reads family > getObservationIOFieldByteLengt"param4": "2025-12-31 23:59:59.000","param5": "2026-01-01 00:00:01.000","param6": 1,+ "param7": 0,},"route": "observation-io-stream-length",- "sql": "SELECT length(toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)))) AS len-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param6:Int64}",+ "sql": "SELECT latest.len+FROM+(+ SELECT+ length(toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)))) AS len,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param6:Int64}+) AS latest+WHERE latest.is_deleted = {param7:UInt8}",},]`;@@ -82,13 +109,21 @@ exports[`golden: events IO payload reads family > getObservationIOFieldByteLengt"param4": "2025-12-31 23:59:59.000","param5": "2026-01-01 00:00:01.000","param6": 1,+ "param7": 0,},"route": "observation-io-stream-length",- "sql": "SELECT length(e.output) AS len-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param6:Int64}",+ "sql": "SELECT latest.len+FROM+(+ SELECT+ length(e.output) AS len,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param6:Int64}+) AS latest+WHERE latest.is_deleted = {param7:UInt8}",},]`;@@ -99,16 +134,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b","golden-span-c",],- "param3": [+ "param4": ["golden-trace","golden-trace-b",],- "param4": [+ "param5": [TupleParam {"values": ["golden-trace",@@ -128,8 +164,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven],},],- "param5": "2025-11-30 23:59:59.000",- "param6": "2026-01-01 00:00:01.000",+ "param6": "2025-11-30 23:59:59.000",+ "param7": "2026-01-01 00:00:01.000",},"route": "unknown","sql": "SELECT@@ -139,7 +175,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEvene.output AS output,mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadataFROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",},]`;@@ -150,16 +186,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b","golden-span-c",],- "param3": [+ "param4": ["golden-trace","golden-trace-b",],- "param4": [+ "param5": [TupleParam {"values": ["golden-trace",@@ -179,8 +216,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven],},],- "param5": "2025-11-30 23:59:59.000",- "param6": "2026-01-01 00:00:01.000",+ "param6": "2025-11-30 23:59:59.000",+ "param7": "2026-01-01 00:00:01.000",},"route": "unknown","sql": "SELECT@@ -195,7 +232,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven) AS experiment_item_metadata,mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadataFROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",},]`;@@ -206,16 +243,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b","golden-span-c",],- "param3": [+ "param4": ["golden-trace","golden-trace-b",],- "param4": [+ "param5": [TupleParam {"values": ["golden-trace",@@ -235,8 +273,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven],},],- "param5": "2025-11-30 23:59:59.000",- "param6": "2026-01-01 00:00:01.000",+ "param6": "2025-11-30 23:59:59.000",+ "param7": "2026-01-01 00:00:01.000",},"route": "unknown","sql": "SELECT@@ -258,7 +296,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEvenarrayReverse(arrayMap(v -> leftUTF8(v, 5000), e.metadata_values))) AS metadataFROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",},]`;@@ -269,16 +307,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b","golden-span-c",],- "param3": [+ "param4": ["golden-trace","golden-trace-b",],- "param4": [+ "param5": [TupleParam {"values": ["golden-trace",@@ -298,8 +337,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven],},],- "param5": "2025-11-30 23:59:59.000",- "param6": "2026-01-01 00:00:01.000",+ "param6": "2025-11-30 23:59:59.000",+ "param7": "2026-01-01 00:00:01.000",},"route": "unknown","sql": "SELECT@@ -309,7 +348,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEvenleftUTF8(e.output, 1000) AS output,mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadataFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",},]`;@@ -320,16 +359,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b","golden-span-c",],- "param3": [+ "param4": ["golden-trace","golden-trace-b",],- "param4": [+ "param5": [TupleParam {"values": ["golden-trace",@@ -349,8 +389,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven],},],- "param5": "2025-11-30 23:59:59.000",- "param6": "2026-01-01 00:00:01.000",+ "param6": "2025-11-30 23:59:59.000",+ "param7": "2026-01-01 00:00:01.000",},"route": "unknown","sql": "SELECT@@ -365,7 +405,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven) AS experiment_item_metadata,mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadataFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",},]`;@@ -376,16 +416,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b","golden-span-c",],- "param3": [+ "param4": ["golden-trace","golden-trace-b",],- "param4": [+ "param5": [TupleParam {"values": ["golden-trace",@@ -405,9 +446,9 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven],},],- "param5": "golden-session",- "param6": "2025-11-30 23:59:59.000",- "param7": "2026-01-01 00:00:01.000",+ "param6": "golden-session",+ "param7": "2025-11-30 23:59:59.000",+ "param8": "2026-01-01 00:00:01.000",},"route": "unknown","sql": "SELECT@@ -417,13 +458,13 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEvenleftUTF8(e.output, 1000) AS output,mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadataFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.trace_id IN (+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.trace_id IN (SELECT trace_idFROM events_core- WHERE (project_id = {param1:String}) AND (trace_id IN ({param3:Array(String)}))+ WHERE (project_id = {param1:String}) AND (events_core.is_deleted = {param2:UInt8}) AND (trace_id IN ({param4:Array(String)}))GROUP BY trace_id- HAVING argMaxIf(session_id, event_ts, session_id != '') = {param5:String}-)) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",+ HAVING argMaxIf(session_id, event_ts, session_id != '') = {param6:String}+)) AND (e.start_time >= {param7:DateTime64(6)}) AND (e.start_time <= {param8:DateTime64(6)})",},]`;@@ -439,13 +480,21 @@ exports[`golden: events IO payload reads family > streamObservationIOFieldFromEv"param4": "2025-12-31 23:59:59.000","param5": "2026-01-01 00:00:01.000","param6": 1,+ "param7": 0,},"route": "observation-io-stream",- "sql": "SELECT e.input AS field-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param6:Int64}+ "sql": "SELECT latest.field+FROM+(+ SELECT+ e.input AS field,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param6:Int64}+) AS latest+WHERE latest.is_deleted = {param7:UInt8}FORMAT RawBLOB",},]@@ -462,13 +511,21 @@ exports[`golden: events IO payload reads family > streamObservationIOFieldFromEv"param4": "2025-12-31 23:59:59.000","param5": "2026-01-01 00:00:01.000","param6": 1,+ "param7": 0,},"route": "observation-io-stream",- "sql": "SELECT toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))) AS field-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param6:Int64}+ "sql": "SELECT latest.field+FROM+(+ SELECT+ toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))) AS field,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param6:Int64}+) AS latest+WHERE latest.is_deleted = {param7:UInt8}FORMAT RawBLOB",},]@@ -485,13 +542,21 @@ exports[`golden: events IO payload reads family > streamObservationIOFieldFromEv"param4": "2025-12-31 23:59:59.000","param5": "2026-01-01 00:00:01.000","param6": 1,+ "param7": 0,},"route": "observation-io-stream",- "sql": "SELECT e.output AS field-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})-ORDER BY e.event_ts DESC-LIMIT {param6:Int64}+ "sql": "SELECT latest.field+FROM+(+ SELECT+ e.output AS field,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT {param6:Int64}+) AS latest+WHERE latest.is_deleted = {param7:UInt8}FORMAT RawBLOB",},]@@ -1662,9 +1727,10 @@ exports[`golden: events point-reads & existence family > getAgentGraphDataFromEv"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": "golden-trace",- "param3": "2026-01-01 00:00:00.000",- "param4": "2026-01-01 01:00:00.000",+ "param2": 0,+ "param3": "golden-trace",+ "param4": "2026-01-01 00:00:00.000",+ "param5": "2026-01-01 01:00:00.000",},"route": "unknown","sql": "SELECT@@ -1677,7 +1743,7 @@ exports[`golden: events point-reads & existence family > getAgentGraphDataFromEvmapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))['langgraph_node'] AS node,mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))['langgraph_step'] AS stepFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.start_time >= {param3:DateTime64(6)}) AND (e.start_time <= {param4:DateTime64(6)})",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.trace_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})",},]`;@@ -1733,63 +1799,112 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE"param1": "golden-project","param2": "golden-span","param3": 1,+ "param4": 0,},"route": "unknown","sql": "SELECT- e.span_id AS id,- e.trace_id AS trace_id,- e.project_id AS project_id,- e.environment AS environment,- e.type AS type,- e.parent_span_id AS parent_observation_id,- e.name AS name,- e.level AS level,- e.status_message AS status_message,- e.version AS version,- e.bookmarked AS bookmarked,- e.public AS public,- e.user_id AS user_id,- e.session_id AS session_id,- ifNull(- coalesce(- nullIf(e.trace_name, ''),- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)- ),- ''- ) AS trace_name,- e.start_time AS start_time,- e.end_time AS end_time,- e.completion_start_time AS completion_start_time,- e.created_at AS created_at,- e.updated_at AS updated_at,- e.event_ts,- e.provided_model_name AS provided_model_name,- e.model_id AS internal_model_id,- e.model_parameters AS model_parameters,- e.provided_usage_details AS provided_usage_details,- e.usage_details AS usage_details,- e.provided_cost_details AS provided_cost_details,- e.cost_details AS cost_details,- e.total_cost AS total_cost,- e.prompt_id AS prompt_id,- e.prompt_name AS prompt_name,- e.prompt_version AS prompt_version,- e.skills_available AS skills_available,- e.skills_resource_loaded AS skills_resource_loaded,- e.tool_definitions AS tool_definitions,- e.tool_calls AS tool_calls,- e.tool_call_names AS tool_call_names,- e.usage_pricing_tier_id AS usage_pricing_tier_id,- e.usage_pricing_tier_name AS usage_pricing_tier_name,- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,- e.tags AS tags,- e.release AS release-FROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})-ORDER BY- start_time DESC,- event_ts DESC-LIMIT {param3:Int64}",+ latest.id,+ latest.trace_id,+ latest.project_id,+ latest.environment,+ latest.type,+ latest.parent_observation_id,+ latest.name,+ latest.level,+ latest.status_message,+ latest.version,+ latest.bookmarked,+ latest.public,+ latest.user_id,+ latest.session_id,+ latest.trace_name,+ latest.start_time,+ latest.end_time,+ latest.completion_start_time,+ latest.created_at,+ latest.updated_at,+ latest.event_ts,+ latest.provided_model_name,+ latest.internal_model_id,+ latest.model_parameters,+ latest.provided_usage_details,+ latest.usage_details,+ latest.provided_cost_details,+ latest.cost_details,+ latest.total_cost,+ latest.prompt_id,+ latest.prompt_name,+ latest.prompt_version,+ latest.skills_available,+ latest.skills_resource_loaded,+ latest.tool_definitions,+ latest.tool_calls,+ latest.tool_call_names,+ latest.usage_pricing_tier_id,+ latest.usage_pricing_tier_name,+ latest.metadata,+ latest.tags,+ latest.release+FROM+(+ SELECT+ e.span_id AS id,+ e.trace_id AS trace_id,+ e.project_id AS project_id,+ e.environment AS environment,+ e.type AS type,+ e.parent_span_id AS parent_observation_id,+ e.name AS name,+ e.level AS level,+ e.status_message AS status_message,+ e.version AS version,+ e.bookmarked AS bookmarked,+ e.public AS public,+ e.user_id AS user_id,+ e.session_id AS session_id,+ ifNull(+ coalesce(+ nullIf(e.trace_name, ''),+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)+ ),+ ''+ ) AS trace_name,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.completion_start_time AS completion_start_time,+ e.created_at AS created_at,+ e.updated_at AS updated_at,+ e.event_ts,+ e.provided_model_name AS provided_model_name,+ e.model_id AS internal_model_id,+ e.model_parameters AS model_parameters,+ e.provided_usage_details AS provided_usage_details,+ e.usage_details AS usage_details,+ e.provided_cost_details AS provided_cost_details,+ e.cost_details AS cost_details,+ e.total_cost AS total_cost,+ e.prompt_id AS prompt_id,+ e.prompt_name AS prompt_name,+ e.prompt_version AS prompt_version,+ e.skills_available AS skills_available,+ e.skills_resource_loaded AS skills_resource_loaded,+ e.tool_definitions AS tool_definitions,+ e.tool_calls AS tool_calls,+ e.tool_call_names AS tool_call_names,+ e.usage_pricing_tier_id AS usage_pricing_tier_id,+ e.usage_pricing_tier_name AS usage_pricing_tier_name,+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,+ e.tags AS tags,+ e.release AS release,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})+ ORDER BY+ start_time DESC,+ event_ts DESC+ LIMIT {param3:Int64}+) AS latest+WHERE latest.is_deleted = {param4:UInt8}",},]`;@@ -1802,65 +1917,116 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE"param1": "golden-project","param2": "golden-span","param3": 1,+ "param4": 0,},"route": "unknown","sql": "SELECT- e.span_id AS id,- e.trace_id AS trace_id,- e.project_id AS project_id,- e.environment AS environment,- e.type AS type,- e.parent_span_id AS parent_observation_id,- e.name AS name,- e.level AS level,- e.status_message AS status_message,- e.version AS version,- e.bookmarked AS bookmarked,- e.public AS public,- e.user_id AS user_id,- e.session_id AS session_id,- ifNull(- coalesce(- nullIf(e.trace_name, ''),- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)- ),- ''- ) AS trace_name,- e.start_time AS start_time,- e.end_time AS end_time,- e.completion_start_time AS completion_start_time,- e.created_at AS created_at,- e.updated_at AS updated_at,- e.event_ts,- e.provided_model_name AS provided_model_name,- e.model_id AS internal_model_id,- e.model_parameters AS model_parameters,- e.provided_usage_details AS provided_usage_details,- e.usage_details AS usage_details,- e.provided_cost_details AS provided_cost_details,- e.cost_details AS cost_details,- e.total_cost AS total_cost,- e.prompt_id AS prompt_id,- e.prompt_name AS prompt_name,- e.prompt_version AS prompt_version,- e.skills_available AS skills_available,- e.skills_resource_loaded AS skills_resource_loaded,- e.tool_definitions AS tool_definitions,- e.tool_calls AS tool_calls,- e.tool_call_names AS tool_call_names,- e.usage_pricing_tier_id AS usage_pricing_tier_id,- e.usage_pricing_tier_name AS usage_pricing_tier_name,- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,- e.tags AS tags,- e.release AS release,- input,- output-FROM events_full AS e-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})-ORDER BY- start_time DESC,- event_ts DESC-LIMIT {param3:Int64}",+ latest.id,+ latest.trace_id,+ latest.project_id,+ latest.environment,+ latest.type,+ latest.parent_observation_id,+ latest.name,+ latest.level,+ latest.status_message,+ latest.version,+ latest.bookmarked,+ latest.public,+ latest.user_id,+ latest.session_id,+ latest.trace_name,+ latest.start_time,+ latest.end_time,+ latest.completion_start_time,+ latest.created_at,+ latest.updated_at,+ latest.event_ts,+ latest.provided_model_name,+ latest.internal_model_id,+ latest.model_parameters,+ latest.provided_usage_details,+ latest.usage_details,+ latest.provided_cost_details,+ latest.cost_details,+ latest.total_cost,+ latest.prompt_id,+ latest.prompt_name,+ latest.prompt_version,+ latest.skills_available,+ latest.skills_resource_loaded,+ latest.tool_definitions,+ latest.tool_calls,+ latest.tool_call_names,+ latest.usage_pricing_tier_id,+ latest.usage_pricing_tier_name,+ latest.metadata,+ latest.tags,+ latest.release,+ latest.input,+ latest.output+FROM+(+ SELECT+ e.span_id AS id,+ e.trace_id AS trace_id,+ e.project_id AS project_id,+ e.environment AS environment,+ e.type AS type,+ e.parent_span_id AS parent_observation_id,+ e.name AS name,+ e.level AS level,+ e.status_message AS status_message,+ e.version AS version,+ e.bookmarked AS bookmarked,+ e.public AS public,+ e.user_id AS user_id,+ e.session_id AS session_id,+ ifNull(+ coalesce(+ nullIf(e.trace_name, ''),+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)+ ),+ ''+ ) AS trace_name,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.completion_start_time AS completion_start_time,+ e.created_at AS created_at,+ e.updated_at AS updated_at,+ e.event_ts,+ e.provided_model_name AS provided_model_name,+ e.model_id AS internal_model_id,+ e.model_parameters AS model_parameters,+ e.provided_usage_details AS provided_usage_details,+ e.usage_details AS usage_details,+ e.provided_cost_details AS provided_cost_details,+ e.cost_details AS cost_details,+ e.total_cost AS total_cost,+ e.prompt_id AS prompt_id,+ e.prompt_name AS prompt_name,+ e.prompt_version AS prompt_version,+ e.skills_available AS skills_available,+ e.skills_resource_loaded AS skills_resource_loaded,+ e.tool_definitions AS tool_definitions,+ e.tool_calls AS tool_calls,+ e.tool_call_names AS tool_call_names,+ e.usage_pricing_tier_id AS usage_pricing_tier_id,+ e.usage_pricing_tier_name AS usage_pricing_tier_name,+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,+ e.tags AS tags,+ e.release AS release,+ input,+ output,+ e.is_deleted AS is_deleted+ FROM events_full AS e+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})+ ORDER BY+ start_time DESC,+ event_ts DESC+ LIMIT {param3:Int64}+) AS latest+WHERE latest.is_deleted = {param4:UInt8}",},]`;@@ -1873,65 +2039,116 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE"param1": "golden-project","param2": "golden-span","param3": 1,+ "param4": 0,},"route": "unknown","sql": "SELECT- e.span_id AS id,- e.trace_id AS trace_id,- e.project_id AS project_id,- e.environment AS environment,- e.type AS type,- e.parent_span_id AS parent_observation_id,- e.name AS name,- e.level AS level,- e.status_message AS status_message,- e.version AS version,- e.bookmarked AS bookmarked,- e.public AS public,- e.user_id AS user_id,- e.session_id AS session_id,- ifNull(- coalesce(- nullIf(e.trace_name, ''),- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)- ),- ''- ) AS trace_name,- e.start_time AS start_time,- e.end_time AS end_time,- e.completion_start_time AS completion_start_time,- e.created_at AS created_at,- e.updated_at AS updated_at,- e.event_ts,- e.provided_model_name AS provided_model_name,- e.model_id AS internal_model_id,- e.model_parameters AS model_parameters,- e.provided_usage_details AS provided_usage_details,- e.usage_details AS usage_details,- e.provided_cost_details AS provided_cost_details,- e.cost_details AS cost_details,- e.total_cost AS total_cost,- e.prompt_id AS prompt_id,- e.prompt_name AS prompt_name,- e.prompt_version AS prompt_version,- e.skills_available AS skills_available,- e.skills_resource_loaded AS skills_resource_loaded,- e.tool_definitions AS tool_definitions,- e.tool_calls AS tool_calls,- e.tool_call_names AS tool_call_names,- e.usage_pricing_tier_id AS usage_pricing_tier_id,- e.usage_pricing_tier_name AS usage_pricing_tier_name,- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,- e.tags AS tags,- e.release AS release,- leftUTF8(input, 1000) AS input,- leftUTF8(output, 1000) AS output-FROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})-ORDER BY- start_time DESC,- event_ts DESC-LIMIT {param3:Int64}",+ latest.id,+ latest.trace_id,+ latest.project_id,+ latest.environment,+ latest.type,+ latest.parent_observation_id,+ latest.name,+ latest.level,+ latest.status_message,+ latest.version,+ latest.bookmarked,+ latest.public,+ latest.user_id,+ latest.session_id,+ latest.trace_name,+ latest.start_time,+ latest.end_time,+ latest.completion_start_time,+ latest.created_at,+ latest.updated_at,+ latest.event_ts,+ latest.provided_model_name,+ latest.internal_model_id,+ latest.model_parameters,+ latest.provided_usage_details,+ latest.usage_details,+ latest.provided_cost_details,+ latest.cost_details,+ latest.total_cost,+ latest.prompt_id,+ latest.prompt_name,+ latest.prompt_version,+ latest.skills_available,+ latest.skills_resource_loaded,+ latest.tool_definitions,+ latest.tool_calls,+ latest.tool_call_names,+ latest.usage_pricing_tier_id,+ latest.usage_pricing_tier_name,+ latest.metadata,+ latest.tags,+ latest.release,+ latest.input,+ latest.output+FROM+(+ SELECT+ e.span_id AS id,+ e.trace_id AS trace_id,+ e.project_id AS project_id,+ e.environment AS environment,+ e.type AS type,+ e.parent_span_id AS parent_observation_id,+ e.name AS name,+ e.level AS level,+ e.status_message AS status_message,+ e.version AS version,+ e.bookmarked AS bookmarked,+ e.public AS public,+ e.user_id AS user_id,+ e.session_id AS session_id,+ ifNull(+ coalesce(+ nullIf(e.trace_name, ''),+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)+ ),+ ''+ ) AS trace_name,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.completion_start_time AS completion_start_time,+ e.created_at AS created_at,+ e.updated_at AS updated_at,+ e.event_ts,+ e.provided_model_name AS provided_model_name,+ e.model_id AS internal_model_id,+ e.model_parameters AS model_parameters,+ e.provided_usage_details AS provided_usage_details,+ e.usage_details AS usage_details,+ e.provided_cost_details AS provided_cost_details,+ e.cost_details AS cost_details,+ e.total_cost AS total_cost,+ e.prompt_id AS prompt_id,+ e.prompt_name AS prompt_name,+ e.prompt_version AS prompt_version,+ e.skills_available AS skills_available,+ e.skills_resource_loaded AS skills_resource_loaded,+ e.tool_definitions AS tool_definitions,+ e.tool_calls AS tool_calls,+ e.tool_call_names AS tool_call_names,+ e.usage_pricing_tier_id AS usage_pricing_tier_id,+ e.usage_pricing_tier_name AS usage_pricing_tier_name,+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,+ e.tags AS tags,+ e.release AS release,+ leftUTF8(input, 1000) AS input,+ leftUTF8(output, 1000) AS output,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})+ ORDER BY+ start_time DESC,+ event_ts DESC+ LIMIT {param3:Int64}+) AS latest+WHERE latest.is_deleted = {param4:UInt8}",},]`;@@ -1948,63 +2165,112 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE"param5": "GENERATION","param6": "golden-trace","param7": 1,+ "param8": 0,},"route": "unknown","sql": "SELECT- e.span_id AS id,- e.trace_id AS trace_id,- e.project_id AS project_id,- e.environment AS environment,- e.type AS type,- e.parent_span_id AS parent_observation_id,- e.name AS name,- e.level AS level,- e.status_message AS status_message,- e.version AS version,- e.bookmarked AS bookmarked,- e.public AS public,- e.user_id AS user_id,- e.session_id AS session_id,- ifNull(- coalesce(- nullIf(e.trace_name, ''),- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)- ),- ''- ) AS trace_name,- e.start_time AS start_time,- e.end_time AS end_time,- e.completion_start_time AS completion_start_time,- e.created_at AS created_at,- e.updated_at AS updated_at,- e.event_ts,- e.provided_model_name AS provided_model_name,- e.model_id AS internal_model_id,- e.model_parameters AS model_parameters,- e.provided_usage_details AS provided_usage_details,- e.usage_details AS usage_details,- e.provided_cost_details AS provided_cost_details,- e.cost_details AS cost_details,- e.total_cost AS total_cost,- e.prompt_id AS prompt_id,- e.prompt_name AS prompt_name,- e.prompt_version AS prompt_version,- e.skills_available AS skills_available,- e.skills_resource_loaded AS skills_resource_loaded,- e.tool_definitions AS tool_definitions,- e.tool_calls AS tool_calls,- e.tool_call_names AS tool_call_names,- e.usage_pricing_tier_id AS usage_pricing_tier_id,- e.usage_pricing_tier_name AS usage_pricing_tier_name,- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,- e.tags AS tags,- e.release AS release-FROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String}) AND (toStartOfMinute(start_time) = toStartOfMinute({param3:DateTime64(6)})) AND (start_time >= {param4:DateTime64(6)}) AND (type = {param5:String}) AND (trace_id = {param6:String})-ORDER BY- start_time DESC,- event_ts DESC-LIMIT {param7:Int64}",+ latest.id,+ latest.trace_id,+ latest.project_id,+ latest.environment,+ latest.type,+ latest.parent_observation_id,+ latest.name,+ latest.level,+ latest.status_message,+ latest.version,+ latest.bookmarked,+ latest.public,+ latest.user_id,+ latest.session_id,+ latest.trace_name,+ latest.start_time,+ latest.end_time,+ latest.completion_start_time,+ latest.created_at,+ latest.updated_at,+ latest.event_ts,+ latest.provided_model_name,+ latest.internal_model_id,+ latest.model_parameters,+ latest.provided_usage_details,+ latest.usage_details,+ latest.provided_cost_details,+ latest.cost_details,+ latest.total_cost,+ latest.prompt_id,+ latest.prompt_name,+ latest.prompt_version,+ latest.skills_available,+ latest.skills_resource_loaded,+ latest.tool_definitions,+ latest.tool_calls,+ latest.tool_call_names,+ latest.usage_pricing_tier_id,+ latest.usage_pricing_tier_name,+ latest.metadata,+ latest.tags,+ latest.release+FROM+(+ SELECT+ e.span_id AS id,+ e.trace_id AS trace_id,+ e.project_id AS project_id,+ e.environment AS environment,+ e.type AS type,+ e.parent_span_id AS parent_observation_id,+ e.name AS name,+ e.level AS level,+ e.status_message AS status_message,+ e.version AS version,+ e.bookmarked AS bookmarked,+ e.public AS public,+ e.user_id AS user_id,+ e.session_id AS session_id,+ ifNull(+ coalesce(+ nullIf(e.trace_name, ''),+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)+ ),+ ''+ ) AS trace_name,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.completion_start_time AS completion_start_time,+ e.created_at AS created_at,+ e.updated_at AS updated_at,+ e.event_ts,+ e.provided_model_name AS provided_model_name,+ e.model_id AS internal_model_id,+ e.model_parameters AS model_parameters,+ e.provided_usage_details AS provided_usage_details,+ e.usage_details AS usage_details,+ e.provided_cost_details AS provided_cost_details,+ e.cost_details AS cost_details,+ e.total_cost AS total_cost,+ e.prompt_id AS prompt_id,+ e.prompt_name AS prompt_name,+ e.prompt_version AS prompt_version,+ e.skills_available AS skills_available,+ e.skills_resource_loaded AS skills_resource_loaded,+ e.tool_definitions AS tool_definitions,+ e.tool_calls AS tool_calls,+ e.tool_call_names AS tool_call_names,+ e.usage_pricing_tier_id AS usage_pricing_tier_id,+ e.usage_pricing_tier_name AS usage_pricing_tier_name,+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,+ e.tags AS tags,+ e.release AS release,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String}) AND (toStartOfMinute(start_time) = toStartOfMinute({param3:DateTime64(6)})) AND (start_time >= {param4:DateTime64(6)}) AND (type = {param5:String}) AND (trace_id = {param6:String})+ ORDER BY+ start_time DESC,+ event_ts DESC+ LIMIT {param7:Int64}+) AS latest+WHERE latest.is_deleted = {param8:UInt8}",},]`;@@ -2015,7 +2281,8 @@ exports[`golden: events point-reads & existence family > getObservationsTraceIds"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-span","golden-span-b",],@@ -2025,7 +2292,7 @@ exports[`golden: events point-reads & existence family > getObservationsTraceIdse.trace_id AS trace_id,e.span_id AS span_idFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)}))",+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)}))",},]`;@@ -2036,10 +2303,11 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-trace",],- "param3": 1,+ "param4": 1,},"route": "unknown","sql": "WITH traces AS@@ -2102,7 +2370,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsTargMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,any(experiment_id) AS experiment_idFROM events_full AS e- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)}))+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)}))GROUP BYtrace_id,project_id@@ -2129,7 +2397,7 @@ SELECTt.outputFROM traces AS tORDER BY t.timestamp DESC-LIMIT {param3:Int64}",+LIMIT {param4:Int64}",},]`;@@ -2140,10 +2408,11 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-trace",],- "param3": 1,+ "param4": 1,},"route": "unknown","sql": "WITH traces AS@@ -2206,7 +2475,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsTargMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,any(experiment_id) AS experiment_idFROM events_full AS e- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)}))+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)}))GROUP BYtrace_id,project_id@@ -2233,7 +2502,7 @@ SELECT'' AS outputFROM traces AS tORDER BY t.timestamp DESC-LIMIT {param3:Int64}",+LIMIT {param4:Int64}",},]`;@@ -2244,12 +2513,13 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-trace",],- "param3": "2025-11-29 00:00:00.000",- "param4": "2026-01-01 00:00:00.000",- "param5": 1,+ "param4": "2025-11-29 00:00:00.000",+ "param5": "2026-01-01 00:00:00.000",+ "param6": 1,},"route": "unknown","sql": "WITH traces AS@@ -2312,7 +2582,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsTargMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,any(experiment_id) AS experiment_idFROM events_full AS e- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)})) AND (start_time >= {param3:DateTime64(6)})+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)})) AND (start_time >= {param4:DateTime64(6)})GROUP BYtrace_id,project_id@@ -2338,9 +2608,9 @@ SELECTt.input,t.outputFROM traces AS t-WHERE toDate(t.timestamp) = toDate({param4:DateTime64(6)})+WHERE toDate(t.timestamp) = toDate({param5:DateTime64(6)})ORDER BY t.timestamp DESC-LIMIT {param5:Int64}",+LIMIT {param6:Int64}",},]`;@@ -2351,10 +2621,11 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-trace",],- "param3": 1,+ "param4": 1,},"route": "unknown","sql": "WITH traces AS@@ -2417,7 +2688,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsTargMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,any(experiment_id) AS experiment_idFROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)}))+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)}))GROUP BYtrace_id,project_id@@ -2444,7 +2715,7 @@ SELECTleftUTF8(t.output, 1000) AS outputFROM traces AS tORDER BY t.timestamp DESC-LIMIT {param3:Int64}",+LIMIT {param4:Int64}",},]`;@@ -2577,30 +2848,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp"param1": "golden-project","param2": "GENERATION","param3": [],- "param4": "input",- "param5": "output",- "param6": 0,+ "param4": 0,+ "param5": "input",+ "param6": "output",},"route": "unknown","sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.prompt_version AS prompt_version,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.start_time AS start_time,- e.end_time AS end_time,- e.usage_details AS usage_details,- e.cost_details AS cost_details,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.prompt_version,+ latest.trace_id,+ latest.span_id,+ latest.start_time,+ latest.end_time,+ latest.usage_details,+ latest.cost_details+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.prompt_version AS prompt_version,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.usage_details AS usage_details,+ e.cost_details AS cost_details,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param4:UInt8})SELECTcount() AS count,@@ -2609,15 +2894,14 @@ SELECTmin(e.start_time) AS first_observation,max(e.start_time) AS last_observation,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param4:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))) AS median_input_usage,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))) AS median_output_usage,medianExact(e.cost_details['total']) AS median_total_cost,medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_msFROM prompt_events AS e-WHERE e.is_deleted = {param6:UInt8}GROUP BYe.prompt_id,e.prompt_version@@ -2638,30 +2922,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp"golden-prompt-id-b",],"param4": "2025-12-01 00:00:00.000",- "param5": "input",- "param6": "output",- "param7": 0,+ "param5": 0,+ "param6": "input",+ "param7": "output",},"route": "unknown","sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.prompt_version AS prompt_version,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.start_time AS start_time,- e.end_time AS end_time,- e.usage_details AS usage_details,- e.cost_details AS cost_details,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.prompt_version,+ latest.trace_id,+ latest.span_id,+ latest.start_time,+ latest.end_time,+ latest.usage_details,+ latest.cost_details+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.prompt_version AS prompt_version,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.usage_details AS usage_details,+ e.cost_details AS cost_details,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param5:UInt8})SELECTcount() AS count,@@ -2670,15 +2968,14 @@ SELECTmin(e.start_time) AS first_observation,max(e.start_time) AS last_observation,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))) AS median_input_usage,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))) AS median_output_usage,medianExact(e.cost_details['total']) AS median_total_cost,medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_msFROM prompt_events AS e-WHERE e.is_deleted = {param7:UInt8}GROUP BYe.prompt_id,e.prompt_version@@ -2700,30 +2997,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp],"param4": "2025-12-01 00:00:00.000","param5": "2026-01-02 00:00:00.000",- "param6": "input",- "param7": "output",- "param8": 0,+ "param6": 0,+ "param7": "input",+ "param8": "output",},"route": "unknown","sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.prompt_version AS prompt_version,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.start_time AS start_time,- e.end_time AS end_time,- e.usage_details AS usage_details,- e.cost_details AS cost_details,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.prompt_version,+ latest.trace_id,+ latest.span_id,+ latest.start_time,+ latest.end_time,+ latest.usage_details,+ latest.cost_details+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.prompt_version AS prompt_version,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.usage_details AS usage_details,+ e.cost_details AS cost_details,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param6:UInt8})SELECTcount() AS count,@@ -2732,15 +3043,14 @@ SELECTmin(e.start_time) AS first_observation,max(e.start_time) AS last_observation,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))) AS median_input_usage,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param8:String}) > 0), e.usage_details)))) AS median_output_usage,medianExact(e.cost_details['total']) AS median_total_cost,medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_msFROM prompt_events AS e-WHERE e.is_deleted = {param8:UInt8}GROUP BYe.prompt_id,e.prompt_version@@ -2761,30 +3071,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp"golden-prompt-id-b",],"param4": "2026-01-02 00:00:00.000",- "param5": "input",- "param6": "output",- "param7": 0,+ "param5": 0,+ "param6": "input",+ "param7": "output",},"route": "unknown","sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.prompt_version AS prompt_version,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.start_time AS start_time,- e.end_time AS end_time,- e.usage_details AS usage_details,- e.cost_details AS cost_details,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time <= {param4:DateTime64(6)})- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.prompt_version,+ latest.trace_id,+ latest.span_id,+ latest.start_time,+ latest.end_time,+ latest.usage_details,+ latest.cost_details+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.prompt_version AS prompt_version,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.usage_details AS usage_details,+ e.cost_details AS cost_details,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time <= {param4:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param5:UInt8})SELECTcount() AS count,@@ -2793,15 +3117,14 @@ SELECTmin(e.start_time) AS first_observation,max(e.start_time) AS last_observation,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))) AS median_input_usage,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))) AS median_output_usage,medianExact(e.cost_details['total']) AS median_total_cost,medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_msFROM prompt_events AS e-WHERE e.is_deleted = {param7:UInt8}GROUP BYe.prompt_id,e.prompt_version@@ -2821,30 +3144,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp"golden-prompt-id","golden-prompt-id-b",],- "param4": "input",- "param5": "output",- "param6": 0,+ "param4": 0,+ "param5": "input",+ "param6": "output",},"route": "unknown","sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.prompt_version AS prompt_version,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.start_time AS start_time,- e.end_time AS end_time,- e.usage_details AS usage_details,- e.cost_details AS cost_details,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.prompt_version,+ latest.trace_id,+ latest.span_id,+ latest.start_time,+ latest.end_time,+ latest.usage_details,+ latest.cost_details+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.prompt_version AS prompt_version,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.start_time AS start_time,+ e.end_time AS end_time,+ e.usage_details AS usage_details,+ e.cost_details AS cost_details,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param4:UInt8})SELECTcount() AS count,@@ -2853,15 +3190,14 @@ SELECTmin(e.start_time) AS first_observation,max(e.start_time) AS last_observation,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param4:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))) AS median_input_usage,medianExact(- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))) AS median_output_usage,medianExact(e.cost_details['total']) AS median_total_cost,medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_msFROM prompt_events AS e-WHERE e.is_deleted = {param6:UInt8}GROUP BYe.prompt_id,e.prompt_version@@ -2999,12 +3335,13 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents f"param1": "input","param2": "output","param3": "golden-project",- "param4": [+ "param4": 0,+ "param5": ["golden-session","golden-session-b",],- "param5": "",- "param6": 2,+ "param6": "",+ "param7": 2,},"route": "unknown","sql": "SELECT@@ -3035,9 +3372,9 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents f) AS session_output_usage,sumMap(usage_details)['total'] AS session_total_usageFROM events_core AS e-WHERE (e.project_id = {param3:String}) AND (session_id IN ({param4:Array(String)})) AND (session_id != {param5:String})+WHERE (e.project_id = {param3:String}) AND (e.is_deleted = {param4:UInt8}) AND (session_id IN ({param5:Array(String)})) AND (session_id != {param6:String})GROUP BY session_id-LIMIT {param6:Int64}",+LIMIT {param7:Int64}",},]`;@@ -3050,21 +3387,22 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents s"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-session","golden-session-b",],- "param3": "",- "param4": 2,+ "param4": "",+ "param5": 2,},"route": "unknown","sql": "SELECTsession_id,uniq(trace_id) AS trace_countFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (session_id IN ({param2:Array(String)})) AND (session_id != {param3:String})+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (session_id IN ({param3:Array(String)})) AND (session_id != {param4:String})GROUP BY session_id-LIMIT {param4:Int64}",+LIMIT {param5:Int64}",},]`;@@ -3075,22 +3413,23 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents s"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-session","golden-session-b",],- "param3": "2025-11-29 00:00:00.000",- "param4": "",- "param5": 2,+ "param4": "2025-11-29 00:00:00.000",+ "param5": "",+ "param6": 2,},"route": "unknown","sql": "SELECTsession_id,uniq(trace_id) AS trace_countFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (session_id IN ({param2:Array(String)})) AND (start_time >= {param3:DateTime64(6)}) AND (session_id != {param4:String})+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (session_id IN ({param3:Array(String)})) AND (start_time >= {param4:DateTime64(6)}) AND (session_id != {param5:String})GROUP BY session_id-LIMIT {param5:Int64}",+LIMIT {param6:Int64}",},]`;diff --git a/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snapindex adc28c90e8..95614e2e7c 100644--- a/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap+++ b/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap@@ -8,11 +8,12 @@ exports[`golden: experiments simple aggregates family > getExperimentMetricsFrom"fn": "queryClickhouse","params": {"param1": "golden-project",- "param2": [+ "param2": 0,+ "param3": ["golden-experiment","golden-experiment-b",],- "param3": "",+ "param4": "",},"route": "unknown","sql": "SELECT@@ -23,7 +24,7 @@ exports[`golden: experiments simple aggregates family > getExperimentMetricsFrom(e.span_id = e.experiment_item_root_span_id) AND (e.end_time IS NOT NULL)) AS latency_avgFROM events_core AS e-WHERE (e.project_id = {param1:String}) AND (e.experiment_id IN ({param2:Array(String)})) AND (e.experiment_id != {param3:String})+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.experiment_id IN ({param3:Array(String)})) AND (e.experiment_id != {param4:String})GROUP BYe.project_id,e.experiment_id",diff --git a/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snapindex 57f58b5b84..0f1739ae82 100644--- a/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap+++ b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap@@ -23,17 +23,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=obser"sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.trace_id,+ latest.span_id+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param6:UInt8})SELECTe.prompt_id AS prompt_id,@@ -45,16 +54,15 @@ SELECTs.data_type AS data_type,s.comment AS comment,s.timestamp AS timestamp,- length(mapKeys(s.metadata)) > {param6:Int64} AS has_metadata+ length(mapKeys(s.metadata)) > {param7:Int64} AS has_metadataFROM scores AS sFINALINNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param7:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param8:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param8:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (SELECTtrace_id,span_idFROM prompt_events- WHERE is_deleted = {param7:UInt8}))GROUP BYe.prompt_id,@@ -91,17 +99,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=obser"sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.trace_id,+ latest.span_id+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param4:UInt8})SELECTe.prompt_id AS prompt_id,@@ -113,16 +130,15 @@ SELECTs.data_type AS data_type,s.comment AS comment,s.timestamp AS timestamp,- length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata+ length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadataFROM scores AS sFINALINNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (SELECTtrace_id,span_idFROM prompt_events- WHERE is_deleted = {param5:UInt8}))GROUP BYe.prompt_id,@@ -160,17 +176,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace"sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.trace_id,+ latest.span_id+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param5:UInt8})SELECTe.prompt_id AS prompt_id,@@ -182,14 +207,13 @@ SELECTs.data_type AS data_type,s.comment AS comment,s.timestamp AS timestamp,- length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadata+ length(mapKeys(s.metadata)) > {param6:Int64} AS has_metadataFROM scores AS sFINALINNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param6:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param7:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param7:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (SELECT trace_idFROM prompt_events- WHERE is_deleted = {param6:UInt8}))GROUP BYe.prompt_id,@@ -226,17 +250,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace"sql": "WITH prompt_events AS(SELECT- e.project_id AS project_id,- e.prompt_id AS prompt_id,- e.trace_id AS trace_id,- e.span_id AS span_id,- e.is_deleted AS is_deleted- FROM events_core AS e- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))- ORDER BY e.event_ts DESC- LIMIT 1 BY- e.span_id,- e.project_id+ latest.project_id,+ latest.prompt_id,+ latest.trace_id,+ latest.span_id+ FROM+ (+ SELECT+ e.project_id AS project_id,+ e.prompt_id AS prompt_id,+ e.trace_id AS trace_id,+ e.span_id AS span_id,+ e.is_deleted AS is_deleted+ FROM events_core AS e+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))+ ORDER BY e.event_ts DESC+ LIMIT 1 BY+ e.span_id,+ e.project_id+ ) AS latest+ WHERE latest.is_deleted = {param4:UInt8})SELECTe.prompt_id AS prompt_id,@@ -248,14 +281,13 @@ SELECTs.data_type AS data_type,s.comment AS comment,s.timestamp AS timestamp,- length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata+ length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadataFROM scores AS sFINALINNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (SELECT trace_idFROM prompt_events- WHERE is_deleted = {param5:UInt8}))GROUP BYe.prompt_id,@@ -276,12 +308,13 @@ exports[`golden: scores.getScoresForExperimentItems > experimentIds set 1`] = `"params": {"param1": 0,"param2": "golden-project",- "param3": [+ "param3": 0,+ "param4": ["exp-a","exp-b",],- "param4": "",- "param5": [+ "param5": "",+ "param6": ["NUMERIC","BOOLEAN","CATEGORICAL",@@ -319,11 +352,11 @@ FROMe.experiment_id,e.trace_idFROM events_core AS e- WHERE (e.project_id = {param2:String}) AND (e.experiment_id IN ({param3:Array(String)})) AND (e.experiment_id != {param4:String})+ WHERE (e.project_id = {param2:String}) AND (e.is_deleted = {param3:UInt8}) AND (e.experiment_id IN ({param4:Array(String)})) AND (e.experiment_id != {param5:String})) AS eINNER JOIN scores AS sFINAL ON (e.trace_id = s.trace_id) AND (e.project_id = s.project_id) AND (s.project_id = {param2:String})-WHERE s.data_type IN ({param5:Array(String)})+WHERE s.data_type IN ({param6:Array(String)})ORDER BY s.event_ts DESCLIMIT 1 BYs.id,diff --git a/packages/shared/src/server/repositories/environments.ts b/packages/shared/src/server/repositories/environments.tsindex 26728d073f..cea6d96b04 100644--- a/packages/shared/src/server/repositories/environments.ts+++ b/packages/shared/src/server/repositories/environments.ts@@ -2,7 +2,6 @@ import { LISTABLE_SCORE_TYPES } from "../../domain/scores";import { env } from "../../env";import { compileClickhouseQuery } from "../query-ast/compile";import { getClickhouseKysely } from "../query-ast/dialect";-import { includeDeleted } from "../query-ast/extensions";import { queryClickhouse } from "./clickhouse";export type EnvironmentFilterProps = {@@ -43,7 +42,6 @@ export const getEnvironmentsForProject = async (): db.selectFrom("events_core")- .$call(includeDeleted()).select("environment").distinct().$if(fromTimestamp != null, (qb) =>diff --git a/packages/shared/src/server/repositories/events.ts b/packages/shared/src/server/repositories/events.tsindex 70f40c42b9..73b8204f03 100644--- a/packages/shared/src/server/repositories/events.ts+++ b/packages/shared/src/server/repositories/events.ts@@ -37,7 +37,7 @@ import {byteLength,count,dateDiff,- includeDeleted,+ softDeleteAfterLimit,leftUTF8,limitBy,literal,@@ -669,7 +669,6 @@ export const getObservationMetricsForPromptsFromEvents = async ("e.end_time as end_time","e.usage_details as usage_details","e.cost_details as cost_details",- "e.is_deleted as is_deleted",],{ fromTimestamp, toTimestamp },),@@ -694,7 +693,6 @@ export const getObservationMetricsForPromptsFromEvents = async (dateDiff("millisecond", eb.ref("e.start_time"), eb.ref("e.end_time")),).as("median_latency_ms"),])- .where("e.is_deleted", "=", 0).groupBy(["e.prompt_id", "e.prompt_version"]).orderBy("e.prompt_version", "desc");const { sql: query, params } = compileClickhouseQuery(builder, projectId);@@ -1104,12 +1102,13 @@ async function getObservationByIdFromEventsTableInternal({const needsFullTable =fetchWithInputOutput && renderingProps.truncated !== true;+ // A soft-deleted latest version reads as not found, not as an older version.const builder = (needsFullTable? db.selectFrom("events_full as e"): db.selectFrom("events_core as e"))- .$call(includeDeleted())+ .$call(softDeleteAfterLimit()).select(OBSERVATION_BY_ID_SELECTS).$if(fetchWithInputOutput, (qb) =>qb.select(observationIoSelects(renderingProps.truncated)),@@ -1197,7 +1196,6 @@ export const getTraceByIdFromEventsTable = async ({? qb.selectFrom("events_core as e"): qb.selectFrom("events_full as e");return from- .$call(includeDeleted()).select(TRACE_AGGREGATION_SELECTS).where("trace_id", "in", [traceId]).$if(fromTimestamp != null, (inner) =>@@ -2641,7 +2639,6 @@ export async function getAgentGraphDataFromEventsTable(params: {const builder = db.selectFrom("events_core as e")- .$call(includeDeleted()).select((eb) => {const metadata = metadataMap(eb.ref("e.metadata_names"),@@ -2814,7 +2811,6 @@ export const getObservationsBatchIOFromEventsTable = async <// for exact trace/observation matching.const builder = getClickhouseKysely().selectFrom(truncated ? "events_core as e" : "events_full as e")- .$call(includeDeleted()).select((eb) => ["e.span_id as id","e.trace_id as trace_id",@@ -2861,7 +2857,6 @@ export const getObservationsBatchIOFromEventsTable = async <qb.where("e.trace_id", "in", (eb) =>eb.selectFrom("events_core")- .$call(includeDeleted()).select("trace_id").where("trace_id", "in", traceIds).groupBy("trace_id")@@ -2962,16 +2957,15 @@ interface ObservationIOReadOpts {// The field expression is chosen from the validated `field` enum — never// string-interpolated user input. The length pre-query and the byte stream share// this builder so they can never drift; a mismatch would make the route's-// Content-Length integrity check wrong.+// Content-Length integrity check wrong. A soft-deleted latest version reads as+// no row.function compileObservationIOFieldRead<A extends string>(opts: ObservationIOReadOpts,select: (field: AliasableExpression<string>) => AliasedExpression<unknown, A>,) {- const db = getClickhouseKysely();-- const query = db+ const query = getClickhouseKysely().selectFrom("events_full as e")- .$call(includeDeleted())+ .$call(softDeleteAfterLimit()).select((eb) =>select(opts.field === "metadata"@@ -3088,9 +3082,10 @@ export const getObservationFullIOForSessionFromEventsTable = async (opts: {const maxTimestamp = new Date(opts.startTime.getTime() + 1000);const db = getClickhouseKysely();+ // A soft-deleted latest version reads as not found.const builder = db.selectFrom("events_full as e")- .$call(includeDeleted())+ .$call(softDeleteAfterLimit()).select((eb) => ["e.span_id as id","e.input as input",@@ -3160,7 +3155,6 @@ export const getObservationsTraceIdsFromEventsTable = async (opts: {const builder = db.selectFrom("events_core as e")- .$call(includeDeleted()).select(["e.trace_id as trace_id", "e.span_id as span_id"]).where("e.span_id", "in", observationIds);@@ -3906,7 +3900,6 @@ export const getSessionMetricsFromEvents = async <const { projectId, sessionIds, fields, queryFromTimestamp } = props;const builder = getClickhouseKysely().selectFrom("events_core as e")- .$call(includeDeleted()).select(fields.map((field) => SESSION_AGGREGATION_SELECTS[field])).where("session_id", "in", sessionIds).$if(queryFromTimestamp != null, (qb) =>diff --git a/packages/shared/src/server/repositories/experiments.ts b/packages/shared/src/server/repositories/experiments.tsindex cc9f35aacb..839f7df8d2 100644--- a/packages/shared/src/server/repositories/experiments.ts+++ b/packages/shared/src/server/repositories/experiments.ts@@ -8,7 +8,7 @@ import { type FilterState } from "../../types";import { convertDateToClickhouseDateTime } from "../clickhouse/client";import { compileClickhouseQuery } from "../query-ast/compile";import { getClickhouseKysely } from "../query-ast/dialect";-import { avgIf, dateDiff, includeDeleted } from "../query-ast/extensions";+import { avgIf, dateDiff } from "../query-ast/extensions";import {FilterList,StringOptionsFilter,@@ -360,7 +360,6 @@ export const getExperimentMetricsFromEvents = async (props: {const { projectId, experimentIds } = props;const builder = getClickhouseKysely().selectFrom("events_core as e")- .$call(includeDeleted()).select((eb) => ["e.experiment_id as experiment_id",eb.fn.sum("e.total_cost").as("total_cost"),diff --git a/packages/shared/src/server/repositories/prompt-events.ts b/packages/shared/src/server/repositories/prompt-events.tsindex 01a64c499e..728b903ba0 100644--- a/packages/shared/src/server/repositories/prompt-events.ts+++ b/packages/shared/src/server/repositories/prompt-events.ts@@ -1,7 +1,7 @@import type { SelectExpression } from "kysely";import { getClickhouseKysely } from "../query-ast/dialect";-import { includeDeleted, limitBy } from "../query-ast/extensions";+import { limitBy, softDeleteAfterLimit } from "../query-ast/extensions";import type { ClickHouseDatabase } from "../query-ast/schema";type EventsAsE = ClickHouseDatabase & { e: ClickHouseDatabase["events_core"] };@@ -9,8 +9,7 @@ type EventsAsE = ClickHouseDatabase & { e: ClickHouseDatabase["events_core"] };/*** Latest version of each generation linked to one of `promptIds`, optionally* within a start-time window. Callers pick the columns and use it as a CTE.- * Soft-deleted rows stay in so a deletion can be the latest version; callers- * select `is_deleted` and drop those rows after the LIMIT BY.+ * A generation whose latest version is soft-deleted is left out.*/export const promptGenerationEvents = <S extends SelectExpression<EventsAsE, "e">,@@ -21,7 +20,7 @@ export const promptGenerationEvents = <) =>getClickhouseKysely().selectFrom("events_core as e")- .$call(includeDeleted())+ .$call(softDeleteAfterLimit()).select(columns).where("e.type", "=", "GENERATION").where("e.prompt_id", "in", promptIds)diff --git a/packages/shared/src/server/repositories/scores.ts b/packages/shared/src/server/repositories/scores.tsindex e55deb81de..e548d19b08 100644--- a/packages/shared/src/server/repositories/scores.ts+++ b/packages/shared/src/server/repositories/scores.ts@@ -76,12 +76,7 @@ import {} from "../queries/clickhouse-sql/query-fragments";import { compileClickhouseQuery } from "../query-ast/compile";import { getClickhouseKysely } from "../query-ast/dialect";-import {- includeDeleted,- limitBy,- mapKeys,- useFinal,-} from "../query-ast/extensions";+import { limitBy, mapKeys, useFinal } from "../query-ast/extensions";import { scoresTableCols } from "../../tableDefinitions/scoresTable";import { promptGenerationEvents } from "./prompt-events";import {@@ -426,7 +421,6 @@ export const getTraceScoresForDatasetRuns = async (const experimentItemEventsQuery = (experimentIds: string[]) =>getClickhouseKysely().selectFrom("events_core as e")- .$call(includeDeleted()).select(["e.project_id", "e.experiment_id", "e.trace_id"]).where("e.experiment_id", "in", experimentIds).where("e.experiment_id", "!=", "");@@ -2255,7 +2249,6 @@ const aggregatedScoresForPromptsFromEventsQuery = ("e.prompt_id as prompt_id","e.trace_id as trace_id","e.span_id as span_id",- "e.is_deleted as is_deleted",],timeWindow,),@@ -2282,17 +2275,13 @@ const aggregatedScoresForPromptsFromEventsQuery = ("s.timestamp as timestamp",eb(eb.fn("length", [mapKeys("s.metadata")]), ">", 0).as("has_metadata"),])- .where("e.is_deleted", "=", 0).where("s.name", "is not", null).where("s.data_type", "in", [...LISTABLE_SCORE_TYPES]).$if(fetchScoreRelation === "trace", (qb) =>qb.where("s.observation_id", "is", null).where("s.trace_id", "in", (eb) =>- eb- .selectFrom("prompt_events")- .select("trace_id")- .where("is_deleted", "=", 0),+ eb.selectFrom("prompt_events").select("trace_id"),),).$if(fetchScoreRelation === "observation", (qb) =>@@ -2304,7 +2293,6 @@ const aggregatedScoresForPromptsFromEventsQuery = ("in",selectFrom("prompt_events").select(["trace_id", "span_id"])- .where("is_deleted", "=", 0).$asTuple("trace_id", "span_id"),),),diff --git a/web/src/__tests__/server/repositories/event-repository.servertest.ts b/web/src/__tests__/server/repositories/event-repository.servertest.tsindex 1f31c3f936..55ed23dbe1 100644--- a/web/src/__tests__/server/repositories/event-repository.servertest.ts+++ b/web/src/__tests__/server/repositories/event-repository.servertest.ts@@ -7,6 +7,7 @@ import {getObservationsCountFromEventsTable,getObservationsCountsFromEventsTable,getObservationByIdFromEventsTable,+ getObservationFullIOForSessionFromEventsTable,getAgentGraphDataFromEventsTable,getObservationsFromEventsTableForPublicApi,getObservationsCountFromEventsTableForPublicApi,@@ -38,6 +39,7 @@ import { prisma } from "@langfuse/shared/src/db";import { randomUUID } from "crypto";import { env } from "@/src/env.mjs";import {+ LangfuseNotFoundError,type EventsTableFilterState,type FilterCondition,type TimeFilter,@@ -1180,6 +1182,41 @@ describe("Clickhouse Events Repository Test", () => {});});+ maybe("latest-version reads of a soft-deleted observation", () => {+ it("read it as missing even though an older live version exists", async () => {+ const traceId = randomUUID();+ const observationId = randomUUID();+ const sessionId = randomUUID();+ const start = Date.now() - 60_000;+ const version = {+ id: observationId,+ span_id: observationId,+ project_id: projectId,+ trace_id: traceId,+ session_id: sessionId,+ start_time: start,+ };++ await createEventsCh([+ createEvent({ ...version, event_ts: start }),+ createEvent({ ...version, event_ts: start + 1000, is_deleted: 1 }),+ ]);++ await expect(+ getObservationByIdFromEventsTable({ id: observationId, projectId }),+ ).rejects.toThrow(LangfuseNotFoundError);+ await expect(+ getObservationFullIOForSessionFromEventsTable({+ projectId,+ sessionId,+ traceId,+ observationId,+ startTime: new Date(start),+ }),+ ).resolves.toBeNull();+ });+ });+maybe("getAgentGraphDataFromEventsTable", () => {it("bounds start_time by Date binds at millisecond precision", async () => {const traceId = randomUUID();diff --git a/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts b/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.tsindex 52caa34443..286059675e 100644--- a/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts+++ b/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts@@ -146,4 +146,36 @@ maybe("streamObservationIOFieldFromEventsTable", () => {});expect(missing).toBeNull();});++ it("reads an observation whose latest version is soft-deleted as missing", async () => {+ const opts = {+ projectId: randomUUID(),+ traceId: randomUUID(),+ observationId: randomUUID(),+ field: "input" as const,+ startTime: new Date(),+ };+ const startMs = opts.startTime.getTime();+ const version = {+ id: opts.observationId,+ span_id: opts.observationId,+ project_id: opts.projectId,+ trace_id: opts.traceId,+ input: "live",+ start_time: startMs * 1000,+ };++ await createEventsCh([+ createEvent({ ...version, event_ts: startMs }),+ createEvent({ ...version, event_ts: startMs + 1000, is_deleted: 1 }),+ ]);++ expect(await getObservationIOFieldByteLengthFromEventsTable(opts)).toBe(+ null,+ );+ const streamed = await streamObservationIOFieldFromEventsTable(opts).then(+ (r) => collectStream(r.stream),+ );+ expect(streamed).toBe("");+ });});