Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(query-ast): hide soft-deleted rows from events reads (#18869)

Niklas Semmler1896859401ef
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index 7898c8521d..4f1d7b5114 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -30,9 +30,9 @@ changing how it emits SQL.
hand-maintained tables that drift.
- `compiler.ts` / `transformer.ts` / `dialect.ts` / `nodes.ts` — the dialect.
- `extensions.ts` — the `$call` helpers (`arrayJoin`, `limitBy`, `useFinal`,
- `includeDeleted`) and typed ClickHouse functions and aggregates (`argMaxIf`,
- `countIf`, `sumMap`, `multiIf`, `metadataValue`, `literal`, …) that take
- `eb.ref(...)` expressions.
+ `includeDeleted`, `softDeleteAfterLimit`) and typed ClickHouse functions and
+ aggregates (`argMaxIf`, `countIf`, `sumMap`, `multiIf`, `metadataValue`,
+ `literal`, …) that take `eb.ref(...)` expressions.
- `views.ts` / `dedup.ts` / `softDelete.ts` — virtual views as CTEs, per-table
dedup lowering, and soft-delete filtering.
- `fixtures/catalog.ts` — test-only catalog of builder queries exercising the
@@ -87,9 +87,15 @@ suffix to run there. Promote it to a required check once it has proven stable.
only `projectId` (see `../repositories/environments.ts`).
- **Never filter `is_deleted` on a physical table yourself.** Tables that
declare `softDelete` in the registry (`events_core`, `events_full`) get
- `is_deleted = 0` on every FROM/JOIN. A select that must see deleted rows
- opts out with `$call(includeDeleted())`, e.g. a latest-version read that
- drops tombstones after its `LIMIT 1 BY`.
+ `is_deleted = 0` on every FROM/JOIN. A latest-version read (ordered by
+ version, then `LIMIT 1` or `LIMIT 1 BY`) marks itself
+ `$call(softDeleteAfterLimit())`, so a deletion (a newer row with the flag
+ set) wins the pick and is dropped after it, instead of an older live
+ version. A select that must see deleted rows opts out with
+ `$call(includeDeleted())`. Reads without a version pick (lists,
+ aggregates) filter each row on its own: like an updated row under dedup
+ `none`, a deleted row's older live version stays visible until a merge
+ collapses the versions.
- **Dedup is declared per table and must be an existing production idiom.**
`events_core` and `events_full` are `none` (immutable at read time — no
LIMIT BY, no FINAL).
@@ -137,10 +143,12 @@ db.selectFrom("observations")
```ts
db.selectFrom("events_core")
+ .$call(softDeleteAfterLimit())
.select(["span_id", "project_id"])
.orderBy("event_ts", "desc")
.$call(limitBy({ count: 1, columns: ["span_id", "project_id"] }));
-// … order by event_ts desc limit 1 by span_id, project_id
+// … order by event_ts desc limit 1 by span_id, project_id, wrapped by the
+// soft-delete pass so a deleted latest version is dropped
```
**Write FINAL** (`useFinal` — ReplacingMergeTree reads; never on `events_core`):
@@ -206,7 +214,10 @@ and it is where tenancy is enforced:
required parameter, so omitting it is also a compile-time type error.
2. `SoftDeleteFilterPlugin` adds `<flag> = 0` for every FROM/JOIN relation
whose table declares `softDelete`, except in selects marked with
- `includeDeleted()`. It runs before tenancy, so the WHERE reads
+ `includeDeleted()`. A select marked `softDeleteAfterLimit()` is left
+ unfiltered, gets the flag added to its outputs, and is wrapped in an outer
+ select over those outputs that keeps rows with `latest.<flag> = 0`. It
+ runs before tenancy, so the WHERE reads
`project_id = … and is_deleted = … and …`.
3. `TenancyInjectionPlugin` walks every FROM/JOIN and injects
`project_id = {projectId}` on each tenanted physical table, unless the tree
diff --git a/packages/shared/src/server/query-ast/extensions.ts b/packages/shared/src/server/query-ast/extensions.ts
index 1a12ae18fa..3948c25df2 100644
--- a/packages/shared/src/server/query-ast/extensions.ts
+++ b/packages/shared/src/server/query-ast/extensions.ts
@@ -782,18 +782,6 @@ export function useFinal(tables: ReadonlyArray<keyof ClickHouseDatabase>) {
): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new FinalPlugin(names));
}
-/** Plugin that marks the select so the soft-delete pass leaves it unfiltered. */
-class IncludeDeletedPlugin implements KyselyPlugin {
- transformQuery(args: PluginTransformQueryArgs): RootOperationNode {
- if (args.node.kind !== "SelectQueryNode") return args.node;
- return { ...args.node, includeDeleted: true } as RootOperationNode;
- }
-
- async transformResult(args: PluginTransformResultArgs) {
- return args.result;
- }
-}
-
/**
* Read soft-deleted rows too: `qb.$call(includeDeleted())`. Applies to this
* select's FROM/JOIN relations only; nested selects are still filtered. A
@@ -802,5 +790,39 @@ class IncludeDeletedPlugin implements KyselyPlugin {
export function includeDeleted() {
return <DB, TB extends keyof DB, O>(
qb: SelectQueryBuilder<DB, TB, O>,
- ): SelectQueryBuilder<DB, TB, O> => qb.withPlugin(new IncludeDeletedPlugin());
+ ): SelectQueryBuilder<DB, TB, O> =>
+ qb.withPlugin(new SoftDeleteMarkerPlugin("includeDeleted"));
+}
+
+/**
+ * Drop soft-deleted rows after this select's ORDER BY / LIMIT / LIMIT BY
+ * instead of before: `qb.$call(softDeleteAfterLimit())`. For a latest-version
+ * pick, where a deletion is a newer version with the flag set; filtering
+ * first would return the older live version. Compiles to
+ * `select latest.<output>, … from (<this select, flag added>) as latest where
+ * latest.<flag> = 0`. The select reads exactly one soft-delete table, has a
+ * LIMIT or LIMIT BY, has no GROUP BY or set operation, names every output (no
+ * `*`, unaliased expressions), and does not select the flag itself.
+ */
+export function softDeleteAfterLimit() {
+ return <DB, TB extends keyof DB, O>(
+ qb: SelectQueryBuilder<DB, TB, O>,
+ ): SelectQueryBuilder<DB, TB, O> =>
+ qb.withPlugin(new SoftDeleteMarkerPlugin("softDeleteAfterLimit"));
+}
+
+/** Sets a soft-delete marker on the select; the soft-delete pass reads it. */
+class SoftDeleteMarkerPlugin implements KyselyPlugin {
+ constructor(
+ private readonly marker: "includeDeleted" | "softDeleteAfterLimit",
+ ) {}
+
+ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {
+ if (args.node.kind !== "SelectQueryNode") return args.node;
+ return { ...args.node, [this.marker]: true } as RootOperationNode;
+ }
+
+ async transformResult(args: PluginTransformResultArgs) {
+ return args.result;
+ }
}
diff --git a/packages/shared/src/server/query-ast/nodes.ts b/packages/shared/src/server/query-ast/nodes.ts
index 74ea1ab4a9..0356d89f12 100644
--- a/packages/shared/src/server/query-ast/nodes.ts
+++ b/packages/shared/src/server/query-ast/nodes.ts
@@ -107,6 +107,11 @@ export type ClickHouseSelectQueryNode = SelectQueryNode & {
readonly limitBy?: LimitByNode;
/** Set by `includeDeleted()`: the soft-delete pass skips this select. */
readonly includeDeleted?: true;
+ /**
+ * Set by `softDeleteAfterLimit()`: the soft-delete pass filters this
+ * select's output instead of its relation.
+ */
+ readonly softDeleteAfterLimit?: true;
};
export function isClickHouseSelectQueryNode(
diff --git a/packages/shared/src/server/query-ast/schema.ts b/packages/shared/src/server/query-ast/schema.ts
index 274f64cf6f..48d466b762 100644
--- a/packages/shared/src/server/query-ast/schema.ts
+++ b/packages/shared/src/server/query-ast/schema.ts
@@ -136,7 +136,7 @@ function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {
/**
* Column that marks a soft-deleted row. The soft-delete pass injects
* `<column> = 0` on every read of this table unless the select opts out
- * with `includeDeleted()`.
+ * with `includeDeleted()`, or after its LIMIT with `softDeleteAfterLimit()`.
*/
softDelete?: keyof Cols & string;
}) {
diff --git a/packages/shared/src/server/query-ast/softDelete.test.ts b/packages/shared/src/server/query-ast/softDelete.test.ts
index 4446cca18f..6363b538a2 100644
--- a/packages/shared/src/server/query-ast/softDelete.test.ts
+++ b/packages/shared/src/server/query-ast/softDelete.test.ts
@@ -9,7 +9,7 @@ import { describe, expect, it } from "vitest";
import { compileClickhouseQuery } from "./compile";
import { getClickhouseKysely } from "./dialect";
import { QueryCompileError } from "./errors";
-import { includeDeleted, limitBy } from "./extensions";
+import { includeDeleted, limitBy, softDeleteAfterLimit } from "./extensions";
import { FinalTableNode } from "./nodes";
import { SOFT_DELETE_COLUMNS } from "./schema";
import { SoftDeleteFilterPlugin } from "./softDelete";
@@ -309,4 +309,153 @@ describe("soft-delete filtering", () => {
),
).toThrow(QueryCompileError);
});
+
+ it("softDeleteAfterLimit() filters after the pick, projecting outputs by name", () => {
+ expect(
+ compile(
+ getClickhouseKysely()
+ .selectFrom("events_full as e")
+ .$call(softDeleteAfterLimit())
+ .select((eb) => [
+ "e.span_id as id",
+ "e.event_ts",
+ "input",
+ eb.fn<number>("length", ["e.output"]).as("len"),
+ ])
+ .where("e.span_id", "=", "o")
+ .orderBy("e.event_ts", "desc")
+ .limit(1),
+ ),
+ ).toMatchInlineSnapshot(
+ `"select latest.id, latest.event_ts, latest.input, latest.len from (select e.span_id as id, e.event_ts, input, length(e.output) as len, e.is_deleted as is_deleted from events_full as e where e.project_id = {p1:String} and e.span_id = {p2:String} order by e.event_ts desc limit {p3:Int64}) as latest where latest.is_deleted = {p4:UInt8}"`,
+ );
+ });
+
+ it("softDeleteAfterLimit() wraps a CTE body and keeps its LIMIT BY inside", () => {
+ expect(
+ compile(
+ getClickhouseKysely()
+ .with("latest_events", () =>
+ getClickhouseKysely()
+ .selectFrom("events_core as e")
+ .$call(softDeleteAfterLimit())
+ .select(["e.span_id as span_id", "e.trace_id as trace_id"])
+ .orderBy("e.event_ts", "desc")
+ .$call(limitBy({ count: 1, columns: ["e.span_id"] })),
+ )
+ .selectFrom("latest_events as e")
+ .select("e.trace_id"),
+ ),
+ ).toMatchInlineSnapshot(
+ `"with latest_events as (select latest.span_id, latest.trace_id from (select e.span_id as span_id, e.trace_id as trace_id, e.is_deleted as is_deleted from events_core as e where e.project_id = {p1:String} order by e.event_ts desc limit 1 by e.span_id) as latest where latest.is_deleted = {p2:UInt8}) select e.trace_id from latest_events as e"`,
+ );
+ });
+
+ it("softDeleteAfterLimit() qualifies the flag by table name on an unaliased FROM", () => {
+ expect(
+ compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .$call(softDeleteAfterLimit())
+ .select("span_id")
+ .limit(1),
+ ),
+ ).toMatchInlineSnapshot(
+ `"select latest.span_id from (select span_id, events_core.is_deleted as is_deleted from events_core where project_id = {p1:String} limit {p2:Int64}) as latest where latest.is_deleted = {p3:UInt8}"`,
+ );
+ });
+
+ it("softDeleteAfterLimit() leaves nested selects filtered", () => {
+ expect(
+ compile(
+ getClickhouseKysely()
+ .selectFrom("events_core as e")
+ .$call(softDeleteAfterLimit())
+ .select("e.span_id")
+ .where("e.trace_id", "in", (eb) =>
+ eb.selectFrom("events_core").select("trace_id"),
+ )
+ .limit(1),
+ ),
+ ).toContain("events_core.is_deleted = {p2:UInt8}");
+ });
+
+ it("rejects softDeleteAfterLimit() on selects it cannot wrap", () => {
+ const marked = () =>
+ getClickhouseKysely()
+ .selectFrom("events_core as e")
+ .$call(softDeleteAfterLimit())
+ .limit(1);
+ const rejects = (
+ builder: Parameters<typeof compileClickhouseQuery>[0],
+ message: RegExp,
+ ) => expect(() => compile(builder)).toThrow(message);
+
+ rejects(marked(), /without outputs/);
+ rejects(
+ getClickhouseKysely()
+ .selectFrom("events_core as e")
+ .$call(softDeleteAfterLimit())
+ .select("e.span_id"),
+ /needs a LIMIT or LIMIT BY/,
+ );
+ rejects(
+ marked().select("e.trace_id").groupBy("e.trace_id"),
+ /cannot wrap a GROUP BY or set operation/,
+ );
+ rejects(
+ marked()
+ .select("e.span_id")
+ .unionAll(
+ getClickhouseKysely().selectFrom("events_core").select("span_id"),
+ ),
+ /cannot wrap a GROUP BY or set operation/,
+ );
+ rejects(marked().selectAll(), /every output named/);
+ rejects(marked().selectAll("e"), /every output named/);
+ rejects(marked().select("e.is_deleted"), /adds `is_deleted` itself/);
+ rejects(
+ marked().select((eb) => [eb.lit(0).as("is_deleted")]),
+ /adds `is_deleted` itself/,
+ );
+ rejects(
+ getClickhouseKysely()
+ .selectFrom("scores")
+ .$call(softDeleteAfterLimit())
+ .select("id"),
+ /reads 0/,
+ );
+ rejects(
+ marked()
+ .innerJoin("events_full as f", (join) =>
+ join.onRef("f.span_id", "=", "e.span_id"),
+ )
+ .select("e.span_id"),
+ /reads 2/,
+ );
+ rejects(
+ getClickhouseKysely()
+ .selectFrom("scores as s")
+ .leftJoin("events_core as e", (join) =>
+ join.onRef("e.trace_id", "=", "s.trace_id"),
+ )
+ .$call(softDeleteAfterLimit())
+ .select("s.id")
+ .limit(1),
+ /cannot wrap a LeftJoin/,
+ );
+ rejects(
+ marked()
+ .rightJoin("scores as s", (join) =>
+ join.onRef("s.trace_id", "=", "e.trace_id"),
+ )
+ .select("s.id"),
+ /cannot wrap a RightJoin/,
+ );
+ rejects(marked().distinct().select("e.span_id"), /cannot wrap a DISTINCT/);
+ rejects(
+ marked().$call(includeDeleted()).select("e.span_id"),
+ /cannot mark the same select/,
+ );
+ });
});
diff --git a/packages/shared/src/server/query-ast/softDelete.ts b/packages/shared/src/server/query-ast/softDelete.ts
index ad24332d93..e89ae881b7 100644
--- a/packages/shared/src/server/query-ast/softDelete.ts
+++ b/packages/shared/src/server/query-ast/softDelete.ts
@@ -6,7 +6,8 @@
* for a JOIN with one, prepended to the WHERE otherwise. A RIGHT or FULL JOIN
* in such a select fails to compile. A select that opts out with
* `$call(includeDeleted())` is left unchanged; nested selects decide for
- * themselves.
+ * themselves. A select marked `$call(softDeleteAfterLimit())` keeps its rows
+ * and is wrapped in an outer select that filters on its output instead.
*
* Runs before tenancy, so the tenancy pass prepends `project_id` ahead of the
* flag: `where project_id = … and is_deleted = … and <caller predicates>`.
@@ -20,6 +21,8 @@ import {
JoinNode,
OperatorNode,
ReferenceNode,
+ SelectionNode,
+ SelectQueryNode,
TableNode,
ValueNode,
WhereNode,
@@ -41,6 +44,8 @@ import { SOFT_DELETE_COLUMNS } from "./schema";
import { CteScopedTransformer } from "./transformer";
const OUTER_PRESERVING_JOINS = new Set<JoinType>(["RightJoin", "FullJoin"]);
+const AFTER_LIMIT_ALIAS = "latest";
+const AFTER_LIMIT_JOINS = new Set<JoinType>(["InnerJoin", "CrossJoin"]);
export class SoftDeleteFilterPlugin implements KyselyPlugin {
transformQuery(args: PluginTransformQueryArgs): RootOperationNode {
@@ -78,6 +83,12 @@ function filterSelect(
const readsSoftDelete =
froms.some(Boolean) || joins.some(({ relation }) => relation);
+ if (node.softDeleteAfterLimit) {
+ return filterAfterLimit(node, [
+ ...froms,
+ ...joins.map(({ relation }) => relation),
+ ]);
+ }
if (node.includeDeleted) {
if (!readsSoftDelete) {
throw new QueryCompileError(
@@ -126,6 +137,142 @@ function filterSelect(
};
}
+/**
+ * `select latest.<output>, … from (<node + flag>) as latest where
+ * latest.<flag> = 0`: the inner select keeps its ORDER BY / LIMIT, so a newer
+ * deleted version wins the pick and the outer filter then drops it.
+ */
+function filterAfterLimit(
+ node: ClickHouseSelectQueryNode,
+ relations: ReadonlyArray<SoftDeleteRelation | undefined>,
+): ClickHouseSelectQueryNode {
+ const { relation, outputs } = checkAfterLimitShape(node, relations);
+ return wrapAfterLimit(node, relation, outputs);
+}
+
+/**
+ * Throws on shapes the wrap would change; returns the soft-delete relation and
+ * the output names to wrap.
+ */
+function checkAfterLimitShape(
+ node: ClickHouseSelectQueryNode,
+ relations: ReadonlyArray<SoftDeleteRelation | undefined>,
+): { relation: SoftDeleteRelation; outputs: string[] } {
+ if (node.includeDeleted) {
+ throw new QueryCompileError(
+ "includeDeleted() and softDeleteAfterLimit() cannot mark the same select.",
+ );
+ }
+ // The soft-delete tables this select reads directly (FROM and JOINs).
+ const read = relations.filter((relation) => relation !== undefined);
+ if (read.length !== 1) {
+ throw new QueryCompileError(
+ `softDeleteAfterLimit() needs a select that reads exactly one soft-delete table; it reads ${read.length}.`,
+ );
+ }
+ const relation = read[0];
+ if (!node.limit && !node.limitBy) {
+ throw new QueryCompileError(
+ "softDeleteAfterLimit() needs a LIMIT or LIMIT BY to filter after.",
+ );
+ }
+ if (node.groupBy || node.setOperations) {
+ throw new QueryCompileError(
+ "softDeleteAfterLimit() cannot wrap a GROUP BY or set operation: the added flag would not match its shape.",
+ );
+ }
+ // The added flag would join the DISTINCT key and change which rows survive.
+ const distinct =
+ node.distinctOn ||
+ node.frontModifiers?.some(({ modifier }) => modifier === "Distinct");
+ if (distinct) {
+ throw new QueryCompileError(
+ "softDeleteAfterLimit() cannot wrap a DISTINCT: the added flag would change its key.",
+ );
+ }
+ // The outer filter would drop rows an outer join preserves; an ON filter keeps them.
+ const outerJoin = node.joins?.find(
+ ({ joinType }) => !AFTER_LIMIT_JOINS.has(joinType),
+ );
+ if (outerJoin) {
+ throw new QueryCompileError(
+ `softDeleteAfterLimit() cannot wrap a ${outerJoin.joinType}: the outer filter would drop its preserved rows.`,
+ );
+ }
+ // Names of the outputs the outer select lists again; `*` is rejected below.
+ const outputs = (node.selections ?? []).map((selection) => {
+ const name = outputName(selection.selection);
+ if (name === undefined) {
+ throw new QueryCompileError(
+ "softDeleteAfterLimit() needs every output named: select columns or alias expressions, not `*`.",
+ );
+ }
+ return name;
+ });
+ if (outputs.length === 0) {
+ throw new QueryCompileError(
+ "softDeleteAfterLimit() was applied to a select without outputs.",
+ );
+ }
+ if (outputs.includes(relation.flag)) {
+ throw new QueryCompileError(
+ `softDeleteAfterLimit() adds \`${relation.flag}\` itself; do not select it.`,
+ );
+ }
+ return { relation, outputs };
+}
+
+function wrapAfterLimit(
+ node: ClickHouseSelectQueryNode,
+ relation: SoftDeleteRelation,
+ outputs: ReadonlyArray<string>,
+): ClickHouseSelectQueryNode {
+ // Adds `<alias>.<flag> AS <flag>` as an output for the outer filter.
+ const inner = SelectQueryNode.cloneWithSelections(node, [
+ SelectionNode.create(
+ AliasNode.create(
+ qualifiedFlag(relation),
+ IdentifierNode.create(relation.flag),
+ ),
+ ),
+ ]);
+ // `select latest.<output>, … from (<inner>) as latest`, without the flag.
+ const outer = SelectQueryNode.cloneWithSelections(
+ SelectQueryNode.createFrom([
+ AliasNode.create(inner, IdentifierNode.create(AFTER_LIMIT_ALIAS)),
+ ]),
+ outputs.map((name) =>
+ SelectionNode.create(
+ ReferenceNode.create(
+ ColumnNode.create(name),
+ TableNode.create(AFTER_LIMIT_ALIAS),
+ ),
+ ),
+ ),
+ );
+ // `where latest.<flag> = 0` drops the pick if it is a deleted version.
+ return {
+ ...outer,
+ where: WhereNode.create(
+ notDeleted({
+ tableName: AFTER_LIMIT_ALIAS,
+ alias: AFTER_LIMIT_ALIAS,
+ flag: relation.flag,
+ }),
+ ),
+ };
+}
+
+/** Name ClickHouse gives a select output: `e.span_id` → `span_id`, `x as y` → `y`. */
+function outputName(node: OperationNode): string | undefined {
+ if (AliasNode.is(node)) {
+ return IdentifierNode.is(node.alias) ? node.alias.name : undefined;
+ }
+ if (ReferenceNode.is(node)) return outputName(node.column);
+ if (ColumnNode.is(node)) return node.column.name;
+ return undefined;
+}
+
function describeRelation(node: OperationNode): SoftDeleteRelation | undefined {
if (FinalTableNode.is(node)) {
return describeRelation(unwrapFinalTable(node));
@@ -147,20 +294,27 @@ function describeRelation(node: OperationNode): SoftDeleteRelation | undefined {
* Always qualified (by alias, else table name): a bare flag column would bind
* to a same-named SELECT alias, e.g. `0 AS is_deleted`.
*/
-function notDeleted(relation: SoftDeleteRelation): OperationNode {
+function notDeleted(relation: FlagColumn): OperationNode {
return BinaryOperationNode.create(
- ReferenceNode.create(
- ColumnNode.create(relation.flag),
- TableNode.create(relation.alias ?? relation.tableName),
- ),
+ qualifiedFlag(relation),
OperatorNode.create("="),
ValueNode.create(0),
);
}
+function qualifiedFlag(relation: FlagColumn): ReferenceNode {
+ return ReferenceNode.create(
+ ColumnNode.create(relation.flag),
+ TableNode.create(relation.alias ?? relation.tableName),
+ );
+}
+
type SoftDeleteRelation = {
tableName: string;
alias?: string;
flag: string;
node: TableNode;
};
+
+/** No `node`: {@link filterAfterLimit} filters a derived table, not a TableNode. */
+type FlagColumn = Pick<SoftDeleteRelation, "tableName" | "alias" | "flag">;
diff --git a/packages/shared/src/server/query-ast/transformer.ts b/packages/shared/src/server/query-ast/transformer.ts
index 2b745427cd..4508812d64 100644
--- a/packages/shared/src/server/query-ast/transformer.ts
+++ b/packages/shared/src/server/query-ast/transformer.ts
@@ -24,8 +24,8 @@ import {
*
* Kysely's default {@link OperationNodeTransformer.transformSelectQuery}
* reconstructs a SelectQueryNode from known slots and drops anything else —
- * so without this override, ARRAY JOIN / LIMIT BY / the `includeDeleted`
- * opt-out would vanish the moment any plugin walked the tree.
+ * so without this override, ARRAY JOIN / LIMIT BY / the soft-delete
+ * markers would vanish the moment any plugin walked the tree.
*/
export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer {
protected override transformNodeImpl<T extends OperationNode>(
@@ -77,6 +77,7 @@ export class ClickHouseOperationNodeTransformer extends OperationNodeTransformer
? { limitBy: this.transformLimitBy(extra.limitBy) }
: {}),
...(extra.includeDeleted ? { includeDeleted: true } : {}),
+ ...(extra.softDeleteAfterLimit ? { softDeleteAfterLimit: true } : {}),
} as ClickHouseSelectQueryNode;
}
diff --git a/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap
index 596cf1763b..55eb40634a 100644
--- a/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap
+++ b/packages/shared/src/server/repositories/__snapshots__/environments.golden.test.ts.snap
@@ -6,12 +6,13 @@ exports[`golden: environments.getEnvironmentsForProject > writeMode=events_only
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": "2026-01-01 00:00:00.000",
+ "param2": 0,
+ "param3": "2026-01-01 00:00:00.000",
},
"route": "environments.getEnvironmentsForProject",
"sql": "SELECT DISTINCT environment
FROM events_core
-WHERE (project_id = {param1:String}) AND (start_time >= {param2:DateTime64(6)})",
+WHERE (project_id = {param1:String}) AND (events_core.is_deleted = {param2:UInt8}) AND (start_time >= {param3:DateTime64(6)})",
},
{
"fn": "queryClickhouse",
@@ -39,11 +40,12 @@ exports[`golden: environments.getEnvironmentsForProject > writeMode=events_only
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
+ "param2": 0,
},
"route": "environments.getEnvironmentsForProject",
"sql": "SELECT DISTINCT environment
FROM events_core
-WHERE project_id = {param1:String}",
+WHERE (project_id = {param1:String}) AND (events_core.is_deleted = {param2:UInt8})",
},
{
"fn": "queryClickhouse",
diff --git a/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap
index 7cb7132074..fd15c54088 100644
--- a/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap
+++ b/packages/shared/src/server/repositories/__snapshots__/events.golden.test.ts.snap
@@ -12,17 +12,28 @@ exports[`golden: events IO payload reads family > getObservationFullIOForSession
"param5": "2025-12-31 23:59:59.000",
"param6": "2026-01-01 00:00:01.000",
"param7": 1,
+ "param8": 0,
},
"route": "unknown",
"sql": "SELECT
- e.span_id AS id,
- e.input AS input,
- e.output AS output,
- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.session_id = {param2:String}) AND (xxHash32(e.trace_id) = xxHash32({param3:String})) AND (e.trace_id = {param3:String}) AND (e.span_id = {param4:String}) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param7:Int64}",
+ latest.id,
+ latest.input,
+ latest.output,
+ latest.metadata
+FROM
+(
+ SELECT
+ e.span_id AS id,
+ e.input AS input,
+ e.output AS output,
+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.session_id = {param2:String}) AND (xxHash32(e.trace_id) = xxHash32({param3:String})) AND (e.trace_id = {param3:String}) AND (e.span_id = {param4:String}) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param7:Int64}
+) AS latest
+WHERE latest.is_deleted = {param8:UInt8}",
},
]
`;
@@ -38,13 +49,21 @@ exports[`golden: events IO payload reads family > getObservationIOFieldByteLengt
"param4": "2025-12-31 23:59:59.000",
"param5": "2026-01-01 00:00:01.000",
"param6": 1,
+ "param7": 0,
},
"route": "observation-io-stream-length",
- "sql": "SELECT length(e.input) AS len
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param6:Int64}",
+ "sql": "SELECT latest.len
+FROM
+(
+ SELECT
+ length(e.input) AS len,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param6:Int64}
+) AS latest
+WHERE latest.is_deleted = {param7:UInt8}",
},
]
`;
@@ -60,13 +79,21 @@ exports[`golden: events IO payload reads family > getObservationIOFieldByteLengt
"param4": "2025-12-31 23:59:59.000",
"param5": "2026-01-01 00:00:01.000",
"param6": 1,
+ "param7": 0,
},
"route": "observation-io-stream-length",
- "sql": "SELECT length(toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)))) AS len
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param6:Int64}",
+ "sql": "SELECT latest.len
+FROM
+(
+ SELECT
+ length(toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)))) AS len,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param6:Int64}
+) AS latest
+WHERE latest.is_deleted = {param7:UInt8}",
},
]
`;
@@ -82,13 +109,21 @@ exports[`golden: events IO payload reads family > getObservationIOFieldByteLengt
"param4": "2025-12-31 23:59:59.000",
"param5": "2026-01-01 00:00:01.000",
"param6": 1,
+ "param7": 0,
},
"route": "observation-io-stream-length",
- "sql": "SELECT length(e.output) AS len
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param6:Int64}",
+ "sql": "SELECT latest.len
+FROM
+(
+ SELECT
+ length(e.output) AS len,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param6:Int64}
+) AS latest
+WHERE latest.is_deleted = {param7:UInt8}",
},
]
`;
@@ -99,16 +134,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
"golden-span-c",
],
- "param3": [
+ "param4": [
"golden-trace",
"golden-trace-b",
],
- "param4": [
+ "param5": [
TupleParam {
"values": [
"golden-trace",
@@ -128,8 +164,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
],
},
],
- "param5": "2025-11-30 23:59:59.000",
- "param6": "2026-01-01 00:00:01.000",
+ "param6": "2025-11-30 23:59:59.000",
+ "param7": "2026-01-01 00:00:01.000",
},
"route": "unknown",
"sql": "SELECT
@@ -139,7 +175,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
e.output AS output,
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata
FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",
},
]
`;
@@ -150,16 +186,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
"golden-span-c",
],
- "param3": [
+ "param4": [
"golden-trace",
"golden-trace-b",
],
- "param4": [
+ "param5": [
TupleParam {
"values": [
"golden-trace",
@@ -179,8 +216,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
],
},
],
- "param5": "2025-11-30 23:59:59.000",
- "param6": "2026-01-01 00:00:01.000",
+ "param6": "2025-11-30 23:59:59.000",
+ "param7": "2026-01-01 00:00:01.000",
},
"route": "unknown",
"sql": "SELECT
@@ -195,7 +232,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
) AS experiment_item_metadata,
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata
FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",
},
]
`;
@@ -206,16 +243,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
"golden-span-c",
],
- "param3": [
+ "param4": [
"golden-trace",
"golden-trace-b",
],
- "param4": [
+ "param5": [
TupleParam {
"values": [
"golden-trace",
@@ -235,8 +273,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
],
},
],
- "param5": "2025-11-30 23:59:59.000",
- "param6": "2026-01-01 00:00:01.000",
+ "param6": "2025-11-30 23:59:59.000",
+ "param7": "2026-01-01 00:00:01.000",
},
"route": "unknown",
"sql": "SELECT
@@ -258,7 +296,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
arrayReverse(arrayMap(v -> leftUTF8(v, 5000), e.metadata_values))
) AS metadata
FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",
},
]
`;
@@ -269,16 +307,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
"golden-span-c",
],
- "param3": [
+ "param4": [
"golden-trace",
"golden-trace-b",
],
- "param4": [
+ "param5": [
TupleParam {
"values": [
"golden-trace",
@@ -298,8 +337,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
],
},
],
- "param5": "2025-11-30 23:59:59.000",
- "param6": "2026-01-01 00:00:01.000",
+ "param6": "2025-11-30 23:59:59.000",
+ "param7": "2026-01-01 00:00:01.000",
},
"route": "unknown",
"sql": "SELECT
@@ -309,7 +348,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
leftUTF8(e.output, 1000) AS output,
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",
},
]
`;
@@ -320,16 +359,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
"golden-span-c",
],
- "param3": [
+ "param4": [
"golden-trace",
"golden-trace-b",
],
- "param4": [
+ "param5": [
TupleParam {
"values": [
"golden-trace",
@@ -349,8 +389,8 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
],
},
],
- "param5": "2025-11-30 23:59:59.000",
- "param6": "2026-01-01 00:00:01.000",
+ "param6": "2025-11-30 23:59:59.000",
+ "param7": "2026-01-01 00:00:01.000",
},
"route": "unknown",
"sql": "SELECT
@@ -365,7 +405,7 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
) AS experiment_item_metadata,
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.start_time >= {param5:DateTime64(6)}) AND (e.start_time <= {param6:DateTime64(6)})",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",
},
]
`;
@@ -376,16 +416,17 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
"golden-span-c",
],
- "param3": [
+ "param4": [
"golden-trace",
"golden-trace-b",
],
- "param4": [
+ "param5": [
TupleParam {
"values": [
"golden-trace",
@@ -405,9 +446,9 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
],
},
],
- "param5": "golden-session",
- "param6": "2025-11-30 23:59:59.000",
- "param7": "2026-01-01 00:00:01.000",
+ "param6": "golden-session",
+ "param7": "2025-11-30 23:59:59.000",
+ "param8": "2026-01-01 00:00:01.000",
},
"route": "unknown",
"sql": "SELECT
@@ -417,13 +458,13 @@ exports[`golden: events IO payload reads family > getObservationsBatchIOFromEven
leftUTF8(e.output, 1000) AS output,
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)})) AND (e.trace_id IN ({param3:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param4:Array(Tuple(String, String))})) AND (e.trace_id IN (
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)})) AND (e.trace_id IN ({param4:Array(String)})) AND ((e.trace_id, e.span_id) IN ({param5:Array(Tuple(String, String))})) AND (e.trace_id IN (
SELECT trace_id
FROM events_core
- WHERE (project_id = {param1:String}) AND (trace_id IN ({param3:Array(String)}))
+ WHERE (project_id = {param1:String}) AND (events_core.is_deleted = {param2:UInt8}) AND (trace_id IN ({param4:Array(String)}))
GROUP BY trace_id
- HAVING argMaxIf(session_id, event_ts, session_id != '') = {param5:String}
-)) AND (e.start_time >= {param6:DateTime64(6)}) AND (e.start_time <= {param7:DateTime64(6)})",
+ HAVING argMaxIf(session_id, event_ts, session_id != '') = {param6:String}
+)) AND (e.start_time >= {param7:DateTime64(6)}) AND (e.start_time <= {param8:DateTime64(6)})",
},
]
`;
@@ -439,13 +480,21 @@ exports[`golden: events IO payload reads family > streamObservationIOFieldFromEv
"param4": "2025-12-31 23:59:59.000",
"param5": "2026-01-01 00:00:01.000",
"param6": 1,
+ "param7": 0,
},
"route": "observation-io-stream",
- "sql": "SELECT e.input AS field
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param6:Int64}
+ "sql": "SELECT latest.field
+FROM
+(
+ SELECT
+ e.input AS field,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param6:Int64}
+) AS latest
+WHERE latest.is_deleted = {param7:UInt8}
FORMAT RawBLOB",
},
]
@@ -462,13 +511,21 @@ exports[`golden: events IO payload reads family > streamObservationIOFieldFromEv
"param4": "2025-12-31 23:59:59.000",
"param5": "2026-01-01 00:00:01.000",
"param6": 1,
+ "param7": 0,
},
"route": "observation-io-stream",
- "sql": "SELECT toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))) AS field
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param6:Int64}
+ "sql": "SELECT latest.field
+FROM
+(
+ SELECT
+ toJSONString(mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))) AS field,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param6:Int64}
+) AS latest
+WHERE latest.is_deleted = {param7:UInt8}
FORMAT RawBLOB",
},
]
@@ -485,13 +542,21 @@ exports[`golden: events IO payload reads family > streamObservationIOFieldFromEv
"param4": "2025-12-31 23:59:59.000",
"param5": "2026-01-01 00:00:01.000",
"param6": 1,
+ "param7": 0,
},
"route": "observation-io-stream",
- "sql": "SELECT e.output AS field
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
-ORDER BY e.event_ts DESC
-LIMIT {param6:Int64}
+ "sql": "SELECT latest.field
+FROM
+(
+ SELECT
+ e.output AS field,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.span_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT {param6:Int64}
+) AS latest
+WHERE latest.is_deleted = {param7:UInt8}
FORMAT RawBLOB",
},
]
@@ -1662,9 +1727,10 @@ exports[`golden: events point-reads & existence family > getAgentGraphDataFromEv
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": "golden-trace",
- "param3": "2026-01-01 00:00:00.000",
- "param4": "2026-01-01 01:00:00.000",
+ "param2": 0,
+ "param3": "golden-trace",
+ "param4": "2026-01-01 00:00:00.000",
+ "param5": "2026-01-01 01:00:00.000",
},
"route": "unknown",
"sql": "SELECT
@@ -1677,7 +1743,7 @@ exports[`golden: events point-reads & existence family > getAgentGraphDataFromEv
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))['langgraph_node'] AS node,
mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))['langgraph_step'] AS step
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (e.trace_id = {param2:String}) AND (e.start_time >= {param3:DateTime64(6)}) AND (e.start_time <= {param4:DateTime64(6)})",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.trace_id = {param3:String}) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})",
},
]
`;
@@ -1733,63 +1799,112 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE
"param1": "golden-project",
"param2": "golden-span",
"param3": 1,
+ "param4": 0,
},
"route": "unknown",
"sql": "SELECT
- e.span_id AS id,
- e.trace_id AS trace_id,
- e.project_id AS project_id,
- e.environment AS environment,
- e.type AS type,
- e.parent_span_id AS parent_observation_id,
- e.name AS name,
- e.level AS level,
- e.status_message AS status_message,
- e.version AS version,
- e.bookmarked AS bookmarked,
- e.public AS public,
- e.user_id AS user_id,
- e.session_id AS session_id,
- ifNull(
- coalesce(
- nullIf(e.trace_name, ''),
- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
- ),
- ''
- ) AS trace_name,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.completion_start_time AS completion_start_time,
- e.created_at AS created_at,
- e.updated_at AS updated_at,
- e.event_ts,
- e.provided_model_name AS provided_model_name,
- e.model_id AS internal_model_id,
- e.model_parameters AS model_parameters,
- e.provided_usage_details AS provided_usage_details,
- e.usage_details AS usage_details,
- e.provided_cost_details AS provided_cost_details,
- e.cost_details AS cost_details,
- e.total_cost AS total_cost,
- e.prompt_id AS prompt_id,
- e.prompt_name AS prompt_name,
- e.prompt_version AS prompt_version,
- e.skills_available AS skills_available,
- e.skills_resource_loaded AS skills_resource_loaded,
- e.tool_definitions AS tool_definitions,
- e.tool_calls AS tool_calls,
- e.tool_call_names AS tool_call_names,
- e.usage_pricing_tier_id AS usage_pricing_tier_id,
- e.usage_pricing_tier_name AS usage_pricing_tier_name,
- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
- e.tags AS tags,
- e.release AS release
-FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})
-ORDER BY
- start_time DESC,
- event_ts DESC
-LIMIT {param3:Int64}",
+ latest.id,
+ latest.trace_id,
+ latest.project_id,
+ latest.environment,
+ latest.type,
+ latest.parent_observation_id,
+ latest.name,
+ latest.level,
+ latest.status_message,
+ latest.version,
+ latest.bookmarked,
+ latest.public,
+ latest.user_id,
+ latest.session_id,
+ latest.trace_name,
+ latest.start_time,
+ latest.end_time,
+ latest.completion_start_time,
+ latest.created_at,
+ latest.updated_at,
+ latest.event_ts,
+ latest.provided_model_name,
+ latest.internal_model_id,
+ latest.model_parameters,
+ latest.provided_usage_details,
+ latest.usage_details,
+ latest.provided_cost_details,
+ latest.cost_details,
+ latest.total_cost,
+ latest.prompt_id,
+ latest.prompt_name,
+ latest.prompt_version,
+ latest.skills_available,
+ latest.skills_resource_loaded,
+ latest.tool_definitions,
+ latest.tool_calls,
+ latest.tool_call_names,
+ latest.usage_pricing_tier_id,
+ latest.usage_pricing_tier_name,
+ latest.metadata,
+ latest.tags,
+ latest.release
+FROM
+(
+ SELECT
+ e.span_id AS id,
+ e.trace_id AS trace_id,
+ e.project_id AS project_id,
+ e.environment AS environment,
+ e.type AS type,
+ e.parent_span_id AS parent_observation_id,
+ e.name AS name,
+ e.level AS level,
+ e.status_message AS status_message,
+ e.version AS version,
+ e.bookmarked AS bookmarked,
+ e.public AS public,
+ e.user_id AS user_id,
+ e.session_id AS session_id,
+ ifNull(
+ coalesce(
+ nullIf(e.trace_name, ''),
+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
+ ),
+ ''
+ ) AS trace_name,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.completion_start_time AS completion_start_time,
+ e.created_at AS created_at,
+ e.updated_at AS updated_at,
+ e.event_ts,
+ e.provided_model_name AS provided_model_name,
+ e.model_id AS internal_model_id,
+ e.model_parameters AS model_parameters,
+ e.provided_usage_details AS provided_usage_details,
+ e.usage_details AS usage_details,
+ e.provided_cost_details AS provided_cost_details,
+ e.cost_details AS cost_details,
+ e.total_cost AS total_cost,
+ e.prompt_id AS prompt_id,
+ e.prompt_name AS prompt_name,
+ e.prompt_version AS prompt_version,
+ e.skills_available AS skills_available,
+ e.skills_resource_loaded AS skills_resource_loaded,
+ e.tool_definitions AS tool_definitions,
+ e.tool_calls AS tool_calls,
+ e.tool_call_names AS tool_call_names,
+ e.usage_pricing_tier_id AS usage_pricing_tier_id,
+ e.usage_pricing_tier_name AS usage_pricing_tier_name,
+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
+ e.tags AS tags,
+ e.release AS release,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})
+ ORDER BY
+ start_time DESC,
+ event_ts DESC
+ LIMIT {param3:Int64}
+) AS latest
+WHERE latest.is_deleted = {param4:UInt8}",
},
]
`;
@@ -1802,65 +1917,116 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE
"param1": "golden-project",
"param2": "golden-span",
"param3": 1,
+ "param4": 0,
},
"route": "unknown",
"sql": "SELECT
- e.span_id AS id,
- e.trace_id AS trace_id,
- e.project_id AS project_id,
- e.environment AS environment,
- e.type AS type,
- e.parent_span_id AS parent_observation_id,
- e.name AS name,
- e.level AS level,
- e.status_message AS status_message,
- e.version AS version,
- e.bookmarked AS bookmarked,
- e.public AS public,
- e.user_id AS user_id,
- e.session_id AS session_id,
- ifNull(
- coalesce(
- nullIf(e.trace_name, ''),
- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
- ),
- ''
- ) AS trace_name,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.completion_start_time AS completion_start_time,
- e.created_at AS created_at,
- e.updated_at AS updated_at,
- e.event_ts,
- e.provided_model_name AS provided_model_name,
- e.model_id AS internal_model_id,
- e.model_parameters AS model_parameters,
- e.provided_usage_details AS provided_usage_details,
- e.usage_details AS usage_details,
- e.provided_cost_details AS provided_cost_details,
- e.cost_details AS cost_details,
- e.total_cost AS total_cost,
- e.prompt_id AS prompt_id,
- e.prompt_name AS prompt_name,
- e.prompt_version AS prompt_version,
- e.skills_available AS skills_available,
- e.skills_resource_loaded AS skills_resource_loaded,
- e.tool_definitions AS tool_definitions,
- e.tool_calls AS tool_calls,
- e.tool_call_names AS tool_call_names,
- e.usage_pricing_tier_id AS usage_pricing_tier_id,
- e.usage_pricing_tier_name AS usage_pricing_tier_name,
- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
- e.tags AS tags,
- e.release AS release,
- input,
- output
-FROM events_full AS e
-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})
-ORDER BY
- start_time DESC,
- event_ts DESC
-LIMIT {param3:Int64}",
+ latest.id,
+ latest.trace_id,
+ latest.project_id,
+ latest.environment,
+ latest.type,
+ latest.parent_observation_id,
+ latest.name,
+ latest.level,
+ latest.status_message,
+ latest.version,
+ latest.bookmarked,
+ latest.public,
+ latest.user_id,
+ latest.session_id,
+ latest.trace_name,
+ latest.start_time,
+ latest.end_time,
+ latest.completion_start_time,
+ latest.created_at,
+ latest.updated_at,
+ latest.event_ts,
+ latest.provided_model_name,
+ latest.internal_model_id,
+ latest.model_parameters,
+ latest.provided_usage_details,
+ latest.usage_details,
+ latest.provided_cost_details,
+ latest.cost_details,
+ latest.total_cost,
+ latest.prompt_id,
+ latest.prompt_name,
+ latest.prompt_version,
+ latest.skills_available,
+ latest.skills_resource_loaded,
+ latest.tool_definitions,
+ latest.tool_calls,
+ latest.tool_call_names,
+ latest.usage_pricing_tier_id,
+ latest.usage_pricing_tier_name,
+ latest.metadata,
+ latest.tags,
+ latest.release,
+ latest.input,
+ latest.output
+FROM
+(
+ SELECT
+ e.span_id AS id,
+ e.trace_id AS trace_id,
+ e.project_id AS project_id,
+ e.environment AS environment,
+ e.type AS type,
+ e.parent_span_id AS parent_observation_id,
+ e.name AS name,
+ e.level AS level,
+ e.status_message AS status_message,
+ e.version AS version,
+ e.bookmarked AS bookmarked,
+ e.public AS public,
+ e.user_id AS user_id,
+ e.session_id AS session_id,
+ ifNull(
+ coalesce(
+ nullIf(e.trace_name, ''),
+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
+ ),
+ ''
+ ) AS trace_name,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.completion_start_time AS completion_start_time,
+ e.created_at AS created_at,
+ e.updated_at AS updated_at,
+ e.event_ts,
+ e.provided_model_name AS provided_model_name,
+ e.model_id AS internal_model_id,
+ e.model_parameters AS model_parameters,
+ e.provided_usage_details AS provided_usage_details,
+ e.usage_details AS usage_details,
+ e.provided_cost_details AS provided_cost_details,
+ e.cost_details AS cost_details,
+ e.total_cost AS total_cost,
+ e.prompt_id AS prompt_id,
+ e.prompt_name AS prompt_name,
+ e.prompt_version AS prompt_version,
+ e.skills_available AS skills_available,
+ e.skills_resource_loaded AS skills_resource_loaded,
+ e.tool_definitions AS tool_definitions,
+ e.tool_calls AS tool_calls,
+ e.tool_call_names AS tool_call_names,
+ e.usage_pricing_tier_id AS usage_pricing_tier_id,
+ e.usage_pricing_tier_name AS usage_pricing_tier_name,
+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
+ e.tags AS tags,
+ e.release AS release,
+ input,
+ output,
+ e.is_deleted AS is_deleted
+ FROM events_full AS e
+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})
+ ORDER BY
+ start_time DESC,
+ event_ts DESC
+ LIMIT {param3:Int64}
+) AS latest
+WHERE latest.is_deleted = {param4:UInt8}",
},
]
`;
@@ -1873,65 +2039,116 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE
"param1": "golden-project",
"param2": "golden-span",
"param3": 1,
+ "param4": 0,
},
"route": "unknown",
"sql": "SELECT
- e.span_id AS id,
- e.trace_id AS trace_id,
- e.project_id AS project_id,
- e.environment AS environment,
- e.type AS type,
- e.parent_span_id AS parent_observation_id,
- e.name AS name,
- e.level AS level,
- e.status_message AS status_message,
- e.version AS version,
- e.bookmarked AS bookmarked,
- e.public AS public,
- e.user_id AS user_id,
- e.session_id AS session_id,
- ifNull(
- coalesce(
- nullIf(e.trace_name, ''),
- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
- ),
- ''
- ) AS trace_name,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.completion_start_time AS completion_start_time,
- e.created_at AS created_at,
- e.updated_at AS updated_at,
- e.event_ts,
- e.provided_model_name AS provided_model_name,
- e.model_id AS internal_model_id,
- e.model_parameters AS model_parameters,
- e.provided_usage_details AS provided_usage_details,
- e.usage_details AS usage_details,
- e.provided_cost_details AS provided_cost_details,
- e.cost_details AS cost_details,
- e.total_cost AS total_cost,
- e.prompt_id AS prompt_id,
- e.prompt_name AS prompt_name,
- e.prompt_version AS prompt_version,
- e.skills_available AS skills_available,
- e.skills_resource_loaded AS skills_resource_loaded,
- e.tool_definitions AS tool_definitions,
- e.tool_calls AS tool_calls,
- e.tool_call_names AS tool_call_names,
- e.usage_pricing_tier_id AS usage_pricing_tier_id,
- e.usage_pricing_tier_name AS usage_pricing_tier_name,
- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
- e.tags AS tags,
- e.release AS release,
- leftUTF8(input, 1000) AS input,
- leftUTF8(output, 1000) AS output
-FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})
-ORDER BY
- start_time DESC,
- event_ts DESC
-LIMIT {param3:Int64}",
+ latest.id,
+ latest.trace_id,
+ latest.project_id,
+ latest.environment,
+ latest.type,
+ latest.parent_observation_id,
+ latest.name,
+ latest.level,
+ latest.status_message,
+ latest.version,
+ latest.bookmarked,
+ latest.public,
+ latest.user_id,
+ latest.session_id,
+ latest.trace_name,
+ latest.start_time,
+ latest.end_time,
+ latest.completion_start_time,
+ latest.created_at,
+ latest.updated_at,
+ latest.event_ts,
+ latest.provided_model_name,
+ latest.internal_model_id,
+ latest.model_parameters,
+ latest.provided_usage_details,
+ latest.usage_details,
+ latest.provided_cost_details,
+ latest.cost_details,
+ latest.total_cost,
+ latest.prompt_id,
+ latest.prompt_name,
+ latest.prompt_version,
+ latest.skills_available,
+ latest.skills_resource_loaded,
+ latest.tool_definitions,
+ latest.tool_calls,
+ latest.tool_call_names,
+ latest.usage_pricing_tier_id,
+ latest.usage_pricing_tier_name,
+ latest.metadata,
+ latest.tags,
+ latest.release,
+ latest.input,
+ latest.output
+FROM
+(
+ SELECT
+ e.span_id AS id,
+ e.trace_id AS trace_id,
+ e.project_id AS project_id,
+ e.environment AS environment,
+ e.type AS type,
+ e.parent_span_id AS parent_observation_id,
+ e.name AS name,
+ e.level AS level,
+ e.status_message AS status_message,
+ e.version AS version,
+ e.bookmarked AS bookmarked,
+ e.public AS public,
+ e.user_id AS user_id,
+ e.session_id AS session_id,
+ ifNull(
+ coalesce(
+ nullIf(e.trace_name, ''),
+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
+ ),
+ ''
+ ) AS trace_name,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.completion_start_time AS completion_start_time,
+ e.created_at AS created_at,
+ e.updated_at AS updated_at,
+ e.event_ts,
+ e.provided_model_name AS provided_model_name,
+ e.model_id AS internal_model_id,
+ e.model_parameters AS model_parameters,
+ e.provided_usage_details AS provided_usage_details,
+ e.usage_details AS usage_details,
+ e.provided_cost_details AS provided_cost_details,
+ e.cost_details AS cost_details,
+ e.total_cost AS total_cost,
+ e.prompt_id AS prompt_id,
+ e.prompt_name AS prompt_name,
+ e.prompt_version AS prompt_version,
+ e.skills_available AS skills_available,
+ e.skills_resource_loaded AS skills_resource_loaded,
+ e.tool_definitions AS tool_definitions,
+ e.tool_calls AS tool_calls,
+ e.tool_call_names AS tool_call_names,
+ e.usage_pricing_tier_id AS usage_pricing_tier_id,
+ e.usage_pricing_tier_name AS usage_pricing_tier_name,
+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
+ e.tags AS tags,
+ e.release AS release,
+ leftUTF8(input, 1000) AS input,
+ leftUTF8(output, 1000) AS output,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String})
+ ORDER BY
+ start_time DESC,
+ event_ts DESC
+ LIMIT {param3:Int64}
+) AS latest
+WHERE latest.is_deleted = {param4:UInt8}",
},
]
`;
@@ -1948,63 +2165,112 @@ exports[`golden: events point-reads & existence family > getObservationByIdFromE
"param5": "GENERATION",
"param6": "golden-trace",
"param7": 1,
+ "param8": 0,
},
"route": "unknown",
"sql": "SELECT
- e.span_id AS id,
- e.trace_id AS trace_id,
- e.project_id AS project_id,
- e.environment AS environment,
- e.type AS type,
- e.parent_span_id AS parent_observation_id,
- e.name AS name,
- e.level AS level,
- e.status_message AS status_message,
- e.version AS version,
- e.bookmarked AS bookmarked,
- e.public AS public,
- e.user_id AS user_id,
- e.session_id AS session_id,
- ifNull(
- coalesce(
- nullIf(e.trace_name, ''),
- if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
- ),
- ''
- ) AS trace_name,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.completion_start_time AS completion_start_time,
- e.created_at AS created_at,
- e.updated_at AS updated_at,
- e.event_ts,
- e.provided_model_name AS provided_model_name,
- e.model_id AS internal_model_id,
- e.model_parameters AS model_parameters,
- e.provided_usage_details AS provided_usage_details,
- e.usage_details AS usage_details,
- e.provided_cost_details AS provided_cost_details,
- e.cost_details AS cost_details,
- e.total_cost AS total_cost,
- e.prompt_id AS prompt_id,
- e.prompt_name AS prompt_name,
- e.prompt_version AS prompt_version,
- e.skills_available AS skills_available,
- e.skills_resource_loaded AS skills_resource_loaded,
- e.tool_definitions AS tool_definitions,
- e.tool_calls AS tool_calls,
- e.tool_call_names AS tool_call_names,
- e.usage_pricing_tier_id AS usage_pricing_tier_id,
- e.usage_pricing_tier_name AS usage_pricing_tier_name,
- mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
- e.tags AS tags,
- e.release AS release
-FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String}) AND (toStartOfMinute(start_time) = toStartOfMinute({param3:DateTime64(6)})) AND (start_time >= {param4:DateTime64(6)}) AND (type = {param5:String}) AND (trace_id = {param6:String})
-ORDER BY
- start_time DESC,
- event_ts DESC
-LIMIT {param7:Int64}",
+ latest.id,
+ latest.trace_id,
+ latest.project_id,
+ latest.environment,
+ latest.type,
+ latest.parent_observation_id,
+ latest.name,
+ latest.level,
+ latest.status_message,
+ latest.version,
+ latest.bookmarked,
+ latest.public,
+ latest.user_id,
+ latest.session_id,
+ latest.trace_name,
+ latest.start_time,
+ latest.end_time,
+ latest.completion_start_time,
+ latest.created_at,
+ latest.updated_at,
+ latest.event_ts,
+ latest.provided_model_name,
+ latest.internal_model_id,
+ latest.model_parameters,
+ latest.provided_usage_details,
+ latest.usage_details,
+ latest.provided_cost_details,
+ latest.cost_details,
+ latest.total_cost,
+ latest.prompt_id,
+ latest.prompt_name,
+ latest.prompt_version,
+ latest.skills_available,
+ latest.skills_resource_loaded,
+ latest.tool_definitions,
+ latest.tool_calls,
+ latest.tool_call_names,
+ latest.usage_pricing_tier_id,
+ latest.usage_pricing_tier_name,
+ latest.metadata,
+ latest.tags,
+ latest.release
+FROM
+(
+ SELECT
+ e.span_id AS id,
+ e.trace_id AS trace_id,
+ e.project_id AS project_id,
+ e.environment AS environment,
+ e.type AS type,
+ e.parent_span_id AS parent_observation_id,
+ e.name AS name,
+ e.level AS level,
+ e.status_message AS status_message,
+ e.version AS version,
+ e.bookmarked AS bookmarked,
+ e.public AS public,
+ e.user_id AS user_id,
+ e.session_id AS session_id,
+ ifNull(
+ coalesce(
+ nullIf(e.trace_name, ''),
+ if((e.parent_span_id = '') OR (e.is_app_root = true), nullIf(e.name, ''), NULL)
+ ),
+ ''
+ ) AS trace_name,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.completion_start_time AS completion_start_time,
+ e.created_at AS created_at,
+ e.updated_at AS updated_at,
+ e.event_ts,
+ e.provided_model_name AS provided_model_name,
+ e.model_id AS internal_model_id,
+ e.model_parameters AS model_parameters,
+ e.provided_usage_details AS provided_usage_details,
+ e.usage_details AS usage_details,
+ e.provided_cost_details AS provided_cost_details,
+ e.cost_details AS cost_details,
+ e.total_cost AS total_cost,
+ e.prompt_id AS prompt_id,
+ e.prompt_name AS prompt_name,
+ e.prompt_version AS prompt_version,
+ e.skills_available AS skills_available,
+ e.skills_resource_loaded AS skills_resource_loaded,
+ e.tool_definitions AS tool_definitions,
+ e.tool_calls AS tool_calls,
+ e.tool_call_names AS tool_call_names,
+ e.usage_pricing_tier_id AS usage_pricing_tier_id,
+ e.usage_pricing_tier_name AS usage_pricing_tier_name,
+ mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values)) AS metadata,
+ e.tags AS tags,
+ e.release AS release,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (span_id = {param2:String}) AND (toStartOfMinute(start_time) = toStartOfMinute({param3:DateTime64(6)})) AND (start_time >= {param4:DateTime64(6)}) AND (type = {param5:String}) AND (trace_id = {param6:String})
+ ORDER BY
+ start_time DESC,
+ event_ts DESC
+ LIMIT {param7:Int64}
+) AS latest
+WHERE latest.is_deleted = {param8:UInt8}",
},
]
`;
@@ -2015,7 +2281,8 @@ exports[`golden: events point-reads & existence family > getObservationsTraceIds
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-span",
"golden-span-b",
],
@@ -2025,7 +2292,7 @@ exports[`golden: events point-reads & existence family > getObservationsTraceIds
e.trace_id AS trace_id,
e.span_id AS span_id
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (e.span_id IN ({param2:Array(String)}))",
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.span_id IN ({param3:Array(String)}))",
},
]
`;
@@ -2036,10 +2303,11 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-trace",
],
- "param3": 1,
+ "param4": 1,
},
"route": "unknown",
"sql": "WITH traces AS
@@ -2102,7 +2370,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
argMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,
any(experiment_id) AS experiment_id
FROM events_full AS e
- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)}))
+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)}))
GROUP BY
trace_id,
project_id
@@ -2129,7 +2397,7 @@ SELECT
t.output
FROM traces AS t
ORDER BY t.timestamp DESC
-LIMIT {param3:Int64}",
+LIMIT {param4:Int64}",
},
]
`;
@@ -2140,10 +2408,11 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-trace",
],
- "param3": 1,
+ "param4": 1,
},
"route": "unknown",
"sql": "WITH traces AS
@@ -2206,7 +2475,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
argMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,
any(experiment_id) AS experiment_id
FROM events_full AS e
- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)}))
+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)}))
GROUP BY
trace_id,
project_id
@@ -2233,7 +2502,7 @@ SELECT
'' AS output
FROM traces AS t
ORDER BY t.timestamp DESC
-LIMIT {param3:Int64}",
+LIMIT {param4:Int64}",
},
]
`;
@@ -2244,12 +2513,13 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-trace",
],
- "param3": "2025-11-29 00:00:00.000",
- "param4": "2026-01-01 00:00:00.000",
- "param5": 1,
+ "param4": "2025-11-29 00:00:00.000",
+ "param5": "2026-01-01 00:00:00.000",
+ "param6": 1,
},
"route": "unknown",
"sql": "WITH traces AS
@@ -2312,7 +2582,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
argMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,
any(experiment_id) AS experiment_id
FROM events_full AS e
- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)})) AND (start_time >= {param3:DateTime64(6)})
+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)})) AND (start_time >= {param4:DateTime64(6)})
GROUP BY
trace_id,
project_id
@@ -2338,9 +2608,9 @@ SELECT
t.input,
t.output
FROM traces AS t
-WHERE toDate(t.timestamp) = toDate({param4:DateTime64(6)})
+WHERE toDate(t.timestamp) = toDate({param5:DateTime64(6)})
ORDER BY t.timestamp DESC
-LIMIT {param5:Int64}",
+LIMIT {param6:Int64}",
},
]
`;
@@ -2351,10 +2621,11 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-trace",
],
- "param3": 1,
+ "param4": 1,
},
"route": "unknown",
"sql": "WITH traces AS
@@ -2417,7 +2688,7 @@ exports[`golden: events point-reads & existence family > getTraceByIdFromEventsT
argMaxIf(evaluation_rule_id, event_ts, evaluation_rule_id != '') AS evaluation_rule_id,
any(experiment_id) AS experiment_id
FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (trace_id IN ({param2:Array(String)}))
+ WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (trace_id IN ({param3:Array(String)}))
GROUP BY
trace_id,
project_id
@@ -2444,7 +2715,7 @@ SELECT
leftUTF8(t.output, 1000) AS output
FROM traces AS t
ORDER BY t.timestamp DESC
-LIMIT {param3:Int64}",
+LIMIT {param4:Int64}",
},
]
`;
@@ -2577,30 +2848,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp
"param1": "golden-project",
"param2": "GENERATION",
"param3": [],
- "param4": "input",
- "param5": "output",
- "param6": 0,
+ "param4": 0,
+ "param5": "input",
+ "param6": "output",
},
"route": "unknown",
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.prompt_version AS prompt_version,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.usage_details AS usage_details,
- e.cost_details AS cost_details,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.prompt_version,
+ latest.trace_id,
+ latest.span_id,
+ latest.start_time,
+ latest.end_time,
+ latest.usage_details,
+ latest.cost_details
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.prompt_version AS prompt_version,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.usage_details AS usage_details,
+ e.cost_details AS cost_details,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param4:UInt8}
)
SELECT
count() AS count,
@@ -2609,15 +2894,14 @@ SELECT
min(e.start_time) AS first_observation,
max(e.start_time) AS last_observation,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param4:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))
) AS median_input_usage,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
) AS median_output_usage,
medianExact(e.cost_details['total']) AS median_total_cost,
medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_ms
FROM prompt_events AS e
-WHERE e.is_deleted = {param6:UInt8}
GROUP BY
e.prompt_id,
e.prompt_version
@@ -2638,30 +2922,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp
"golden-prompt-id-b",
],
"param4": "2025-12-01 00:00:00.000",
- "param5": "input",
- "param6": "output",
- "param7": 0,
+ "param5": 0,
+ "param6": "input",
+ "param7": "output",
},
"route": "unknown",
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.prompt_version AS prompt_version,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.usage_details AS usage_details,
- e.cost_details AS cost_details,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.prompt_version,
+ latest.trace_id,
+ latest.span_id,
+ latest.start_time,
+ latest.end_time,
+ latest.usage_details,
+ latest.cost_details
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.prompt_version AS prompt_version,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.usage_details AS usage_details,
+ e.cost_details AS cost_details,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param5:UInt8}
)
SELECT
count() AS count,
@@ -2670,15 +2968,14 @@ SELECT
min(e.start_time) AS first_observation,
max(e.start_time) AS last_observation,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
) AS median_input_usage,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))
) AS median_output_usage,
medianExact(e.cost_details['total']) AS median_total_cost,
medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_ms
FROM prompt_events AS e
-WHERE e.is_deleted = {param7:UInt8}
GROUP BY
e.prompt_id,
e.prompt_version
@@ -2700,30 +2997,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp
],
"param4": "2025-12-01 00:00:00.000",
"param5": "2026-01-02 00:00:00.000",
- "param6": "input",
- "param7": "output",
- "param8": 0,
+ "param6": 0,
+ "param7": "input",
+ "param8": "output",
},
"route": "unknown",
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.prompt_version AS prompt_version,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.usage_details AS usage_details,
- e.cost_details AS cost_details,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.prompt_version,
+ latest.trace_id,
+ latest.span_id,
+ latest.start_time,
+ latest.end_time,
+ latest.usage_details,
+ latest.cost_details
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.prompt_version AS prompt_version,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.usage_details AS usage_details,
+ e.cost_details AS cost_details,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param6:UInt8}
)
SELECT
count() AS count,
@@ -2732,15 +3043,14 @@ SELECT
min(e.start_time) AS first_observation,
max(e.start_time) AS last_observation,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))
) AS median_input_usage,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param8:String}) > 0), e.usage_details)))
) AS median_output_usage,
medianExact(e.cost_details['total']) AS median_total_cost,
medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_ms
FROM prompt_events AS e
-WHERE e.is_deleted = {param8:UInt8}
GROUP BY
e.prompt_id,
e.prompt_version
@@ -2761,30 +3071,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp
"golden-prompt-id-b",
],
"param4": "2026-01-02 00:00:00.000",
- "param5": "input",
- "param6": "output",
- "param7": 0,
+ "param5": 0,
+ "param6": "input",
+ "param7": "output",
},
"route": "unknown",
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.prompt_version AS prompt_version,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.usage_details AS usage_details,
- e.cost_details AS cost_details,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time <= {param4:DateTime64(6)})
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.prompt_version,
+ latest.trace_id,
+ latest.span_id,
+ latest.start_time,
+ latest.end_time,
+ latest.usage_details,
+ latest.cost_details
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.prompt_version AS prompt_version,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.usage_details AS usage_details,
+ e.cost_details AS cost_details,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time <= {param4:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param5:UInt8}
)
SELECT
count() AS count,
@@ -2793,15 +3117,14 @@ SELECT
min(e.start_time) AS first_observation,
max(e.start_time) AS last_observation,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
) AS median_input_usage,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param7:String}) > 0), e.usage_details)))
) AS median_output_usage,
medianExact(e.cost_details['total']) AS median_total_cost,
medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_ms
FROM prompt_events AS e
-WHERE e.is_deleted = {param7:UInt8}
GROUP BY
e.prompt_id,
e.prompt_version
@@ -2821,30 +3144,44 @@ exports[`golden: events simple aggregates family > getObservationMetricsForPromp
"golden-prompt-id",
"golden-prompt-id-b",
],
- "param4": "input",
- "param5": "output",
- "param6": 0,
+ "param4": 0,
+ "param5": "input",
+ "param6": "output",
},
"route": "unknown",
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.prompt_version AS prompt_version,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.start_time AS start_time,
- e.end_time AS end_time,
- e.usage_details AS usage_details,
- e.cost_details AS cost_details,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.prompt_version,
+ latest.trace_id,
+ latest.span_id,
+ latest.start_time,
+ latest.end_time,
+ latest.usage_details,
+ latest.cost_details
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.prompt_version AS prompt_version,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.start_time AS start_time,
+ e.end_time AS end_time,
+ e.usage_details AS usage_details,
+ e.cost_details AS cost_details,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param4:UInt8}
)
SELECT
count() AS count,
@@ -2853,15 +3190,14 @@ SELECT
min(e.start_time) AS first_observation,
max(e.start_time) AS last_observation,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param4:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))
) AS median_input_usage,
medianExact(
- arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param5:String}) > 0), e.usage_details)))
+ arraySum(mapValues(mapFilter(x -> (positionCaseInsensitive(x.1, {param6:String}) > 0), e.usage_details)))
) AS median_output_usage,
medianExact(e.cost_details['total']) AS median_total_cost,
medianExact(dateDiff('millisecond', e.start_time, e.end_time)) AS median_latency_ms
FROM prompt_events AS e
-WHERE e.is_deleted = {param6:UInt8}
GROUP BY
e.prompt_id,
e.prompt_version
@@ -2999,12 +3335,13 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents f
"param1": "input",
"param2": "output",
"param3": "golden-project",
- "param4": [
+ "param4": 0,
+ "param5": [
"golden-session",
"golden-session-b",
],
- "param5": "",
- "param6": 2,
+ "param6": "",
+ "param7": 2,
},
"route": "unknown",
"sql": "SELECT
@@ -3035,9 +3372,9 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents f
) AS session_output_usage,
sumMap(usage_details)['total'] AS session_total_usage
FROM events_core AS e
-WHERE (e.project_id = {param3:String}) AND (session_id IN ({param4:Array(String)})) AND (session_id != {param5:String})
+WHERE (e.project_id = {param3:String}) AND (e.is_deleted = {param4:UInt8}) AND (session_id IN ({param5:Array(String)})) AND (session_id != {param6:String})
GROUP BY session_id
-LIMIT {param6:Int64}",
+LIMIT {param7:Int64}",
},
]
`;
@@ -3050,21 +3387,22 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents s
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-session",
"golden-session-b",
],
- "param3": "",
- "param4": 2,
+ "param4": "",
+ "param5": 2,
},
"route": "unknown",
"sql": "SELECT
session_id,
uniq(trace_id) AS trace_count
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (session_id IN ({param2:Array(String)})) AND (session_id != {param3:String})
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (session_id IN ({param3:Array(String)})) AND (session_id != {param4:String})
GROUP BY session_id
-LIMIT {param4:Int64}",
+LIMIT {param5:Int64}",
},
]
`;
@@ -3075,22 +3413,23 @@ exports[`golden: events simple aggregates family > getSessionMetricsFromEvents s
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-session",
"golden-session-b",
],
- "param3": "2025-11-29 00:00:00.000",
- "param4": "",
- "param5": 2,
+ "param4": "2025-11-29 00:00:00.000",
+ "param5": "",
+ "param6": 2,
},
"route": "unknown",
"sql": "SELECT
session_id,
uniq(trace_id) AS trace_count
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (session_id IN ({param2:Array(String)})) AND (start_time >= {param3:DateTime64(6)}) AND (session_id != {param4:String})
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (session_id IN ({param3:Array(String)})) AND (start_time >= {param4:DateTime64(6)}) AND (session_id != {param5:String})
GROUP BY session_id
-LIMIT {param5:Int64}",
+LIMIT {param6:Int64}",
},
]
`;
diff --git a/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap
index adc28c90e8..95614e2e7c 100644
--- a/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap
+++ b/packages/shared/src/server/repositories/__snapshots__/experiments.golden.test.ts.snap
@@ -8,11 +8,12 @@ exports[`golden: experiments simple aggregates family > getExperimentMetricsFrom
"fn": "queryClickhouse",
"params": {
"param1": "golden-project",
- "param2": [
+ "param2": 0,
+ "param3": [
"golden-experiment",
"golden-experiment-b",
],
- "param3": "",
+ "param4": "",
},
"route": "unknown",
"sql": "SELECT
@@ -23,7 +24,7 @@ exports[`golden: experiments simple aggregates family > getExperimentMetricsFrom
(e.span_id = e.experiment_item_root_span_id) AND (e.end_time IS NOT NULL)
) AS latency_avg
FROM events_core AS e
-WHERE (e.project_id = {param1:String}) AND (e.experiment_id IN ({param2:Array(String)})) AND (e.experiment_id != {param3:String})
+WHERE (e.project_id = {param1:String}) AND (e.is_deleted = {param2:UInt8}) AND (e.experiment_id IN ({param3:Array(String)})) AND (e.experiment_id != {param4:String})
GROUP BY
e.project_id,
e.experiment_id",
diff --git a/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap
index 57f58b5b84..0f1739ae82 100644
--- a/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap
+++ b/packages/shared/src/server/repositories/__snapshots__/scores.golden.test.ts.snap
@@ -23,17 +23,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=obser
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.trace_id,
+ latest.span_id
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)}) AND (e.start_time <= {param5:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param6:UInt8}
)
SELECT
e.prompt_id AS prompt_id,
@@ -45,16 +54,15 @@ SELECT
s.data_type AS data_type,
s.comment AS comment,
s.timestamp AS timestamp,
- length(mapKeys(s.metadata)) > {param6:Int64} AS has_metadata
+ length(mapKeys(s.metadata)) > {param7:Int64} AS has_metadata
FROM scores AS s
FINAL
INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)
-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param7:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param8:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (
+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param8:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (
SELECT
trace_id,
span_id
FROM prompt_events
- WHERE is_deleted = {param7:UInt8}
))
GROUP BY
e.prompt_id,
@@ -91,17 +99,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=obser
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.trace_id,
+ latest.span_id
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param4:UInt8}
)
SELECT
e.prompt_id AS prompt_id,
@@ -113,16 +130,15 @@ SELECT
s.data_type AS data_type,
s.comment AS comment,
s.timestamp AS timestamp,
- length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata
+ length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadata
FROM scores AS s
FINAL
INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id) AND (s.observation_id = e.span_id)
-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (
+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NOT NULL) AND ((s.trace_id, s.observation_id) IN (
SELECT
trace_id,
span_id
FROM prompt_events
- WHERE is_deleted = {param5:UInt8}
))
GROUP BY
e.prompt_id,
@@ -160,17 +176,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.trace_id,
+ latest.span_id
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)})) AND (e.start_time >= {param4:DateTime64(6)})
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param5:UInt8}
)
SELECT
e.prompt_id AS prompt_id,
@@ -182,14 +207,13 @@ SELECT
s.data_type AS data_type,
s.comment AS comment,
s.timestamp AS timestamp,
- length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadata
+ length(mapKeys(s.metadata)) > {param6:Int64} AS has_metadata
FROM scores AS s
FINAL
INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)
-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param6:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param7:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (
+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param7:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (
SELECT trace_id
FROM prompt_events
- WHERE is_deleted = {param6:UInt8}
))
GROUP BY
e.prompt_id,
@@ -226,17 +250,26 @@ exports[`golden: scores.getAggregatedScoresForPromptsFromEvents > relation=trace
"sql": "WITH prompt_events AS
(
SELECT
- e.project_id AS project_id,
- e.prompt_id AS prompt_id,
- e.trace_id AS trace_id,
- e.span_id AS span_id,
- e.is_deleted AS is_deleted
- FROM events_core AS e
- WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
- ORDER BY e.event_ts DESC
- LIMIT 1 BY
- e.span_id,
- e.project_id
+ latest.project_id,
+ latest.prompt_id,
+ latest.trace_id,
+ latest.span_id
+ FROM
+ (
+ SELECT
+ e.project_id AS project_id,
+ e.prompt_id AS prompt_id,
+ e.trace_id AS trace_id,
+ e.span_id AS span_id,
+ e.is_deleted AS is_deleted
+ FROM events_core AS e
+ WHERE (e.project_id = {param1:String}) AND (e.type = {param2:String}) AND (e.prompt_id IN ({param3:Array(String)}))
+ ORDER BY e.event_ts DESC
+ LIMIT 1 BY
+ e.span_id,
+ e.project_id
+ ) AS latest
+ WHERE latest.is_deleted = {param4:UInt8}
)
SELECT
e.prompt_id AS prompt_id,
@@ -248,14 +281,13 @@ SELECT
s.data_type AS data_type,
s.comment AS comment,
s.timestamp AS timestamp,
- length(mapKeys(s.metadata)) > {param4:Int64} AS has_metadata
+ length(mapKeys(s.metadata)) > {param5:Int64} AS has_metadata
FROM scores AS s
FINAL
INNER JOIN prompt_events AS e ON (s.project_id = e.project_id) AND (s.trace_id = e.trace_id)
-WHERE (s.project_id = {param1:String}) AND (e.is_deleted = {param5:UInt8}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (
+WHERE (s.project_id = {param1:String}) AND (s.name IS NOT NULL) AND (s.data_type IN ({param6:Array(String)})) AND (s.observation_id IS NULL) AND (s.trace_id IN (
SELECT trace_id
FROM prompt_events
- WHERE is_deleted = {param5:UInt8}
))
GROUP BY
e.prompt_id,
@@ -276,12 +308,13 @@ exports[`golden: scores.getScoresForExperimentItems > experimentIds set 1`] = `
"params": {
"param1": 0,
"param2": "golden-project",
- "param3": [
+ "param3": 0,
+ "param4": [
"exp-a",
"exp-b",
],
- "param4": "",
- "param5": [
+ "param5": "",
+ "param6": [
"NUMERIC",
"BOOLEAN",
"CATEGORICAL",
@@ -319,11 +352,11 @@ FROM
e.experiment_id,
e.trace_id
FROM events_core AS e
- WHERE (e.project_id = {param2:String}) AND (e.experiment_id IN ({param3:Array(String)})) AND (e.experiment_id != {param4:String})
+ WHERE (e.project_id = {param2:String}) AND (e.is_deleted = {param3:UInt8}) AND (e.experiment_id IN ({param4:Array(String)})) AND (e.experiment_id != {param5:String})
) AS e
INNER JOIN scores AS s
FINAL ON (e.trace_id = s.trace_id) AND (e.project_id = s.project_id) AND (s.project_id = {param2:String})
-WHERE s.data_type IN ({param5:Array(String)})
+WHERE s.data_type IN ({param6:Array(String)})
ORDER BY s.event_ts DESC
LIMIT 1 BY
s.id,
diff --git a/packages/shared/src/server/repositories/environments.ts b/packages/shared/src/server/repositories/environments.ts
index 26728d073f..cea6d96b04 100644
--- a/packages/shared/src/server/repositories/environments.ts
+++ b/packages/shared/src/server/repositories/environments.ts
@@ -2,7 +2,6 @@ import { LISTABLE_SCORE_TYPES } from "../../domain/scores";
import { env } from "../../env";
import { compileClickhouseQuery } from "../query-ast/compile";
import { getClickhouseKysely } from "../query-ast/dialect";
-import { includeDeleted } from "../query-ast/extensions";
import { queryClickhouse } from "./clickhouse";
export type EnvironmentFilterProps = {
@@ -43,7 +42,6 @@ export const getEnvironmentsForProject = async (
)
: db
.selectFrom("events_core")
- .$call(includeDeleted())
.select("environment")
.distinct()
.$if(fromTimestamp != null, (qb) =>
diff --git a/packages/shared/src/server/repositories/events.ts b/packages/shared/src/server/repositories/events.ts
index 70f40c42b9..73b8204f03 100644
--- a/packages/shared/src/server/repositories/events.ts
+++ b/packages/shared/src/server/repositories/events.ts
@@ -37,7 +37,7 @@ import {
byteLength,
count,
dateDiff,
- includeDeleted,
+ softDeleteAfterLimit,
leftUTF8,
limitBy,
literal,
@@ -669,7 +669,6 @@ export const getObservationMetricsForPromptsFromEvents = async (
"e.end_time as end_time",
"e.usage_details as usage_details",
"e.cost_details as cost_details",
- "e.is_deleted as is_deleted",
],
{ fromTimestamp, toTimestamp },
),
@@ -694,7 +693,6 @@ export const getObservationMetricsForPromptsFromEvents = async (
dateDiff("millisecond", eb.ref("e.start_time"), eb.ref("e.end_time")),
).as("median_latency_ms"),
])
- .where("e.is_deleted", "=", 0)
.groupBy(["e.prompt_id", "e.prompt_version"])
.orderBy("e.prompt_version", "desc");
const { sql: query, params } = compileClickhouseQuery(builder, projectId);
@@ -1104,12 +1102,13 @@ async function getObservationByIdFromEventsTableInternal({
const needsFullTable =
fetchWithInputOutput && renderingProps.truncated !== true;
+ // A soft-deleted latest version reads as not found, not as an older version.
const builder = (
needsFullTable
? db.selectFrom("events_full as e")
: db.selectFrom("events_core as e")
)
- .$call(includeDeleted())
+ .$call(softDeleteAfterLimit())
.select(OBSERVATION_BY_ID_SELECTS)
.$if(fetchWithInputOutput, (qb) =>
qb.select(observationIoSelects(renderingProps.truncated)),
@@ -1197,7 +1196,6 @@ export const getTraceByIdFromEventsTable = async ({
? qb.selectFrom("events_core as e")
: qb.selectFrom("events_full as e");
return from
- .$call(includeDeleted())
.select(TRACE_AGGREGATION_SELECTS)
.where("trace_id", "in", [traceId])
.$if(fromTimestamp != null, (inner) =>
@@ -2641,7 +2639,6 @@ export async function getAgentGraphDataFromEventsTable(params: {
const builder = db
.selectFrom("events_core as e")
- .$call(includeDeleted())
.select((eb) => {
const metadata = metadataMap(
eb.ref("e.metadata_names"),
@@ -2814,7 +2811,6 @@ export const getObservationsBatchIOFromEventsTable = async <
// for exact trace/observation matching.
const builder = getClickhouseKysely()
.selectFrom(truncated ? "events_core as e" : "events_full as e")
- .$call(includeDeleted())
.select((eb) => [
"e.span_id as id",
"e.trace_id as trace_id",
@@ -2861,7 +2857,6 @@ export const getObservationsBatchIOFromEventsTable = async <
qb.where("e.trace_id", "in", (eb) =>
eb
.selectFrom("events_core")
- .$call(includeDeleted())
.select("trace_id")
.where("trace_id", "in", traceIds)
.groupBy("trace_id")
@@ -2962,16 +2957,15 @@ interface ObservationIOReadOpts {
// The field expression is chosen from the validated `field` enum — never
// string-interpolated user input. The length pre-query and the byte stream share
// this builder so they can never drift; a mismatch would make the route's
-// Content-Length integrity check wrong.
+// Content-Length integrity check wrong. A soft-deleted latest version reads as
+// no row.
function compileObservationIOFieldRead<A extends string>(
opts: ObservationIOReadOpts,
select: (field: AliasableExpression<string>) => AliasedExpression<unknown, A>,
) {
- const db = getClickhouseKysely();
-
- const query = db
+ const query = getClickhouseKysely()
.selectFrom("events_full as e")
- .$call(includeDeleted())
+ .$call(softDeleteAfterLimit())
.select((eb) =>
select(
opts.field === "metadata"
@@ -3088,9 +3082,10 @@ export const getObservationFullIOForSessionFromEventsTable = async (opts: {
const maxTimestamp = new Date(opts.startTime.getTime() + 1000);
const db = getClickhouseKysely();
+ // A soft-deleted latest version reads as not found.
const builder = db
.selectFrom("events_full as e")
- .$call(includeDeleted())
+ .$call(softDeleteAfterLimit())
.select((eb) => [
"e.span_id as id",
"e.input as input",
@@ -3160,7 +3155,6 @@ export const getObservationsTraceIdsFromEventsTable = async (opts: {
const builder = db
.selectFrom("events_core as e")
- .$call(includeDeleted())
.select(["e.trace_id as trace_id", "e.span_id as span_id"])
.where("e.span_id", "in", observationIds);
@@ -3906,7 +3900,6 @@ export const getSessionMetricsFromEvents = async <
const { projectId, sessionIds, fields, queryFromTimestamp } = props;
const builder = getClickhouseKysely()
.selectFrom("events_core as e")
- .$call(includeDeleted())
.select(fields.map((field) => SESSION_AGGREGATION_SELECTS[field]))
.where("session_id", "in", sessionIds)
.$if(queryFromTimestamp != null, (qb) =>
diff --git a/packages/shared/src/server/repositories/experiments.ts b/packages/shared/src/server/repositories/experiments.ts
index cc9f35aacb..839f7df8d2 100644
--- a/packages/shared/src/server/repositories/experiments.ts
+++ b/packages/shared/src/server/repositories/experiments.ts
@@ -8,7 +8,7 @@ import { type FilterState } from "../../types";
import { convertDateToClickhouseDateTime } from "../clickhouse/client";
import { compileClickhouseQuery } from "../query-ast/compile";
import { getClickhouseKysely } from "../query-ast/dialect";
-import { avgIf, dateDiff, includeDeleted } from "../query-ast/extensions";
+import { avgIf, dateDiff } from "../query-ast/extensions";
import {
FilterList,
StringOptionsFilter,
@@ -360,7 +360,6 @@ export const getExperimentMetricsFromEvents = async (props: {
const { projectId, experimentIds } = props;
const builder = getClickhouseKysely()
.selectFrom("events_core as e")
- .$call(includeDeleted())
.select((eb) => [
"e.experiment_id as experiment_id",
eb.fn.sum("e.total_cost").as("total_cost"),
diff --git a/packages/shared/src/server/repositories/prompt-events.ts b/packages/shared/src/server/repositories/prompt-events.ts
index 01a64c499e..728b903ba0 100644
--- a/packages/shared/src/server/repositories/prompt-events.ts
+++ b/packages/shared/src/server/repositories/prompt-events.ts
@@ -1,7 +1,7 @@
import type { SelectExpression } from "kysely";
import { getClickhouseKysely } from "../query-ast/dialect";
-import { includeDeleted, limitBy } from "../query-ast/extensions";
+import { limitBy, softDeleteAfterLimit } from "../query-ast/extensions";
import type { ClickHouseDatabase } from "../query-ast/schema";
type EventsAsE = ClickHouseDatabase & { e: ClickHouseDatabase["events_core"] };
@@ -9,8 +9,7 @@ type EventsAsE = ClickHouseDatabase & { e: ClickHouseDatabase["events_core"] };
/**
* Latest version of each generation linked to one of `promptIds`, optionally
* within a start-time window. Callers pick the columns and use it as a CTE.
- * Soft-deleted rows stay in so a deletion can be the latest version; callers
- * select `is_deleted` and drop those rows after the LIMIT BY.
+ * A generation whose latest version is soft-deleted is left out.
*/
export const promptGenerationEvents = <
S extends SelectExpression<EventsAsE, "e">,
@@ -21,7 +20,7 @@ export const promptGenerationEvents = <
) =>
getClickhouseKysely()
.selectFrom("events_core as e")
- .$call(includeDeleted())
+ .$call(softDeleteAfterLimit())
.select(columns)
.where("e.type", "=", "GENERATION")
.where("e.prompt_id", "in", promptIds)
diff --git a/packages/shared/src/server/repositories/scores.ts b/packages/shared/src/server/repositories/scores.ts
index e55deb81de..e548d19b08 100644
--- a/packages/shared/src/server/repositories/scores.ts
+++ b/packages/shared/src/server/repositories/scores.ts
@@ -76,12 +76,7 @@ import {
} from "../queries/clickhouse-sql/query-fragments";
import { compileClickhouseQuery } from "../query-ast/compile";
import { getClickhouseKysely } from "../query-ast/dialect";
-import {
- includeDeleted,
- limitBy,
- mapKeys,
- useFinal,
-} from "../query-ast/extensions";
+import { limitBy, mapKeys, useFinal } from "../query-ast/extensions";
import { scoresTableCols } from "../../tableDefinitions/scoresTable";
import { promptGenerationEvents } from "./prompt-events";
import {
@@ -426,7 +421,6 @@ export const getTraceScoresForDatasetRuns = async (
const experimentItemEventsQuery = (experimentIds: string[]) =>
getClickhouseKysely()
.selectFrom("events_core as e")
- .$call(includeDeleted())
.select(["e.project_id", "e.experiment_id", "e.trace_id"])
.where("e.experiment_id", "in", experimentIds)
.where("e.experiment_id", "!=", "");
@@ -2255,7 +2249,6 @@ const aggregatedScoresForPromptsFromEventsQuery = (
"e.prompt_id as prompt_id",
"e.trace_id as trace_id",
"e.span_id as span_id",
- "e.is_deleted as is_deleted",
],
timeWindow,
),
@@ -2282,17 +2275,13 @@ const aggregatedScoresForPromptsFromEventsQuery = (
"s.timestamp as timestamp",
eb(eb.fn("length", [mapKeys("s.metadata")]), ">", 0).as("has_metadata"),
])
- .where("e.is_deleted", "=", 0)
.where("s.name", "is not", null)
.where("s.data_type", "in", [...LISTABLE_SCORE_TYPES])
.$if(fetchScoreRelation === "trace", (qb) =>
qb
.where("s.observation_id", "is", null)
.where("s.trace_id", "in", (eb) =>
- eb
- .selectFrom("prompt_events")
- .select("trace_id")
- .where("is_deleted", "=", 0),
+ eb.selectFrom("prompt_events").select("trace_id"),
),
)
.$if(fetchScoreRelation === "observation", (qb) =>
@@ -2304,7 +2293,6 @@ const aggregatedScoresForPromptsFromEventsQuery = (
"in",
selectFrom("prompt_events")
.select(["trace_id", "span_id"])
- .where("is_deleted", "=", 0)
.$asTuple("trace_id", "span_id"),
),
),
diff --git a/web/src/__tests__/server/repositories/event-repository.servertest.ts b/web/src/__tests__/server/repositories/event-repository.servertest.ts
index 1f31c3f936..55ed23dbe1 100644
--- a/web/src/__tests__/server/repositories/event-repository.servertest.ts
+++ b/web/src/__tests__/server/repositories/event-repository.servertest.ts
@@ -7,6 +7,7 @@ import {
getObservationsCountFromEventsTable,
getObservationsCountsFromEventsTable,
getObservationByIdFromEventsTable,
+ getObservationFullIOForSessionFromEventsTable,
getAgentGraphDataFromEventsTable,
getObservationsFromEventsTableForPublicApi,
getObservationsCountFromEventsTableForPublicApi,
@@ -38,6 +39,7 @@ import { prisma } from "@langfuse/shared/src/db";
import { randomUUID } from "crypto";
import { env } from "@/src/env.mjs";
import {
+ LangfuseNotFoundError,
type EventsTableFilterState,
type FilterCondition,
type TimeFilter,
@@ -1180,6 +1182,41 @@ describe("Clickhouse Events Repository Test", () => {
});
});
+ maybe("latest-version reads of a soft-deleted observation", () => {
+ it("read it as missing even though an older live version exists", async () => {
+ const traceId = randomUUID();
+ const observationId = randomUUID();
+ const sessionId = randomUUID();
+ const start = Date.now() - 60_000;
+ const version = {
+ id: observationId,
+ span_id: observationId,
+ project_id: projectId,
+ trace_id: traceId,
+ session_id: sessionId,
+ start_time: start,
+ };
+
+ await createEventsCh([
+ createEvent({ ...version, event_ts: start }),
+ createEvent({ ...version, event_ts: start + 1000, is_deleted: 1 }),
+ ]);
+
+ await expect(
+ getObservationByIdFromEventsTable({ id: observationId, projectId }),
+ ).rejects.toThrow(LangfuseNotFoundError);
+ await expect(
+ getObservationFullIOForSessionFromEventsTable({
+ projectId,
+ sessionId,
+ traceId,
+ observationId,
+ startTime: new Date(start),
+ }),
+ ).resolves.toBeNull();
+ });
+ });
+
maybe("getAgentGraphDataFromEventsTable", () => {
it("bounds start_time by Date binds at millisecond precision", async () => {
const traceId = randomUUID();
diff --git a/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts b/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts
index 52caa34443..286059675e 100644
--- a/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts
+++ b/web/src/__tests__/server/repositories/stream-observation-io-repository.servertest.ts
@@ -146,4 +146,36 @@ maybe("streamObservationIOFieldFromEventsTable", () => {
});
expect(missing).toBeNull();
});
+
+ it("reads an observation whose latest version is soft-deleted as missing", async () => {
+ const opts = {
+ projectId: randomUUID(),
+ traceId: randomUUID(),
+ observationId: randomUUID(),
+ field: "input" as const,
+ startTime: new Date(),
+ };
+ const startMs = opts.startTime.getTime();
+ const version = {
+ id: opts.observationId,
+ span_id: opts.observationId,
+ project_id: opts.projectId,
+ trace_id: opts.traceId,
+ input: "live",
+ start_time: startMs * 1000,
+ };
+
+ await createEventsCh([
+ createEvent({ ...version, event_ts: startMs }),
+ createEvent({ ...version, event_ts: startMs + 1000, is_deleted: 1 }),
+ ]);
+
+ expect(await getObservationIOFieldByteLengthFromEventsTable(opts)).toBe(
+ null,
+ );
+ const streamed = await streamObservationIOFieldFromEventsTable(opts).then(
+ (r) => collectStream(r.stream),
+ );
+ expect(streamed).toBe("");
+ });
});