Skip to content

πŸͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(query-ast): declare per-table dedup without events LIMIT BY (#16981)

Niklas Semmlere2cd635b49de
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.md
index 993d3e87c9..55a6c4ccc5 100644
--- a/packages/shared/AGENTS.md
+++ b/packages/shared/AGENTS.md
@@ -31,7 +31,7 @@
`clickhouse format` for snapshot comparison. Every migrated call site is
proven against its baseline here. The Kysely ClickHouse dialect (ARRAY JOIN /
LIMIT BY / metadata indexOf nodes, `ExecutionContext` tenancy injection,
- typed selection, virtual views, catalog parity) lives under
+ per-table dedup lowering, virtual views, catalog parity) lives under
`src/server/query-ast/kysely/`.
- Postgres schema: `prisma/schema.prisma`
- Prisma migrations: `prisma/migrations/*`
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index e1e57352d0..ddc03f515f 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -17,13 +17,14 @@ sugar, semantic field metadata, `FilterState` embedding) will land under
- `kysely/` β€” compile-only ClickHouse dialect on Kysely 0.28. Real
`OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf`
subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;
- schema-typed selection (TS + runtime validation); virtual views as WITH
- CTEs; catalog parity. Library-specific code lives only in this folder.
+ per-table dedup lowering from the registry; schema-typed selection; virtual
+ views as WITH CTEs; catalog parity. Library-specific code lives only in
+ this folder.
- `kysely/schema.ts` β€” the physical table registry: one `defineTable`
- declaration per relation drives all three downstream views β€” the Kysely row
- types (`ClickHouseDatabase`), the runtime column-type map the type-check pass
- consults (`COLUMN_DATA_TYPES`), and the set the tenancy pass scopes
- (`TENANTED_TABLES`) β€” instead of three hand-maintained tables that drift.
+ declaration per relation drives the Kysely row types (`ClickHouseDatabase`),
+ the runtime column-type map (`COLUMN_DATA_TYPES`), the tenanted-table set
+ (`TENANTED_TABLES`), and the per-table dedup specs (`DEDUP_SPECS`) β€”
+ instead of hand-maintained tables that drift.
Regenerate baselines with `-u` after an intentional SQL change:
@@ -56,6 +57,11 @@ a required check once it has proven stable.
- **Never filter `project_id` yourself.** The compile step injects
`project_id = {projectId}` into every tenanted relation, so call sites pass
only `{ projectId }` (see `repositories/environments.ts`).
+- **Dedup is declared per table and must be an existing production idiom.**
+ `events_core` is `none` (immutable at read time β€” no LIMIT BY, no FINAL).
+ `limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` already
+ used on traces / observations / scores. `$call(limitBy(...))` remains for
+ explicit non-version LIMIT BY.
- **ClickHouse-only clauses use `$call(helper())`** β€” not fluent builder
methods, so they compose inside CTEs, subqueries, and views. See the recipes
below.
diff --git a/packages/shared/src/server/query-ast/kysely/README.md b/packages/shared/src/server/query-ast/kysely/README.md
index 148a395afd..0401a1cc7c 100644
--- a/packages/shared/src/server/query-ast/kysely/README.md
+++ b/packages/shared/src/server/query-ast/kysely/README.md
@@ -50,9 +50,15 @@ is where tenancy is enforced:
name (if not) β€” so `scores AS traces` joined to `traces AS t` still scopes
both. It then identity-stamps the tree (`WeakSet`); a copied
`langfuseTenancy` property is not a valid stamp.
-3. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is
+3. `DedupLoweringPlugin` applies the table's declared read idiom
+ (`none` / `limitBy` / `final`). `events_core` is `none` β€” immutable at
+ read time, so the pass does not inject LIMIT BY or FINAL. `limitBy` is
+ the existing legacy `ORDER BY <version> DESC LIMIT 1 BY <key>`.
+ `final` is fail-closed until an emitter exists. The pass restamps the
+ rewritten root.
+4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is
present, so `qb.compile()` without the plugin also fails.
-4. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a
+5. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a
`SELECT`/`FROM`/`JOIN` in SELECT/WHERE throw `UnscopedRelationError`. Kysely's
own keyword fragments (`asc`/`desc`) are not relations.
diff --git a/packages/shared/src/server/query-ast/kysely/compile.ts b/packages/shared/src/server/query-ast/kysely/compile.ts
index 88f5840fa4..d60734c533 100644
--- a/packages/shared/src/server/query-ast/kysely/compile.ts
+++ b/packages/shared/src/server/query-ast/kysely/compile.ts
@@ -6,6 +6,7 @@ import {
import type { ExecutionContext } from "../executionContext";
import { ClickHouseQueryCompiler } from "./compiler";
+import { DedupLoweringPlugin } from "./dedup";
import { TenancyInjectionPlugin, requireExecutionContext } from "./tenancy";
import { validateTypeCompatibility } from "./typecheck";
@@ -27,11 +28,12 @@ export type ClickhouseCompilable = {
* The choke point. Every query that should produce ClickHouse SQL goes
* through here: the `ctx` parameter is required, so a caller cannot compile
* without a tenancy scope β€” omitting it is a compile-time type error. Tenancy
- * is injected from that scope, and the dialect compiler refuses to emit SQL
- * unless the tree was identity-stamped by that pass (a copied property is not
- * enough). The runtime `requireExecutionContext` guard remains as
- * defense-in-depth for the one case the type cannot express (an empty
- * `projectId`) and for callers reaching this through an `any` boundary.
+ * is injected from that scope, per-table dedup is applied from the
+ * registry, and the dialect compiler refuses to emit SQL unless the tree was
+ * identity-stamped by those passes (a copied property is not enough). The
+ * runtime `requireExecutionContext` guard remains as defense-in-depth for the
+ * one case the type cannot express (an empty `projectId`) and for callers
+ * reaching this through an `any` boundary.
*/
export function compileClickhouseQuery(
query: ClickhouseCompilable,
@@ -40,6 +42,7 @@ export function compileClickhouseQuery(
const scope = requireExecutionContext(ctx);
const stamped = query
.withPlugin(new TenancyInjectionPlugin(scope))
+ .withPlugin(new DedupLoweringPlugin())
.toOperationNode();
const compiler = new ClickHouseQueryCompiler();
validateTypeCompatibility(stamped);
diff --git a/packages/shared/src/server/query-ast/kysely/dedup.test.ts b/packages/shared/src/server/query-ast/kysely/dedup.test.ts
new file mode 100644
index 0000000000..52993dd254
--- /dev/null
+++ b/packages/shared/src/server/query-ast/kysely/dedup.test.ts
@@ -0,0 +1,76 @@
+import { describe, expect, it } from "vitest";
+
+import { compileClickhouseQuery } from "./compile";
+import { getClickhouseKysely } from "./dialect";
+import { limitBy } from "./extensions";
+import { DEDUP_SPECS } from "./schema";
+
+const ctx = { projectId: "proj-1" };
+
+function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {
+ return compileClickhouseQuery(builder, ctx);
+}
+
+describe("per-table dedup lowering", () => {
+ it("declares events_core as immutable (no query-time collapse)", () => {
+ expect(DEDUP_SPECS.events_core).toEqual({ strategy: "none" });
+ expect(DEDUP_SPECS.traces).toBeUndefined();
+ expect(DEDUP_SPECS.observations).toBeUndefined();
+ expect(DEDUP_SPECS.scores).toBeUndefined();
+ });
+
+ it("does not inject LIMIT BY or FINAL on events_core row reads", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select(["span_id", "project_id"]),
+ );
+ const lower = sql.toLowerCase();
+ expect(lower).not.toContain("limit 1 by");
+ expect(lower).not.toMatch(/\bfinal\b/);
+ });
+
+ it("does not wrap events_core DISTINCT", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select("environment")
+ .distinct(),
+ );
+ const lower = sql.toLowerCase();
+ expect(lower).not.toContain("limit 1 by");
+ expect(lower).not.toContain("from (select *");
+ });
+
+ it("does not wrap events_core aggregations", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select((eb) => ["environment", eb.fn.countAll().as("n")])
+ .groupBy("environment"),
+ );
+ const lower = sql.toLowerCase();
+ expect(lower).not.toContain("limit 1 by");
+ expect(lower).not.toContain("from (select *");
+ expect(lower).toContain("group by environment");
+ });
+
+ it("still emits an explicit caller LIMIT BY", () => {
+ const { sql } = compile(
+ getClickhouseKysely()
+ .selectFrom("events_core")
+ .select("span_id")
+ .orderBy("event_ts", "desc")
+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] })),
+ );
+ expect(sql.toLowerCase().split("limit 1 by").length - 1).toBe(1);
+ });
+
+ it("does not rewrite undeclared tables", () => {
+ const { sql } = compile(
+ getClickhouseKysely().selectFrom("traces").select("id"),
+ );
+ expect(sql.toLowerCase()).not.toContain("limit 1 by");
+ expect(sql.toLowerCase()).not.toMatch(/\bfinal\b/);
+ });
+});
diff --git a/packages/shared/src/server/query-ast/kysely/dedup.ts b/packages/shared/src/server/query-ast/kysely/dedup.ts
new file mode 100644
index 0000000000..b088ca61b3
--- /dev/null
+++ b/packages/shared/src/server/query-ast/kysely/dedup.ts
@@ -0,0 +1,353 @@
+/**
+ * Per-table dedup lowering. Always-on: every compile walks this pass.
+ * The physical idiom is the one already used in production SQL for that
+ * table ({@link DEDUP_SPECS}) β€” never a new collapse invented here.
+ *
+ * - **none** β€” immutable at read time (`events_core` / `events_full`).
+ * Leave the query unchanged. No FINAL, no LIMIT BY.
+ * - **limitBy** β€” existing legacy idiom: `ORDER BY <version> DESC`
+ * `LIMIT 1 BY <key>` on row reads; wrap the FROM first for
+ * aggregates / DISTINCT so LIMIT BY runs on input rows. Used by
+ * traces / observations / scores point and list reads. A caller
+ * ORDER BY on another column wraps so the version sort is the
+ * LIMIT BY input order.
+ * - **final** β€” existing `FROM <table> FINAL` idiom. Declaring it
+ * without an emitter is a compile error (fail-closed).
+ *
+ * Omit the spec until a family is migrated. `$call(limitBy)` remains
+ * for non-version LIMIT BY the registry does not own.
+ */
+import {
+ AggregateFunctionNode,
+ AliasNode,
+ ColumnNode,
+ FromNode,
+ IdentifierNode,
+ OrderByItemNode,
+ OrderByNode,
+ QueryNode,
+ RawNode,
+ ReferenceNode,
+ SelectModifierNode,
+ SelectQueryNode,
+ SelectionNode,
+ TableNode,
+ ValueNode,
+ type KyselyPlugin,
+ type OperationNode,
+ type PluginTransformQueryArgs,
+ type PluginTransformResultArgs,
+ type QueryId,
+ type RootOperationNode,
+} from "kysely";
+
+import { QueryCompileError } from "./errors";
+import { LimitByNode, type ClickHouseSelectQueryNode } from "./nodes";
+import { DEDUP_SPECS, type DedupSpec } from "./schema";
+import { stampCompiledTree } from "./tenancy";
+import { ClickHouseOperationNodeTransformer } from "./transformer";
+
+type PhysicalFrom = {
+ tableName: string;
+ alias?: string;
+ fromItem: OperationNode;
+};
+
+type LimitBySpec = Extract<DedupSpec, { strategy: "limitBy" }>;
+
+const DOUBLE_COUNT_AGGREGATES = new Set(["count", "sum", "avg"]);
+
+export class DedupLoweringPlugin implements KyselyPlugin {
+ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {
+ const transformer = new DedupLoweringTransformer();
+ const lowered = transformer.transformNode(args.node) as RootOperationNode;
+ // A rewritten root is a new object; restamp so the compiler will emit.
+ return stampCompiledTree(lowered);
+ }
+
+ async transformResult(args: PluginTransformResultArgs) {
+ return args.result;
+ }
+}
+
+class DedupLoweringTransformer extends ClickHouseOperationNodeTransformer {
+ protected override transformSelectQuery(
+ node: SelectQueryNode,
+ queryId?: QueryId,
+ ): SelectQueryNode {
+ const withChildren = super.transformSelectQuery(
+ node,
+ queryId,
+ ) as ClickHouseSelectQueryNode;
+ return lowerSelect(withChildren, collectCteNames(withChildren));
+ }
+}
+
+function collectCteNames(node: SelectQueryNode): Set<string> {
+ const names = new Set<string>();
+ for (const expr of node.with?.expressions ?? []) {
+ names.add(expr.name.table.table.identifier.name);
+ }
+ return names;
+}
+
+function lowerSelect(
+ node: ClickHouseSelectQueryNode,
+ cteNames: Set<string>,
+): ClickHouseSelectQueryNode {
+ const from = singlePhysicalFrom(node, cteNames);
+ if (!from) return node;
+
+ const spec = DEDUP_SPECS[from.tableName];
+ if (!spec || spec.strategy === "none") return node;
+
+ if (spec.strategy === "final") {
+ throw new QueryCompileError(
+ `Dedup strategy "final" is declared on ${from.tableName} but has no emitter. Use strategy "limitBy" or implement FINAL lowering before declaring it.`,
+ );
+ }
+
+ if (node.limitBy) {
+ // Caller already attached LIMIT BY. Still pin the version sort so the
+ // clause keeps the latest row when they forgot ORDER BY <version>.
+ return ensureVersionOrder(node, from, spec);
+ }
+
+ const shape = classifyShape(node);
+ if (shape === "skip") return node;
+ if (shape === "aggregate") return wrapPhysicalFrom(node, from, spec);
+ return attachRowDedup(node, from, spec);
+}
+
+function singlePhysicalFrom(
+ node: SelectQueryNode,
+ cteNames: Set<string>,
+): PhysicalFrom | undefined {
+ if (node.joins?.length) return undefined;
+ const froms = node.from?.froms ?? [];
+ if (froms.length !== 1) return undefined;
+ const described = describeFrom(froms[0]);
+ if (!described) return undefined;
+ if (cteNames.has(described.tableName)) return undefined;
+ return described;
+}
+
+function describeFrom(node: OperationNode): PhysicalFrom | undefined {
+ if (AliasNode.is(node)) {
+ const inner = describeFrom(node.node);
+ if (!inner) return undefined;
+ const alias = IdentifierNode.is(node.alias) ? node.alias.name : undefined;
+ return { ...inner, alias, fromItem: node };
+ }
+ if (TableNode.is(node)) {
+ return {
+ tableName: node.table.identifier.name,
+ fromItem: node,
+ };
+ }
+ return undefined;
+}
+
+type QueryShape = "row" | "aggregate" | "skip";
+
+function classifyShape(node: SelectQueryNode): QueryShape {
+ if (isExistence(node)) return "skip";
+ if (
+ isDistinctOnly(node) ||
+ node.groupBy ||
+ node.having ||
+ hasWrappingAggregate(node)
+ ) {
+ return "aggregate";
+ }
+ // Scalar max/min/argMax without GROUP BY already collapse versions;
+ // attaching ORDER BY / LIMIT BY after the aggregate is illegal SQL.
+ if (hasAnyAggregate(node)) return "skip";
+ return "row";
+}
+
+function isDistinctOnly(node: SelectQueryNode): boolean {
+ return (
+ node.frontModifiers?.some(
+ (modifier) =>
+ SelectModifierNode.is(modifier) && modifier.modifier === "Distinct",
+ ) ?? false
+ );
+}
+
+function isExistence(node: SelectQueryNode): boolean {
+ if (!isLimitOne(node) || node.groupBy) return false;
+ const selections = node.selections ?? [];
+ if (selections.length !== 1) return false;
+ const expr = unwrapSelection(selections[0]);
+ if (!expr) return false;
+ if (ValueNode.is(expr) && expr.value === 1) return true;
+ return AggregateFunctionNode.is(expr) && isCountFunc(expr.func);
+}
+
+function isLimitOne(node: SelectQueryNode): boolean {
+ const limit = node.limit?.limit;
+ return limit !== undefined && ValueNode.is(limit) && limit.value === 1;
+}
+
+function hasWrappingAggregate(node: SelectQueryNode): boolean {
+ let found = false;
+ walk(node.selections, (child) => {
+ if (!AggregateFunctionNode.is(child)) return;
+ if (child.over || DOUBLE_COUNT_AGGREGATES.has(child.func.toLowerCase())) {
+ found = true;
+ }
+ });
+ return found;
+}
+
+function hasAnyAggregate(node: SelectQueryNode): boolean {
+ let found = false;
+ walk(node.selections, (child) => {
+ if (AggregateFunctionNode.is(child)) found = true;
+ });
+ return found;
+}
+
+function isCountFunc(func: string): boolean {
+ return func.toLowerCase() === "count";
+}
+
+function unwrapSelection(selection: OperationNode): OperationNode | undefined {
+ if (!SelectionNode.is(selection)) return undefined;
+ const inner = selection.selection;
+ if (AliasNode.is(inner)) return inner.node;
+ return inner;
+}
+
+function attachRowDedup(
+ node: ClickHouseSelectQueryNode,
+ from: PhysicalFrom,
+ spec: LimitBySpec,
+): ClickHouseSelectQueryNode {
+ // LIMIT BY keeps the first row per key in the query's full sort order.
+ // A caller ORDER BY on any other column would make the version a
+ // tiebreaker and could keep a stale row β€” wrap so collapse is inner.
+ if (node.orderBy && !hasLeadingVersionDesc(node, spec)) {
+ return wrapPhysicalFrom(node, from, spec);
+ }
+ return {
+ ...ensureVersionOrder(node, from, spec),
+ limitBy: limitByNode(from, spec),
+ };
+}
+
+function wrapPhysicalFrom(
+ node: ClickHouseSelectQueryNode,
+ from: PhysicalFrom,
+ spec: LimitBySpec,
+): ClickHouseSelectQueryNode {
+ const inner: ClickHouseSelectQueryNode = {
+ kind: "SelectQueryNode",
+ from: FromNode.create([from.fromItem]),
+ selections: [SelectionNode.createSelectAll()],
+ ...(node.where ? { where: node.where } : {}),
+ orderBy: versionOrderBy(from, spec),
+ limitBy: limitByNode(from, spec),
+ };
+ const alias = IdentifierNode.create(from.alias ?? from.tableName);
+ return {
+ ...node,
+ from: FromNode.create([AliasNode.create(inner, alias)]),
+ where: undefined,
+ };
+}
+
+function ensureVersionOrder(
+ node: ClickHouseSelectQueryNode,
+ from: PhysicalFrom,
+ spec: LimitBySpec,
+): ClickHouseSelectQueryNode {
+ if (orderByHasColumn(node.orderBy, spec.version)) return node;
+ return QueryNode.cloneWithOrderByItems(node, [
+ versionOrderItem(from, spec),
+ ]) as ClickHouseSelectQueryNode;
+}
+
+function versionOrderBy(from: PhysicalFrom, spec: LimitBySpec): OrderByNode {
+ return OrderByNode.create([versionOrderItem(from, spec)]);
+}
+
+function versionOrderItem(
+ from: PhysicalFrom,
+ spec: LimitBySpec,
+): OrderByItemNode {
+ return OrderByItemNode.create(
+ qualifyColumn(from, spec.version),
+ RawNode.createWithSql("desc"),
+ );
+}
+
+function limitByNode(from: PhysicalFrom, spec: LimitBySpec): LimitByNode {
+ return LimitByNode.create(
+ ValueNode.createImmediate(1),
+ spec.key.map((column) => qualifyColumn(from, column)),
+ );
+}
+
+function qualifyColumn(from: PhysicalFrom, column: string): OperationNode {
+ const col = ColumnNode.create(column);
+ return from.alias
+ ? ReferenceNode.create(col, TableNode.create(from.alias))
+ : col;
+}
+
+function hasLeadingVersionDesc(
+ node: SelectQueryNode,
+ spec: LimitBySpec,
+): boolean {
+ const first = node.orderBy?.items[0];
+ return (
+ first !== undefined &&
+ columnNameOf(first.orderBy) === spec.version &&
+ isDescending(first)
+ );
+}
+
+function isDescending(item: OrderByItemNode): boolean {
+ const direction = item.direction;
+ if (!direction || !RawNode.is(direction)) return false;
+ return direction.sqlFragments.join("").trim().toLowerCase() === "desc";
+}
+
+function orderByHasColumn(
+ orderBy: OrderByNode | undefined,
+ column: string,
+): boolean {
+ if (!orderBy) return false;
+ return orderBy.items.some((item) => columnNameOf(item.orderBy) === column);
+}
+
+function columnNameOf(node: OperationNode): string | undefined {
+ if (ColumnNode.is(node)) return node.column.name;
+ if (ReferenceNode.is(node) && ColumnNode.is(node.column)) {
+ return node.column.column.name;
+ }
+ return undefined;
+}
+
+function walk(value: unknown, visit: (node: OperationNode) => void): void {
+ if (!value || typeof value !== "object") return;
+ if (
+ "kind" in value &&
+ typeof (value as { kind: unknown }).kind === "string"
+ ) {
+ const node = value as OperationNode;
+ visit(node);
+ // Nested selects have their own shape; do not attribute their
+ // aggregates to the outer query.
+ if (SelectQueryNode.is(node)) return;
+ }
+ for (const child of Object.values(value)) {
+ if (Array.isArray(child)) {
+ for (const item of child) walk(item, visit);
+ } else {
+ walk(child, visit);
+ }
+ }
+}
diff --git a/packages/shared/src/server/query-ast/kysely/schema.test.ts b/packages/shared/src/server/query-ast/kysely/schema.test.ts
index 815486231c..4d502ed386 100644
--- a/packages/shared/src/server/query-ast/kysely/schema.test.ts
+++ b/packages/shared/src/server/query-ast/kysely/schema.test.ts
@@ -1,6 +1,6 @@
import { describe, expect, it } from "vitest";
-import { COLUMN_DATA_TYPES, TENANTED_TABLES } from "./schema";
+import { COLUMN_DATA_TYPES, DEDUP_SPECS, TENANTED_TABLES } from "./schema";
import { schemaTypeAssertions, extensionTypeAssertions } from "./types.assert";
describe("table registry derivation", () => {
@@ -27,6 +27,13 @@ describe("table registry derivation", () => {
expect(COLUMN_DATA_TYPES.cost_details).toBe("map");
});
+ it("declares events_core as immutable and leaves legacy tables undeclared", () => {
+ expect(DEDUP_SPECS.events_core).toEqual({ strategy: "none" });
+ expect(DEDUP_SPECS.traces).toBeUndefined();
+ expect(DEDUP_SPECS.observations).toBeUndefined();
+ expect(DEDUP_SPECS.scores).toBeUndefined();
+ });
+
// The assertions themselves are compile-time (`tsc` is the test); this only
// anchors the file so its `@ts-expect-error` checks stay in the build graph.
it("keeps the compile-time type assertions in the build graph", () => {
diff --git a/packages/shared/src/server/query-ast/kysely/schema.ts b/packages/shared/src/server/query-ast/kysely/schema.ts
index 0fd812f599..a4548799a5 100644
--- a/packages/shared/src/server/query-ast/kysely/schema.ts
+++ b/packages/shared/src/server/query-ast/kysely/schema.ts
@@ -1,6 +1,6 @@
/**
* Single source of truth for the physical ClickHouse relations the Kysely
- * compiler targets. One table declaration drives all three downstream views:
+ * compiler targets. One table declaration drives the downstream views:
*
* - `ClickHouseDatabase` β€” the Kysely row types (column autocomplete and typed
* comparisons).
@@ -8,12 +8,14 @@
* pass consults (`typecheck.ts`).
* - `TENANTED_TABLES` β€” the relations the tenancy pass must scope
* (`tenancy.ts`).
+ * - `DEDUP_SPECS` β€” the per-table read idiom the lowering pass applies
+ * (`dedup.ts`). Each table names an existing production mechanism; the
+ * pass does not invent a new one.
*
- * Deriving all three from one declaration keeps them from drifting apart.
+ * Deriving every view from one declaration keeps them from drifting apart.
*
* Column sets cover the relations the compiler targets, not a full schema
- * dump. Physical tuning metadata (partition / sort / dedup keys) is not
- * modeled here; dedup lowering will extend this same declaration.
+ * dump. Partition / sort keys stay unmodeled until a pass needs them.
*/
/**
@@ -50,6 +52,24 @@ const RUNTIME_TYPE: Record<ChColumnType, ColumnDataType> = {
"Map(String, Float)": "map",
};
+/**
+ * Per-table read idiom. Must match a mechanism already used in production
+ * SQL for that table β€” the pass applies the declaration, it does not invent
+ * a new collapse.
+ *
+ * - `none` β€” rows are treated as immutable at read time. No FINAL, no
+ * LIMIT BY. Used for `events_core` / `events_full`.
+ * - `limitBy` β€” the existing legacy idiom
+ * `ORDER BY <version> DESC LIMIT 1 BY <key>` (traces / observations /
+ * scores point and list reads).
+ * - `final` β€” the existing `FROM <table> FINAL` idiom. Declaring it
+ * without an emitter is a compile error (fail-closed).
+ */
+export type DedupSpec =
+ | { strategy: "none" }
+ | { strategy: "limitBy"; key: readonly string[]; version: string }
+ | { strategy: "final" };
+
function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {
columns: Cols;
/**
@@ -59,8 +79,17 @@ function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {
* only for a genuinely global relation.
*/
tenant?: boolean;
+ /**
+ * Existing read-time collapse for this table. Omit until the family is
+ * migrated; `none` is an explicit "do not collapse" (immutable rows).
+ */
+ dedup?: DedupSpec;
}) {
- return { columns: spec.columns, tenant: spec.tenant ?? true };
+ return {
+ columns: spec.columns,
+ tenant: spec.tenant ?? true,
+ dedup: spec.dedup,
+ };
}
const TABLE_REGISTRY = {
@@ -95,6 +124,8 @@ const TABLE_REGISTRY = {
metadata_names: "Array(String)",
metadata_values: "Array(String)",
},
+ // Immutable at read time. Do not inject LIMIT BY or FINAL.
+ dedup: { strategy: "none" },
}),
scores: defineTable({
columns: {
@@ -138,3 +169,10 @@ export const TENANTED_TABLES = new Set<string>(
.filter(([, spec]) => spec.tenant)
.map(([name]) => name),
);
+
+/** Derived per-table read idioms, keyed by table name. */
+export const DEDUP_SPECS: Record<string, DedupSpec> = Object.fromEntries(
+ Object.entries(TABLE_REGISTRY).flatMap(([name, spec]) =>
+ spec.dedup ? [[name, spec.dedup]] : [],
+ ),
+);
diff --git a/packages/shared/src/server/query-ast/kysely/tenancy.ts b/packages/shared/src/server/query-ast/kysely/tenancy.ts
index 60dbd7fb83..b2f2c3636b 100644
--- a/packages/shared/src/server/query-ast/kysely/tenancy.ts
+++ b/packages/shared/src/server/query-ast/kysely/tenancy.ts
@@ -315,6 +315,16 @@ function stampTenancy<T extends object>(node: T): T {
return node;
}
+/**
+ * Re-stamp a tree after a later rewrite pass. Identity-based: a copied
+ * property is not a valid stamp. Dedup lowering (and any future rewrite)
+ * must call this on the node it returns, because a new root object is not
+ * the object the tenancy pass stamped.
+ */
+export function stampCompiledTree<T extends object>(node: T): T {
+ return stampTenancy(node);
+}
+
export function assertTenancyStamped(node: RootOperationNode): void {
if (!TENANCY_STAMPED.has(node)) {
throw new QueryCompileError(