πͺ’ Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(query-ast): declare per-table dedup without events LIMIT BY (#16981)
Niklas Semmlere2cd635b49de
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.mdindex 993d3e87c9..55a6c4ccc5 100644--- a/packages/shared/AGENTS.md+++ b/packages/shared/AGENTS.md@@ -31,7 +31,7 @@`clickhouse format` for snapshot comparison. Every migrated call site isproven against its baseline here. The Kysely ClickHouse dialect (ARRAY JOIN /LIMIT BY / metadata indexOf nodes, `ExecutionContext` tenancy injection,- typed selection, virtual views, catalog parity) lives under+ per-table dedup lowering, virtual views, catalog parity) lives under`src/server/query-ast/kysely/`.- Postgres schema: `prisma/schema.prisma`- Prisma migrations: `prisma/migrations/*`diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.mdindex e1e57352d0..ddc03f515f 100644--- a/packages/shared/src/server/query-ast/README.md+++ b/packages/shared/src/server/query-ast/README.md@@ -17,13 +17,14 @@ sugar, semantic field metadata, `FilterState` embedding) will land under- `kysely/` β compile-only ClickHouse dialect on Kysely 0.28. Real`OperationNode`s for ARRAY JOIN, LIMIT BY, and metadata `indexOf`subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;- schema-typed selection (TS + runtime validation); virtual views as WITH- CTEs; catalog parity. Library-specific code lives only in this folder.+ per-table dedup lowering from the registry; schema-typed selection; virtual+ views as WITH CTEs; catalog parity. Library-specific code lives only in+ this folder.- `kysely/schema.ts` β the physical table registry: one `defineTable`- declaration per relation drives all three downstream views β the Kysely row- types (`ClickHouseDatabase`), the runtime column-type map the type-check pass- consults (`COLUMN_DATA_TYPES`), and the set the tenancy pass scopes- (`TENANTED_TABLES`) β instead of three hand-maintained tables that drift.+ declaration per relation drives the Kysely row types (`ClickHouseDatabase`),+ the runtime column-type map (`COLUMN_DATA_TYPES`), the tenanted-table set+ (`TENANTED_TABLES`), and the per-table dedup specs (`DEDUP_SPECS`) β+ instead of hand-maintained tables that drift.Regenerate baselines with `-u` after an intentional SQL change:@@ -56,6 +57,11 @@ a required check once it has proven stable.- **Never filter `project_id` yourself.** The compile step injects`project_id = {projectId}` into every tenanted relation, so call sites passonly `{ projectId }` (see `repositories/environments.ts`).+- **Dedup is declared per table and must be an existing production idiom.**+ `events_core` is `none` (immutable at read time β no LIMIT BY, no FINAL).+ `limitBy` is the legacy `ORDER BY <version> DESC LIMIT 1 BY <key>` already+ used on traces / observations / scores. `$call(limitBy(...))` remains for+ explicit non-version LIMIT BY.- **ClickHouse-only clauses use `$call(helper())`** β not fluent buildermethods, so they compose inside CTEs, subqueries, and views. See the recipesbelow.diff --git a/packages/shared/src/server/query-ast/kysely/README.md b/packages/shared/src/server/query-ast/kysely/README.mdindex 148a395afd..0401a1cc7c 100644--- a/packages/shared/src/server/query-ast/kysely/README.md+++ b/packages/shared/src/server/query-ast/kysely/README.md@@ -50,9 +50,15 @@ is where tenancy is enforced:name (if not) β so `scores AS traces` joined to `traces AS t` still scopesboth. It then identity-stamps the tree (`WeakSet`); a copied`langfuseTenancy` property is not a valid stamp.-3. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp is+3. `DedupLoweringPlugin` applies the table's declared read idiom+ (`none` / `limitBy` / `final`). `events_core` is `none` β immutable at+ read time, so the pass does not inject LIMIT BY or FINAL. `limitBy` is+ the existing legacy `ORDER BY <version> DESC LIMIT 1 BY <key>`.+ `final` is fail-closed until an emitter exists. The pass restamps the+ rewritten root.+4. `ClickHouseQueryCompiler` refuses to emit SQL unless that identity stamp ispresent, so `qb.compile()` without the plugin also fails.-4. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a+5. Raw-SQL table sources (`selectFrom(sql\`...\`)`) and raw fragments embedding a`SELECT`/`FROM`/`JOIN` in SELECT/WHERE throw `UnscopedRelationError`. Kysely'sown keyword fragments (`asc`/`desc`) are not relations.diff --git a/packages/shared/src/server/query-ast/kysely/compile.ts b/packages/shared/src/server/query-ast/kysely/compile.tsindex 88f5840fa4..d60734c533 100644--- a/packages/shared/src/server/query-ast/kysely/compile.ts+++ b/packages/shared/src/server/query-ast/kysely/compile.ts@@ -6,6 +6,7 @@ import {import type { ExecutionContext } from "../executionContext";import { ClickHouseQueryCompiler } from "./compiler";+import { DedupLoweringPlugin } from "./dedup";import { TenancyInjectionPlugin, requireExecutionContext } from "./tenancy";import { validateTypeCompatibility } from "./typecheck";@@ -27,11 +28,12 @@ export type ClickhouseCompilable = {* The choke point. Every query that should produce ClickHouse SQL goes* through here: the `ctx` parameter is required, so a caller cannot compile* without a tenancy scope β omitting it is a compile-time type error. Tenancy- * is injected from that scope, and the dialect compiler refuses to emit SQL- * unless the tree was identity-stamped by that pass (a copied property is not- * enough). The runtime `requireExecutionContext` guard remains as- * defense-in-depth for the one case the type cannot express (an empty- * `projectId`) and for callers reaching this through an `any` boundary.+ * is injected from that scope, per-table dedup is applied from the+ * registry, and the dialect compiler refuses to emit SQL unless the tree was+ * identity-stamped by those passes (a copied property is not enough). The+ * runtime `requireExecutionContext` guard remains as defense-in-depth for the+ * one case the type cannot express (an empty `projectId`) and for callers+ * reaching this through an `any` boundary.*/export function compileClickhouseQuery(query: ClickhouseCompilable,@@ -40,6 +42,7 @@ export function compileClickhouseQuery(const scope = requireExecutionContext(ctx);const stamped = query.withPlugin(new TenancyInjectionPlugin(scope))+ .withPlugin(new DedupLoweringPlugin()).toOperationNode();const compiler = new ClickHouseQueryCompiler();validateTypeCompatibility(stamped);diff --git a/packages/shared/src/server/query-ast/kysely/dedup.test.ts b/packages/shared/src/server/query-ast/kysely/dedup.test.tsnew file mode 100644index 0000000000..52993dd254--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/dedup.test.ts@@ -0,0 +1,76 @@+import { describe, expect, it } from "vitest";++import { compileClickhouseQuery } from "./compile";+import { getClickhouseKysely } from "./dialect";+import { limitBy } from "./extensions";+import { DEDUP_SPECS } from "./schema";++const ctx = { projectId: "proj-1" };++function compile(builder: Parameters<typeof compileClickhouseQuery>[0]) {+ return compileClickhouseQuery(builder, ctx);+}++describe("per-table dedup lowering", () => {+ it("declares events_core as immutable (no query-time collapse)", () => {+ expect(DEDUP_SPECS.events_core).toEqual({ strategy: "none" });+ expect(DEDUP_SPECS.traces).toBeUndefined();+ expect(DEDUP_SPECS.observations).toBeUndefined();+ expect(DEDUP_SPECS.scores).toBeUndefined();+ });++ it("does not inject LIMIT BY or FINAL on events_core row reads", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select(["span_id", "project_id"]),+ );+ const lower = sql.toLowerCase();+ expect(lower).not.toContain("limit 1 by");+ expect(lower).not.toMatch(/\bfinal\b/);+ });++ it("does not wrap events_core DISTINCT", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("environment")+ .distinct(),+ );+ const lower = sql.toLowerCase();+ expect(lower).not.toContain("limit 1 by");+ expect(lower).not.toContain("from (select *");+ });++ it("does not wrap events_core aggregations", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select((eb) => ["environment", eb.fn.countAll().as("n")])+ .groupBy("environment"),+ );+ const lower = sql.toLowerCase();+ expect(lower).not.toContain("limit 1 by");+ expect(lower).not.toContain("from (select *");+ expect(lower).toContain("group by environment");+ });++ it("still emits an explicit caller LIMIT BY", () => {+ const { sql } = compile(+ getClickhouseKysely()+ .selectFrom("events_core")+ .select("span_id")+ .orderBy("event_ts", "desc")+ .$call(limitBy({ count: 1, columns: ["span_id", "project_id"] })),+ );+ expect(sql.toLowerCase().split("limit 1 by").length - 1).toBe(1);+ });++ it("does not rewrite undeclared tables", () => {+ const { sql } = compile(+ getClickhouseKysely().selectFrom("traces").select("id"),+ );+ expect(sql.toLowerCase()).not.toContain("limit 1 by");+ expect(sql.toLowerCase()).not.toMatch(/\bfinal\b/);+ });+});diff --git a/packages/shared/src/server/query-ast/kysely/dedup.ts b/packages/shared/src/server/query-ast/kysely/dedup.tsnew file mode 100644index 0000000000..b088ca61b3--- /dev/null+++ b/packages/shared/src/server/query-ast/kysely/dedup.ts@@ -0,0 +1,353 @@+/**+ * Per-table dedup lowering. Always-on: every compile walks this pass.+ * The physical idiom is the one already used in production SQL for that+ * table ({@link DEDUP_SPECS}) β never a new collapse invented here.+ *+ * - **none** β immutable at read time (`events_core` / `events_full`).+ * Leave the query unchanged. No FINAL, no LIMIT BY.+ * - **limitBy** β existing legacy idiom: `ORDER BY <version> DESC`+ * `LIMIT 1 BY <key>` on row reads; wrap the FROM first for+ * aggregates / DISTINCT so LIMIT BY runs on input rows. Used by+ * traces / observations / scores point and list reads. A caller+ * ORDER BY on another column wraps so the version sort is the+ * LIMIT BY input order.+ * - **final** β existing `FROM <table> FINAL` idiom. Declaring it+ * without an emitter is a compile error (fail-closed).+ *+ * Omit the spec until a family is migrated. `$call(limitBy)` remains+ * for non-version LIMIT BY the registry does not own.+ */+import {+ AggregateFunctionNode,+ AliasNode,+ ColumnNode,+ FromNode,+ IdentifierNode,+ OrderByItemNode,+ OrderByNode,+ QueryNode,+ RawNode,+ ReferenceNode,+ SelectModifierNode,+ SelectQueryNode,+ SelectionNode,+ TableNode,+ ValueNode,+ type KyselyPlugin,+ type OperationNode,+ type PluginTransformQueryArgs,+ type PluginTransformResultArgs,+ type QueryId,+ type RootOperationNode,+} from "kysely";++import { QueryCompileError } from "./errors";+import { LimitByNode, type ClickHouseSelectQueryNode } from "./nodes";+import { DEDUP_SPECS, type DedupSpec } from "./schema";+import { stampCompiledTree } from "./tenancy";+import { ClickHouseOperationNodeTransformer } from "./transformer";++type PhysicalFrom = {+ tableName: string;+ alias?: string;+ fromItem: OperationNode;+};++type LimitBySpec = Extract<DedupSpec, { strategy: "limitBy" }>;++const DOUBLE_COUNT_AGGREGATES = new Set(["count", "sum", "avg"]);++export class DedupLoweringPlugin implements KyselyPlugin {+ transformQuery(args: PluginTransformQueryArgs): RootOperationNode {+ const transformer = new DedupLoweringTransformer();+ const lowered = transformer.transformNode(args.node) as RootOperationNode;+ // A rewritten root is a new object; restamp so the compiler will emit.+ return stampCompiledTree(lowered);+ }++ async transformResult(args: PluginTransformResultArgs) {+ return args.result;+ }+}++class DedupLoweringTransformer extends ClickHouseOperationNodeTransformer {+ protected override transformSelectQuery(+ node: SelectQueryNode,+ queryId?: QueryId,+ ): SelectQueryNode {+ const withChildren = super.transformSelectQuery(+ node,+ queryId,+ ) as ClickHouseSelectQueryNode;+ return lowerSelect(withChildren, collectCteNames(withChildren));+ }+}++function collectCteNames(node: SelectQueryNode): Set<string> {+ const names = new Set<string>();+ for (const expr of node.with?.expressions ?? []) {+ names.add(expr.name.table.table.identifier.name);+ }+ return names;+}++function lowerSelect(+ node: ClickHouseSelectQueryNode,+ cteNames: Set<string>,+): ClickHouseSelectQueryNode {+ const from = singlePhysicalFrom(node, cteNames);+ if (!from) return node;++ const spec = DEDUP_SPECS[from.tableName];+ if (!spec || spec.strategy === "none") return node;++ if (spec.strategy === "final") {+ throw new QueryCompileError(+ `Dedup strategy "final" is declared on ${from.tableName} but has no emitter. Use strategy "limitBy" or implement FINAL lowering before declaring it.`,+ );+ }++ if (node.limitBy) {+ // Caller already attached LIMIT BY. Still pin the version sort so the+ // clause keeps the latest row when they forgot ORDER BY <version>.+ return ensureVersionOrder(node, from, spec);+ }++ const shape = classifyShape(node);+ if (shape === "skip") return node;+ if (shape === "aggregate") return wrapPhysicalFrom(node, from, spec);+ return attachRowDedup(node, from, spec);+}++function singlePhysicalFrom(+ node: SelectQueryNode,+ cteNames: Set<string>,+): PhysicalFrom | undefined {+ if (node.joins?.length) return undefined;+ const froms = node.from?.froms ?? [];+ if (froms.length !== 1) return undefined;+ const described = describeFrom(froms[0]);+ if (!described) return undefined;+ if (cteNames.has(described.tableName)) return undefined;+ return described;+}++function describeFrom(node: OperationNode): PhysicalFrom | undefined {+ if (AliasNode.is(node)) {+ const inner = describeFrom(node.node);+ if (!inner) return undefined;+ const alias = IdentifierNode.is(node.alias) ? node.alias.name : undefined;+ return { ...inner, alias, fromItem: node };+ }+ if (TableNode.is(node)) {+ return {+ tableName: node.table.identifier.name,+ fromItem: node,+ };+ }+ return undefined;+}++type QueryShape = "row" | "aggregate" | "skip";++function classifyShape(node: SelectQueryNode): QueryShape {+ if (isExistence(node)) return "skip";+ if (+ isDistinctOnly(node) ||+ node.groupBy ||+ node.having ||+ hasWrappingAggregate(node)+ ) {+ return "aggregate";+ }+ // Scalar max/min/argMax without GROUP BY already collapse versions;+ // attaching ORDER BY / LIMIT BY after the aggregate is illegal SQL.+ if (hasAnyAggregate(node)) return "skip";+ return "row";+}++function isDistinctOnly(node: SelectQueryNode): boolean {+ return (+ node.frontModifiers?.some(+ (modifier) =>+ SelectModifierNode.is(modifier) && modifier.modifier === "Distinct",+ ) ?? false+ );+}++function isExistence(node: SelectQueryNode): boolean {+ if (!isLimitOne(node) || node.groupBy) return false;+ const selections = node.selections ?? [];+ if (selections.length !== 1) return false;+ const expr = unwrapSelection(selections[0]);+ if (!expr) return false;+ if (ValueNode.is(expr) && expr.value === 1) return true;+ return AggregateFunctionNode.is(expr) && isCountFunc(expr.func);+}++function isLimitOne(node: SelectQueryNode): boolean {+ const limit = node.limit?.limit;+ return limit !== undefined && ValueNode.is(limit) && limit.value === 1;+}++function hasWrappingAggregate(node: SelectQueryNode): boolean {+ let found = false;+ walk(node.selections, (child) => {+ if (!AggregateFunctionNode.is(child)) return;+ if (child.over || DOUBLE_COUNT_AGGREGATES.has(child.func.toLowerCase())) {+ found = true;+ }+ });+ return found;+}++function hasAnyAggregate(node: SelectQueryNode): boolean {+ let found = false;+ walk(node.selections, (child) => {+ if (AggregateFunctionNode.is(child)) found = true;+ });+ return found;+}++function isCountFunc(func: string): boolean {+ return func.toLowerCase() === "count";+}++function unwrapSelection(selection: OperationNode): OperationNode | undefined {+ if (!SelectionNode.is(selection)) return undefined;+ const inner = selection.selection;+ if (AliasNode.is(inner)) return inner.node;+ return inner;+}++function attachRowDedup(+ node: ClickHouseSelectQueryNode,+ from: PhysicalFrom,+ spec: LimitBySpec,+): ClickHouseSelectQueryNode {+ // LIMIT BY keeps the first row per key in the query's full sort order.+ // A caller ORDER BY on any other column would make the version a+ // tiebreaker and could keep a stale row β wrap so collapse is inner.+ if (node.orderBy && !hasLeadingVersionDesc(node, spec)) {+ return wrapPhysicalFrom(node, from, spec);+ }+ return {+ ...ensureVersionOrder(node, from, spec),+ limitBy: limitByNode(from, spec),+ };+}++function wrapPhysicalFrom(+ node: ClickHouseSelectQueryNode,+ from: PhysicalFrom,+ spec: LimitBySpec,+): ClickHouseSelectQueryNode {+ const inner: ClickHouseSelectQueryNode = {+ kind: "SelectQueryNode",+ from: FromNode.create([from.fromItem]),+ selections: [SelectionNode.createSelectAll()],+ ...(node.where ? { where: node.where } : {}),+ orderBy: versionOrderBy(from, spec),+ limitBy: limitByNode(from, spec),+ };+ const alias = IdentifierNode.create(from.alias ?? from.tableName);+ return {+ ...node,+ from: FromNode.create([AliasNode.create(inner, alias)]),+ where: undefined,+ };+}++function ensureVersionOrder(+ node: ClickHouseSelectQueryNode,+ from: PhysicalFrom,+ spec: LimitBySpec,+): ClickHouseSelectQueryNode {+ if (orderByHasColumn(node.orderBy, spec.version)) return node;+ return QueryNode.cloneWithOrderByItems(node, [+ versionOrderItem(from, spec),+ ]) as ClickHouseSelectQueryNode;+}++function versionOrderBy(from: PhysicalFrom, spec: LimitBySpec): OrderByNode {+ return OrderByNode.create([versionOrderItem(from, spec)]);+}++function versionOrderItem(+ from: PhysicalFrom,+ spec: LimitBySpec,+): OrderByItemNode {+ return OrderByItemNode.create(+ qualifyColumn(from, spec.version),+ RawNode.createWithSql("desc"),+ );+}++function limitByNode(from: PhysicalFrom, spec: LimitBySpec): LimitByNode {+ return LimitByNode.create(+ ValueNode.createImmediate(1),+ spec.key.map((column) => qualifyColumn(from, column)),+ );+}++function qualifyColumn(from: PhysicalFrom, column: string): OperationNode {+ const col = ColumnNode.create(column);+ return from.alias+ ? ReferenceNode.create(col, TableNode.create(from.alias))+ : col;+}++function hasLeadingVersionDesc(+ node: SelectQueryNode,+ spec: LimitBySpec,+): boolean {+ const first = node.orderBy?.items[0];+ return (+ first !== undefined &&+ columnNameOf(first.orderBy) === spec.version &&+ isDescending(first)+ );+}++function isDescending(item: OrderByItemNode): boolean {+ const direction = item.direction;+ if (!direction || !RawNode.is(direction)) return false;+ return direction.sqlFragments.join("").trim().toLowerCase() === "desc";+}++function orderByHasColumn(+ orderBy: OrderByNode | undefined,+ column: string,+): boolean {+ if (!orderBy) return false;+ return orderBy.items.some((item) => columnNameOf(item.orderBy) === column);+}++function columnNameOf(node: OperationNode): string | undefined {+ if (ColumnNode.is(node)) return node.column.name;+ if (ReferenceNode.is(node) && ColumnNode.is(node.column)) {+ return node.column.column.name;+ }+ return undefined;+}++function walk(value: unknown, visit: (node: OperationNode) => void): void {+ if (!value || typeof value !== "object") return;+ if (+ "kind" in value &&+ typeof (value as { kind: unknown }).kind === "string"+ ) {+ const node = value as OperationNode;+ visit(node);+ // Nested selects have their own shape; do not attribute their+ // aggregates to the outer query.+ if (SelectQueryNode.is(node)) return;+ }+ for (const child of Object.values(value)) {+ if (Array.isArray(child)) {+ for (const item of child) walk(item, visit);+ } else {+ walk(child, visit);+ }+ }+}diff --git a/packages/shared/src/server/query-ast/kysely/schema.test.ts b/packages/shared/src/server/query-ast/kysely/schema.test.tsindex 815486231c..4d502ed386 100644--- a/packages/shared/src/server/query-ast/kysely/schema.test.ts+++ b/packages/shared/src/server/query-ast/kysely/schema.test.ts@@ -1,6 +1,6 @@import { describe, expect, it } from "vitest";-import { COLUMN_DATA_TYPES, TENANTED_TABLES } from "./schema";+import { COLUMN_DATA_TYPES, DEDUP_SPECS, TENANTED_TABLES } from "./schema";import { schemaTypeAssertions, extensionTypeAssertions } from "./types.assert";describe("table registry derivation", () => {@@ -27,6 +27,13 @@ describe("table registry derivation", () => {expect(COLUMN_DATA_TYPES.cost_details).toBe("map");});+ it("declares events_core as immutable and leaves legacy tables undeclared", () => {+ expect(DEDUP_SPECS.events_core).toEqual({ strategy: "none" });+ expect(DEDUP_SPECS.traces).toBeUndefined();+ expect(DEDUP_SPECS.observations).toBeUndefined();+ expect(DEDUP_SPECS.scores).toBeUndefined();+ });+// The assertions themselves are compile-time (`tsc` is the test); this only// anchors the file so its `@ts-expect-error` checks stay in the build graph.it("keeps the compile-time type assertions in the build graph", () => {diff --git a/packages/shared/src/server/query-ast/kysely/schema.ts b/packages/shared/src/server/query-ast/kysely/schema.tsindex 0fd812f599..a4548799a5 100644--- a/packages/shared/src/server/query-ast/kysely/schema.ts+++ b/packages/shared/src/server/query-ast/kysely/schema.ts@@ -1,6 +1,6 @@/*** Single source of truth for the physical ClickHouse relations the Kysely- * compiler targets. One table declaration drives all three downstream views:+ * compiler targets. One table declaration drives the downstream views:** - `ClickHouseDatabase` β the Kysely row types (column autocomplete and typed* comparisons).@@ -8,12 +8,14 @@* pass consults (`typecheck.ts`).* - `TENANTED_TABLES` β the relations the tenancy pass must scope* (`tenancy.ts`).+ * - `DEDUP_SPECS` β the per-table read idiom the lowering pass applies+ * (`dedup.ts`). Each table names an existing production mechanism; the+ * pass does not invent a new one.*- * Deriving all three from one declaration keeps them from drifting apart.+ * Deriving every view from one declaration keeps them from drifting apart.** Column sets cover the relations the compiler targets, not a full schema- * dump. Physical tuning metadata (partition / sort / dedup keys) is not- * modeled here; dedup lowering will extend this same declaration.+ * dump. Partition / sort keys stay unmodeled until a pass needs them.*//**@@ -50,6 +52,24 @@ const RUNTIME_TYPE: Record<ChColumnType, ColumnDataType> = {"Map(String, Float)": "map",};+/**+ * Per-table read idiom. Must match a mechanism already used in production+ * SQL for that table β the pass applies the declaration, it does not invent+ * a new collapse.+ *+ * - `none` β rows are treated as immutable at read time. No FINAL, no+ * LIMIT BY. Used for `events_core` / `events_full`.+ * - `limitBy` β the existing legacy idiom+ * `ORDER BY <version> DESC LIMIT 1 BY <key>` (traces / observations /+ * scores point and list reads).+ * - `final` β the existing `FROM <table> FINAL` idiom. Declaring it+ * without an emitter is a compile error (fail-closed).+ */+export type DedupSpec =+ | { strategy: "none" }+ | { strategy: "limitBy"; key: readonly string[]; version: string }+ | { strategy: "final" };+function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {columns: Cols;/**@@ -59,8 +79,17 @@ function defineTable<const Cols extends Record<string, ChColumnType>>(spec: {* only for a genuinely global relation.*/tenant?: boolean;+ /**+ * Existing read-time collapse for this table. Omit until the family is+ * migrated; `none` is an explicit "do not collapse" (immutable rows).+ */+ dedup?: DedupSpec;}) {- return { columns: spec.columns, tenant: spec.tenant ?? true };+ return {+ columns: spec.columns,+ tenant: spec.tenant ?? true,+ dedup: spec.dedup,+ };}const TABLE_REGISTRY = {@@ -95,6 +124,8 @@ const TABLE_REGISTRY = {metadata_names: "Array(String)",metadata_values: "Array(String)",},+ // Immutable at read time. Do not inject LIMIT BY or FINAL.+ dedup: { strategy: "none" },}),scores: defineTable({columns: {@@ -138,3 +169,10 @@ export const TENANTED_TABLES = new Set<string>(.filter(([, spec]) => spec.tenant).map(([name]) => name),);++/** Derived per-table read idioms, keyed by table name. */+export const DEDUP_SPECS: Record<string, DedupSpec> = Object.fromEntries(+ Object.entries(TABLE_REGISTRY).flatMap(([name, spec]) =>+ spec.dedup ? [[name, spec.dedup]] : [],+ ),+);diff --git a/packages/shared/src/server/query-ast/kysely/tenancy.ts b/packages/shared/src/server/query-ast/kysely/tenancy.tsindex 60dbd7fb83..b2f2c3636b 100644--- a/packages/shared/src/server/query-ast/kysely/tenancy.ts+++ b/packages/shared/src/server/query-ast/kysely/tenancy.ts@@ -315,6 +315,16 @@ function stampTenancy<T extends object>(node: T): T {return node;}+/**+ * Re-stamp a tree after a later rewrite pass. Identity-based: a copied+ * property is not a valid stamp. Dedup lowering (and any future rewrite)+ * must call this on the node it returns, because a new root object is not+ * the object the tenancy pass stamped.+ */+export function stampCompiledTree<T extends object>(node: T): T {+ return stampTenancy(node);+}+export function assertTenancyStamped(node: RootOperationNode): void {if (!TENANCY_STAMPED.has(node)) {throw new QueryCompileError(