Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(query-ast): take the last value of a duplicated metadata key in metadataValue (#18808)

Niklas Semmler2c13b3fa2efd
diff --git a/packages/shared/AGENTS.md b/packages/shared/AGENTS.md
index d03b0e40a0..456d721fbe 100644
--- a/packages/shared/AGENTS.md
+++ b/packages/shared/AGENTS.md
@@ -47,7 +47,7 @@
- Code evaluator dispatcher/error contract: `src/server/evals/codeEvalDispatcherTypes.ts`. Keep provider mappings, user-visible messages, and worker terminal-outcome classification aligned when adding an error code.
- Dashboard/monitor query feature (data model + server-only builder/executor): `src/features/query/*`
- Query-builder AST (server half, WIP): `src/server/query-ast/*` — the Kysely
- ClickHouse dialect (ARRAY JOIN / LIMIT BY / FINAL / metadata indexOf nodes,
+ ClickHouse dialect (ARRAY JOIN / LIMIT BY / FINAL / metadata subscript nodes,
`ExecutionContext` tenancy injection, per-table dedup lowering, virtual views,
catalog parity). Compile only through `compileClickhouseQuery` in
`src/server/query-ast/compile.ts`. SQL correctness is proven by a golden-SQL
diff --git a/packages/shared/src/server/query-ast/README.md b/packages/shared/src/server/query-ast/README.md
index ce6f798df1..bed030cc7c 100644
--- a/packages/shared/src/server/query-ast/README.md
+++ b/packages/shared/src/server/query-ast/README.md
@@ -8,7 +8,7 @@ metadata, `FilterState` embedding) will land under
`packages/shared/src/query-ast/`.
This folder is a compile-only ClickHouse dialect on Kysely 0.28: real
-`OperationNode`s for ARRAY JOIN, LIMIT BY, FINAL, and metadata `indexOf`
+`OperationNode`s for ARRAY JOIN, LIMIT BY, FINAL, and metadata map
subscripts; a mandatory tenancy injection pass keyed on `ExecutionContext`;
per-table dedup lowering from the registry; schema-typed selection; virtual
views as WITH CTEs; catalog parity.
@@ -136,13 +136,13 @@ db.selectFrom("scores as s")
// … from scores as s final
```
-**Select a metadata value** (`metadataValue` — lowers `metadata[key]` to a bound `indexOf` subscript):
+**Select a metadata value** (`metadataValue` — lowers `metadata[key]` to a bound subscript of `metadataMap`, so the last duplicate key wins):
```ts
db.selectFrom("events_core as e")
.select((eb) => [metadataValue("e", "my_key").as("my_val")])
- .where((eb) => eb(metadataValue("e", "my_key"), ">", 2));
-// select metadata_values[indexof(e.metadata_names, {p:String})] as my_val …
+ .where((eb) => eb(metadataValue("e", "my_key"), "=", "x"));
+// select mapfromarrays(arrayreverse(e.metadata_names), arrayreverse(e.metadata_values))[{p:String}] as my_val …
```
## Developing the dialect
@@ -162,7 +162,7 @@ preserve it:
| ARRAY JOIN | Plugin attaches `ArrayJoinNode` as an extra field on `SelectQueryNode`. `ClickHouseOperationNodeTransformer` preserves it. `ClickHouseQueryCompiler.visitSelectQuery` emits it after JOINs / before WHERE. | no |
| LIMIT BY | Plugin attaches `LimitByNode` the same way. Compiler emits it after ORDER BY / before LIMIT. | no |
| FINAL | Plugin wraps a FROM/JOIN table expression in `FinalTableNode`. Compiler emits `final` after that expression (`scores as s final`). `$call(useFinal(["scores"]))`. Tables declared `dedup: none` throw. | no |
-| metadata `indexOf` | Helper builds an `ArrayIndexNode` whose index child is a `FunctionNode` (`indexOf`) over a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no |
+| metadata subscript | Helper builds an `ArrayIndexNode` over the `metadataMap` `FunctionNode`, indexed by a bound `ValueNode` key. Transformer + compiler special-case the node. No plugin. | no |
| Virtual view | Plugin rewrites `selectFrom(viewName)` into a WITH CTE. Outer types only expose the view's selected columns. | no |
These are real node objects whose children are traced Kysely
diff --git a/packages/shared/src/server/query-ast/catalog.golden.test.ts b/packages/shared/src/server/query-ast/catalog.golden.test.ts
index 22bb737d25..f9485f6c6b 100644
--- a/packages/shared/src/server/query-ast/catalog.golden.test.ts
+++ b/packages/shared/src/server/query-ast/catalog.golden.test.ts
@@ -161,3 +161,22 @@ describeWithClickhouseLocal("catalog tier 5-7 execution", () => {
});
}
});
+
+describeWithClickhouseLocal("catalog metadata semantics", () => {
+ it("metadata_group_by takes the last value of a duplicated metadata key", () => {
+ const entry = CATALOG.find((e) => e.id === "metadata_group_by");
+ expect(entry).toBeDefined();
+ if (!entry) return;
+
+ const compiled = compileClickhouseQuery(entry.build(), {
+ projectId: CATALOG_PROJECT_ID,
+ });
+ const executable = substituteNamedParams(compiled.sql, compiled.params);
+ const output = executeClickhouseLocal(`${CATALOG_MEMORY_DDL}
+ INSERT INTO events_core (project_id, metadata_names, metadata_values)
+ VALUES ('${CATALOG_PROJECT_ID}', ['model', 'other', 'model'], ['first', 'x', 'last']);
+ ${executable}`);
+
+ expect(output).toBe("last\t1");
+ });
+});
diff --git a/packages/shared/src/server/query-ast/extensions.test.ts b/packages/shared/src/server/query-ast/extensions.test.ts
index 6ca2018da5..a0fd7c1845 100644
--- a/packages/shared/src/server/query-ast/extensions.test.ts
+++ b/packages/shared/src/server/query-ast/extensions.test.ts
@@ -46,26 +46,26 @@ describe("schema-typed selection (condition 7)", () => {
});
});
-describe("metadata indexOf access (condition 7b)", () => {
+describe("metadata subscript access (condition 7b)", () => {
// These tests use a two-layer strategy. This first test inspects the AST
// itself (serialized to JSON) to prove `metadata[key]` is *lowered* to real
- // operation nodes — an `ArrayIndexNode` subscript around an `indexOf(...)`
+ // operation nodes — an `ArrayIndexNode` subscript over a `mapFromArrays(...)`
// `FunctionNode` — and never a `RawNode` (a raw SQL string). The sibling
// tests below assert the SQL the compiler then emits. Separating them
// isolates "the builder produced the right tree" (composable, escaped,
// parameter-bound) from "the compiler rendered that tree correctly".
- it("lowers metadata[key] to ArrayIndexNode + indexOf FunctionNode, not RawNode", () => {
+ it("lowers metadata[key] to ArrayIndexNode + mapFromArrays FunctionNode, not RawNode", () => {
const qb = getClickhouseKysely()
.selectFrom("events_core as e")
.select((_eb) => [metadataValue("e", "a").as("a")])
- .where((eb) => eb(metadataValue("e", "a"), ">", 2));
+ .where((eb) => eb(metadataValue("e", "a"), "=", "x"));
const node = qb
.withPlugin(new TenancyInjectionPlugin(ctx))
.toOperationNode();
const json = JSON.stringify(node);
expect(json).toContain('"kind":"ArrayIndexNode"');
- expect(json).toContain('"func":"indexOf"');
+ expect(json).toContain('"func":"mapFromArrays"');
expect(json).not.toMatch(/"kind":"RawNode"/);
// `.as("a")` wraps the expression in an AliasNode, so the first selection's
@@ -80,15 +80,17 @@ describe("metadata indexOf access (condition 7b)", () => {
const qb = getClickhouseKysely()
.selectFrom("events_core as e")
.select((_eb) => [metadataValue("e", "a").as("a")])
- .where((eb) => eb(metadataValue("e", "a"), ">", 2));
+ .where((eb) => eb(metadataValue("e", "a"), "=", "x"));
const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);
const lower = compiled.toLowerCase();
- expect(lower).toContain("metadata_values[indexof(e.metadata_names");
+ expect(lower).toContain(
+ "mapfromarrays(arrayreverse(e.metadata_names), arrayreverse(e.metadata_values))[",
+ );
expect(lower).toContain("as a");
expect(Object.values(params)).toContain("a");
- expect(Object.values(params)).toContain(2);
- expect(compiled).not.toMatch(/metadata_names,\s*'a'/);
+ expect(Object.values(params)).toContain("x");
+ expect(compiled).not.toMatch(/\['a'\]/);
});
it("emits the same lowering in HAVING", () => {
@@ -96,12 +98,12 @@ describe("metadata indexOf access (condition 7b)", () => {
.selectFrom("events_core as e")
.select("environment")
.groupBy("environment")
- .having((eb) => eb(metadataValue("e", "a"), ">", 2));
+ .having((eb) => eb(metadataValue("e", "a"), "=", "x"));
const { sql: compiled, params } = compileClickhouseQuery(qb, ctx);
expect(compiled.toLowerCase()).toContain("having");
expect(compiled.toLowerCase()).toContain(
- "metadata_values[indexof(e.metadata_names",
+ "mapfromarrays(arrayreverse(e.metadata_names), arrayreverse(e.metadata_values))[",
);
expect(Object.values(params)).toContain("a");
});
diff --git a/packages/shared/src/server/query-ast/extensions.ts b/packages/shared/src/server/query-ast/extensions.ts
index 5e698c34ea..141ef0a373 100644
--- a/packages/shared/src/server/query-ast/extensions.ts
+++ b/packages/shared/src/server/query-ast/extensions.ts
@@ -468,21 +468,21 @@ export function traceSessionId(): ChExpression<string> {
}
/**
- * Lower `metadata[key]` to a traced array-subscript + `indexOf` node.
- * The key is a bound `ValueNode`, not a SQL literal.
+ * Lower `metadata[key]` to a subscript of {@link metadataMap}, so the last
+ * entry of a duplicated key wins. The key is a bound `ValueNode`, not a SQL
+ * literal.
*/
export function metadataValue(
tableAlias: string,
key: string,
-): ExpressionWrapper<ClickHouseDatabase, "events_core", string | number> {
- return new ExpressionWrapper(
- ArrayIndexNode.create(
- qualifiedColumn(tableAlias, "metadata_values"),
- FunctionNode.create("indexOf", [
- qualifiedColumn(tableAlias, "metadata_names"),
- ValueNode.create(key),
- ]),
- ) as unknown as OperationNode,
+): ChExpression<string> {
+ const column = (name: string) =>
+ new ExpressionWrapper<ClickHouseDatabase, "events_core", string[]>(
+ qualifiedColumn(tableAlias, name),
+ );
+ return mapValue<string>(
+ metadataMap(column("metadata_names"), column("metadata_values")),
+ key,
);
}
diff --git a/packages/shared/src/server/query-ast/fixtures/catalog.ts b/packages/shared/src/server/query-ast/fixtures/catalog.ts
index 54d78d9922..761ff1addc 100644
--- a/packages/shared/src/server/query-ast/fixtures/catalog.ts
+++ b/packages/shared/src/server/query-ast/fixtures/catalog.ts
@@ -333,12 +333,12 @@ export const CATALOG: CatalogEntry[] = [
},
{
// Metadata-map access in projection and GROUP BY. `metadata[key]` lowers to
- // the `metadata_values[indexOf(metadata_names, {key})]` array-pair subscript
- // (ArrayIndexNode), aliased and then grouped by that alias.
+ // a subscript of the reversed name/value map (ArrayIndexNode), so the last
+ // duplicate key wins, aliased and then grouped by that alias.
id: "metadata_group_by",
tier: 6,
referenceSql: `
- SELECT e.metadata_values[indexOf(e.metadata_names, {p1:String})] AS model, count(*) AS n
+ SELECT mapFromArrays(arrayReverse(e.metadata_names), arrayReverse(e.metadata_values))[{p1:String}] AS model, count(*) AS n
FROM events_core AS e
WHERE e.project_id = {p2:String}
GROUP BY model
diff --git a/packages/shared/src/server/query-ast/nodes.ts b/packages/shared/src/server/query-ast/nodes.ts
index f5ef78b6ae..9f2f5c4493 100644
--- a/packages/shared/src/server/query-ast/nodes.ts
+++ b/packages/shared/src/server/query-ast/nodes.ts
@@ -71,7 +71,7 @@ export const LimitByNode = {
/**
* ClickHouse array subscript `arr[index]`. Children are traced nodes
- * (typically a column ref and an `indexOf(...)` FunctionNode), not raw SQL.
+ * (typically a `mapFromArrays(...)` FunctionNode and a bound key), not raw SQL.
*
* Same closed-`kind` constraint as ARRAY JOIN: not a Kysely visitor-map
* kind. The dialect compiler and transformer special-case this object.