Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

[MBQL lib] Start on user docs: top-level and expressions (#70375)

Braden Shepherdson3c152db0a1f5
diff --git a/src/metabase/lib/core.cljc b/src/metabase/lib/core.cljc
index e5c1cc5ccdb..f08809781bc 100644
--- a/src/metabase/lib/core.cljc
+++ b/src/metabase/lib/core.cljc
@@ -1,6 +1,54 @@
(ns metabase.lib.core
- "Currently this is mostly a convenience namespace for REPL and test usage. We'll probably have a slightly different
- version of this for namespace for QB and QP usage in the future -- TBD."
+ "Exported interface for MBQL lib (formerly \"MLv2\").
+
+ This namespace re-exports or wraps all the functions which are intentionally exported for use outside of
+ `metabase.lib.*`. If you need something that this namespace doesn't provide, ask the Querying Platform team.
+
+ ## Purpose
+
+ This library exists to encapsulate the myriad details it takes to work with MBQL *correctly*. Queries are plain
+ Clojure data structures, but they should not be directly read or modified outside of this library. It's too easy to
+ introduce a bug by missing cases (*oops, I only handled `:field` refs and not `:expression` refs*), creating malformed
+ MBQL, or making inconsistent edits.
+
+ ### Guiding Principles
+
+ - MBQL created by the lib is always correctly structured.
+ - Attempting to construct an invalid query is rejected with errors.
+ - Focus on high-level, meaningful functions that return a consistent query.
+ - E.g. \"remove this expression\" includes a cascading delete of all clauses that referenced it, transitively.
+
+ ## Terminology
+
+ Clarifying some terminology that's used with care in these docs:
+
+ - **query** refers to both MBQL and native queries.
+ - **column** refers to any column in a query
+ - **field** is the subset of columns which come directly from a table in the user's DWH
+ - **metadata** is a map of information about a table, card, measure, segment, or *column*.
+ - **Column metadata** is by far the most significant kind of metadata; much of the lib is concerned with this.
+ - **clause** refers to a fragment of MBQL, describing e.g. and aggregation, breakout, join, etc.
+ - Most clauses take the form of a vector like this: `[:operator-keyword {:options :map} ,,,]`.
+ - **ref** means a reference to a column. Much of the lib is concerned with these.
+ - We sometimes call these \"field refs\", since they're often represented as a `[:field ...]` clause, but that's
+ too specific - `:field` clauses come in several flavours, and there are `:expression` and `:aggregation` refs.
+ - *Refs are a code smell.* They are an internal detail of MBQL structures and it would be better if they had not
+ leaked out of the lib so extensively that they're a major part of the API surface.
+
+ ## Code health
+
+ This API surface grew haphazardly, and there's a pile of functions exported from here that should not be. Rather than
+ try to fix everything up front, we document the *health* of each exported function in its docstring, to guide usage.
+
+ - **Healthy:** No issues; use these functions without concern.
+ - **Smelly:** This function isn't going away, but it needs cleanup or improvement. Perhaps it's badly named, or
+ somewhat duplicated.
+ - **Single use:** Exists to support a specific use case; new calls should be avoided. Ask via code review if you have
+ a legitimate use. In time Kondo will flag these as discouraged.
+ - **Leak:** This function is legitimate internally, but it should not have been exported. Avoid new calls; remove
+ existing calls when practical. Will be unexported when there are no callers left. Docstrings should suggest an
+ alternative to calling these functions.
+ - **Deprecated:** Stronger than Leak - the function should be removed altogether, not just unexported."
(:refer-clojure :exclude [filter remove replace and or not = < <= > ->> >= not-empty case count distinct max min
+ - * / time abs concat replace ref var float])
(:require
@@ -51,9 +99,11 @@
[metabase.lib.query.test-spec :as lib.query.test-spec]
[metabase.lib.ref :as lib.ref]
[metabase.lib.remove-replace :as lib.remove-replace]
- [metabase.lib.schema]
+ [metabase.lib.schema :as lib.schema]
+ [metabase.lib.schema.common :as lib.schema.common]
[metabase.lib.schema.expression :as lib.schema.expression]
[metabase.lib.schema.id :as lib.schema.id]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
[metabase.lib.schema.util]
[metabase.lib.segment :as lib.segment]
[metabase.lib.serialize]
@@ -69,6 +119,46 @@
[metabase.util.malli :as mu]
[metabase.util.namespaces :as shared.ns]))
+;;; # Principle: Stages
+;;; Queries have 1 or more **stages**. Each stage roughly corresponds to a SQL `SELECT`'s several parts:
+;;;
+;;; - A primary **source** - for stage 0 this is a table or card; for later stages it's implicitly "the previous stage".
+;;; - Explicit **joins**, which add new sources and their columns to the query
+;;; - **Expressions** (or "custom columns" in the UI) which compute a new column from other columns
+;;; - **Filters** are boolean expressions, implicitly `AND`ed together to filter the rows of a query
+;;; - **Aggregations** and **breakouts** together form a _summary_ step
+;;; - Breakouts generate SQL `GROUP BY`s and `ORDER BY`s; they divide the rows into groups which have all breakout
+;;; values in common.
+;;; - Aggregations compute a single value from a set of rows - `COUNT`, `SUM(some_column)`, etc.
+;;; - **Order by** controls the order of output rows, either ascending or descending
+;;; - Leftmost is the most significant, the others are cascading tiebreakers (like SQL `ORDER BY`)
+;;; - For an aggregated query, these refer to the *breakouts and aggregations*, not to input columns.
+;;; - **Limit** restricts the number of rows returned from the stage
+;;; - **Fields** lists allow a query to return only a subset of the available columns.
+;;; - In the UI, this is the drop-down list on each source that lets you choose a subset of the columns.
+;;; - For a summarized query, you can't choose a subset: all breakouts and aggregations are always returned.
+;;;
+;;; Inside the lib, stages are addressed by **stage number**. Stages are numbered 0, 1, 2, etc. Negative numbers
+;;; count backwards, in the style of Python indexes: **-1 is the last stage**.
+;;;
+;;; Since the last stage is by far the most common target, many lib functions have an arity which omits the
+;;; `stage-number`. It defaults to `-1`, ie. the last stage.
+;;;
+;;; Note that in the Notebook UI, you cannot arbitrarily add extra stages. The UI only adds stages when necessary,
+;;; such as when adding a pre-aggregation operation like an expression or filter *after* the summary step. The lib
+;;; is more flexible: **stages can be nested arbitrarily**, and can even be empty.
+;;;
+;;; # Principle: Implicit joins
+;;; MBQL queries support _implicit joins_. These are offered in the UI when picking a column for e.g. a filter.
+;;; There is a group of columns for each *foreign key* column that's in scope. Each group contains the columns of the
+;;; target table of the FK. The groups are named after the FK, rather than the table - that's important if there are
+;;; multiple FKs to the same table, consider Github issues which have `reporter` and `assignee`, both FKs to users but
+;;; with a very different meaning.
+;;;
+;;; Of course databases don't support implicit joins, so the QP constructs explicit join clauses for them transparently.
+;;;
+;;; Implicit joins are only currently supported where the target is a Field on a Table, not for columns generally.
+;;; There's some support for the FK to come from a model and target a Field, but it isn't a first class use case.
(comment lib.aggregation/keep-me
lib.binning/keep-me
lib.breakout/keep-me
@@ -113,7 +203,7 @@
lib.query.test-spec/keep-me
lib.ref/keep-me
lib.remove-replace/keep-me
- metabase.lib.schema/keep-me
+ lib.schema/keep-me
metabase.lib.schema.util/keep-me
lib.segment/keep-me
lib.measure/keep-me
@@ -127,6 +217,178 @@
metabase.lib.util.unique-name-generator/keep-me
metabase.lib.walk.util/keep-me)
+;;; # Core APIs for building queries
+;;; These are the heart of the Lib API: adding clauses to queries, getting a list of clauses on a stage, removing
+;;; and replacing clauses, and getting lists of columns.
+
+;;; For each part of a query - joins, expressions, aggregations, etc. - the lib has a set of functions that follow
+;;; a naming convention:
+;;;
+;;; - Adding something new uses an imperative verb (usually):
+;;; - `filter`, `aggregate`, `breakout`, `join`, `order-by`, etc.
+;;; - `expression` breaks the pattern
+;;; - Getting the list of current ones (on a stage) is a plural noun
+;;; - `filters`, `aggregations`, `expressions`, etc.
+;;; - The relevant list of columns can be fetched with `fooable-columns`
+;;; - `expressionable-columns`, `filterable-columns`, `orderable-columns`
+;;;
+;;; There is no corresponding verb to delete or update a clause; instead the generic `remove-clause` and
+;;; `replace-clause` functions are provided.
+
+;;; ## Expressions
+;;; Expressions are written as formulas in the Notebook UI, and given an arbitrary *name* by the user.
+;;; The spreadsheet formula is parsed to a tree of `[:operator {} args...]` clauses, where the args can be nested
+;;; operators, references to columns, and constants.
+
+(mu/defn expression :- ::lib.schema/query
+ "Appends a new expression to the specified stage of `a-query`. The `expression-name` is a required parameter.
+ The `expressionable` can be a constant, a column, a column ref, or an arbitrarily nested tree of clauses.
+
+ The columns which are in scope for an expression can be fetched with [[expressionable-columns]]; see that function
+ for details.
+
+ Call this if you already have an `expr-clause` and want to attach it to the query directly. For interop with the
+ expression parser and pretty-printer in the Notebook editor, see [[expression-clause]] and [[expression-parts]].
+
+ **Code Health:** Healthy. This is a core API."
+ ([a-query expression-name expressionable]
+ (expression a-query -1 expression-name expressionable))
+ ([a-query :- ::lib.schema/query
+ stage-number :- [:maybe :int]
+ expression-name :- ::lib.schema.common/non-blank-string
+ expressionable]
+ (lib.expression/expression a-query stage-number expression-name expressionable)))
+; TODO: (bshepherdson, 2026-03-03) There's a third arity in `lib.expression/expression` that supports an options map
+; but it's a bit "gory details" so it's omitted here. I think it's only called by QP.
+
+(mu/defn expressions :- [:maybe ::lib.schema.expression/expressions]
+ "Get the vector of expressions defined on the specified stage of `a-query`, or `nil` if there are no expressions.
+
+ The expressions are represented as possibly nested clauses, the same as can be passed to [[expression]] when creating
+ a new expression.
+
+ **Code Health:** Healthy. This is a core API."
+ ([a-query] (expressions a-query -1))
+ ([a-query :- ::lib.schema/query
+ stage-number :- [:maybe :int]]
+ (lib.expression/expressions a-query stage-number)))
+
+(mu/defn expressionable-columns :- [:sequential ::lib.schema.metadata/column]
+ "Returns column metadata for all the columns that can be used in expressions at the target stage of `a-query`.
+
+ The `expression-position` is needed because *earlier* expressions are visible to *later* expressions. This should
+ be the index of an expression being edited, or nil if adding a new expression at the end.
+
+ Columns in scope for an expression:
+
+ 1. Columns from the stage's primary source
+ 2. Columns from all explicit joins
+ 3. All earlier expressions on this stage, ie. those with a lower `expression-position`.
+ 4. All columns which are *implicitly joinable* through any FK in the above.
+
+ **Code Health:** Healthy. This is a core API."
+ ([a-query expression-position] (expressionable-columns a-query -1 expression-position))
+ ([a-query :- ::lib.schema/query
+ stage-number :- [:maybe :int]
+ expression-position :- [:maybe ::lib.schema.common/int-greater-than-or-equal-to-zero]]
+ (lib.expression/expressionable-columns a-query stage-number expression-position)))
+
+(mu/defn expressions-metadata :- [:maybe [:sequential ::lib.schema.metadata/column]]
+ "Similar to [[expressions]], but it returns the column metadata for the *output* columns of each expression on the
+ stage, rather than the clauses which *define* those expressions.
+
+ **Code Health:** Leak. Expression columns are properly included in e.g. [[filterable-columns]]; there should not be
+ cause to care about the expression columns specifically outside of Lib."
+ ([a-query] (expressions-metadata a-query -1))
+ ([a-query :- ::lib.schema/query
+ stage-number :- [:maybe :int]]
+ (lib.expression/expressions-metadata a-query stage-number)))
+
+(mu/defn expression-ref :- :mbql.clause/expression
+ "Given `a-query` and an `expression-name`, look up the corresponding expression in the target stage and return a ref
+ to it.
+
+ **Code Health:** Deprecated. This is a test helper that crept into the public API."
+ ([a-query expression-name] (expression-ref a-query -1 expression-name))
+ ([a-query :- ::lib.schema/query
+ stage-number :- [:maybe :int]
+ expression-name :- ::lib.schema.common/non-blank-string]
+ (lib.expression/expression-ref a-query stage-number expression-name)))
+
+(mu/defn with-expression-name :- ::lib.schema.expression/expression
+ "Given `an-expression-clause`, returns an updated version of it with its expression name set to `new-name`.
+ By *expression name* is meant the user-provided name at the bottom of the editor the user sees for an expression or
+ custom aggregation.
+
+ The provided clause can be either an expression or aggregation. For aggregations, this sets the `:display-name`
+ option.
+
+ **Code Health:** Healthy."
+ [an-expression-clause :- ::lib.schema.expression/expression
+ new-name :- :string]
+ (lib.expression/with-expression-name an-expression-clause new-name))
+
+;; ### Expression Functions
+;; These functions are quite generic, so they are re-exported directly. Each of these functions takes a number of
+;; arguments and returns an expression clause.
+;;
+;; **Code Health:** Leak. Nearly all of these are only used in tests and should Move to a test helper namespace.
+(shared.ns/import-fns
+ [lib.expression
+ +
+ -
+ *
+ /
+ case
+ coalesce
+ abs
+ log
+ exp
+ sqrt
+ ceil
+ floor
+ round
+ power
+ date
+ datetime
+ interval
+ relative-datetime
+ time
+ absolute-datetime
+ now
+ convert-timezone
+ get-week
+ get-year
+ get-month
+ get-day
+ get-day-of-week
+ get-hour
+ get-minute
+ get-second
+ get-quarter
+ datetime-add
+ datetime-subtract
+ concat
+ substring
+ replace
+ regex-match-first
+ length
+ trim
+ ltrim
+ rtrim
+ upper
+ lower
+ offset
+ text
+ today
+ split-part
+ collate
+ integer
+ float])
+
+;;; # IGNORE ME!
+;;; <img src="https://i.redd.it/1b9z1mv805e61.jpg" />
+;;; *These are the leftovers which are not properly wrapped in user documentation yet.*
(shared.ns/import-fns
[lib.aggregation
aggregable-columns
@@ -198,64 +460,7 @@
[lib.equality
find-column-for-legacy-ref
find-matching-column]
- [lib.expression
- expression
- expressions
- expressions-metadata
- expressionable-columns
- expression-ref
- resolve-expression
- with-expression-name
- +
- -
- *
- /
- case
- coalesce
- abs
- log
- exp
- sqrt
- ceil
- floor
- round
- power
- date
- datetime
- interval
- relative-datetime
- time
- absolute-datetime
- now
- convert-timezone
- get-week
- get-year
- get-month
- get-day
- get-day-of-week
- get-hour
- get-minute
- get-second
- get-quarter
- datetime-add
- datetime-subtract
- concat
- substring
- replace
- regex-match-first
- length
- trim
- ltrim
- rtrim
- upper
- lower
- offset
- text
- today
- split-part
- collate
- integer
- float]
+
[lib.extraction
column-extractions
extract
@@ -468,7 +673,7 @@
rename-join
replace-clause
replace-join]
- [metabase.lib.schema
+ [lib.schema
native-only-query?]
[metabase.lib.schema.util
remove-lib-uuids]
@@ -564,7 +769,7 @@
by `a-query`.
**Code Health:** Discouraged; there are few legitimate use cases for working with raw table IDs outside lib."
- [a-query :- :metabase.lib.schema/query]
+ [a-query :- ::lib.schema/query]
(lib.util/source-table-id a-query))
(mu/defn primary-source-card-id :- [:maybe ::lib.schema.id/card]
@@ -577,7 +782,7 @@
by `a-query`.
**Code Health:** Discouraged; there are few legitimate use cases for working with raw card IDs outside lib."
- [a-query :- :metabase.lib.schema/query]
+ [a-query :- ::lib.schema/query]
(lib.util/source-card-id a-query))
(mu/defn display-name-without-id :- :string
diff --git a/src/metabase/query_processor/pivot.clj b/src/metabase/query_processor/pivot.clj
index 99de9d5cfc8..a34f922ea2d 100644
--- a/src/metabase/query_processor/pivot.clj
+++ b/src/metabase/query_processor/pivot.clj
@@ -9,6 +9,7 @@
[metabase.lib-be.core :as lib-be]
[metabase.lib.core :as lib]
[metabase.lib.equality :as lib.equality]
+ [metabase.lib.expression :as lib.expression]
[metabase.lib.schema :as lib.schema]
[metabase.lib.schema.common :as lib.schema.common]
[metabase.lib.schema.id :as lib.schema.id]
@@ -167,8 +168,10 @@
[query :- ::lib.schema/query
bitmask :- ::bitmask]
(as-> query query
+ ;; [[lib/expression]] is exported but it doesn't include the 5th arg for options, since it's an MBQL internal
+ ;; detail. So this imports and calls [[lib.expression/expression]] directly.
;;TODO: replace this value with a bitmask or something to indicate the source better
- (lib/expression query -1 "pivot-grouping" (lib/abs bitmask) {:add-to-fields? false})
+ (lib.expression/expression query -1 "pivot-grouping" (lib/abs bitmask) {:add-to-fields? false})
;; in PostgreSQL and most other databases, all the expressions must be present in the breakouts. Add a pivot
;; grouping expression ref to the breakouts
(lib/breakout query (-> (lib/expression-ref query "pivot-grouping")