The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
[MBQL lib] Start on user docs: top-level and expressions (#70375)
Braden Shepherdson3c152db0a1f5
diff --git a/src/metabase/lib/core.cljc b/src/metabase/lib/core.cljcindex e5c1cc5ccdb..f08809781bc 100644--- a/src/metabase/lib/core.cljc+++ b/src/metabase/lib/core.cljc@@ -1,6 +1,54 @@(ns metabase.lib.core- "Currently this is mostly a convenience namespace for REPL and test usage. We'll probably have a slightly different- version of this for namespace for QB and QP usage in the future -- TBD."+ "Exported interface for MBQL lib (formerly \"MLv2\").++ This namespace re-exports or wraps all the functions which are intentionally exported for use outside of+ `metabase.lib.*`. If you need something that this namespace doesn't provide, ask the Querying Platform team.++ ## Purpose++ This library exists to encapsulate the myriad details it takes to work with MBQL *correctly*. Queries are plain+ Clojure data structures, but they should not be directly read or modified outside of this library. It's too easy to+ introduce a bug by missing cases (*oops, I only handled `:field` refs and not `:expression` refs*), creating malformed+ MBQL, or making inconsistent edits.++ ### Guiding Principles++ - MBQL created by the lib is always correctly structured.+ - Attempting to construct an invalid query is rejected with errors.+ - Focus on high-level, meaningful functions that return a consistent query.+ - E.g. \"remove this expression\" includes a cascading delete of all clauses that referenced it, transitively.++ ## Terminology++ Clarifying some terminology that's used with care in these docs:++ - **query** refers to both MBQL and native queries.+ - **column** refers to any column in a query+ - **field** is the subset of columns which come directly from a table in the user's DWH+ - **metadata** is a map of information about a table, card, measure, segment, or *column*.+ - **Column metadata** is by far the most significant kind of metadata; much of the lib is concerned with this.+ - **clause** refers to a fragment of MBQL, describing e.g. and aggregation, breakout, join, etc.+ - Most clauses take the form of a vector like this: `[:operator-keyword {:options :map} ,,,]`.+ - **ref** means a reference to a column. Much of the lib is concerned with these.+ - We sometimes call these \"field refs\", since they're often represented as a `[:field ...]` clause, but that's+ too specific - `:field` clauses come in several flavours, and there are `:expression` and `:aggregation` refs.+ - *Refs are a code smell.* They are an internal detail of MBQL structures and it would be better if they had not+ leaked out of the lib so extensively that they're a major part of the API surface.++ ## Code health++ This API surface grew haphazardly, and there's a pile of functions exported from here that should not be. Rather than+ try to fix everything up front, we document the *health* of each exported function in its docstring, to guide usage.++ - **Healthy:** No issues; use these functions without concern.+ - **Smelly:** This function isn't going away, but it needs cleanup or improvement. Perhaps it's badly named, or+ somewhat duplicated.+ - **Single use:** Exists to support a specific use case; new calls should be avoided. Ask via code review if you have+ a legitimate use. In time Kondo will flag these as discouraged.+ - **Leak:** This function is legitimate internally, but it should not have been exported. Avoid new calls; remove+ existing calls when practical. Will be unexported when there are no callers left. Docstrings should suggest an+ alternative to calling these functions.+ - **Deprecated:** Stronger than Leak - the function should be removed altogether, not just unexported."(:refer-clojure :exclude [filter remove replace and or not = < <= > ->> >= not-empty case count distinct max min+ - * / time abs concat replace ref var float])(:require@@ -51,9 +99,11 @@[metabase.lib.query.test-spec :as lib.query.test-spec][metabase.lib.ref :as lib.ref][metabase.lib.remove-replace :as lib.remove-replace]- [metabase.lib.schema]+ [metabase.lib.schema :as lib.schema]+ [metabase.lib.schema.common :as lib.schema.common][metabase.lib.schema.expression :as lib.schema.expression][metabase.lib.schema.id :as lib.schema.id]+ [metabase.lib.schema.metadata :as lib.schema.metadata][metabase.lib.schema.util][metabase.lib.segment :as lib.segment][metabase.lib.serialize]@@ -69,6 +119,46 @@[metabase.util.malli :as mu][metabase.util.namespaces :as shared.ns]))+;;; # Principle: Stages+;;; Queries have 1 or more **stages**. Each stage roughly corresponds to a SQL `SELECT`'s several parts:+;;;+;;; - A primary **source** - for stage 0 this is a table or card; for later stages it's implicitly "the previous stage".+;;; - Explicit **joins**, which add new sources and their columns to the query+;;; - **Expressions** (or "custom columns" in the UI) which compute a new column from other columns+;;; - **Filters** are boolean expressions, implicitly `AND`ed together to filter the rows of a query+;;; - **Aggregations** and **breakouts** together form a _summary_ step+;;; - Breakouts generate SQL `GROUP BY`s and `ORDER BY`s; they divide the rows into groups which have all breakout+;;; values in common.+;;; - Aggregations compute a single value from a set of rows - `COUNT`, `SUM(some_column)`, etc.+;;; - **Order by** controls the order of output rows, either ascending or descending+;;; - Leftmost is the most significant, the others are cascading tiebreakers (like SQL `ORDER BY`)+;;; - For an aggregated query, these refer to the *breakouts and aggregations*, not to input columns.+;;; - **Limit** restricts the number of rows returned from the stage+;;; - **Fields** lists allow a query to return only a subset of the available columns.+;;; - In the UI, this is the drop-down list on each source that lets you choose a subset of the columns.+;;; - For a summarized query, you can't choose a subset: all breakouts and aggregations are always returned.+;;;+;;; Inside the lib, stages are addressed by **stage number**. Stages are numbered 0, 1, 2, etc. Negative numbers+;;; count backwards, in the style of Python indexes: **-1 is the last stage**.+;;;+;;; Since the last stage is by far the most common target, many lib functions have an arity which omits the+;;; `stage-number`. It defaults to `-1`, ie. the last stage.+;;;+;;; Note that in the Notebook UI, you cannot arbitrarily add extra stages. The UI only adds stages when necessary,+;;; such as when adding a pre-aggregation operation like an expression or filter *after* the summary step. The lib+;;; is more flexible: **stages can be nested arbitrarily**, and can even be empty.+;;;+;;; # Principle: Implicit joins+;;; MBQL queries support _implicit joins_. These are offered in the UI when picking a column for e.g. a filter.+;;; There is a group of columns for each *foreign key* column that's in scope. Each group contains the columns of the+;;; target table of the FK. The groups are named after the FK, rather than the table - that's important if there are+;;; multiple FKs to the same table, consider Github issues which have `reporter` and `assignee`, both FKs to users but+;;; with a very different meaning.+;;;+;;; Of course databases don't support implicit joins, so the QP constructs explicit join clauses for them transparently.+;;;+;;; Implicit joins are only currently supported where the target is a Field on a Table, not for columns generally.+;;; There's some support for the FK to come from a model and target a Field, but it isn't a first class use case.(comment lib.aggregation/keep-melib.binning/keep-melib.breakout/keep-me@@ -113,7 +203,7 @@lib.query.test-spec/keep-melib.ref/keep-melib.remove-replace/keep-me- metabase.lib.schema/keep-me+ lib.schema/keep-memetabase.lib.schema.util/keep-melib.segment/keep-melib.measure/keep-me@@ -127,6 +217,178 @@metabase.lib.util.unique-name-generator/keep-memetabase.lib.walk.util/keep-me)+;;; # Core APIs for building queries+;;; These are the heart of the Lib API: adding clauses to queries, getting a list of clauses on a stage, removing+;;; and replacing clauses, and getting lists of columns.++;;; For each part of a query - joins, expressions, aggregations, etc. - the lib has a set of functions that follow+;;; a naming convention:+;;;+;;; - Adding something new uses an imperative verb (usually):+;;; - `filter`, `aggregate`, `breakout`, `join`, `order-by`, etc.+;;; - `expression` breaks the pattern+;;; - Getting the list of current ones (on a stage) is a plural noun+;;; - `filters`, `aggregations`, `expressions`, etc.+;;; - The relevant list of columns can be fetched with `fooable-columns`+;;; - `expressionable-columns`, `filterable-columns`, `orderable-columns`+;;;+;;; There is no corresponding verb to delete or update a clause; instead the generic `remove-clause` and+;;; `replace-clause` functions are provided.++;;; ## Expressions+;;; Expressions are written as formulas in the Notebook UI, and given an arbitrary *name* by the user.+;;; The spreadsheet formula is parsed to a tree of `[:operator {} args...]` clauses, where the args can be nested+;;; operators, references to columns, and constants.++(mu/defn expression :- ::lib.schema/query+ "Appends a new expression to the specified stage of `a-query`. The `expression-name` is a required parameter.+ The `expressionable` can be a constant, a column, a column ref, or an arbitrarily nested tree of clauses.++ The columns which are in scope for an expression can be fetched with [[expressionable-columns]]; see that function+ for details.++ Call this if you already have an `expr-clause` and want to attach it to the query directly. For interop with the+ expression parser and pretty-printer in the Notebook editor, see [[expression-clause]] and [[expression-parts]].++ **Code Health:** Healthy. This is a core API."+ ([a-query expression-name expressionable]+ (expression a-query -1 expression-name expressionable))+ ([a-query :- ::lib.schema/query+ stage-number :- [:maybe :int]+ expression-name :- ::lib.schema.common/non-blank-string+ expressionable]+ (lib.expression/expression a-query stage-number expression-name expressionable)))+; TODO: (bshepherdson, 2026-03-03) There's a third arity in `lib.expression/expression` that supports an options map+; but it's a bit "gory details" so it's omitted here. I think it's only called by QP.++(mu/defn expressions :- [:maybe ::lib.schema.expression/expressions]+ "Get the vector of expressions defined on the specified stage of `a-query`, or `nil` if there are no expressions.++ The expressions are represented as possibly nested clauses, the same as can be passed to [[expression]] when creating+ a new expression.++ **Code Health:** Healthy. This is a core API."+ ([a-query] (expressions a-query -1))+ ([a-query :- ::lib.schema/query+ stage-number :- [:maybe :int]]+ (lib.expression/expressions a-query stage-number)))++(mu/defn expressionable-columns :- [:sequential ::lib.schema.metadata/column]+ "Returns column metadata for all the columns that can be used in expressions at the target stage of `a-query`.++ The `expression-position` is needed because *earlier* expressions are visible to *later* expressions. This should+ be the index of an expression being edited, or nil if adding a new expression at the end.++ Columns in scope for an expression:++ 1. Columns from the stage's primary source+ 2. Columns from all explicit joins+ 3. All earlier expressions on this stage, ie. those with a lower `expression-position`.+ 4. All columns which are *implicitly joinable* through any FK in the above.++ **Code Health:** Healthy. This is a core API."+ ([a-query expression-position] (expressionable-columns a-query -1 expression-position))+ ([a-query :- ::lib.schema/query+ stage-number :- [:maybe :int]+ expression-position :- [:maybe ::lib.schema.common/int-greater-than-or-equal-to-zero]]+ (lib.expression/expressionable-columns a-query stage-number expression-position)))++(mu/defn expressions-metadata :- [:maybe [:sequential ::lib.schema.metadata/column]]+ "Similar to [[expressions]], but it returns the column metadata for the *output* columns of each expression on the+ stage, rather than the clauses which *define* those expressions.++ **Code Health:** Leak. Expression columns are properly included in e.g. [[filterable-columns]]; there should not be+ cause to care about the expression columns specifically outside of Lib."+ ([a-query] (expressions-metadata a-query -1))+ ([a-query :- ::lib.schema/query+ stage-number :- [:maybe :int]]+ (lib.expression/expressions-metadata a-query stage-number)))++(mu/defn expression-ref :- :mbql.clause/expression+ "Given `a-query` and an `expression-name`, look up the corresponding expression in the target stage and return a ref+ to it.++ **Code Health:** Deprecated. This is a test helper that crept into the public API."+ ([a-query expression-name] (expression-ref a-query -1 expression-name))+ ([a-query :- ::lib.schema/query+ stage-number :- [:maybe :int]+ expression-name :- ::lib.schema.common/non-blank-string]+ (lib.expression/expression-ref a-query stage-number expression-name)))++(mu/defn with-expression-name :- ::lib.schema.expression/expression+ "Given `an-expression-clause`, returns an updated version of it with its expression name set to `new-name`.+ By *expression name* is meant the user-provided name at the bottom of the editor the user sees for an expression or+ custom aggregation.++ The provided clause can be either an expression or aggregation. For aggregations, this sets the `:display-name`+ option.++ **Code Health:** Healthy."+ [an-expression-clause :- ::lib.schema.expression/expression+ new-name :- :string]+ (lib.expression/with-expression-name an-expression-clause new-name))++;; ### Expression Functions+;; These functions are quite generic, so they are re-exported directly. Each of these functions takes a number of+;; arguments and returns an expression clause.+;;+;; **Code Health:** Leak. Nearly all of these are only used in tests and should Move to a test helper namespace.+(shared.ns/import-fns+ [lib.expression+ ++ -+ *+ /+ case+ coalesce+ abs+ log+ exp+ sqrt+ ceil+ floor+ round+ power+ date+ datetime+ interval+ relative-datetime+ time+ absolute-datetime+ now+ convert-timezone+ get-week+ get-year+ get-month+ get-day+ get-day-of-week+ get-hour+ get-minute+ get-second+ get-quarter+ datetime-add+ datetime-subtract+ concat+ substring+ replace+ regex-match-first+ length+ trim+ ltrim+ rtrim+ upper+ lower+ offset+ text+ today+ split-part+ collate+ integer+ float])++;;; # IGNORE ME!+;;; <img src="https://i.redd.it/1b9z1mv805e61.jpg" />+;;; *These are the leftovers which are not properly wrapped in user documentation yet.*(shared.ns/import-fns[lib.aggregationaggregable-columns@@ -198,64 +460,7 @@[lib.equalityfind-column-for-legacy-reffind-matching-column]- [lib.expression- expression- expressions- expressions-metadata- expressionable-columns- expression-ref- resolve-expression- with-expression-name- +- -- *- /- case- coalesce- abs- log- exp- sqrt- ceil- floor- round- power- date- datetime- interval- relative-datetime- time- absolute-datetime- now- convert-timezone- get-week- get-year- get-month- get-day- get-day-of-week- get-hour- get-minute- get-second- get-quarter- datetime-add- datetime-subtract- concat- substring- replace- regex-match-first- length- trim- ltrim- rtrim- upper- lower- offset- text- today- split-part- collate- integer- float]+[lib.extractioncolumn-extractionsextract@@ -468,7 +673,7 @@rename-joinreplace-clausereplace-join]- [metabase.lib.schema+ [lib.schemanative-only-query?][metabase.lib.schema.utilremove-lib-uuids]@@ -564,7 +769,7 @@by `a-query`.**Code Health:** Discouraged; there are few legitimate use cases for working with raw table IDs outside lib."- [a-query :- :metabase.lib.schema/query]+ [a-query :- ::lib.schema/query](lib.util/source-table-id a-query))(mu/defn primary-source-card-id :- [:maybe ::lib.schema.id/card]@@ -577,7 +782,7 @@by `a-query`.**Code Health:** Discouraged; there are few legitimate use cases for working with raw card IDs outside lib."- [a-query :- :metabase.lib.schema/query]+ [a-query :- ::lib.schema/query](lib.util/source-card-id a-query))(mu/defn display-name-without-id :- :stringdiff --git a/src/metabase/query_processor/pivot.clj b/src/metabase/query_processor/pivot.cljindex 99de9d5cfc8..a34f922ea2d 100644--- a/src/metabase/query_processor/pivot.clj+++ b/src/metabase/query_processor/pivot.clj@@ -9,6 +9,7 @@[metabase.lib-be.core :as lib-be][metabase.lib.core :as lib][metabase.lib.equality :as lib.equality]+ [metabase.lib.expression :as lib.expression][metabase.lib.schema :as lib.schema][metabase.lib.schema.common :as lib.schema.common][metabase.lib.schema.id :as lib.schema.id]@@ -167,8 +168,10 @@[query :- ::lib.schema/querybitmask :- ::bitmask](as-> query query+ ;; [[lib/expression]] is exported but it doesn't include the 5th arg for options, since it's an MBQL internal+ ;; detail. So this imports and calls [[lib.expression/expression]] directly.;;TODO: replace this value with a bitmask or something to indicate the source better- (lib/expression query -1 "pivot-grouping" (lib/abs bitmask) {:add-to-fields? false})+ (lib.expression/expression query -1 "pivot-grouping" (lib/abs bitmask) {:add-to-fields? false});; in PostgreSQL and most other databases, all the expressions must be present in the breakouts. Add a pivot;; grouping expression ref to the breakouts(lib/breakout query (-> (lib/expression-ref query "pivot-grouping")