Database governance built for humans and agents — controlling changes and access across every major database.
Go cicddata-governancedata-maskingdata-securitydatabase-accessdatabase-governancedevopsdevsecopsflywaygitopsliquibasemongodbmysqloraclepostgresqlschema-migrationssecuritysql-clientsqlservertidb
feat(sql-editor): draw Databricks plans, the last engine explaining without one (#21679)
Danny Xu0af79a27def7
diff --git a/docs/design/query-plan-visualizer.md b/docs/design/query-plan-visualizer.mdindex 4799779851..8f74b33c56 100644--- a/docs/design/query-plan-visualizer.md+++ b/docs/design/query-plan-visualizer.md@@ -28,10 +28,9 @@ This ordering is a judgment call, not a measurement.## 2. Background / problemThe editor explained on 12 engines and drew a picture for 3 (PostgreSQL, SQL-Server, Spanner); it now explains on 18 and draws for 18 (§3.4) — not the same-eighteen, since Databricks explains with no parser to draw its plan, and-BigQuery draws from a plan that arrives with an ordinary run rather than with an-explain. Reaching the+Server, Spanner); it now explains on 18 and draws for 19 (§3.4): every engine+it explains, and BigQuery besides, whose plan arrives with an ordinary run+rather than with an explain. Reaching thepicture takes five steps: Ctrl/Cmd+E → plan rows in the grid → click a small"Visualize Explain" link → Bytebase runs `EXPLAIN` a second time for JSON → aseparate browser tab opens.@@ -359,7 +358,7 @@ regression.| **Doris** | one `Query()`, on demand, cached | graph, drawn from the text plan by the StarRocks parser | none — Doris has no `EXPLAIN ANALYZE` || **Snowflake** | one `Query()`, on demand, cached | graph, with pruning in place of estimates | none — `EXPLAIN` compiles without running, whichever format it names || **Hive** | one `Query()`, on demand, cached | graph, with rows and row widths, no costs | none in the editor |-| **Databricks** | one `Query()`, on demand, cached | text: Spark's `EXPLAIN FORMATTED` has no parser yet | none — `EXPLAIN` never executes what it plans |+| **Databricks** | one `Query()`, on demand, cached | graph, drawn from Spark's FORMATTED plan; shape and detail, no cost or rows | none — `EXPLAIN` never executes what it plans |A plan opened from a result is always an estimate, on every engine. Measuredplans come from a typed `EXPLAIN ANALYZE` or its engine's equivalent, which is a@@ -402,15 +401,26 @@ statistics, and if the slow run was caused by stale statistics that have sincerefreshed, the plan shown is the *good* one, and the reader concludes the querywas never the problem. That is the inverted failure this wording exists to stop.-**One of the eighteen engines the editor explains has no parser**, so there the-plan is text. Databricks is that one, and it is the engine this section had-listed as explaining nothing at all: its driver runs whatever SQL it is handed,-so the capability table refused explain requests to keep an unplanned statement-from running — an explain request skips the read-only validation an ordinary-query gets. It now asks for Spark's `EXPLAIN FORMATTED`, a numbered tree-followed by a block of detail per operator, which is the shape the Redshift and-StarRocks parsers already read, so the graph is a parser away rather than a-question to answer.+**Every engine the editor explains now draws.** Databricks was the last one+without a parser, and the engine this section had first listed as explaining+nothing at all: its driver runs whatever SQL it is handed, so the capability+table refused explain requests to keep an unplanned statement from running — an+explain request skips the read-only validation an ordinary query gets. It asks+for Spark's `EXPLAIN FORMATTED`, a numbered tree followed by a block of detail+per operator, and its parser puts the two back together.++What that plan does not state is cost or rows. Spark reports those from+`EXPLAIN COST`, and only for the logical plan, which is not the plan that ran,+so a Databricks graph ranks nothing: what it carries is the shape and the+detail — whether a filter reached the scan, which partitions it skipped,+whether a join broadcast or shuffled, and where the exchanges are. That is what+a Spark plan is read for.++The engines that remain state no plan at all: CQL has no `EXPLAIN`, and the+document and key-value APIs publish none. Elasticsearch is the one with+something to draw — its Profile API returns a measured tree — but reaching it+means rewriting a request body rather than planning a statement, which is a+different shape of request from every explain here.**Hive and Snowflake draw, and neither publishes what this section assumed.**Hive's `EXPLAIN FORMATTED` answers with one JSON object whose `CBOPlan` isdiff --git a/frontend/src/apps/explain-visualizer/QueryPlanView.test.tsx b/frontend/src/apps/explain-visualizer/QueryPlanView.test.tsxindex a4f400b3e8..06050c11ed 100644--- a/frontend/src/apps/explain-visualizer/QueryPlanView.test.tsx+++ b/frontend/src/apps/explain-visualizer/QueryPlanView.test.tsx@@ -42,6 +42,7 @@ describe("QueryPlanView", () => {[Engine.OCEANBASE]: "oceanbase",[Engine.MONGODB]: "mongodb",[Engine.COCKROACHDB]: "cockroachdb",+ [Engine.DATABRICKS]: "databricks",[Engine.SNOWFLAKE]: "snowflake",[Engine.HIVE]: "hive",[Engine.REDSHIFT]: "redshift",diff --git a/frontend/src/apps/explain-visualizer/QueryPlanView.tsx b/frontend/src/apps/explain-visualizer/QueryPlanView.tsxindex 7d737a430f..949577f19d 100644--- a/frontend/src/apps/explain-visualizer/QueryPlanView.tsx+++ b/frontend/src/apps/explain-visualizer/QueryPlanView.tsx@@ -17,6 +17,10 @@ import {COCKROACHDB_PLAN_INVALID_JSON_MESSAGE,COCKROACHDB_PLAN_NO_PLAN_MESSAGE,} from "./cockroachdb-plan";+import {+ DATABRICKS_PLAN_EMPTY_MESSAGE,+ DATABRICKS_PLAN_NO_PLAN_MESSAGE,+} from "./databricks-plan";import {HIVE_PLAN_EMPTY_MESSAGE,HIVE_PLAN_INVALID_JSON_MESSAGE,@@ -141,6 +145,8 @@ const PARSE_ERROR_KEYS = new Map<string, QueryPlanTranslationKey>([[HIVE_PLAN_EMPTY_MESSAGE, "error.hive-empty"],[HIVE_PLAN_INVALID_JSON_MESSAGE, "error.hive-invalid"],[HIVE_PLAN_NO_PLAN_MESSAGE, "error.hive-no-plan"],+ [DATABRICKS_PLAN_EMPTY_MESSAGE, "error.databricks-empty"],+ [DATABRICKS_PLAN_NO_PLAN_MESSAGE, "error.databricks-no-plan"],[COCKROACHDB_PLAN_EMPTY_MESSAGE, "error.cockroachdb-empty"],[COCKROACHDB_PLAN_INVALID_JSON_MESSAGE, "error.cockroachdb-invalid"],[COCKROACHDB_PLAN_NO_PLAN_MESSAGE, "error.cockroachdb-no-plan"],diff --git a/frontend/src/apps/explain-visualizer/databricks-plan.test.ts b/frontend/src/apps/explain-visualizer/databricks-plan.test.tsnew file mode 100644index 0000000000..391e548cea--- /dev/null+++ b/frontend/src/apps/explain-visualizer/databricks-plan.test.ts@@ -0,0 +1,188 @@+import { describe, expect, test } from "vitest";+import {+ DATABRICKS_PLAN_EMPTY_MESSAGE,+ DATABRICKS_PLAN_NO_PLAN_MESSAGE,+ parseDatabricksPlan,+} from "./databricks-plan";+import {+ PLAN_TOO_DEEP_MESSAGE,+ type PlanNode,+ type PlanTree,+} from "./plan-model";+import broadcastJoin from "./test-data/databricks/broadcast-join.txt?raw";+import existsSubquery from "./test-data/databricks/exists-subquery.txt?raw";+import partitionFilter from "./test-data/databricks/partition-filter.txt?raw";+import singleScan from "./test-data/databricks/single-scan.txt?raw";+import sortLimit from "./test-data/databricks/sort-limit.txt?raw";+import unionAll from "./test-data/databricks/union-all.txt?raw";++/** A chain of `depth` operators, which is what drives the depth limit. */+const nestedPlan = (depth: number): string => {+ const tree = [`${" ".repeat(0)}Project (${depth})`];+ for (let level = 1; level < depth; level += 1) {+ tree.push(`${" ".repeat(level * 3)}+- Filter (${depth - level})`);+ }+ return ["== Physical Plan ==", ...tree].join("\n");+};++const parse = (source: string): PlanTree => {+ const result = parseDatabricksPlan(source);+ if (!result.ok)+ throw new Error(`expected a parsable plan: ${result.message}`);+ return result.tree;+};++const outline = (tree: PlanTree): string[] =>+ tree.nodes.map(+ (node) => `${node.id} ${node.nodeType} ${node.subject ?? "-"}`+ );++const propertyValue = (node: PlanNode, label: string): string | undefined =>+ node.properties.find((property) => property.label === label)?.value;++describe("parseDatabricksPlan", () => {+ test("reads the tree and the detail block that belongs to each operator", () => {+ const tree = parse(singleScan);+ expect(outline(tree)).toEqual([+ "0 Filter -",+ "0.0 ColumnarToRow -",+ "0.0.0 Scan parquet spark_catalog.default.customers",+ ]);+ const scan = tree.nodes[2];+ // What the read pushed down, which is the first thing a Spark plan is read+ // for.+ expect(propertyValue(scan, "Pushed filters")).toBe(+ "[IsNotNull(city), EqualTo(city,berlin)]"+ );+ expect(propertyValue(scan, "Read schema")).toBe("struct<city:string>");+ // Spark counts what it lists in the label; the count is in the value, and+ // the column carries the expression id of the session that planned it.+ expect(propertyValue(scan, "Output")).toMatch(/^\[city#\d+\]$/);+ expect(propertyValue(tree.root, "Condition")).toMatch(+ /^\(isnotnull\(city#\d+\) AND \(city#\d+ = berlin\)\)$/+ );+ // The operator number Spark's own output prints, and the stage it fused it+ // into.+ expect(propertyValue(tree.root, "Operator")).toBe("3");+ expect(propertyValue(tree.root, "Codegen")).toBe("codegen id : 1");+ });++ test("resolves a read's table from the name Spark qualifies", () => {+ const tree = parse(singleScan);+ expect(tree.nodes[2].relation).toEqual({+ database: "spark_catalog",+ schema: "default",+ table: "customers",+ });+ // An operator that reads no table has none to look up.+ expect(tree.root.relation).toBeUndefined();+ });++ test("follows both sides of a join, and the exchange under one", () => {+ const tree = parse(broadcastJoin);+ expect(outline(tree)).toEqual([+ // FORMATTED prints the operator alone on the tree line; the isFinalPlan+ // flag a plain EXPLAIN puts beside it is in the detail block instead,+ // which the node carries as a property.+ "0 AdaptiveSparkPlan -",+ "0.0 HashAggregate -",+ "0.0.0 Exchange -",+ "0.0.0.0 HashAggregate -",+ "0.0.0.0.0 Project -",+ "0.0.0.0.0.0 BroadcastHashJoin Inner BuildLeft",+ "0.0.0.0.0.0.0 BroadcastExchange -",+ "0.0.0.0.0.0.0.0 Filter -",+ "0.0.0.0.0.0.0.0.0 Scan parquet spark_catalog.default.customers",+ "0.0.0.0.0.0.1 Filter -",+ "0.0.0.0.0.0.1.0 Scan parquet spark_catalog.default.orders",+ ]);+ const join = tree.nodes[5];+ // Spark suffixes a column with the expression id of the session that+ // planned it, so the shape is what is worth pinning.+ expect(propertyValue(join, "Left keys")).toMatch(/^\[id#\d+\]$/);+ expect(propertyValue(join, "Join type")).toBe("Inner");+ expect(propertyValue(tree.root, "Arguments")).toBe("isFinalPlan=false");+ });++ test("draws both branches of a union", () => {+ const tree = parse(unionAll);+ expect(tree.nodes[1].nodeType).toBe("Union");+ expect(tree.nodes[1].children).toHaveLength(2);+ expect(+ tree.nodes.filter((node) => node.nodeType === "Scan parquet")+ ).toHaveLength(2);+ });++ test("keeps the partitions a read skipped", () => {+ const tree = parse(partitionFilter);+ const scan = tree.nodes.find((node) => node.relation?.table === "events");+ expect(propertyValue(scan ?? tree.root, "Partition filters")).toContain(+ "day"+ );+ });++ test("reads a semi join, and a sort with its fetch", () => {+ expect(+ outline(parse(existsSubquery)).some((line) =>+ line.includes("BroadcastHashJoin LeftSemi BuildRight")+ )+ ).toBe(true);+ expect(+ outline(parse(sortLimit)).some((line) =>+ line.includes("TakeOrderedAndProject")+ )+ ).toBe(true);+ });++ test("ranks nothing, since FORMATTED states neither cost nor rows", () => {+ const tree = parse(broadcastJoin);+ expect(tree.estimates.cost).toBe(false);+ expect(tree.estimates.rows).toBe(false);+ expect(tree.measured.rows).toBe(false);+ expect(tree.root.totalCost).toBeUndefined();+ expect(tree.root.rows).toBeUndefined();+ });++ test("reads the indent from the markers, not from the operator's own text", () => {+ // An operator can print an expression holding the same characters a branch+ // does, and taking that for an indent would leave the read below it out of+ // the tree.+ const tree = parse(+ [+ "== Physical Plan ==",+ "Project [cast((a+-b) as int) AS x] (2)",+ "+- Scan parquet spark_catalog.default.t (1)",+ ].join("\n")+ );+ expect(outline(tree)).toEqual([+ "0 Project [cast((a+-b) as int) AS x]",+ "0.0 Scan parquet spark_catalog.default.t",+ ]);+ });++ test("says so when there is no physical plan to read", () => {+ expect(parseDatabricksPlan("")).toEqual({+ ok: false,+ message: DATABRICKS_PLAN_EMPTY_MESSAGE,+ });+ expect(parseDatabricksPlan("not a plan")).toEqual({+ ok: false,+ message: DATABRICKS_PLAN_NO_PLAN_MESSAGE,+ });+ expect(parseDatabricksPlan("{}")).toEqual({+ ok: false,+ message: DATABRICKS_PLAN_NO_PLAN_MESSAGE,+ });+ // The header alone, with every operator line missing its number.+ expect(+ parseDatabricksPlan("== Physical Plan ==\n* Filter\n+- Scan parquet t\n")+ ).toEqual({ ok: false, message: DATABRICKS_PLAN_NO_PLAN_MESSAGE });+ });++ test("refuses a plan deeper than the viewer draws", () => {+ expect(parseDatabricksPlan(nestedPlan(600))).toEqual({+ ok: false,+ message: PLAN_TOO_DEEP_MESSAGE,+ });+ });+});diff --git a/frontend/src/apps/explain-visualizer/databricks-plan.ts b/frontend/src/apps/explain-visualizer/databricks-plan.tsnew file mode 100644index 0000000000..0055181f5f--- /dev/null+++ b/frontend/src/apps/explain-visualizer/databricks-plan.ts@@ -0,0 +1,223 @@+import {+ addPlanProperty,+ checkPlanDepth,+ type PlanNode,+ type PlanParseResult,+ type PlanProperty,+ type PlanRelation,+ parseWithDepthLimit,+ planLabel,+} from "./plan-model";++/**+ * Maps Spark's `EXPLAIN FORMATTED` output onto the engine-neutral plan model.+ *+ * Databricks SQL is Spark SQL, and FORMATTED answers in two parts: a tree of+ * operators, each tagged with a number, and then a block of detail per number.+ * The tree gives the shape, and the blocks give what each operator does — the+ * filters pushed into a read, the partitions it skipped, the keys a join+ * matched on. This parser puts the two back together.+ *+ * It states neither cost nor rows. Spark reports those from `EXPLAIN COST`,+ * and only for the logical plan, which is not the plan that ran; so nothing+ * here is ranked and what a reader gets is the shape and the detail. That is+ * what a Spark plan is read for anyway: whether a filter reached the scan,+ * whether a join broadcast or shuffled, and where the exchanges are.+ */++const PLAN_HEADER = "== Physical Plan ==";++/**+ * An operator in the tree, ending in the number its detail block carries:+ * `+- * ColumnarToRow (2)`. The `*` marks an operator Spark fused into a+ * whole-stage codegen block.+ */+const TREE_OPERATOR = /^([\s:+|-]*)(\*\s)?\s*(\S.*?)\s+\((\d+)\)\s*$/;++/** Where a tree line branches, which is how deep its operator sits. */+const BRANCH = /[+:]-/;++/** A detail block's header: `(3) Filter [codegen id : 1]`. */+const DETAIL_HEADER = /^\((\d+)\)\s+(\S.*?)(?:\s+\[([^\]]*)\])?\s*$/;++/**+ * A line of detail: `PushedFilters: [IsNotNull(city)]`. Spark counts what it+ * lists in the label — `Output [2]:` — and the count is already in the value it+ * introduces, so the label keeps the name alone.+ */+const DETAIL_LINE = /^([A-Z][A-Za-z ]*?)(?:\s*\[[^\]]*\])?\s*:\s*(.*)$/;++/** A read, which names its format and then its table: `Scan parquet db.t`. */+const SCAN = /^Scan\s+(\S+)\s+(\S+)$/;++/** Spark writes a table as `catalog.schema.table`. */+const QUALIFIED_TABLE = /^([^.]+)\.([^.]+)\.([^.]+)$/;++export const DATABRICKS_PLAN_EMPTY_MESSAGE =+ "Databricks returned no query plan for this statement.";+export const DATABRICKS_PLAN_NO_PLAN_MESSAGE =+ "The query plan has no physical plan in it. EXPLAIN FORMATTED output is expected.";++interface Detail {+ readonly name: string;+ readonly codegen?: string;+ readonly properties: PlanProperty[];+}++interface Parsed {+ readonly indent: number;+ readonly id: string;+ readonly nodeType: string;+ readonly subject?: string;+ readonly relation?: PlanRelation;+ readonly children: Parsed[];+}++/**+ * How deep a tree line sits: where its branch marker is, or above everything+ * when it has none, which is the root. Only the markers before the operator are+ * read, so an operator whose own text held one could not be taken for a node+ * deeper than it is.+ */+function indentOf(prefix: string): number {+ const branch = prefix.search(BRANCH);+ return branch >= 0 ? branch : -1;+}++/** The table a read names, where Spark qualifies it fully. */+function toRelation(table: string): PlanRelation | undefined {+ const qualified = QUALIFIED_TABLE.exec(table);+ if (!qualified) return undefined;+ // Spark's catalog holds schemas, which hold tables; Databricks calls the+ // first two a catalog and a schema as well.+ return { database: qualified[1], schema: qualified[2], table: qualified[3] };+}++function parseOperator(line: string): Parsed | undefined {+ const matched = TREE_OPERATOR.exec(line);+ if (!matched) return undefined;+ const [, prefix, , text, id] = matched;++ const scan = SCAN.exec(text);+ if (scan) {+ // A read says which format it read and which table, and both belong on the+ // card: the format is how the operator differs from another read.+ return {+ indent: indentOf(prefix),+ id,+ nodeType: `Scan ${scan[1]}`,+ subject: scan[2],+ relation: toRelation(scan[2]),+ children: [],+ };+ }+ // Everything else is an operator name, and then whatever Spark prints beside+ // it: a join's type and which side it built.+ const space = text.indexOf(" ");+ return {+ indent: indentOf(prefix),+ id,+ nodeType: space > 0 ? text.slice(0, space) : text,+ subject: space > 0 ? text.slice(space + 1) : undefined,+ children: [],+ };+}++/** The detail blocks, by the number their operator carries in the tree. */+function parseDetails(lines: string[]): Map<string, Detail> {+ const details = new Map<string, Detail>();+ let current: Detail | undefined;+ for (const line of lines) {+ const header = DETAIL_HEADER.exec(line);+ if (header) {+ current = { name: header[2], codegen: header[3], properties: [] };+ details.set(header[1], current);+ continue;+ }+ if (!current || !line.trim()) continue;+ const detail = DETAIL_LINE.exec(line);+ if (detail) {+ addPlanProperty(+ current.properties,+ planLabel(detail[1].trim()),+ detail[2].trim()+ );+ } else {+ // A line the block continues onto, such as the rest of a long condition.+ addPlanProperty(current.properties, "Details", line.trim());+ }+ }+ return details;+}++function toPlanNode(+ parsed: Parsed,+ details: Map<string, Detail>,+ id: string,+ depth: number+): PlanNode {+ checkPlanDepth(depth);+ const children = parsed.children.map((child, index) =>+ toPlanNode(child, details, `${id}.${index}`, depth + 1)+ );+ const detail = details.get(parsed.id);+ const properties: PlanProperty[] = [];+ // The number Spark's own plan prints for this operator, so a reader can line+ // the node up with the output they may also be looking at.+ addPlanProperty(properties, "Operator", parsed.id);+ if (detail?.codegen) {+ addPlanProperty(properties, "Codegen", detail.codegen);+ }+ for (const property of detail?.properties ?? []) {+ addPlanProperty(properties, property.label, property.value);+ }+ return {+ id,+ nodeType: parsed.nodeType,+ subject: parsed.subject,+ relation: parsed.relation,+ properties,+ warnings: [],+ children,+ };+}++export function parseDatabricksPlan(source: string): PlanParseResult {+ if (!source.trim()) {+ return { ok: false, message: DATABRICKS_PLAN_EMPTY_MESSAGE };+ }++ const lines = source.split("\n");+ const start = lines.findIndex((line) => line.trim() === PLAN_HEADER);+ if (start < 0) {+ return { ok: false, message: DATABRICKS_PLAN_NO_PLAN_MESSAGE };+ }++ let root: Parsed | undefined;+ // The operators from the root down to the one being read, so a line's indent+ // says which of them it belongs under.+ const open: Parsed[] = [];+ let at = start + 1;+ for (; at < lines.length; at++) {+ const line = lines[at];+ if (!line.trim()) continue;+ // The tree ends where the detail blocks begin.+ if (DETAIL_HEADER.test(line)) break;+ const parsed = parseOperator(line);+ if (!parsed) continue;+ while (open.length > 0 && open[open.length - 1].indent >= parsed.indent) {+ open.pop();+ }+ const parent = open[open.length - 1];+ if (parent) parent.children.push(parsed);+ else if (!root) root = parsed;+ else break;+ open.push(parsed);+ }+ if (!root) {+ return { ok: false, message: DATABRICKS_PLAN_NO_PLAN_MESSAGE };+ }++ const details = parseDetails(lines.slice(at));+ return parseWithDepthLimit(() => toPlanNode(root, details, "0", 0));+}diff --git a/frontend/src/apps/explain-visualizer/plan-i18n.tsx b/frontend/src/apps/explain-visualizer/plan-i18n.tsxindex 3bcc5ea39f..828497715c 100644--- a/frontend/src/apps/explain-visualizer/plan-i18n.tsx+++ b/frontend/src/apps/explain-visualizer/plan-i18n.tsx@@ -34,6 +34,10 @@ const ENGLISH = {"The query plan is not valid JSON. EXPLAIN json = 1 output is expected.","error.clickhouse-no-plan":'The ClickHouse query plan JSON does not contain a "Plan" object.',+ "error.databricks-empty":+ "Databricks returned no query plan for this statement.",+ "error.databricks-no-plan":+ "The query plan has no physical plan in it. EXPLAIN FORMATTED output is expected.","error.cockroachdb-empty":"CockroachDB returned no query plan for this statement.","error.cockroachdb-invalid":diff --git a/frontend/src/apps/explain-visualizer/plan-parsers.ts b/frontend/src/apps/explain-visualizer/plan-parsers.tsindex 2016065301..feeca409b9 100644--- a/frontend/src/apps/explain-visualizer/plan-parsers.ts+++ b/frontend/src/apps/explain-visualizer/plan-parsers.ts@@ -3,6 +3,7 @@ import type { VisualizerEngine } from "@/utils/queryPlanEngines";import { parseBigqueryPlan } from "./bigquery-plan";import { parseClickhousePlan } from "./clickhouse-plan";import { parseCockroachdbPlan } from "./cockroachdb-plan";+import { parseDatabricksPlan } from "./databricks-plan";import { parseHivePlan } from "./hive-plan";import { parseMongodbPlan } from "./mongodb-plan";import { parseMssqlPlan } from "./mssql-plan";@@ -58,6 +59,9 @@ const PARSERS: Record<VisualizerEngine, (source: string) => PlanParseResult> = {// anything else. StarRocks forked from Doris and their plans still agree on// the fragments and the tree, so one parser reads both.[Engine.REDSHIFT]: parseRedshiftPlan,+ // Databricks is Spark, whose FORMATTED plan is a tree of numbered operators+ // and then a block of detail per number.+ [Engine.DATABRICKS]: parseDatabricksPlan,[Engine.STARROCKS]: parseStarrocksPlan,[Engine.DORIS]: parseStarrocksPlan,};diff --git a/frontend/src/apps/explain-visualizer/test-data/README.md b/frontend/src/apps/explain-visualizer/test-data/README.mdindex 1237a566af..d412fd30f8 100644--- a/frontend/src/apps/explain-visualizer/test-data/README.md+++ b/frontend/src/apps/explain-visualizer/test-data/README.md@@ -377,3 +377,27 @@ statement it has not run.| File | Shape it covers || --- | --- || `filtered-aggregate.json` | Two stages: the input reads a table, aggregates and writes a shuffle, and the output reads that shuffle and writes the query's rows |++## `databricks/`++`EXPLAIN FORMATTED` output from Spark 3.5.3, which is what Databricks SQL runs,+against `customers`, `orders` and a partitioned `events`. Databricks itself is+reachable from neither a dev box nor CI; the statement is identical and the+layout is Spark's own, though a Databricks runtime may name operators these+fixtures do not show. The plan states neither cost nor rows: `EXPLAIN COST`+reports those, and only for the logical plan.++| File | Shape it covers |+| --- | --- |+| `single-scan.txt` | A filter over a read, with the predicate pushed into it |+| `broadcast-join.txt` | A broadcast hash join under two aggregates and an exchange, with the build side's own exchange |+| `union-all.txt` | A union of two reads, the one tree with two branches |+| `sort-limit.txt` | A sort with its fetch, as `TakeOrderedAndProject` |+| `partition-filter.txt` | A read of a partitioned table, stating the partitions it skipped |+| `exists-subquery.txt` | An EXISTS turned into a left semi join |++To add one, `EXPLAIN FORMATTED <statement>` through the same client. Spark+suffixes each column with the expression id of the session that planned it, so+a re-capture renumbers them; the tests match the shape of those values rather+than the numbers.+diff --git a/frontend/src/apps/explain-visualizer/test-data/databricks/broadcast-join.txt b/frontend/src/apps/explain-visualizer/test-data/databricks/broadcast-join.txtnew file mode 100644index 0000000000..394d140f52--- /dev/null+++ b/frontend/src/apps/explain-visualizer/test-data/databricks/broadcast-join.txt@@ -0,0 +1,71 @@+== Physical Plan ==+AdaptiveSparkPlan (11)++- HashAggregate (10)+ +- Exchange (9)+ +- HashAggregate (8)+ +- Project (7)+ +- BroadcastHashJoin Inner BuildLeft (6)+ :- BroadcastExchange (3)+ : +- Filter (2)+ : +- Scan parquet spark_catalog.default.customers (1)+ +- Filter (5)+ +- Scan parquet spark_catalog.default.orders (4)+++(1) Scan parquet spark_catalog.default.customers+Output [2]: [id#47, city#49]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/customers]+PushedFilters: [IsNotNull(city), EqualTo(city,berlin), IsNotNull(id)]+ReadSchema: struct<id:int,city:string>++(2) Filter+Input [2]: [id#47, city#49]+Condition : ((isnotnull(city#49) AND (city#49 = berlin)) AND isnotnull(id#47))++(3) BroadcastExchange+Input [2]: [id#47, city#49]+Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=151]++(4) Scan parquet spark_catalog.default.orders+Output [1]: [customer_id#60]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/orders]+PushedFilters: [IsNotNull(customer_id)]+ReadSchema: struct<customer_id:int>++(5) Filter+Input [1]: [customer_id#60]+Condition : isnotnull(customer_id#60)++(6) BroadcastHashJoin+Left keys [1]: [id#47]+Right keys [1]: [customer_id#60]+Join type: Inner+Join condition: None++(7) Project+Output [1]: [city#49]+Input [3]: [id#47, city#49, customer_id#60]++(8) HashAggregate+Input [1]: [city#49]+Keys [1]: [city#49]+Functions [1]: [partial_count(1)]+Aggregate Attributes [1]: [count#64L]+Results [2]: [city#49, count#65L]++(9) Exchange+Input [2]: [city#49, count#65L]+Arguments: hashpartitioning(city#49, 200), ENSURE_REQUIREMENTS, [plan_id=156]++(10) HashAggregate+Input [2]: [city#49, count#65L]+Keys [1]: [city#49]+Functions [1]: [count(1)]+Aggregate Attributes [1]: [count(1)#58L]+Results [2]: [city#49, count(1)#58L AS count(1)#62L]++(11) AdaptiveSparkPlan+Output [2]: [city#49, count(1)#62L]+Arguments: isFinalPlan=falsediff --git a/frontend/src/apps/explain-visualizer/test-data/databricks/exists-subquery.txt b/frontend/src/apps/explain-visualizer/test-data/databricks/exists-subquery.txtnew file mode 100644index 0000000000..958558a29d--- /dev/null+++ b/frontend/src/apps/explain-visualizer/test-data/databricks/exists-subquery.txt@@ -0,0 +1,33 @@+== Physical Plan ==+AdaptiveSparkPlan (5)++- BroadcastHashJoin LeftSemi BuildRight (4)+ :- Scan parquet spark_catalog.default.customers (1)+ +- BroadcastExchange (3)+ +- Scan parquet spark_catalog.default.orders (2)+++(1) Scan parquet spark_catalog.default.customers+Output [1]: [id#47]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/customers]+ReadSchema: struct<id:int>++(2) Scan parquet spark_catalog.default.orders+Output [1]: [customer_id#60]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/orders]+ReadSchema: struct<customer_id:int>++(3) BroadcastExchange+Input [1]: [customer_id#60]+Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=275]++(4) BroadcastHashJoin+Left keys [1]: [id#47]+Right keys [1]: [customer_id#60]+Join type: LeftSemi+Join condition: None++(5) AdaptiveSparkPlan+Output [1]: [id#47]+Arguments: isFinalPlan=falsediff --git a/frontend/src/apps/explain-visualizer/test-data/databricks/partition-filter.txt b/frontend/src/apps/explain-visualizer/test-data/databricks/partition-filter.txtnew file mode 100644index 0000000000..a10a1cc865--- /dev/null+++ b/frontend/src/apps/explain-visualizer/test-data/databricks/partition-filter.txt@@ -0,0 +1,19 @@+== Physical Plan ==+* Project (3)++- * ColumnarToRow (2)+ +- Scan parquet spark_catalog.default.events (1)+++(1) Scan parquet spark_catalog.default.events+Output [3]: [id#91, kind#92, day#93]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/events/day=2026-10-27]+PartitionFilters: [isnotnull(day#93), (day#93 = 2026-10-27)]+ReadSchema: struct<id:int,kind:string>++(2) ColumnarToRow [codegen id : 1]+Input [3]: [id#91, kind#92, day#93]++(3) Project [codegen id : 1]+Output [2]: [id#91, kind#92]+Input [3]: [id#91, kind#92, day#93]diff --git a/frontend/src/apps/explain-visualizer/test-data/databricks/single-scan.txt b/frontend/src/apps/explain-visualizer/test-data/databricks/single-scan.txtnew file mode 100644index 0000000000..ee7fcc2760--- /dev/null+++ b/frontend/src/apps/explain-visualizer/test-data/databricks/single-scan.txt@@ -0,0 +1,19 @@+== Physical Plan ==+* Filter (3)++- * ColumnarToRow (2)+ +- Scan parquet spark_catalog.default.customers (1)+++(1) Scan parquet spark_catalog.default.customers+Output [1]: [city#49]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/customers]+PushedFilters: [IsNotNull(city), EqualTo(city,berlin)]+ReadSchema: struct<city:string>++(2) ColumnarToRow [codegen id : 1]+Input [1]: [city#49]++(3) Filter [codegen id : 1]+Input [1]: [city#49]+Condition : (isnotnull(city#49) AND (city#49 = berlin))diff --git a/frontend/src/apps/explain-visualizer/test-data/databricks/sort-limit.txt b/frontend/src/apps/explain-visualizer/test-data/databricks/sort-limit.txtnew file mode 100644index 0000000000..f1b4690842--- /dev/null+++ b/frontend/src/apps/explain-visualizer/test-data/databricks/sort-limit.txt@@ -0,0 +1,18 @@+== Physical Plan ==+TakeOrderedAndProject (3)++- * ColumnarToRow (2)+ +- Scan parquet spark_catalog.default.customers (1)+++(1) Scan parquet spark_catalog.default.customers+Output [1]: [city#49]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/customers]+ReadSchema: struct<city:string>++(2) ColumnarToRow [codegen id : 1]+Input [1]: [city#49]++(3) TakeOrderedAndProject+Input [1]: [city#49]+Arguments: 5, [city#49 ASC NULLS FIRST], [city#49]diff --git a/frontend/src/apps/explain-visualizer/test-data/databricks/union-all.txt b/frontend/src/apps/explain-visualizer/test-data/databricks/union-all.txtnew file mode 100644index 0000000000..cb59fe2541--- /dev/null+++ b/frontend/src/apps/explain-visualizer/test-data/databricks/union-all.txt@@ -0,0 +1,23 @@+== Physical Plan ==+* ColumnarToRow (4)++- Union (3)+ :- Scan parquet spark_catalog.default.customers (1)+ +- Scan parquet spark_catalog.default.customers (2)+++(1) Scan parquet spark_catalog.default.customers+Output [1]: [city#49]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/customers]+ReadSchema: struct<city:string>++(2) Scan parquet spark_catalog.default.customers+Output [1]: [name#81]+Batched: true+Location: InMemoryFileIndex [file:/tmp/warehouse2/customers]+ReadSchema: struct<name:string>++(3) Union++(4) ColumnarToRow [codegen id : 1]+Input [1]: [city#49]diff --git a/frontend/src/locales/en-US.json b/frontend/src/locales/en-US.jsonindex 00a5f11c20..1e0f6da45c 100644--- a/frontend/src/locales/en-US.json+++ b/frontend/src/locales/en-US.json@@ -3611,6 +3611,8 @@"cockroachdb-empty": "CockroachDB returned no query plan for this statement.","cockroachdb-invalid": "The query plan is not valid JSON. The diagram EXPLAIN (DISTSQL) encodes is expected.","cockroachdb-no-plan": "The query plan JSON contains no \"processors\" to draw.",+ "databricks-empty": "Databricks returned no query plan for this statement.",+ "databricks-no-plan": "The query plan has no physical plan in it. EXPLAIN FORMATTED output is expected.","hive-empty": "Hive returned no query plan for this statement.","hive-invalid": "The query plan is not valid JSON. Hive's EXPLAIN FORMATTED output is expected.","hive-no-plan": "The query plan has no CBO plan to draw. Hive states one only when its cost-based optimizer planned the statement.",diff --git a/frontend/src/locales/es-ES.json b/frontend/src/locales/es-ES.jsonindex 48d8ec6193..99e7763a3c 100644--- a/frontend/src/locales/es-ES.json+++ b/frontend/src/locales/es-ES.json@@ -3611,6 +3611,8 @@"cockroachdb-empty": "CockroachDB no devolvió ningún plan de consulta para esta sentencia.","cockroachdb-invalid": "El plan de consulta no es JSON válido. Se espera el diagrama que codifica EXPLAIN (DISTSQL).","cockroachdb-no-plan": "El JSON del plan de consulta no contiene \"processors\" que dibujar.",+ "databricks-empty": "Databricks no devolvió ningún plan de consulta para esta sentencia.",+ "databricks-no-plan": "El plan de consulta no contiene un plan físico. Se espera la salida de EXPLAIN FORMATTED.","hive-empty": "Hive no devolvió ningún plan de consulta para esta sentencia.","hive-invalid": "El plan de consulta no es JSON válido. Se espera la salida de EXPLAIN FORMATTED de Hive.","hive-no-plan": "El plan de consulta no tiene un plan de CBO que dibujar. Hive solo lo indica cuando su optimizador basado en costes planificó la sentencia.",diff --git a/frontend/src/locales/ja-JP.json b/frontend/src/locales/ja-JP.jsonindex d4df2f353a..0cc9a98d44 100644--- a/frontend/src/locales/ja-JP.json+++ b/frontend/src/locales/ja-JP.json@@ -3611,6 +3611,8 @@"cockroachdb-empty": "CockroachDB はこのステートメントのクエリプランを返しませんでした。","cockroachdb-invalid": "クエリプランが有効な JSON ではありません。EXPLAIN (DISTSQL) が埋め込む図が必要です。","cockroachdb-no-plan": "クエリプラン JSON に描画できる \"processors\" がありません。",+ "databricks-empty": "Databricks はこのステートメントのクエリプランを返しませんでした。",+ "databricks-no-plan": "クエリプランに物理プランがありません。EXPLAIN FORMATTED の出力が必要です。","hive-empty": "Hive はこのステートメントのクエリプランを返しませんでした。","hive-invalid": "クエリプランが有効な JSON ではありません。Hive の EXPLAIN FORMATTED の出力が必要です。","hive-no-plan": "クエリプランに描画できる CBO プランがありません。Hive はコストベースオプティマイザーがステートメントをプランした場合にのみ出力します。",diff --git a/frontend/src/locales/vi-VN.json b/frontend/src/locales/vi-VN.jsonindex 853cf2c8f6..277d4251d2 100644--- a/frontend/src/locales/vi-VN.json+++ b/frontend/src/locales/vi-VN.json@@ -3611,6 +3611,8 @@"cockroachdb-empty": "CockroachDB không trả về kế hoạch truy vấn cho câu lệnh này.","cockroachdb-invalid": "Kế hoạch truy vấn không phải JSON hợp lệ. Cần sơ đồ mà EXPLAIN (DISTSQL) mã hoá.","cockroachdb-no-plan": "JSON của kế hoạch truy vấn không chứa \"processors\" nào để vẽ.",+ "databricks-empty": "Databricks không trả về kế hoạch truy vấn cho câu lệnh này.",+ "databricks-no-plan": "Kế hoạch truy vấn không có kế hoạch vật lý. Cần đầu ra EXPLAIN FORMATTED.","hive-empty": "Hive không trả về kế hoạch truy vấn cho câu lệnh này.","hive-invalid": "Kế hoạch truy vấn không phải JSON hợp lệ. Cần đầu ra EXPLAIN FORMATTED của Hive.","hive-no-plan": "Kế hoạch truy vấn không có kế hoạch CBO nào để vẽ. Hive chỉ nêu kế hoạch này khi bộ tối ưu hóa dựa trên chi phí lập kế hoạch cho câu lệnh.",diff --git a/frontend/src/locales/zh-CN.json b/frontend/src/locales/zh-CN.jsonindex ed163fb445..3738401a09 100644--- a/frontend/src/locales/zh-CN.json+++ b/frontend/src/locales/zh-CN.json@@ -3611,6 +3611,8 @@"cockroachdb-empty": "CockroachDB 未返回此语句的查询计划。","cockroachdb-invalid": "查询计划不是有效的 JSON。需要 EXPLAIN (DISTSQL) 编码的图示。","cockroachdb-no-plan": "查询计划 JSON 不包含可绘制的 \"processors\"。",+ "databricks-empty": "Databricks 未返回此语句的查询计划。",+ "databricks-no-plan": "查询计划中没有物理计划。需要 EXPLAIN FORMATTED 输出。","hive-empty": "Hive 未返回此语句的查询计划。","hive-invalid": "查询计划不是有效的 JSON。需要 Hive EXPLAIN FORMATTED 输出。","hive-no-plan": "查询计划中没有可绘制的 CBO 计划。只有基于代价的优化器为该语句生成计划时,Hive 才会输出它。",diff --git a/frontend/src/modules/sql-editor/components/ResultView/SingleResultView.test.tsx b/frontend/src/modules/sql-editor/components/ResultView/SingleResultView.test.tsxindex a7d3a1a4a6..b40cf6b89b 100644--- a/frontend/src/modules/sql-editor/components/ResultView/SingleResultView.test.tsx+++ b/frontend/src/modules/sql-editor/components/ResultView/SingleResultView.test.tsx@@ -1105,8 +1105,9 @@ describe("SingleResultView explain visualizer", () => {);test("keeps multi-column explain output in the result table", () => {- // Databricks explains in Spark's own FORMATTED text, which has no parser,- // so there is no drawable plan to fetch and the grid is all there is.+ // Databricks' plan is text and comes back in one column, which is what its+ // parser reads; a grid of several columns is not that plan, and there is no+ // other format to fetch, so the grid is all there is.tabularPlan(Engine.DATABRICKS, QueryOption_ExplainFormat.TEXT);expect(screen.queryByTestId("query-plan-result")).toBeNull();diff --git a/frontend/src/utils/queryPlanEngines.test.ts b/frontend/src/utils/queryPlanEngines.test.tsindex 8b7d3b2298..c7e7f8ff35 100644--- a/frontend/src/utils/queryPlanEngines.test.ts+++ b/frontend/src/utils/queryPlanEngines.test.ts@@ -30,12 +30,14 @@ describe("isVisualizerEngine", () => {Engine.REDSHIFT,Engine.STARROCKS,Engine.DORIS,+ Engine.DATABRICKS,].sort());expect(isVisualizerEngine(Engine.POSTGRES)).toBe(true);expect(isVisualizerEngine(Engine.MYSQL)).toBe(true);expect(isVisualizerEngine(Engine.SNOWFLAKE)).toBe(true);- expect(isVisualizerEngine(Engine.DATABRICKS)).toBe(false);+ expect(isVisualizerEngine(Engine.DATABRICKS)).toBe(true);+ expect(isVisualizerEngine(Engine.CASSANDRA)).toBe(false);});test("names only engines whose plan the editor can reach", () => {@@ -65,7 +67,12 @@ describe("explainQueryOption", () => {);// These plans are text; the viewer draws them because these engines// publish nothing else.- for (const engine of [Engine.REDSHIFT, Engine.STARROCKS, Engine.DORIS]) {+ for (const engine of [+ Engine.REDSHIFT,+ Engine.STARROCKS,+ Engine.DORIS,+ Engine.DATABRICKS,+ ]) {expect(explainQueryOption(engine).explainFormat).toBe(QueryOption_ExplainFormat.TEXT);@@ -100,12 +107,18 @@ describe("explainQueryOption", () => {});test("leaves an engine the viewer cannot draw on its own default", () => {- // Databricks explains, in Spark's own FORMATTED text, which has no reader.- expect(explainQueryOption(Engine.DATABRICKS).explainFormat).toBe(- QueryOption_ExplainFormat.EXPLAIN_FORMAT_UNSPECIFIED- );- expect(explainQueryOption(Engine.HIVE).explainFormat).not.toBe(- QueryOption_ExplainFormat.EXPLAIN_FORMAT_UNSPECIFIED- );+ // What is left are the engines that stated no plan to begin with: CQL has+ // no EXPLAIN, and the document and key-value APIs publish none.+ for (const engine of [+ Engine.CASSANDRA,+ Engine.REDIS,+ Engine.DYNAMODB,+ Engine.COSMOSDB,+ Engine.ELASTICSEARCH,+ ]) {+ expect(explainQueryOption(engine).explainFormat, Engine[engine]).toBe(+ QueryOption_ExplainFormat.EXPLAIN_FORMAT_UNSPECIFIED+ );+ }});});diff --git a/frontend/src/utils/queryPlanEngines.ts b/frontend/src/utils/queryPlanEngines.tsindex 039f6551b2..3cf889c1e3 100644--- a/frontend/src/utils/queryPlanEngines.ts+++ b/frontend/src/utils/queryPlanEngines.ts@@ -29,11 +29,13 @@ export const VISUALIZER_EXPLAIN_FORMATS = {// BigQuery states no plan for a statement it has not run. What it reports is// the stages of a job that did, which the result carries beside its rows.[Engine.BIGQUERY]: "JSON",- // These three publish no plan as a document; their text plan is what there- // is.+ // These four publish no plan as a document; their text plan is what there+ // is. Databricks prints Spark's FORMATTED plan: a tree of operators and then+ // a block of detail for each.[Engine.REDSHIFT]: "TEXT",[Engine.STARROCKS]: "TEXT",[Engine.DORIS]: "TEXT",+ [Engine.DATABRICKS]: "TEXT",} as const;export type VisualizerEngine = keyof typeof VISUALIZER_EXPLAIN_FORMATS;