Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Add OpenTelemetry Tracing module (#69295)

Anton Kostenko5765b0df156c
diff --git a/.claude/skills/add-tracing/SKILL.md b/.claude/skills/add-tracing/SKILL.md
new file mode 100644
index 00000000000..ffa661f21d8
--- /dev/null
+++ b/.claude/skills/add-tracing/SKILL.md
@@ -0,0 +1,357 @@
+--b-
+name: add-tracing
+description: Add OpenTelemetry tracing spans to Clojure code following Metabase tracing conventions. Use when instrumenting backend code with trace coverage.
+---
+
+# Add Tracing Spans to Clojure Code
+
+This skill helps you add OpenTelemetry (OTel) tracing spans to the Metabase backend codebase using the custom `tracing/with-span` macro.
+
+## Reference Files
+
+- `src/metabase/tracing/core.clj` - `with-span` macro, group registry, SDK lifecycle, `best-effort-sanitize-sql`, Pyroscope integration
+- `src/metabase/task/impl.clj` - `defjob` macro that wraps Quartz jobs with root spans
+- `.clj-kondo/config/modules/config.edn` - Module boundary configuration
+
+## Module Architecture
+
+The tracing module has a deliberately minimal API surface. **Only 2 namespaces are public** (listed in `:api` in the module config):
+
+| Namespace | Role | Status |
+|---|---|---|
+| `tracing.core` | Primary API: `with-span`, groups, SDK lifecycle, Pyroscope, MDC, `best-effort-sanitize-sql` | **Public API** |
+| `tracing.init` | Side-effect loader for `quartz` and `settings` | **Public API** (init convention) |
+| `tracing.attributes` | `best-effort-sanitize-sql` implementation (re-exported via `tracing.core`) | Internal |
+| `tracing.settings` | Setting definitions (`MB_TRACING_*` env vars) | Internal |
+| `tracing.quartz` | Quartz JDBC proxy + JobListener | Internal |
+
+**Rules:**
+- Only require `[metabase.tracing.core :as tracing]` from outside the module. `tracing/best-effort-sanitize-sql` and all other public functions are available from this single namespace.
+- Do not add new API namespaces. Add new public functions to `tracing.core` instead.
+- Do not require internal namespaces (`tracing.attributes`, `tracing.settings`, `tracing.quartz`) from outside the module.
+- `:uses :any` on the `core` module does NOT bypass the target module's `:api` check — internal namespaces are still enforced.
+
+### Cyclic Dependency Avoidance
+
+`tracing/core.clj` is required by many modules across the codebase. It **must NOT** compile-time require `tracing.settings`, as this creates transitive cyclic load dependencies (e.g., `settings/core -> tracing/settings -> tracing/core -> events/impl -> events/core`).
+
+Instead, `tracing/core.clj` uses `requiring-resolve` for settings access:
+
+```clojure
+;; CORRECT — lazy runtime resolution, no compile-time dependency
+((requiring-resolve 'metabase.tracing.settings/tracing-enabled))
+
+;; WRONG — creates cyclic load dependency
+(require '[metabase.tracing.settings :as settings])
+(settings/tracing-enabled)
+```
+
+External library namespaces (clj-otel API, SDK, exporters) are safe to require normally — they don't participate in Metabase namespace cycles.
+
+**Important:** `requiring-resolve` must use **literal quoted symbols**. Kondo hooks validate that `required-namespaces` are all simple symbols, so dynamic construction fails:
+
+```clojure
+;; CORRECT — literal quoted symbol
+(requiring-resolve 'metabase.tracing.settings/tracing-endpoint)
+
+;; WRONG — kondo hook rejects this: "Assert failed: (every? simple-symbol? required-namespaces)"
+(requiring-resolve (symbol "metabase.tracing.settings" "tracing-endpoint"))
+```
+
+## Quick Checklist
+
+When adding tracing spans:
+
+- [ ] Module has `tracing` in its `:uses` set in `.clj-kondo/config/modules/config.edn`
+- [ ] Added `[metabase.tracing.core :as tracing]` to ns requires (alphabetically sorted)
+- [ ] Span wraps a meaningful I/O boundary (not pure computation)
+- [ ] Group matches the domain (check `src/metabase/tracing/core.clj` for registered groups; add a new one if none fit)
+- [ ] Span name follows dot-notation convention (`"domain.subsystem.operation"`)
+- [ ] Attributes use namespaced keywords (`:search/query-length`, `:db/id`)
+- [ ] No sensitive data in attributes (use `best-effort-sanitize-sql` for HoneySQL, never raw SQL)
+- [ ] No new tracing namespaces created (add to `tracing.core` instead)
+- [ ] No `DO_NOT_ADD_NEW_FILES_HERE.txt` violations in the target directory
+- [ ] Run `clj-kondo --lint <files>` to verify 0 errors, 0 warnings
+- [ ] Add or update tests in the corresponding `test/` path (see Testing section below)
+- [ ] Run tests: `clojure -X:dev:test :only <test-ns>`
+
+## The `with-span` Macro
+
+```clojure
+(tracing/with-span group span-name attrs & body)
+```
+
+- **group** - A keyword selecting which trace group this span belongs to (e.g., `:tasks`, `:sync`)
+- **span-name** - A string identifying the span in traces (e.g., `"search.execute"`)
+- **attrs** - A map of span attributes (e.g., `{:db/id 42}`)
+- **body** - The code to execute inside the span
+
+**When disabled:** zero overhead -- single atom deref + boolean check, body runs directly.
+**When enabled:** creates OTel span AND injects `trace_id`/`span_id` into Log4j2 MDC for log-to-trace correlation.
+
+## Trace Groups
+
+Groups are registered in `src/metabase/tracing/core.clj`. Check that file for the current list. The general rule: **match the group to the domain, not the call site.** If code runs inside a Quartz job but is logically search work, use `:search`, not `:tasks`.
+
+To add a new group:
+
+```clojure
+;; In src/metabase/tracing/core.clj
+(register-group! :my-domain "Description of what this covers")
+```
+
+Users enable groups via `MB_TRACING_GROUPS=tasks,search,sync` (comma-separated, or `"all"`).
+
+## Naming Conventions
+
+### Span Names
+
+Use dot-separated hierarchical names: `"domain.subsystem.operation"`. The domain prefix should match the group name:
+
+```
+search.execute -- `:search` group
+sync.fingerprint.table -- `:sync` group
+task.session-cleanup.delete -- `:tasks` group
+db-app.collection-items -- `:db-app` group
+```
+
+### Attributes
+
+Use namespaced keywords. The namespace groups related attributes:
+
+```clojure
+:db/id -- Database ID (integer)
+:db/engine -- Database engine name (string)
+:db/statement -- Sanitized SQL (string, via best-effort-sanitize-sql)
+:search/engine -- Search engine name (string)
+:search/query-length -- Query string length (integer)
+:sync/table -- Table name (string)
+:sync/step -- Sync step name (string)
+:task/name -- Task name (string)
+:http/method -- HTTP method (string)
+:http/url -- Request URL (string)
+```
+
+Invent new namespaced attributes as needed (e.g., `:pulse/id`, `:transform/count`). Keep values as primitives (strings, numbers, booleans) -- no maps or collections.
+
+## Step-by-Step: Adding a Span
+
+### 1. Check module boundaries
+
+Look up the module for your namespace in `.clj-kondo/config/modules/config.edn`. If `tracing` is not in the module's `:uses` set, add it (keep alphabetically sorted):
+
+```edn
+my-module
+{:team "MyTeam"
+ :uses #{analytics config tracing util}}
+```
+
+### 2. Add the require
+
+```clojure
+(ns metabase.my-module.thing
+ (:require
+ [metabase.tracing.core :as tracing]
+ [metabase.util :as u]))
+```
+
+`best-effort-sanitize-sql` is available from `tracing.core` — no additional require needed.
+
+### 3. Identify the I/O boundary
+
+Only wrap code at **meaningful I/O boundaries**:
+
+**DO trace:**
+- External API calls (embedding APIs, metabot, webhooks)
+- Database queries (both app DB and user DB)
+- Network requests (HTTP calls to external services)
+- Heavy batch processing (batch indexing, batch embedding)
+- Top-level orchestration functions that coordinate multiple sub-operations
+
+**DO NOT trace:**
+- Pure computation (sorting, filtering, mapping)
+- Simple single-row lookups (`t2/select-one :model/Setting :key k`)
+- Every function in a call chain (only boundaries matter)
+- Trivial operations (string formatting, hash calculations)
+
+### 4. Wrap with `with-span`
+
+```clojure
+;; Simple span (no attributes needed)
+(tracing/with-span :search "search.init-index" {}
+ (do-expensive-thing))
+
+;; Span with static attributes
+(tracing/with-span :sync "sync.fingerprint.table"
+ {:db/id (:db_id table)
+ :sync/table (:name table)}
+ (fingerprint-fields! table fields))
+
+;; Span with computed attributes
+(tracing/with-span :search "search.execute"
+ {:search/engine (name (:search-engine ctx))
+ :search/query-length (count (:search-string ctx))}
+ (search.engine/results ctx))
+
+;; Span with sanitized SQL (for dynamic HoneySQL queries)
+(let [hsql {:delete-from [(t2/table-name :model/Session)]
+ :where [:< :created_at oldest-allowed]}]
+ (tracing/with-span :tasks "task.session-cleanup.delete"
+ {:db/statement (tracing/best-effort-sanitize-sql hsql)}
+ (t2/query-one hsql)))
+
+;; Sub-spans breaking a function into I/O phases
+(let [embedding (tracing/with-span :search "search.semantic.embedding"
+ {:search.semantic/provider (:provider model)}
+ (get-embedding model search-string))
+ results (tracing/with-span :search "search.semantic.db-query" {}
+ (into [] xform reducible))]
+ (process results))
+
+;; Per-item iteration spans
+(doseq [e (search.engine/active-engines)]
+ (tracing/with-span :search "search.ingestion.update" {:search/engine (name e)}
+ (search.engine/update! e batch)))
+```
+
+### 5. Add tests
+
+Create or update tests in the corresponding `test/` path. Follow the patterns in existing tracing tests:
+
+- **Reference tests:** `test/metabase/tracing/quartz_test.clj`, `test/metabase/server/middleware/trace_test.clj`
+- Use `tracing/init-enabled-groups!` / `tracing/shutdown-groups!` with `try`/`finally` to manage group lifecycle
+- Test both enabled and disabled paths (verify zero overhead when group is off)
+- Use `reify` mocks for Java interfaces (Connection, PreparedStatement, JobListener, etc.)
+- Add `(set! *warn-on-reflection* true)` and type-hint proxy/reify calls to avoid reflection warnings
+
+```clojure
+(deftest my-span-enabled-test
+ (testing "when group is enabled, span is created"
+ (try
+ (tracing/init-enabled-groups! "my-group" "INFO")
+ ;; ... test that span behavior occurs ...
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest my-span-disabled-test
+ (testing "when group is disabled, code runs without tracing"
+ (tracing/shutdown-groups!)
+ ;; ... test that code still works, no wrapping applied ...
+ ))
+```
+
+### 6. Lint and run tests
+
+```bash
+# Lint modified source and test files — expect 0 errors, 0 warnings
+clj-kondo --lint path/to/modified/file.clj path/to/test/file.clj
+
+# Run tests (requires Java 21+)
+ clojure -X:dev:test :only my-ns.test-ns
+```
+
+Expect: all tests pass, 0 failures, 0 errors, no reflection warnings from your files.
+
+## Sanitizing SQL for Attributes
+
+When including SQL in span attributes, **always** use `tracing/best-effort-sanitize-sql`. This converts HoneySQL maps to parameterized SQL strings where values become `?` placeholders -- no data leaks.
+
+```clojure
+(let [hsql {:delete-from [:core_session]
+ :where [:< :created_at some-timestamp]}]
+ (tracing/with-span :tasks "task.cleanup.delete"
+ {:db/statement (tracing/best-effort-sanitize-sql hsql)}
+ (t2/query-one hsql)))
+;; Trace attribute: db/statement = "DELETE FROM core_session WHERE created_at < ?"
+```
+
+**Rules:**
+- Never put raw SQL strings or user-provided values in attributes
+- Use `best-effort-sanitize-sql` only for app DB (HoneySQL) queries
+- For external/user DB queries, trace only timing and counts, not SQL content
+
+## Defjob and Root Spans
+
+The `defjob` macro in `metabase.task.impl` automatically wraps every Quartz job with a `:tasks` root span:
+
+```clojure
+(task/defjob ^{DisallowConcurrentExecution true} SessionCleanup [_]
+ (cleanup-sessions!))
+;; Automatically creates span: "task.SessionCleanup" {:task/name "SessionCleanup"}
+```
+
+You do NOT need a root span inside `defjob` bodies. Add **child spans** for I/O inside the job.
+
+For code on plain `Thread`s (not Quartz), add the root span manually:
+
+```clojure
+(defn init! []
+ (tracing/with-span :search "search.task.init" {}
+ (search/init-index!)))
+```
+
+## What NOT to Do
+
+### Span Usage Mistakes
+
+```clojure
+;; WRONG - pure computation, no I/O
+(tracing/with-span :search "search.format-results" {}
+ (map format-result results))
+
+;; WRONG - trivial single-row lookup
+(tracing/with-span :db-app "db-app.get-setting" {}
+ (t2/select-one :model/Setting :key "my-setting"))
+
+;; WRONG - raw SQL in attributes (data leak)
+(tracing/with-span :tasks "task.cleanup" {:db/statement raw-sql-string}
+ (execute! raw-sql-string))
+
+;; WRONG - wrong group (search work should use :search, not :tasks)
+(tracing/with-span :tasks "search.execute" {} ...)
+
+;; WRONG - redundant nesting (do-search already has a span)
+(tracing/with-span :search "search.process" {}
+ (let [results (do-search ctx)]
+ (tracing/with-span :search "search.format" {}
+ (format-results results))))
+```
+
+### Architecture Mistakes
+
+```clojure
+;; WRONG - creating a new tracing namespace
+(ns metabase.tracing.my-feature ...)
+
+;; WRONG - requiring internal tracing namespaces from outside the module
+(ns metabase.my-module.thing
+ (:require [metabase.tracing.attributes :as trace-attrs] ;; internal!
+ [metabase.tracing.settings :as tracing.settings] ;; internal!
+ [metabase.tracing.quartz :as tracing.quartz])) ;; internal!
+
+;; WRONG - adding compile-time requires to tracing/core.clj for settings or SDK
+;; This creates cyclic load dependencies because tracing/core is widely required
+(ns metabase.tracing.core
+ (:require [metabase.tracing.settings :as settings])) ;; causes cycle!
+
+;; WRONG - dynamic symbol construction with requiring-resolve (kondo rejects it)
+(requiring-resolve (symbol "metabase.tracing.settings" "tracing-enabled"))
+```
+
+## Configuration
+
+All settings are env-var-only (defined in `src/metabase/tracing/settings.clj`):
+
+```bash
+# Core
+MB_TRACING_ENABLED=true # Enable tracing (default: false)
+MB_TRACING_ENDPOINT=host:4317 # OTLP collector endpoint (default: http://localhost:4317)
+MB_TRACING_GROUPS=tasks,search,sync # Comma-separated groups or "all" (default: all)
+MB_TRACING_SERVICE_NAME=metabase # Service name in traces (default: hostname)
+MB_TRACING_LOG_LEVEL=DEBUG # Log threshold for traced threads: TRACE/DEBUG/INFO (default: INFO)
+
+# Batch span processor tuning
+MB_TRACING_MAX_QUEUE_SIZE=2048 # Max spans queued for export; drops when full (default: 2048)
+MB_TRACING_EXPORT_TIMEOUT_MS=10000 # Max wait for batch export to complete (default: 10000)
+MB_TRACING_SCHEDULE_DELAY_MS=5000 # Delay between consecutive batch exports (default: 5000)
+```
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.edn
index 3ef710719b6..a7c939bd67d 100644
--- a/.clj-kondo/config/modules/config.edn
+++ b/.clj-kondo/config/modules/config.edn
@@ -215,6 +215,7 @@
settings
task
task-history
+ tracing
util}}
auth-identity
@@ -397,6 +398,7 @@
request
revisions
settings
+ tracing
transforms
upload
util}}
@@ -589,6 +591,7 @@
settings
sql-tools
system
+ tracing
util
warehouse-schema
warehouses}}
@@ -667,7 +670,8 @@
:api #{metabase.events.core
metabase.events.init}
:uses #{models
- util}}
+ util}
+ :friends #{tracing}}
formatter
{:team "UX West"
@@ -891,6 +895,7 @@
system
task
task-history
+ tracing
util}}
models
@@ -964,6 +969,7 @@
system
task
task-history
+ tracing
users
util}}
@@ -1007,6 +1013,7 @@
premium-features
remote-sync
settings
+ tracing
util
enterprise/advanced-permissions}}
@@ -1052,6 +1059,7 @@
settings
startup
task
+ tracing
util}}
product-feedback
@@ -1119,6 +1127,7 @@
request
task
task-history
+ tracing
util
enterprise/advanced-permissions
enterprise/sandbox}}
@@ -1262,6 +1271,7 @@
server
settings
system
+ tracing
users
util}}
@@ -1343,6 +1353,7 @@
settings
startup
task
+ tracing
transforms
util}}
@@ -1389,6 +1400,7 @@
session
settings
system
+ tracing
users
util
enterprise/sso}}
@@ -1410,6 +1422,7 @@
sso
system
task
+ tracing
util}}
settings
@@ -1532,6 +1545,7 @@
settings
task
task-history
+ tracing
util
warehouse-schema
warehouses}}
@@ -1548,6 +1562,7 @@
metabase.task.core}
:uses #{app-db
classloader
+ tracing
util}}
task-history
@@ -1564,6 +1579,7 @@
query-processor
request
task
+ tracing
util}}
tenants
@@ -1611,6 +1627,15 @@
models
util}}
+ tracing
+ {:team "DevEx"
+ :api #{metabase.tracing.core
+ metabase.tracing.init}
+ :uses #{events
+ settings
+ task
+ util}}
+
transforms
{:team "Querying"
:api #{metabase.transforms.core
@@ -1633,6 +1658,7 @@
settings
task
task-history
+ tracing
transforms-base
util}
:friends #{models
@@ -2434,6 +2460,7 @@
search
settings
task
+ tracing
util
enterprise/llm
enterprise/metabot-v3}}
diff --git a/deps.edn b/deps.edn
index 9067a65462e..c79389008c0 100644
--- a/deps.edn
+++ b/deps.edn
@@ -47,6 +47,8 @@
com.github.seancorfield/honeysql {:mvn/version "2.7.1350"} ; Honey SQL 2. SQL generation from Clojure data maps
com.github.seancorfield/next.jdbc {:mvn/version "1.3.1070"} ; Talk to JDBC DBs
com.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.8"} ; Telemetry library
+ com.github.steffan-westcott/clj-otel-exporter-otlp {:mvn/version "0.2.8"} ; OTLP span exporter (gRPC and HTTP)
+ com.github.steffan-westcott/clj-otel-sdk {:mvn/version "0.2.8"} ; OpenTelemetry SDK for programmatic configuration
com.github.vertical-blank/sql-formatter {:mvn/version "2.0.5"} ; Java SQL formatting library https://github.com/vertical-blank/sql-formatter
com.google.guava/guava {:mvn/version "33.5.0-jre"} ; dep for BigQuery, Spark, and GA. Require here rather than letting different dep versions stomp on each other — see comments on #9697
com.h2database/h2 ^:antq/exclude ; https://metaboat.slack.com/archives/CKZEMT1MJ/p1727191010259979
diff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj
index 0e989215cae..d9683ca4fbe 100644
--- a/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj
+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj
@@ -12,6 +12,7 @@
[metabase.analytics.core :as analytics]
[metabase.premium-features.core :refer [defenterprise]]
[metabase.search.engine :as search.engine]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[toucan2.realize :as t2.realize]))
@@ -38,53 +39,54 @@
when semantic search returns too few results and some results were filtered out (e.g. due to permission checks)."
:feature :semantic-search
[search-ctx]
- (try
- (let [{:keys [results raw-count]}
- (semantic.pgvector-api/query (semantic.env/get-pgvector-datasource!)
- (semantic.env/get-index-metadata)
- search-ctx)
- final-count (count results)
- threshold (semantic.settings/semantic-search-min-results-threshold)]
- (if (or (>= final-count threshold)
- (and (zero? raw-count)
- ;; :search-string is nil when using search to populate the list of tables for a given database in
- ;; the native query editor. Semantic search doesn't support this, so fallback in this case.
- (not (str/blank? (:search-string search-ctx)))))
- results
- ;; Fallback: semantic search found results but some were filtered out (e.g. due to permission checks), so try to
- ;; supplement with appdb search.
- (let [fallback (fallback-engine)]
- (log/debugf "Semantic search returned %d final results (< %d) from %d raw results, supplementing with %s search"
- final-count threshold raw-count fallback)
- (analytics/inc! :metabase-search/semantic-fallback-triggered {:fallback-engine fallback})
- (analytics/observe! :metabase-search/semantic-results-before-fallback final-count)
+ (tracing/with-span :search "search.semantic.execute" {:search/query-length (count (:search-string search-ctx))}
+ (try
+ (let [{:keys [results raw-count]}
+ (semantic.pgvector-api/query (semantic.env/get-pgvector-datasource!)
+ (semantic.env/get-index-metadata)
+ search-ctx)
+ final-count (count results)
+ threshold (semantic.settings/semantic-search-min-results-threshold)]
+ (if (or (>= final-count threshold)
+ (and (zero? raw-count)
+ ;; :search-string is nil when using search to populate the list of tables for a given database in
+ ;; the native query editor. Semantic search doesn't support this, so fallback in this case.
+ (not (str/blank? (:search-string search-ctx)))))
+ results
+ ;; Fallback: semantic search found results but some were filtered out (e.g. due to permission checks), so try to
+ ;; supplement with appdb search.
+ (let [fallback (fallback-engine)]
+ (log/debugf "Semantic search returned %d final results (< %d) from %d raw results, supplementing with %s search"
+ final-count threshold raw-count fallback)
+ (analytics/inc! :metabase-search/semantic-fallback-triggered {:fallback-engine fallback})
+ (analytics/observe! :metabase-search/semantic-results-before-fallback final-count)
- (when (some-> (:offset-int search-ctx) pos?)
- (log/warn "Using an offset with semantic search will produce strange results, e.g. missing expected results, or duplicating them across pages"))
+ (when (some-> (:offset-int search-ctx) pos?)
+ (log/warn "Using an offset with semantic search will produce strange results, e.g. missing expected results, or duplicating them across pages"))
- (let [total-limit (semantic.settings/semantic-search-results-limit)
- fallback-results (try
- (cond->> (search.engine/results (assoc search-ctx :search-engine fallback))
- ;; The in-place engine returns a reducible (but not seqable) result that needs to
- ;; be realized before we concat and dedup with the semantic engine results.
- (= :search.engine/in-place fallback)
- (into [] (comp (map t2.realize/realize)
- (take total-limit))))
- (catch Throwable t
- (log/warn t "Semantic search fallback errored, ignoring")
- []))
- fallback-results (take total-limit fallback-results)
- _ (analytics/observe! :metabase-search/semantic-fallback-results-usage (count fallback-results))
- combined-results (concat results fallback-results)
- deduped-results (m/distinct-by (juxt :model :id) combined-results)]
- (take total-limit deduped-results)))))
- (catch Exception e
- (log/error e "Error executing semantic search, falling back to appdb")
- (let [fallback (fallback-engine)]
- (analytics/inc! :metabase-search/semantic-error-fallback {:fallback-engine fallback})
- (if fallback
- (search.engine/results (assoc search-ctx :search-engine fallback))
- (throw (ex-info "Error executing semantic search" {:type :semantic-search-error} e)))))))
+ (let [total-limit (semantic.settings/semantic-search-results-limit)
+ fallback-results (try
+ (cond->> (search.engine/results (assoc search-ctx :search-engine fallback))
+ ;; The in-place engine returns a reducible (but not seqable) result that needs to
+ ;; be realized before we concat and dedup with the semantic engine results.
+ (= :search.engine/in-place fallback)
+ (into [] (comp (map t2.realize/realize)
+ (take total-limit))))
+ (catch Throwable t
+ (log/warn t "Semantic search fallback errored, ignoring")
+ []))
+ fallback-results (take total-limit fallback-results)
+ _ (analytics/observe! :metabase-search/semantic-fallback-results-usage (count fallback-results))
+ combined-results (concat results fallback-results)
+ deduped-results (m/distinct-by (juxt :model :id) combined-results)]
+ (take total-limit deduped-results)))))
+ (catch Exception e
+ (log/error e "Error executing semantic search, falling back to appdb")
+ (let [fallback (fallback-engine)]
+ (analytics/inc! :metabase-search/semantic-error-fallback {:fallback-engine fallback})
+ (if fallback
+ (search.engine/results (assoc search-ctx :search-engine fallback))
+ (throw (ex-info "Error executing semantic search" {:type :semantic-search-error} e))))))))
(defenterprise update-index!
"Enterprise implementation of semantic index updating."
diff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj
index db03fb74511..8a4eac2acba 100644
--- a/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj
+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj
@@ -6,6 +6,7 @@
[metabase-enterprise.semantic-search.models.token-tracking :as semantic.models.token-tracking]
[metabase-enterprise.semantic-search.settings :as semantic-settings]
[metabase.analytics.core :as analytics]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.json :as json]
[metabase.util.log :as log])
@@ -279,26 +280,30 @@
[embedding-model texts process-fn & {:as opts}]
(when (seq texts)
(let [{:keys [model-name provider vector-dimensions]} embedding-model]
- (u/profile (str "Generating embeddings " {:model model-name
- :dimensions vector-dimensions
- :texts (calc-token-metrics texts)})
- (if (= "openai" provider)
- (let [max-tokens-per-batch (semantic-settings/openai-max-tokens-per-batch)
- batches (create-batches max-tokens-per-batch count-tokens texts)
-
- process-batch
- (fn [batch-idx batch-texts]
- (let [embeddings (u/profile (format "Embedding batch %d/%d %s"
- (inc batch-idx) (count batches) (str (calc-token-metrics batch-texts)))
- (openai-get-embeddings-batch embedding-model batch-texts opts))
- text-embedding-map (zipmap batch-texts embeddings)]
- (process-fn text-embedding-map)))]
-
- (transduce (map-indexed process-batch) (partial merge-with +) batches))
-
- (let [embeddings (get-embeddings-batch embedding-model texts opts)
- text-embedding-map (zipmap texts embeddings)]
- (process-fn text-embedding-map)))))))
+ (tracing/with-span :search "search.semantic.embeddings-batch"
+ {:search.semantic/provider provider
+ :search.semantic/model-name model-name
+ :search.semantic/text-count (count texts)}
+ (u/profile (str "Generating embeddings " {:model model-name
+ :dimensions vector-dimensions
+ :texts (calc-token-metrics texts)})
+ (if (= "openai" provider)
+ (let [max-tokens-per-batch (semantic-settings/openai-max-tokens-per-batch)
+ batches (create-batches max-tokens-per-batch count-tokens texts)
+
+ process-batch
+ (fn [batch-idx batch-texts]
+ (let [embeddings (u/profile (format "Embedding batch %d/%d %s"
+ (inc batch-idx) (count batches) (str (calc-token-metrics batch-texts)))
+ (openai-get-embeddings-batch embedding-model batch-texts opts))
+ text-embedding-map (zipmap batch-texts embeddings)]
+ (process-fn text-embedding-map)))]
+
+ (transduce (map-indexed process-batch) (partial merge-with +) batches))
+
+ (let [embeddings (get-embeddings-batch embedding-model texts opts)
+ text-embedding-map (zipmap texts embeddings)]
+ (process-fn text-embedding-map))))))))
(comment
;; Configuration:
diff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj
index 94cf79185fe..c01613f4a41 100644
--- a/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj
+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj
@@ -16,6 +16,7 @@
[metabase.models.interface :as mi]
[metabase.search.config :as search.config]
[metabase.search.core :as search]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.json :as json]
[metabase.util.log :as log]
@@ -389,20 +390,21 @@
model + model_id already exists, it will be replaced. Parallelizes batch insertion
using a shared thread pool with a configurable thread count (default: 2)."
([connectable index documents-reducible & {:keys [serial?] :or {serial? false}}]
- (not-empty
- (let [pool @index-update-executor
- results (transduce
- (comp (partition-all *batch-size*)
- (map (if serial?
- #(upsert-index-batch! connectable index % {:type :index})
- #(upsert-index-pooled! pool connectable index % {:type :index}))))
- conj
- documents-reducible)]
- (reduce (fn [update-counts result]
- (let [value (if (future? result) @result result)]
- (merge-with + update-counts (when value value))))
- {}
- results)))))
+ (tracing/with-span :search "search.semantic.index-upsert" {}
+ (not-empty
+ (let [pool @index-update-executor
+ results (transduce
+ (comp (partition-all *batch-size*)
+ (map (if serial?
+ #(upsert-index-batch! connectable index % {:type :index})
+ #(upsert-index-pooled! pool connectable index % {:type :index}))))
+ conj
+ documents-reducible)]
+ (reduce (fn [update-counts result]
+ (let [value (if (future? result) @result result)]
+ (merge-with + update-counts (when value value))))
+ {}
+ results))))))
(defn- drop-index-table-sql
[{:keys [table-name]}]
@@ -803,7 +805,10 @@
{:results [] :raw-count 0}
(let [timer (u/start-timer)
- embedding (embedding/get-embedding embedding-model search-string {:type :query})
+ embedding (tracing/with-span :search "search.semantic.embedding"
+ {:search.semantic/provider (:provider embedding-model)
+ :search.semantic/model-name (:model-name embedding-model)}
+ (embedding/get-embedding embedding-model search-string {:type :query}))
embedding-time-ms (u/since-ms timer)
db-timer (u/start-timer)
@@ -813,14 +818,18 @@
xform (comp (map decode-metadata)
(map (partial legacy-input-with-score weights (keys scorers))))
reducible (reducible-search-query db query)
- raw-results (into [] xform reducible)
+ raw-results (tracing/with-span :search "search.semantic.db-query"
+ {:search/query-length (count search-string)}
+ (into [] xform reducible))
db-query-time-ms (u/since-ms db-timer)
filter-timer (u/start-timer)
- filtered-results (->> raw-results
- filter-read-permitted
- (apply-collection-id-filter search-context)
- (mapv search/collapse-id))
+ filtered-results (tracing/with-span :search "search.semantic.permission-filter"
+ {:search.semantic/raw-count (count raw-results)}
+ (->> raw-results
+ filter-read-permitted
+ (apply-collection-id-filter search-context)
+ (mapv search/collapse-id)))
filter-time-ms (u/since-ms filter-timer)
appdb-scorers (scoring/appdb-scorers search-context)
diff --git a/frontend/src/metabase-types/api/mocks/settings.ts b/frontend/src/metabase-types/api/mocks/settings.ts
index c3fbf4fbb73..d370f40d252 100644
--- a/frontend/src/metabase-types/api/mocks/settings.ts
+++ b/frontend/src/metabase-types/api/mocks/settings.ts
@@ -385,6 +385,7 @@ export const createMockSettings = (
"subscription-allowed-domains": null,
"token-features": createMockTokenFeatures(),
"token-status": null,
+ "tracing-enabled": false,
"transforms-enabled": false,
version: createMockVersion(),
"version-info": createMockVersionInfo(),
diff --git a/frontend/src/metabase-types/api/settings.ts b/frontend/src/metabase-types/api/settings.ts
index 848989f6e20..acc25fd40ef 100644
--- a/frontend/src/metabase-types/api/settings.ts
+++ b/frontend/src/metabase-types/api/settings.ts
@@ -599,6 +599,7 @@ interface PublicSettings {
"snowplow-url": string;
"start-of-week": DayOfWeekId;
"token-features": TokenFeatures;
+ "tracing-enabled": boolean;
"transforms-enabled": boolean;
version: Version;
"version-info-last-checked": string | null;
diff --git a/frontend/src/metabase/app.js b/frontend/src/metabase/app.js
index 01db6526077..0013ed7d05e 100644
--- a/frontend/src/metabase/app.js
+++ b/frontend/src/metabase/app.js
@@ -39,6 +39,7 @@ import { createTracker } from "metabase/lib/analytics";
import api from "metabase/lib/api";
import { initializeEmbedding } from "metabase/lib/embed";
import { captureConsoleErrors } from "metabase/lib/errors";
+import { initTracing, rotateTraceId } from "metabase/lib/otel";
import { MetabaseReduxProvider } from "metabase/lib/redux/custom-context";
import MetabaseSettings from "metabase/lib/settings";
import { PLUGIN_APP_INIT_FUNCTIONS, PLUGIN_METABOT } from "metabase/plugins";
@@ -72,6 +73,15 @@ function _init(reducers, getRoutes, callback) {
createTracker(store);
+ // Initialize distributed tracing if enabled via MB_TRACING_ENABLED.
+ // Uses bootstrap data so it's available before the first API call.
+ if (window.MetabaseBootstrap?.["tracing-enabled"]) {
+ initTracing();
+ // Rotate trace ID on route changes so all API calls within
+ // a single page view share one trace.
+ syncedHistory.listen(() => rotateTraceId());
+ }
+
initializeEmbedding(store);
const root = createRoot(document.getElementById("root"));
diff --git a/frontend/src/metabase/lib/api.js b/frontend/src/metabase/lib/api.js
index acdb86284ab..3f6ba628ec5 100644
--- a/frontend/src/metabase/lib/api.js
+++ b/frontend/src/metabase/lib/api.js
@@ -6,6 +6,7 @@ import { isEmbeddingSdk } from "metabase/embedding-sdk/config";
import { isTest } from "metabase/env";
import { isWithinIframe } from "metabase/lib/dom";
import { IFRAMED_IN_SELF } from "metabase/lib/iframe";
+import { getTraceparentHeader } from "metabase/lib/otel";
import { delay } from "metabase/lib/promise";
import { PLUGIN_API, PLUGIN_EMBEDDING_SDK } from "metabase/plugins";
@@ -103,6 +104,11 @@ export class Api extends EventEmitter {
DEFAULT_OPTIONS.headers["X-Metabase-Locale"];
}
+ const traceparent = getTraceparentHeader();
+ if (traceparent) {
+ headers["traceparent"] = traceparent;
+ }
+
return headers;
}
diff --git a/frontend/src/metabase/lib/otel.ts b/frontend/src/metabase/lib/otel.ts
new file mode 100644
index 00000000000..0849e4b42de
--- /dev/null
+++ b/frontend/src/metabase/lib/otel.ts
@@ -0,0 +1,106 @@
+/**
+ * Lightweight W3C Trace Context propagation for distributed tracing.
+ *
+ * Generates `traceparent` headers (https://www.w3.org/TR/trace-context/)
+ * without requiring the full OpenTelemetry browser SDK.
+ *
+ * How it works:
+ * - A new trace ID is generated for each "activity burst" — a group of
+ * API requests that happen close together (e.g., a dashboard load,
+ * a filter application, a query run).
+ * - After a short idle period (no API calls), the next request starts
+ * a new trace. This naturally groups related requests together.
+ * - Route changes always start a new trace immediately.
+ * - Each individual API request gets a unique span ID (parent-id).
+ * - The backend extracts the `traceparent` header and creates child spans.
+ *
+ * traceparent format: {version}-{trace-id}-{parent-id}-{trace-flags}
+ * version: "00"
+ * trace-id: 32 hex chars (128-bit)
+ * parent-id: 16 hex chars (64-bit)
+ * trace-flags: "01" (sampled)
+ */
+
+let currentTraceId: string | null = null;
+let enabled = false;
+let idleTimer: ReturnType<typeof setTimeout> | null = null;
+let traceExpired = false;
+
+/** Idle time in ms before the current trace expires and the next request starts a new one. */
+const IDLE_TIMEOUT_MS = 5000;
+
+function generateHexId(byteLength: number): string {
+ const buf = new Uint8Array(byteLength);
+ crypto.getRandomValues(buf);
+ return Array.from(buf, (b) => b.toString(16).padStart(2, "0")).join("");
+}
+
+function newTraceId(): void {
+ currentTraceId = generateHexId(16); // 128-bit = 32 hex chars
+ traceExpired = false;
+}
+
+function resetIdleTimer(): void {
+ if (idleTimer !== null) {
+ clearTimeout(idleTimer);
+ }
+ idleTimer = setTimeout(() => {
+ traceExpired = true;
+ }, IDLE_TIMEOUT_MS);
+}
+
+/**
+ * Enable tracing and generate the first trace ID.
+ * Called once at app startup when `tracing-enabled` is true.
+ */
+export function initTracing(): void {
+ enabled = true;
+ newTraceId();
+}
+
+/**
+ * Force a new trace ID immediately. Called on route changes
+ * so navigating between pages always starts a fresh trace.
+ */
+export function rotateTraceId(): void {
+ if (enabled) {
+ newTraceId();
+ }
+}
+
+/**
+ * Get a W3C `traceparent` header value for the current request.
+ * Returns null when tracing is disabled.
+ *
+ * If the idle timeout has elapsed since the last API call, a new
+ * trace ID is generated automatically — so each "burst" of related
+ * requests (dashboard load, filter click, etc.) gets its own trace.
+ *
+ * Each call generates a unique span ID (parent-id) so every
+ * API request is a distinct span within the shared trace.
+ */
+/** Reset all state. Exported only for tests. */
+export function _resetForTesting(): void {
+ currentTraceId = null;
+ enabled = false;
+ traceExpired = false;
+ if (idleTimer !== null) {
+ clearTimeout(idleTimer);
+ idleTimer = null;
+ }
+}
+
+export function getTraceparentHeader(): string | null {
+ if (!enabled || !currentTraceId) {
+ return null;
+ }
+
+ if (traceExpired) {
+ newTraceId();
+ }
+
+ resetIdleTimer();
+
+ const spanId = generateHexId(8); // 64-bit = 16 hex chars
+ return `00-${currentTraceId}-${spanId}-01`;
+}
diff --git a/frontend/src/metabase/lib/otel.unit.spec.ts b/frontend/src/metabase/lib/otel.unit.spec.ts
new file mode 100644
index 00000000000..c7f785f87b0
--- /dev/null
+++ b/frontend/src/metabase/lib/otel.unit.spec.ts
@@ -0,0 +1,133 @@
+import {
+ _resetForTesting,
+ getTraceparentHeader,
+ initTracing,
+ rotateTraceId,
+} from "metabase/lib/otel";
+
+// W3C traceparent: 00-{32 hex}-{16 hex}-01
+const TRACEPARENT_REGEX = /^00-[0-9a-f]{32}-[0-9a-f]{16}-01$/;
+
+describe("otel", () => {
+ beforeEach(() => {
+ _resetForTesting();
+ jest.useFakeTimers();
+ });
+
+ afterEach(() => {
+ jest.useRealTimers();
+ });
+
+ describe("getTraceparentHeader", () => {
+ it("returns null before initialization", () => {
+ expect(getTraceparentHeader()).toBeNull();
+ });
+
+ it("returns a valid W3C traceparent after init", () => {
+ initTracing();
+ const header = getTraceparentHeader();
+ expect(header).toMatch(TRACEPARENT_REGEX);
+ });
+
+ it("returns the same trace ID for consecutive calls within idle timeout", () => {
+ initTracing();
+ const header1 = getTraceparentHeader()!;
+ const header2 = getTraceparentHeader()!;
+
+ const traceId1 = header1.split("-")[1];
+ const traceId2 = header2.split("-")[1];
+ expect(traceId1).toBe(traceId2);
+ });
+
+ it("generates unique span IDs for each call", () => {
+ initTracing();
+ const header1 = getTraceparentHeader()!;
+ const header2 = getTraceparentHeader()!;
+
+ const spanId1 = header1.split("-")[2];
+ const spanId2 = header2.split("-")[2];
+ expect(spanId1).not.toBe(spanId2);
+ });
+ });
+
+ describe("rotateTraceId", () => {
+ it("changes the trace ID immediately", () => {
+ initTracing();
+ const before = getTraceparentHeader()!.split("-")[1];
+ rotateTraceId();
+ const after = getTraceparentHeader()!.split("-")[1];
+
+ expect(before).not.toBe(after);
+ });
+
+ it("is a no-op when tracing is not initialized", () => {
+ rotateTraceId();
+ expect(getTraceparentHeader()).toBeNull();
+ });
+ });
+
+ describe("idle timeout", () => {
+ it("rotates trace ID after idle period expires", () => {
+ initTracing();
+ const first = getTraceparentHeader()!.split("-")[1];
+
+ // Advance past the 5s idle timeout
+ jest.advanceTimersByTime(6000);
+
+ const second = getTraceparentHeader()!.split("-")[1];
+ expect(first).not.toBe(second);
+ });
+
+ it("keeps the same trace ID if requests happen within idle window", () => {
+ initTracing();
+ const first = getTraceparentHeader()!.split("-")[1];
+
+ // Make a request at 3s (resets the timer)
+ jest.advanceTimersByTime(3000);
+ const second = getTraceparentHeader()!.split("-")[1];
+ expect(first).toBe(second);
+
+ // Make another at 3s later (6s total, but only 3s since last call)
+ jest.advanceTimersByTime(3000);
+ const third = getTraceparentHeader()!.split("-")[1];
+ expect(first).toBe(third);
+ });
+
+ it("rotates after idle even without route change", () => {
+ initTracing();
+ const first = getTraceparentHeader()!.split("-")[1];
+
+ // Simulate user inactivity then a new action
+ jest.advanceTimersByTime(6000);
+ const second = getTraceparentHeader()!.split("-")[1];
+
+ jest.advanceTimersByTime(6000);
+ const third = getTraceparentHeader()!.split("-")[1];
+
+ expect(first).not.toBe(second);
+ expect(second).not.toBe(third);
+ });
+ });
+
+ describe("traceparent format", () => {
+ it("has version 00", () => {
+ initTracing();
+ const header = getTraceparentHeader()!;
+ expect(header.startsWith("00-")).toBe(true);
+ });
+
+ it("has trace-flags 01 (sampled)", () => {
+ initTracing();
+ const header = getTraceparentHeader()!;
+ expect(header.endsWith("-01")).toBe(true);
+ });
+
+ it("has a 32-char trace ID and 16-char span ID", () => {
+ initTracing();
+ const parts = getTraceparentHeader()!.split("-");
+ expect(parts).toHaveLength(4);
+ expect(parts[1]).toHaveLength(32);
+ expect(parts[2]).toHaveLength(16);
+ });
+ });
+});
diff --git a/mage/src/mage/modules.clj b/mage/src/mage/modules.clj
index 3af63fdab1c..9c22f5cd006 100644
--- a/mage/src/mage/modules.clj
+++ b/mage/src/mage/modules.clj
@@ -132,6 +132,7 @@
task
task-history
timeline
+ tracing
types
users
util
diff --git a/src/metabase/audit_app/task/truncate_audit_tables.clj b/src/metabase/audit_app/task/truncate_audit_tables.clj
index 88df8678282..b24607a193e 100644
--- a/src/metabase/audit_app/task/truncate_audit_tables.clj
+++ b/src/metabase/audit_app/task/truncate_audit_tables.clj
@@ -11,6 +11,7 @@
[metabase.premium-features.core :refer [defenterprise]]
[metabase.task-history.core :as task-history]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.log :as log]
[toucan2.core :as t2]))
@@ -44,18 +45,21 @@
[model time-column]
(when-not (infinite? (audit-app.settings/audit-max-retention-days))
(let [table-name (name (t2/table-name model))]
- (try
- (log/infof "Cleaning up %s table" table-name)
- (loop [total-rows-deleted 0]
- (let [batch-rows-deleted (truncate-table-batched! table-name time-column)]
- ;; Only try to delete another batch if the last batch was full
- (if (= batch-rows-deleted (audit-app.settings/audit-table-truncation-batch-size))
- (recur (+ total-rows-deleted (long batch-rows-deleted)))
- (if (not= total-rows-deleted 0)
- (log/infof "%s cleanup successful, %d rows were deleted" table-name total-rows-deleted)
- (log/infof "%s cleanup successful, no rows were deleted" table-name)))))
- (catch Throwable e
- (log/errorf e "%s cleanup failed" table-name))))))
+ (tracing/with-span :tasks "task.audit-cleanup.table" {:audit/table table-name
+ :audit/time-column (name time-column)
+ :db/type (name (mdb/db-type))}
+ (try
+ (log/infof "Cleaning up %s table" table-name)
+ (loop [total-rows-deleted 0]
+ (let [batch-rows-deleted (truncate-table-batched! table-name time-column)]
+ ;; Only try to delete another batch if the last batch was full
+ (if (= batch-rows-deleted (audit-app.settings/audit-table-truncation-batch-size))
+ (recur (+ total-rows-deleted (long batch-rows-deleted)))
+ (if (not= total-rows-deleted 0)
+ (log/infof "%s cleanup successful, %d rows were deleted" table-name total-rows-deleted)
+ (log/infof "%s cleanup successful, no rows were deleted" table-name)))))
+ (catch Throwable e
+ (log/errorf e "%s cleanup failed" table-name)))))))
(defenterprise audit-models-to-truncate
"List of models to truncate. OSS implementation only truncates `query_execution` table."
diff --git a/src/metabase/collections_rest/api.clj b/src/metabase/collections_rest/api.clj
index f0ee92bc1bd..7b943b574da 100644
--- a/src/metabase/collections_rest/api.clj
+++ b/src/metabase/collections_rest/api.clj
@@ -27,6 +27,7 @@
[metabase.queries.core :as queries]
[metabase.request.core :as request]
[metabase.revisions.core :as revisions]
+ [metabase.tracing.core :as tracing]
[metabase.transforms.feature-gating :as transforms.gating]
[metabase.transforms.util :as transforms.u]
[metabase.upload.core :as upload]
@@ -1102,11 +1103,13 @@
;; :total_count
:limit (if (zero? limit) 1 limit)
:offset offset))
- rows (mdb/query limit-query)
+ rows (tracing/with-span :db-app "db-app.collection-items-query" {:collection/id (:id collection)}
+ (mdb/query limit-query))
res {:total (->> rows first :total_count)
:data (if (= limit 0)
[]
- (post-process-rows options collection rows))
+ (tracing/with-span :db-app "db-app.collection-items-post-process" {:collection/id (:id collection)}
+ (post-process-rows options collection rows)))
:models models}
limit-res (assoc res
:limit (request/limit)
diff --git a/src/metabase/core/core.clj b/src/metabase/core/core.clj
index 14499530da1..3050b0e1ce4 100644
--- a/src/metabase/core/core.clj
+++ b/src/metabase/core/core.clj
@@ -32,6 +32,7 @@
[metabase.startup.core :as startup]
[metabase.system.core :as system]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.queue :as queue]
@@ -106,6 +107,7 @@
(queue/stop-listeners!)
(task/stop-scheduler!)
(server/stop-web-server!)
+ (tracing/shutdown!)
(analytics/shutdown!)
(notification/shutdown!)
;; This timeout was chosen based on a 30s default termination grace period in Kubernetes.
@@ -167,6 +169,8 @@
(init-status/set-progress! 0.2)
;; Ensure the classloader is installed as soon as possible.
(classloader/the-classloader)
+ ;; Initialize OpenTelemetry tracing early (before plugins, no DB dependency)
+ (tracing/init!)
;; load any plugins as needed
(plugins/load-plugins!)
(init-status/set-progress! 0.3)
diff --git a/src/metabase/core/init.clj b/src/metabase/core/init.clj
index 3e546ff3f53..64cd03d38be 100644
--- a/src/metabase/core/init.clj
+++ b/src/metabase/core/init.clj
@@ -65,6 +65,7 @@
[metabase.task-history.init]
[metabase.testing-api.init]
[metabase.tiles.init]
+ [metabase.tracing.init]
[metabase.transforms-base.init]
[metabase.transforms.init]
[metabase.types.init]
diff --git a/src/metabase/driver/sql_jdbc/execute.clj b/src/metabase/driver/sql_jdbc/execute.clj
index d535e7fd2d9..c4233dd9ebb 100644
--- a/src/metabase/driver/sql_jdbc/execute.clj
+++ b/src/metabase/driver/sql_jdbc/execute.clj
@@ -22,6 +22,7 @@
[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface]
[metabase.lib.schema.info :as lib.schema.info]
[metabase.premium-features.core :refer [defenterprise]]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.i18n :refer [tru]]
[metabase.util.log :as log]
@@ -773,48 +774,49 @@
{:added "0.35.0", :arglists '([driver query context respond])}
[driver {{sql :query, params :params} :native, :as outer-query} _context respond]
{:pre [(string? sql) (seq sql)]}
- (let [database (driver-api/database (driver-api/metadata-provider))
- sql (if (get-in (driver.conn/effective-details database) [:include-user-id-and-hash] true)
- (->> (driver-api/query->remark driver outer-query)
- (inject-remark driver sql))
- sql)
- max-rows (driver-api/determine-query-max-rows outer-query)]
- (do-with-connection-with-options
- driver
- (driver-api/database (driver-api/metadata-provider))
- {:session-timezone (driver-api/report-timezone-id-if-supported driver (driver-api/database (driver-api/metadata-provider)))
- :download? (download? (-> outer-query :info :context))}
- (fn [^Connection conn]
- (with-open [stmt (statement-or-prepared-statement driver conn sql params (driver-api/canceled-chan))
- ^ResultSet rs (try
- (execute-statement-or-prepared-statement! driver stmt max-rows params sql)
- (catch Throwable e
- (throw (ex-info (tru "Error executing query: {0}" (ex-message e))
- (cond-> {:driver driver
- :sql (str/split-lines (driver/prettify-native-form driver sql))
- :params params
- :type driver-api/qp.error-type.invalid-query}
- (driver/query-canceled? driver e)
- (assoc :query/query-canceled? true))
- e))))]
- (let [rsmeta (.getMetaData rs)
- results-metadata {:cols (column-metadata driver rsmeta)}]
- (try (respond results-metadata (reducible-rows driver rs rsmeta (driver-api/canceled-chan)))
+ (tracing/with-span :db-user "db-user.execute-query" {:db/engine (name driver)}
+ (let [database (driver-api/database (driver-api/metadata-provider))
+ sql (if (get-in (driver.conn/effective-details database) [:include-user-id-and-hash] true)
+ (->> (driver-api/query->remark driver outer-query)
+ (inject-remark driver sql))
+ sql)
+ max-rows (driver-api/determine-query-max-rows outer-query)]
+ (do-with-connection-with-options
+ driver
+ (driver-api/database (driver-api/metadata-provider))
+ {:session-timezone (driver-api/report-timezone-id-if-supported driver (driver-api/database (driver-api/metadata-provider)))
+ :download? (download? (-> outer-query :info :context))}
+ (fn [^Connection conn]
+ (with-open [stmt (statement-or-prepared-statement driver conn sql params (driver-api/canceled-chan))
+ ^ResultSet rs (try
+ (execute-statement-or-prepared-statement! driver stmt max-rows params sql)
+ (catch Throwable e
+ (throw (ex-info (tru "Error executing query: {0}" (ex-message e))
+ (cond-> {:driver driver
+ :sql (str/split-lines (driver/prettify-native-form driver sql))
+ :params params
+ :type driver-api/qp.error-type.invalid-query}
+ (driver/query-canceled? driver e)
+ (assoc :query/query-canceled? true))
+ e))))]
+ (let [rsmeta (.getMetaData rs)
+ results-metadata {:cols (column-metadata driver rsmeta)}]
+ (try (respond results-metadata (reducible-rows driver rs rsmeta (driver-api/canceled-chan)))
;; Following cancels the statement on the dbms side.
;; It avoids blocking `.close` call, in case we reduced the results subset eg. by means of
;; [[metabase.query-processor.middleware.limit/limit-xform]] middleware, while statement is still
;; in progress. This problem was encountered on Redshift. For details see the issue #39018.
;; It also handles situation where query is canceled through [[driver-api/canceled-chan]] (#41448).
- (finally
+ (finally
;; TODO: Following `when` is in place just to find out if vertica is flaking because of cancelations.
;; It should be removed afterwards!
- (when-not (= :vertica driver)
- (try (.cancel stmt)
- (catch SQLFeatureNotSupportedException _
- (log/warnf "Statemet's `.cancel` method is not supported by the `%s` driver."
- (name driver)))
- (catch Throwable _
- (log/warn "Statement cancelation failed."))))))))))))
+ (when-not (= :vertica driver)
+ (try (.cancel stmt)
+ (catch SQLFeatureNotSupportedException _
+ (log/warnf "Statemet's `.cancel` method is not supported by the `%s` driver."
+ (name driver)))
+ (catch Throwable _
+ (log/warn "Statement cancelation failed.")))))))))))))
(defn reducible-query
"Returns a reducible collection of rows as maps from `db` and a given SQL query. This is similar to [[jdbc/reducible-query]] but reuses the
diff --git a/src/metabase/model_persistence/task/persist_refresh.clj b/src/metabase/model_persistence/task/persist_refresh.clj
index a66b3d5da18..8345b5d9553 100644
--- a/src/metabase/model_persistence/task/persist_refresh.clj
+++ b/src/metabase/model_persistence/task/persist_refresh.clj
@@ -17,6 +17,7 @@
[metabase.query-processor.timezone :as qp.timezone]
[metabase.task-history.core :as task-history]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.honey-sql-2 :as h2x]
[metabase.util.log :as log]
@@ -56,36 +57,39 @@
;; Since this could be long running, double check state just before refreshing
(driver.conn/with-write-connection
(when (contains? (persisted-info/refreshable-states) (t2/select-one-fn :state :model/PersistedInfo :id (:id persisted-info)))
- (log/infof "Attempting to refresh persisted model %s." (:card_id persisted-info))
- (let [card (t2/select-one :model/Card :id (:card_id persisted-info))
- definition (persisted-info/metadata->definition (:result_metadata card)
- (:table_name persisted-info))
- _ (t2/update! :model/PersistedInfo (u/the-id persisted-info)
- {:definition definition,
- :query_hash (persisted-info/query-hash (:dataset_query card))
- :active false,
- :refresh_begin :%now,
- :refresh_end nil,
- :state "refreshing"
- :state_change_at :%now})
- {:keys [state error]} (try
- (refresh! refresher database definition card)
- (catch Exception e
- (log/infof e "Error refreshing persisting model with card-id %s"
- (:card_id persisted-info))
- {:state :error :error (ex-message e)}))]
- (t2/update! :model/PersistedInfo (u/the-id persisted-info)
- {:active (= state :success),
- :refresh_end :%now,
- :state (if (= state :success) "persisted" "error")
- :state_change_at :%now
- :error (when (= state :error) error)})
- (if (= :success state)
- (update stats :success inc)
- (-> stats
- (update :error-details conj {:persisted-info-id (:id persisted-info)
- :error error})
- (update :error inc)))))))
+ (tracing/with-span :tasks "task.persist.refresh-model" {:db/id (u/the-id database)
+ :persist/card-id (:card_id persisted-info)
+ :persist/table (:table_name persisted-info)}
+ (log/infof "Attempting to refresh persisted model %s." (:card_id persisted-info))
+ (let [card (t2/select-one :model/Card :id (:card_id persisted-info))
+ definition (persisted-info/metadata->definition (:result_metadata card)
+ (:table_name persisted-info))
+ _ (t2/update! :model/PersistedInfo (u/the-id persisted-info)
+ {:definition definition,
+ :query_hash (persisted-info/query-hash (:dataset_query card))
+ :active false,
+ :refresh_begin :%now,
+ :refresh_end nil,
+ :state "refreshing"
+ :state_change_at :%now})
+ {:keys [state error]} (try
+ (refresh! refresher database definition card)
+ (catch Exception e
+ (log/infof e "Error refreshing persisting model with card-id %s"
+ (:card_id persisted-info))
+ {:state :error :error (ex-message e)}))]
+ (t2/update! :model/PersistedInfo (u/the-id persisted-info)
+ {:active (= state :success),
+ :refresh_end :%now,
+ :state (if (= state :success) "persisted" "error")
+ :state_change_at :%now
+ :error (when (= state :error) error)})
+ (if (= :success state)
+ (update stats :success inc)
+ (-> stats
+ (update :error-details conj {:persisted-info-id (:id persisted-info)
+ :error error})
+ (update :error inc))))))))
(defn- error-details
[results]
@@ -139,14 +143,16 @@
(not= (:type card-info) :model))
(let [database (-> persisted-info :database_id db-id->db)]
(log/infof "Unpersisting model with card-id %s" (:card_id persisted-info))
- (try
- (unpersist! refresher database persisted-info)
- (when-not (= "off" current-state)
- (t2/delete! :model/PersistedInfo :id (:id persisted-info)))
- (update stats :success inc)
- (catch Exception e
- (log/infof e "Error unpersisting model with card-id %s" (:card_id persisted-info))
- (update stats :error inc))))
+ (tracing/with-span :tasks "task.persist.unpersist-model" {:db/id (:database_id persisted-info)
+ :persist/card-id (:card_id persisted-info)}
+ (try
+ (unpersist! refresher database persisted-info)
+ (when-not (= "off" current-state)
+ (t2/delete! :model/PersistedInfo :id (:id persisted-info)))
+ (update stats :success inc)
+ (catch Exception e
+ (log/infof e "Error unpersisting model with card-id %s" (:card_id persisted-info))
+ (update stats :error inc)))))
(update stats :skipped inc))))
{:success 0, :error 0, :skipped 0}
deletables))]
@@ -158,8 +164,7 @@
persisted info records pointing to cards that are no longer models, archived cards/models, and all records where the corresponding
card or database has been permanently deleted."
[]
- (t2/select :model/PersistedInfo
- {:select [:p.*]
+ (let [hsql {:select [:p.*]
:from [[:persisted_info :p]]
:left-join [[:report_card :c] [:= :c.id :p.card_id]]
:where [:or
@@ -174,20 +179,23 @@
[:= :c.type "question"]
[:= :c.archived true]
;; card_id is set to null when the corresponding card is deleted
- [:= :p.card_id nil]]}))
+ [:= :p.card_id nil]]}]
+ (tracing/with-span :tasks "task.persist.find-deletable" {:db/statement (tracing/best-effort-sanitize-sql hsql)}
+ (t2/select :model/PersistedInfo hsql))))
(defn- refreshable-models
"Returns refreshable models for a database id. Must still be models and not archived."
[database-id]
- (t2/select :model/PersistedInfo
- {:select [:p.* :c.type :c.archived :c.name]
+ (let [hsql {:select [:p.* :c.type :c.archived :c.name]
:from [[:persisted_info :p]]
:left-join [[:report_card :c] [:= :c.id :p.card_id]]
:where [:and
[:= :p.database_id database-id]
[:in :p.state (persisted-info/refreshable-states)]
[:= :c.archived false]
- [:= :c.type "model"]]}))
+ [:= :c.type "model"]]}]
+ (tracing/with-span :tasks "task.persist.find-refreshable" {:db/id database-id :db/statement (tracing/best-effort-sanitize-sql hsql)}
+ (t2/select :model/PersistedInfo hsql))))
(defn- prune-all-deletable!
"Prunes all deletable PersistInfos, should not be called from tests as
diff --git a/src/metabase/notification/task/send.clj b/src/metabase/notification/task/send.clj
index 5670afaf0a5..9a9103847c2 100644
--- a/src/metabase/notification/task/send.clj
+++ b/src/metabase/notification/task/send.clj
@@ -10,6 +10,7 @@
[metabase.query-processor.timezone :as qp.timezone]
[metabase.task-history.core :as task-history]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.log :as log]
[toucan2.core :as t2])
(:import
@@ -107,19 +108,21 @@
(cond
(:active notification)
- (task-history/with-task-run (some-> (notification.send/notification->task-run-info notification) (assoc :auto-complete false))
- (try
- (log/info "Submitting to the notification queue")
- (task-history/with-task-history {:task "notification-trigger"
- :task_details {:trigger_type :notification-subscription/cron
- :notification_subscription_id subscription-id
- :cron_schedule (:cron_schedule subscription)
- :notification_ids [notification-id]}}
- (notification.send/send-notification! (assoc notification :triggering_subscription subscription)))
- (log/info "Submitted to the notification queue")
- (catch Exception e
- (log/error e "Failed to submit to the notification queue")
- (throw e))))
+ (tracing/with-span :tasks "task.notification.send" {:notification/subscription-id subscription-id
+ :notification/id notification-id}
+ (task-history/with-task-run (some-> (notification.send/notification->task-run-info notification) (assoc :auto-complete false))
+ (try
+ (log/info "Submitting to the notification queue")
+ (task-history/with-task-history {:task "notification-trigger"
+ :task_details {:trigger_type :notification-subscription/cron
+ :notification_subscription_id subscription-id
+ :cron_schedule (:cron_schedule subscription)
+ :notification_ids [notification-id]}}
+ (notification.send/send-notification! (assoc notification :triggering_subscription subscription)))
+ (log/info "Submitted to the notification queue")
+ (catch Exception e
+ (log/error e "Failed to submit to the notification queue")
+ (throw e)))))
(nil? notification)
(do
diff --git a/src/metabase/permissions/user.clj b/src/metabase/permissions/user.clj
index e1117a396a7..f46264e9a43 100644
--- a/src/metabase/permissions/user.clj
+++ b/src/metabase/permissions/user.clj
@@ -4,6 +4,7 @@
[metabase.permissions.models.data-permissions :as data-perms]
[metabase.permissions.path :as permissions.path]
[metabase.premium-features.core :refer [defenterprise]]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]))
(defenterprise user->tenant-collection-and-descendant-ids
@@ -15,30 +16,31 @@
(defn user-permissions-set
"Return a set of all permissions object paths that `user-or-id` has been granted access to. (2 DB Calls)"
[user-or-id]
- (let [s
- (set (when-let [user-id (u/the-id user-or-id)]
- (concat
+ (tracing/with-span :db-app "db-app.permissions-load" {}
+ (let [s
+ (set (when-let [user-id (u/the-id user-or-id)]
+ (concat
;; Current User always gets readwrite perms for their Personal Collection and for its descendants! (1 DB
;; Call)
- (map permissions.path/collection-readwrite-path
- ((requiring-resolve 'metabase.collections.models.collection/user->personal-collection-and-descendant-ids)
- user-or-id))
+ (map permissions.path/collection-readwrite-path
+ ((requiring-resolve 'metabase.collections.models.collection/user->personal-collection-and-descendant-ids)
+ user-or-id))
;; Current User always gets readwrite perms for their Tenant Collection and for its descendants! (3 DB Calls)
- (map permissions.path/collection-readwrite-path
- (user->tenant-collection-and-descendant-ids user-or-id))
+ (map permissions.path/collection-readwrite-path
+ (user->tenant-collection-and-descendant-ids user-or-id))
;; Current User always gets read perms for Transforms if they are an analyst (1 DB Call)
- (when (or (data-perms/is-data-analyst? user-id) (data-perms/is-superuser? user-id))
- (concat ["/collection/namespace/transforms/root/"]
- (map permissions.path/collection-readwrite-path ((requiring-resolve 'metabase.collections.models.collection/collections-in-namespace)
- :transforms))))
+ (when (or (data-perms/is-data-analyst? user-id) (data-perms/is-superuser? user-id))
+ (concat ["/collection/namespace/transforms/root/"]
+ (map permissions.path/collection-readwrite-path ((requiring-resolve 'metabase.collections.models.collection/collections-in-namespace)
+ :transforms))))
;; include the other Perms entries for any Group this User is in (1 DB Call)
- (map :object (app-db/query {:select [:p.object]
- :from [[:permissions_group_membership :pgm]]
- :join [[:permissions_group :pg] [:= :pgm.group_id :pg.id]
- [:permissions :p] [:= :p.group_id :pg.id]]
- :where [:= :pgm.user_id user-id]})))))]
- ;; Append permissions as a vector for more efficient iteration in checks that go over each permission linearly.
- (with-meta s {:as-vec (vec s)})))
+ (map :object (app-db/query {:select [:p.object]
+ :from [[:permissions_group_membership :pgm]]
+ :join [[:permissions_group :pg] [:= :pgm.group_id :pg.id]
+ [:permissions :p] [:= :p.group_id :pg.id]]
+ :where [:= :pgm.user_id user-id]})))))]
+ ;; Append permissions as a vector for more efficient iteration in checks that go over each permission linearly.
+ (with-meta s {:as-vec (vec s)}))))
diff --git a/src/metabase/premium_features/token_check.clj b/src/metabase/premium_features/token_check.clj
index 372286b9af8..c5ab6ec4c66 100644
--- a/src/metabase/premium_features/token_check.clj
+++ b/src/metabase/premium_features/token_check.clj
@@ -23,6 +23,7 @@
[metabase.premium-features.defenterprise :refer [defenterprise]]
[metabase.premium-features.settings :as premium-features.settings]
[metabase.settings.core :as setting]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.i18n :refer [trs tru]]
[metabase.util.json :as json]
@@ -218,19 +219,20 @@
[]
(when-let [token (premium-features.settings/premium-embedding-token)]
(when (mr/validate [:re RemoteCheckedToken] token)
- (let [site-uuid (premium-features.settings/site-uuid-for-premium-features-token-checks)
- stats (-> (metering-stats)
- ;; for backwards compatibility, we send values as strings
- (update-vals str))]
- (try
- (http/post (metering-url token token-check-url)
- {:body (json/encode (merge stats
- {:site-uuid site-uuid
- :mb-version (:tag config/mb-version-info)}))
- :content-type :json
- :throw-exceptions false})
- (catch Throwable e
- (log/error e "Error sending metering events")))))))
+ (tracing/with-span :tasks "metering.send-events" {}
+ (let [site-uuid (premium-features.settings/site-uuid-for-premium-features-token-checks)
+ stats (-> (metering-stats)
+ ;; for backwards compatibility, we send values as strings
+ (update-vals str))]
+ (try
+ (http/post (metering-url token token-check-url)
+ {:body (json/encode (merge stats
+ {:site-uuid site-uuid
+ :mb-version (:tag config/mb-version-info)}))
+ :content-type :json
+ :throw-exceptions false})
+ (catch Throwable e
+ (log/error e "Error sending metering events"))))))))
;;;;;;;;;;;;;;;;;;;; Airgap Tokens ;;;;;;;;;;;;;;;;;;;;
diff --git a/src/metabase/pulse/task/send_pulses.clj b/src/metabase/pulse/task/send_pulses.clj
index 765c6852a7a..0f236d6411c 100644
--- a/src/metabase/pulse/task/send_pulses.clj
+++ b/src/metabase/pulse/task/send_pulses.clj
@@ -17,6 +17,7 @@
[metabase.query-processor.timezone :as qp.timezone]
[metabase.task-history.core :as task-history]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.cron :as u.cron]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -48,22 +49,24 @@
(defn- send-pulse!
[pulse-id channel-ids]
- (try
- (if-let [pulse (models.pulse/retrieve-notification pulse-id
- :archived false
- ;; alerts should all be migrated to notifications by now
- :alert_condition nil)]
- (task-history/with-task-run (some-> (pulse.send/pulse->task-run-info pulse) (assoc :auto-complete false))
- (task-history/with-task-history {:task "send-pulse"
- :task_details {:pulse-id pulse-id
- :channel-ids (seq channel-ids)}}
- (log/debugf "Starting Pulse Execution: %d" pulse-id)
- (pulse.send/send-pulse! pulse :channel-ids channel-ids :async? true)
- (log/debugf "Finished Pulse Execution: %d" pulse-id)
- :done))
- (log/debugf "Pulse %d not found, Skipping." pulse-id))
- (catch Throwable e
- (log/errorf e "Error sending Pulse %d to channel ids: %s" pulse-id (str/join ", " channel-ids)))))
+ (tracing/with-span :tasks "task.pulse.send" {:pulse/id pulse-id
+ :pulse/channel-count (count channel-ids)}
+ (try
+ (if-let [pulse (models.pulse/retrieve-notification pulse-id
+ :archived false
+ ;; alerts should all be migrated to notifications by now
+ :alert_condition nil)]
+ (task-history/with-task-run (some-> (pulse.send/pulse->task-run-info pulse) (assoc :auto-complete false))
+ (task-history/with-task-history {:task "send-pulse"
+ :task_details {:pulse-id pulse-id
+ :channel-ids (seq channel-ids)}}
+ (log/debugf "Starting Pulse Execution: %d" pulse-id)
+ (pulse.send/send-pulse! pulse :channel-ids channel-ids :async? true)
+ (log/debugf "Finished Pulse Execution: %d" pulse-id)
+ :done))
+ (log/debugf "Pulse %d not found, Skipping." pulse-id))
+ (catch Throwable e
+ (log/errorf e "Error sending Pulse %d to channel ids: %s" pulse-id (str/join ", " channel-ids))))))
(defn- send-trigger-timezone
[]
@@ -104,24 +107,25 @@
(defn- clear-pulse-channels-no-recipients!
"Delete PulseChannels that have no recipients and no channel set for a pulse, returns the channel ids that were deleted."
[pulse-id]
- (when-let [ids-to-delete (seq
- (for [channel (t2/select [:model/PulseChannel :id :details :channel_id :channel_type]
- :pulse_id pulse-id
- :id [:not-in {:select [[:pulse_channel_id :id]]
- :from :pulse_channel_recipient
- :group-by [:pulse_channel_id]
- :having [:>= :%count.* [:raw 1]]}])
- :when (case (:channel_type channel)
- :email
- (empty? (get-in channel [:details :emails]))
- :slack
- (empty? (get-in channel [:details :channel]))
- :http
- (nil? (:channel_id channel)))]
- (:id channel)))]
- (log/infof "Deleting %d PulseChannels with id: %s due to having no recipients" (count ids-to-delete) (str/join ", " ids-to-delete))
- (t2/delete! :model/PulseChannel :id [:in ids-to-delete])
- (set ids-to-delete)))
+ (tracing/with-span :tasks "task.pulse.clear-orphan-channels" {:pulse/id pulse-id}
+ (when-let [ids-to-delete (seq
+ (for [channel (t2/select [:model/PulseChannel :id :details :channel_id :channel_type]
+ :pulse_id pulse-id
+ :id [:not-in {:select [[:pulse_channel_id :id]]
+ :from :pulse_channel_recipient
+ :group-by [:pulse_channel_id]
+ :having [:>= :%count.* [:raw 1]]}])
+ :when (case (:channel_type channel)
+ :email
+ (empty? (get-in channel [:details :emails]))
+ :slack
+ (empty? (get-in channel [:details :channel]))
+ :http
+ (nil? (:channel_id channel)))]
+ (:id channel)))]
+ (log/infof "Deleting %d PulseChannels with id: %s due to having no recipients" (count ids-to-delete) (str/join ", " ids-to-delete))
+ (t2/delete! :model/PulseChannel :id [:in ids-to-delete])
+ (set ids-to-delete))))
(defn- send-pulse!*
"Do several things:
diff --git a/src/metabase/query_processor.clj b/src/metabase/query_processor.clj
index fad0c7688a5..35bb3b0fa78 100644
--- a/src/metabase/query_processor.clj
+++ b/src/metabase/query_processor.clj
@@ -19,6 +19,7 @@
[metabase.query-processor.reducible :as qp.reducible]
[metabase.query-processor.schema :as qp.schema]
[metabase.query-processor.setup :as qp.setup]
+ [metabase.tracing.core :as tracing]
[metabase.util.log :as log]
[metabase.util.malli :as mu]))
@@ -44,10 +45,14 @@
(defn- process-query** [query rff]
(qp.debug/debug> (list `process-query query))
- (let [preprocessed (qp.preprocess/preprocess query)
- compiled (qp.compile/attach-compiled-query preprocessed)
- rff (qp.postprocess/post-processing-rff preprocessed rff)]
- (qp.execute/execute compiled rff)))
+ (tracing/with-span :qp "qp.pipeline" {:query/type (some-> (:type query) name)
+ :db/id (:database query)}
+ (let [preprocessed (tracing/with-span :qp "qp.preprocess" {}
+ (qp.preprocess/preprocess query))
+ compiled (qp.compile/attach-compiled-query preprocessed)
+ rff (qp.postprocess/post-processing-rff preprocessed rff)]
+ (tracing/with-span :qp "qp.execute" {}
+ (qp.execute/execute compiled rff)))))
(def ^:private ^{:arglists '([query rff])} process-query* nil)
@@ -76,9 +81,11 @@
([query :- ::qp.schema/any-query
rff :- [:maybe ::qp.schema/rff]]
- (qp.setup/with-qp-setup [query query]
- (let [rff (or rff qp.reducible/default-rff)]
- (process-query* query rff)))))
+ (tracing/with-span :qp "qp.process-query" {:query/type (some-> (:type query) name)
+ :db/id (:database query)}
+ (qp.setup/with-qp-setup [query query]
+ (let [rff (or rff qp.reducible/default-rff)]
+ (process-query* query rff))))))
(mu/defn userland-query :- ::qp.schema/any-query
"Add middleware options and `:info` to a `query` so it is ran as a 'userland' query, which slightly changes the QP
diff --git a/src/metabase/query_processor/middleware/cache.clj b/src/metabase/query_processor/middleware/cache.clj
index c89d52e5715..50e90cfe759 100644
--- a/src/metabase/query_processor/middleware/cache.clj
+++ b/src/metabase/query_processor/middleware/cache.clj
@@ -21,6 +21,7 @@
[metabase.query-processor.pipeline :as qp.pipeline]
[metabase.query-processor.schema :as qp.schema]
[metabase.query-processor.util :as qp.util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -266,5 +267,6 @@
(let [cacheable? (is-cacheable? query)]
(log/tracef "Query is %scacheable: %s" (if-not cacheable? "not " "") (get-cache-eligibility-description query))
(if cacheable?
- (run-query-with-cache qp query rff)
+ (tracing/with-span :qp "qp.cache" {:cache/eligible true}
+ (run-query-with-cache qp query rff))
(qp query rff)))))
diff --git a/src/metabase/query_processor/middleware/process_userland_query.clj b/src/metabase/query_processor/middleware/process_userland_query.clj
index 94672a35f6c..71a20d400b2 100644
--- a/src/metabase/query_processor/middleware/process_userland_query.clj
+++ b/src/metabase/query_processor/middleware/process_userland_query.clj
@@ -14,6 +14,7 @@
[metabase.queries.models.query :as query]
[metabase.query-processor.schema :as qp.schema]
[metabase.query-processor.util :as qp.util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -42,11 +43,12 @@
(defn- save-execution-metadata!*
"Save a `QueryExecution` and update the average execution time for the corresponding `Query`."
[{query :json_query, query-hash :hash, running-time :running_time, context :context :as query-execution}]
- (when-not (:cache_hit query-execution)
- (query/save-query-and-update-average-execution-time! query query-hash running-time))
- (if-not context
- (log/warn "Cannot save QueryExecution, missing :context")
- (t2/insert-returning-pk! :model/QueryExecution (dissoc query-execution :json_query))))
+ (tracing/with-span :db-app "db-app.save-query-execution" {}
+ (when-not (:cache_hit query-execution)
+ (query/save-query-and-update-average-execution-time! query query-hash running-time))
+ (if-not context
+ (log/warn "Cannot save QueryExecution, missing :context")
+ (t2/insert-returning-pk! :model/QueryExecution (dissoc query-execution :json_query)))))
(defn- save-execution-metadata!
"Save a `QueryExecution` row containing `execution-info`. Done asynchronously when a query is finished."
diff --git a/src/metabase/search/appdb/core.clj b/src/metabase/search/appdb/core.clj
index 7fdf12e9e91..899ba2c4dd6 100644
--- a/src/metabase/search/appdb/core.clj
+++ b/src/metabase/search/appdb/core.clj
@@ -19,6 +19,7 @@
[metabase.search.spec :as search.spec]
[metabase.search.util :as search.util]
[metabase.settings.core :as setting]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.i18n :as i18n]
[metabase.util.json :as json]
@@ -145,33 +146,34 @@
:index-state-after @@#'search.index/*indexes*
:index-metadata (t2/select :model/SearchIndexMetadata :engine :appdb)}))))
- (try
- (when (setting/string->boolean (:mb-experimental-search-block-on-queue env/env))
- ;; wait for a bit for the queue to be drained
- (let [pending-updates #(.size ^Queue @#'search.ingestion/queue)]
- (when-not (u/poll {:thunk pending-updates
- :done? zero?
- :timeout-ms 2000
- :interval-ms 100})
- (log/warn "Returning search results even though they may be stale. Queue size:" (pending-updates)))))
-
- (let [weights (search.config/weights search-ctx)
- scorers (search.scoring/scorers search-ctx)
- query (->> (search.index/search-query search-string search-ctx [:legacy_input])
- (add-collection-join-and-where-clauses search-ctx)
- (add-table-where-clauses search-ctx)
- (#(sql.helpers/where % (search.filter/transform-source-type-where-clause
- search-ctx
- :search_index.model
- :search_index.source_type)))
- (search.scoring/with-scores search-ctx scorers)
- (search.filter/with-filters search-ctx))]
- (->> (t2/query query)
- (map (partial rehydrate weights (keys scorers)))))
- (catch Exception e
- ;; Rule out the error coming from stale index metadata.
- (#'search.index/sync-tracking-atoms!)
- (throw e))))
+ (tracing/with-span :search "search.appdb.query" {:search/query-length (count search-string)}
+ (try
+ (when (setting/string->boolean (:mb-experimental-search-block-on-queue env/env))
+ ;; wait for a bit for the queue to be drained
+ (let [pending-updates #(.size ^Queue @#'search.ingestion/queue)]
+ (when-not (u/poll {:thunk pending-updates
+ :done? zero?
+ :timeout-ms 2000
+ :interval-ms 100})
+ (log/warn "Returning search results even though they may be stale. Queue size:" (pending-updates)))))
+
+ (let [weights (search.config/weights search-ctx)
+ scorers (search.scoring/scorers search-ctx)
+ query (->> (search.index/search-query search-string search-ctx [:legacy_input])
+ (add-collection-join-and-where-clauses search-ctx)
+ (add-table-where-clauses search-ctx)
+ (#(sql.helpers/where % (search.filter/transform-source-type-where-clause
+ search-ctx
+ :search_index.model
+ :search_index.source_type)))
+ (search.scoring/with-scores search-ctx scorers)
+ (search.filter/with-filters search-ctx))]
+ (->> (t2/query query)
+ (map (partial rehydrate weights (keys scorers)))))
+ (catch Exception e
+ ;; Rule out the error coming from stale index metadata.
+ (#'search.index/sync-tracking-atoms!)
+ (throw e)))))
(defmethod search.engine/results :search.engine/appdb
[search-ctx]
diff --git a/src/metabase/search/appdb/index.clj b/src/metabase/search/appdb/index.clj
index 377ee08fc30..8ac64def192 100644
--- a/src/metabase/search/appdb/index.clj
+++ b/src/metabase/search/appdb/index.clj
@@ -15,6 +15,7 @@
[metabase.search.ingestion :as search.ingestion]
[metabase.search.models.search-index-metadata :as search-index-metadata]
[metabase.search.spec :as search.spec]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.i18n :as i18n]
[metabase.util.json :as json]
@@ -316,10 +317,11 @@
"Indexes the documents. The context should be :search/updating or :search/reindexing.
Context should be :search/updating or :search/reindexing to help control how to manage the updates"
[context document-reducible]
- (transduce (comp (partition-all insert-batch-size)
- (map (partial batch-update! context)))
- (partial merge-with +)
- document-reducible))
+ (tracing/with-span :search "search.appdb.index-docs" {:search/context (name context)}
+ (transduce (comp (partition-all insert-batch-size)
+ (map (partial batch-update! context)))
+ (partial merge-with +)
+ document-reducible)))
(defmethod search.engine/update! :search.engine/appdb [_engine document-reducible]
(index-docs! :search/updating document-reducible))
diff --git a/src/metabase/search/core.clj b/src/metabase/search/core.clj
index 2623ebd8a55..7f7518ea36a 100644
--- a/src/metabase/search/core.clj
+++ b/src/metabase/search/core.clj
@@ -10,6 +10,7 @@
[metabase.search.ingestion :as search.ingestion]
[metabase.search.spec :as search.spec]
[metabase.search.util :as search.util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[potemkin :as p]))
@@ -87,48 +88,50 @@
[& {:as opts}]
(when (supports-index?)
(log/info "Initializing search indexes")
- (lib-be/with-metadata-provider-cache
- ;; If there are multiple indexes, return the peak inserted for each type. In practice, they should all be the same.
- (try
- (let [timer (u/start-timer)
- report (reduce (partial merge-with max)
- nil
- (for [e (search.engine/active-engines)]
- (search.engine/init! e opts)))
- duration (u/since-ms timer)]
- (if (seq report)
- (do
- (analytics/inc! :metabase-search/index-reindex-ms duration)
- (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)
- (doseq [[model cnt] report]
- (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))
- (log/infof "Index initialized in %.0fms %s" duration (sort-by (comp - val) report))
- report)
- (log/info "Found existing search index, and using it.")))
- (catch Exception e
- (analytics/inc! :metabase-search/index-error)
- (throw e))))))
+ (tracing/with-span :search "search.init-index" {}
+ (lib-be/with-metadata-provider-cache
+ ;; If there are multiple indexes, return the peak inserted for each type. In practice, they should all be the same.
+ (try
+ (let [timer (u/start-timer)
+ report (reduce (partial merge-with max)
+ nil
+ (for [e (search.engine/active-engines)]
+ (search.engine/init! e opts)))
+ duration (u/since-ms timer)]
+ (if (seq report)
+ (do
+ (analytics/inc! :metabase-search/index-reindex-ms duration)
+ (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)
+ (doseq [[model cnt] report]
+ (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))
+ (log/infof "Index initialized in %.0fms %s" duration (sort-by (comp - val) report))
+ report)
+ (log/info "Found existing search index, and using it.")))
+ (catch Exception e
+ (analytics/inc! :metabase-search/index-error)
+ (throw e)))))))
(defn- reindex-logic! [opts]
(when (supports-index?)
- (lib-be/with-metadata-provider-cache
- (try
- (log/info "Reindexing searchable entities")
- (let [timer (u/start-timer)
- report (reduce (partial merge-with max)
- nil
- (for [e (search.engine/active-engines)]
- (search.engine/reindex! e opts)))
- duration (u/since-ms timer)]
- (analytics/inc! :metabase-search/index-reindex-ms duration)
- (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)
- (doseq [[model cnt] report]
- (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))
- (log/infof "Done reindexing in %.0fms %s" duration (sort-by (comp - val) report))
- report)
- (catch Exception e
- (analytics/inc! :metabase-search/index-error)
- (throw e))))))
+ (tracing/with-span :search "search.reindex" {}
+ (lib-be/with-metadata-provider-cache
+ (try
+ (log/info "Reindexing searchable entities")
+ (let [timer (u/start-timer)
+ report (reduce (partial merge-with max)
+ nil
+ (for [e (search.engine/active-engines)]
+ (search.engine/reindex! e opts)))
+ duration (u/since-ms timer)]
+ (analytics/inc! :metabase-search/index-reindex-ms duration)
+ (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)
+ (doseq [[model cnt] report]
+ (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))
+ (log/infof "Done reindexing in %.0fms %s" duration (sort-by (comp - val) report))
+ report)
+ (catch Exception e
+ (analytics/inc! :metabase-search/index-error)
+ (throw e)))))))
(defn reindex!
"Populate a new index, and make it active. Simultaneously updates the current index.
diff --git a/src/metabase/search/impl.clj b/src/metabase/search/impl.clj
index a89c0831685..1d43fefa3c9 100644
--- a/src/metabase/search/impl.clj
+++ b/src/metabase/search/impl.clj
@@ -11,6 +11,7 @@
[metabase.search.filter :as search.filter]
[metabase.search.in-place.filter :as search.in-place.filter]
[metabase.search.in-place.scoring :as scoring]
+ [metabase.tracing.core :as tracing]
[metabase.transforms.feature-gating :as transforms.gating]
[metabase.util :as u]
[metabase.util.i18n :refer [deferred-tru tru]]
@@ -440,19 +441,22 @@
(mu/defn search
"Builds a search query that includes all the searchable entities, and runs it."
[search-ctx :- search.config/SearchContext]
- (let [reducible-results (search.engine/results search-ctx)
- scoring-ctx (select-keys search-ctx [:search-engine :search-string :search-native-query])
- xf (comp
- (take search.config/*db-max-results*)
- (map normalize-result)
- (filter (partial check-permissions-for-model search-ctx))
- (map (partial normalize-result-more search-ctx))
- (keep #(search.engine/score scoring-ctx %)))
- total-results (cond->> (scoring/top-results reducible-results search.config/max-filtered-results xf)
- true hydrate-dashboards
- true hydrate-user-metadata
- (:include-metadata? search-ctx) (add-metadata)
- (:model-ancestors? search-ctx) (add-dataset-collection-hierarchy)
- true (add-collection-effective-location)
- true (map serialize))]
- (search-results search-ctx search.engine/model-set total-results)))
+ (tracing/with-span :search "search.execute" {:search/engine (name (:search-engine search-ctx))
+ :search/query-length (count (:search-string search-ctx))
+ :search/model-count (count (:models search-ctx))}
+ (let [reducible-results (search.engine/results search-ctx)
+ scoring-ctx (select-keys search-ctx [:search-engine :search-string :search-native-query])
+ xf (comp
+ (take search.config/*db-max-results*)
+ (map normalize-result)
+ (filter (partial check-permissions-for-model search-ctx))
+ (map (partial normalize-result-more search-ctx))
+ (keep #(search.engine/score scoring-ctx %)))
+ total-results (cond->> (scoring/top-results reducible-results search.config/max-filtered-results xf)
+ true hydrate-dashboards
+ true hydrate-user-metadata
+ (:include-metadata? search-ctx) (add-metadata)
+ (:model-ancestors? search-ctx) (add-dataset-collection-hierarchy)
+ true (add-collection-effective-location)
+ true (map serialize))]
+ (search-results search-ctx search.engine/model-set total-results))))
diff --git a/src/metabase/search/ingestion.clj b/src/metabase/search/ingestion.clj
index 355fd7a1b16..5494df087d9 100644
--- a/src/metabase/search/ingestion.clj
+++ b/src/metabase/search/ingestion.clj
@@ -9,6 +9,7 @@
[metabase.lib-be.core :as lib-be]
[metabase.search.engine :as search.engine]
[metabase.search.spec :as search.spec]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.performance :as perf]
@@ -231,22 +232,23 @@
;; We are partitioning the documents into batches at this level and sending each batch to all the engines
;; to avoid having to retain the head of the sequences as we work through all the documents.
;; Individual engines may also partition the documents further if they prefer
- (let [timer (u/start-timer)
- update-report (reduce (fn [_ batch] (search.engine/update! e batch)) nil
- (eduction (partition-all 150) documents-reducible))
- delete-report (reduce (fn [acc batch]
- (->> batch
- (remove nil?)
- (u/group-by first second)
- (map (fn [[group ids]] (search.engine/delete! e group ids)))
- (apply merge-with + acc))) {}
- (eduction (partition-all 1000) removed-models-reducible))
- duration (u/since-ms timer)]
- (log/debugf "Updated search entries in %.0fms Updated: %s Deleted: %s" duration (sort-by (comp - val) update-report) (sort-by (comp - val) delete-report))
- (analytics/inc! :metabase-search/index-update-ms duration)
- (prometheus/observe! :metabase-search/index-update-duration-ms duration)
- (doseq [[model cnt] (merge-with + update-report delete-report)]
- (analytics/inc! :metabase-search/index-updates {:model model} cnt)))))
+ (tracing/with-span :search "search.ingestion.update" {:search/engine (name e)}
+ (let [timer (u/start-timer)
+ update-report (reduce (fn [_ batch] (search.engine/update! e batch)) nil
+ (eduction (partition-all 150) documents-reducible))
+ delete-report (reduce (fn [acc batch]
+ (->> batch
+ (remove nil?)
+ (u/group-by first second)
+ (map (fn [[group ids]] (search.engine/delete! e group ids)))
+ (apply merge-with + acc))) {}
+ (eduction (partition-all 1000) removed-models-reducible))
+ duration (u/since-ms timer)]
+ (log/debugf "Updated search entries in %.0fms Updated: %s Deleted: %s" duration (sort-by (comp - val) update-report) (sort-by (comp - val) delete-report))
+ (analytics/inc! :metabase-search/index-update-ms duration)
+ (prometheus/observe! :metabase-search/index-update-duration-ms duration)
+ (doseq [[model cnt] (merge-with + update-report delete-report)]
+ (analytics/inc! :metabase-search/index-updates {:model model} cnt))))))
(comment
(u/group-by first second [["metric" 124] ["dataset" 124] ["metric" 124] ["other" 5]]))
@@ -271,24 +273,25 @@
(defn bulk-ingest!
"Process the given search model updates."
[updates]
- (lib-be/with-metadata-provider-cache
- (if (seq (search.engine/active-engines))
- (let [documents (->> (for [[search-model where-clauses] (u/group-by first second updates)]
- (spec-index-reducible search-model (into [:or] (distinct where-clauses))))
- ;; init collection is only for clj-kondo, as we know that the list is non-empty
- (reduce u/rconcat [])
- query->documents)
- passed-documents (map extract-model-and-id updates)
- indexed-documents (map (juxt :model (comp str :id)) (into [] documents))
- ;; TODO: The list of documents to delete is not completely accurate.
- ;; We are attempting to figure it out based on the ids that are passed in to be indexed vs. the ids of the rows that were actually indexed.
- ;; This will not work for cases like indexed-entries with compound PKs,
- ;; but it's fine for now because that model doesn't have a where clause so never needs to be purged during an update.
- ;; Long-term, we should find a better approach to knowing what to purge.
- to-delete (remove (set indexed-documents) passed-documents)]
-
- (update! documents to-delete))
- {})))
+ (tracing/with-span :search "search.ingestion.bulk-ingest" {:search/update-count (count updates)}
+ (lib-be/with-metadata-provider-cache
+ (if (seq (search.engine/active-engines))
+ (let [documents (->> (for [[search-model where-clauses] (u/group-by first second updates)]
+ (spec-index-reducible search-model (into [:or] (distinct where-clauses))))
+ ;; init collection is only for clj-kondo, as we know that the list is non-empty
+ (reduce u/rconcat [])
+ query->documents)
+ passed-documents (map extract-model-and-id updates)
+ indexed-documents (map (juxt :model (comp str :id)) (into [] documents))
+ ;; TODO: The list of documents to delete is not completely accurate.
+ ;; We are attempting to figure it out based on the ids that are passed in to be indexed vs. the ids of the rows that were actually indexed.
+ ;; This will not work for cases like indexed-entries with compound PKs,
+ ;; but it's fine for now because that model doesn't have a where clause so never needs to be purged during an update.
+ ;; Long-term, we should find a better approach to knowing what to purge.
+ to-delete (remove (set indexed-documents) passed-documents)]
+
+ (update! documents to-delete))
+ {}))))
(defn- track-queue-size! []
(analytics/set! :metabase-search/queue-size (.size queue)))
diff --git a/src/metabase/search/task/search_index.clj b/src/metabase/search/task/search_index.clj
index 5ed31f890e1..c6826673097 100644
--- a/src/metabase/search/task/search_index.clj
+++ b/src/metabase/search/task/search_index.clj
@@ -8,6 +8,7 @@
[metabase.search.ingestion :as ingestion]
[metabase.startup.core :as startup]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.queue :as queue])
(:import
(java.time Instant)
@@ -34,8 +35,9 @@
"Create a new index, if necessary"
[]
(when (search/supports-index?)
- (cluster-lock/with-cluster-lock cluster-lock-name
- (search/init-index! {:force-reset? false, :re-populate? false}))))
+ (tracing/with-span :search "search.task.init" {}
+ (cluster-lock/with-cluster-lock cluster-lock-name
+ (search/init-index! {:force-reset? false, :re-populate? false})))))
(task/defjob ^{DisallowConcurrentExecution true
:doc "Populate a new Search Index"}
diff --git a/src/metabase/server/handler.clj b/src/metabase/server/handler.clj
index 798d6a57e15..18b090e4473 100644
--- a/src/metabase/server/handler.clj
+++ b/src/metabase/server/handler.clj
@@ -18,6 +18,7 @@
[metabase.server.middleware.session :as mw.session]
[metabase.server.middleware.settings-cache :as mw.settings-cache]
[metabase.server.middleware.ssl :as mw.ssl]
+ [metabase.server.middleware.trace :as mw.trace]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
[ring.core.protocols :as ring.protocols]
@@ -94,6 +95,7 @@
#'mw.misc/add-content-type ; Adds a Content-Type header for any response that doesn't already have one
#'mw.misc/disable-streaming-buffering ; Add header to streaming (async) responses so nginx doesn't buffer keepalive bytes
#'wrap-gzip ; GZIP response if client can handle it
+ #'mw.trace/wrap-trace ; Create root OpenTelemetry span per request (after request-id is available)
#'mw.request-id/wrap-request-id ; Add a unique request ID to the request
#'mw.misc/bind-request ; bind `metabase.middleware.misc/*request*` for the duration of the request
#'mw.ssl/redirect-to-https-middleware
diff --git a/src/metabase/server/middleware/log.clj b/src/metabase/server/middleware/log.clj
index 2fa57b13a73..3a44d2f3da6 100644
--- a/src/metabase/server/middleware/log.clj
+++ b/src/metabase/server/middleware/log.clj
@@ -19,6 +19,7 @@
(clojure.core.async.impl.channels ManyToManyChannel)
(com.mchange.v2.c3p0 PoolBackedDataSource)
(metabase.server.streaming_response StreamingResponse)
+ (org.apache.logging.log4j ThreadContext)
(org.eclipse.jetty.util.thread QueuedThreadPool)))
(set! *warn-on-reflection* true)
@@ -159,29 +160,42 @@
(defn- log-core-async-response
"For async responses that return a `core.async` channel, wait for the channel to return a response before logging the
- API request info."
- [{{chan :body, :as _response} :response, :as info}]
+ API request info. `mdc-ctx` is the MDC snapshot from the request thread — restored before logging
+ so trace_id/span_id appear in async log lines."
+ [{{chan :body, :as _response} :response, :as info} mdc-ctx]
{:pre [(async.u/promise-chan? chan)]}
;; [async] wait for the pipe to close the canceled/finished channel and log the API response
(a/go
(let [result (a/<! chan)]
- (log-info (assoc info :async-status (if (nil? result) "canceled" "completed"))))))
-
-(defn- log-streaming-response [{{streaming-response :body, :as _response} :response, :as info}]
+ (try
+ (ThreadContext/putAll mdc-ctx)
+ (log-info (assoc info :async-status (if (nil? result) "canceled" "completed")))
+ (finally
+ (ThreadContext/removeAll (keys mdc-ctx)))))))
+
+(defn- log-streaming-response
+ "Log a streaming response after it completes. `mdc-ctx` is the MDC snapshot from the request thread."
+ [{{streaming-response :body, :as _response} :response, :as info} mdc-ctx]
;; [async] wait for the streaming response to be canceled/finished channel and log the API response
(let [finished-chan (streaming-response/finished-chan streaming-response)]
(a/go
(let [result (a/<! finished-chan)]
- (log-info (assoc info :async-status (name result)))))))
+ (try
+ (ThreadContext/putAll mdc-ctx)
+ (log-info (assoc info :async-status (name result)))
+ (finally
+ (ThreadContext/removeAll (keys mdc-ctx))))))))
(defn- logged-response
"Log an API response. Returns response, possibly modified (i.e., core.async channels will be wrapped); this value
- should be passed to the normal `respond` function."
+ should be passed to the normal `respond` function.
+ Captures MDC context from the request thread so that async log lines include trace_id/span_id."
[{{:keys [body], :as response} :response, :as info}]
- (condp instance? body
- ManyToManyChannel (log-core-async-response info)
- StreamingResponse (log-streaming-response info)
- (log-info info))
+ (let [mdc-ctx (ThreadContext/getImmutableContext)]
+ (condp instance? body
+ ManyToManyChannel (log-core-async-response info mdc-ctx)
+ StreamingResponse (log-streaming-response info mdc-ctx)
+ (log-info info)))
response)
;;; +----------------------------------------------------------------------------------------------------------------+
diff --git a/src/metabase/server/middleware/session.clj b/src/metabase/server/middleware/session.clj
index 9bcb15aaa89..94b4fb28482 100644
--- a/src/metabase/server/middleware/session.clj
+++ b/src/metabase/server/middleware/session.clj
@@ -29,6 +29,7 @@
[metabase.request.schema :as request.schema]
[metabase.session.core :as session]
[metabase.settings.core :as setting]
+ [metabase.tracing.core :as tracing]
[metabase.util.honey-sql-2 :as h2x]
[metabase.util.i18n :as i18n]
[metabase.util.log :as log]
@@ -246,7 +247,9 @@
token OR a valid API key was passed."
[handler]
(fn [request respond raise]
- (handler (merge-current-user-info request) respond raise)))
+ (let [request' (tracing/with-span :db-app "db-app.session-lookup" {}
+ (merge-current-user-info request))]
+ (handler request' respond raise))))
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | bind-current-user |
diff --git a/src/metabase/server/middleware/trace.clj b/src/metabase/server/middleware/trace.clj
new file mode 100644
index 00000000000..fd73044b54d
--- /dev/null
+++ b/src/metabase/server/middleware/trace.clj
@@ -0,0 +1,116 @@
+(ns metabase.server.middleware.trace
+ "Ring middleware that creates a root OpenTelemetry span per HTTP request.
+
+ When a frontend `traceparent` header is present, the trace ID is extracted
+ and forced onto the root span via a custom IdGenerator (in tracing.core).
+ This correlates frontend and backend spans under the same trace WITHOUT
+ creating a parent-child link to a non-existent browser span — avoiding
+ the 'root span not yet received' message in Tempo.
+
+ Uses clj-otel's `new-span!`/`end-span!` (manual span lifecycle) instead of
+ `with-span!` (auto-close) because Metabase Ring handlers are async — the handler
+ returns immediately and calls `respond` later on a different thread. The span
+ must stay open until `respond` or `raise` is called."
+ (:require
+ [metabase.tracing.core :as tracing]
+ [metabase.util.log :as log]
+ [steffan-westcott.clj-otel.api.trace.span :as span])
+ (:import
+ (io.opentelemetry.api.trace Span StatusCode)
+ (io.opentelemetry.context Context Scope)))
+
+(set! *warn-on-reflection* true)
+
+(defn- parse-frontend-trace-id
+ "Extract the trace ID from a W3C traceparent header.
+ Returns nil if the header is missing or malformed.
+ Format: 00-{32-hex-trace-id}-{16-hex-parent-id}-{2-hex-flags}"
+ ^String [request]
+ (when-let [tp (get-in request [:headers "traceparent"])]
+ (second (re-matches #"00-([0-9a-f]{32})-[0-9a-f]{16}-[0-9a-f]{2}" tp))))
+
+(defn- end-span-with-status!
+ "End a span from its context, setting HTTP status code and OTel status.
+ Sets OK for 1xx-4xx, ERROR for 5xx (per OTel HTTP server span conventions)."
+ [^Span span ^Context span-ctx status-code]
+ (when status-code
+ (.setAttribute span "http.status_code" (long status-code))
+ (if (>= (long status-code) 500)
+ (.setStatus span StatusCode/ERROR)
+ (.setStatus span StatusCode/OK)))
+ (span/end-span! {:context span-ctx}))
+
+(defn- api-request?
+ "Returns true for API requests (/api/*) which are the only requests worth tracing.
+ Excludes static files (/app/*, /favicon*), the root page (/), and any other
+ non-API routes that would create noise as independent traces in the collector."
+ [request]
+ (.startsWith (str (:uri request)) "/api/"))
+
+(defn wrap-trace
+ "Middleware that wraps each HTTP request in an OpenTelemetry span (`:api` group).
+ Extracts trace ID from the frontend's traceparent header for correlation.
+ Includes HTTP method, URI, and request ID as span attributes.
+ Skips static file requests (/app/*, /favicon*) to avoid trace noise.
+
+ Uses clj-otel's `new-span!`/`end-span!` for manual span lifecycle to correctly
+ handle async Ring handlers — the span stays open until `respond` or `raise` is called."
+ [handler]
+ (fn [request respond raise]
+ (if (and (tracing/group-enabled? :api)
+ (api-request? request))
+ (do
+ ;; If frontend sent a traceparent, force its trace ID for this thread.
+ ;; The custom IdGenerator in tracing.core will return it instead of a random one.
+ (when-let [tid (parse-frontend-trace-id request)]
+ (tracing/force-trace-id! tid))
+ (let [;; Create as root span — no parent context means no parent-child link
+ ;; to the browser's phantom span. If a frontend trace ID was forced,
+ ;; the IdGenerator returns it; otherwise a random trace ID is generated.
+ span-ctx (span/new-span! {:name "api.request"
+ :parent nil
+ :span-kind :server
+ :attributes (cond-> {:http/method (some-> (:request-method request) name)
+ :http/url (:uri request)
+ :http/request-id (:metabase-request-id request)
+ :http/referer (get-in request [:headers "referer"])}
+ (:query-string request)
+ (assoc :http/query-string (:query-string request)))})
+ ^Span span (span/get-span span-ctx)
+ ^Scope scope (.makeCurrent ^Context span-ctx)
+ span-id (.getSpanId (.getSpanContext span))]
+ (tracing/inject-trace-id-into-mdc!)
+ ;; Tag profiling samples with this span's ID and set pyroscope.profile.id
+ ;; attribute on the span (no-op without Pyroscope agent).
+ (tracing/set-pyroscope-context! span span-id "api.request")
+ (try
+ (handler request
+ (fn trace-respond [response]
+ (try
+ (end-span-with-status! span span-ctx (:status response))
+ (catch Throwable t
+ (log/error t "Error ending trace span")))
+ (respond response))
+ (fn trace-raise [exception]
+ (try
+ (.setStatus span StatusCode/ERROR)
+ (.recordException span exception)
+ (span/end-span! {:context span-ctx})
+ (catch Throwable t
+ (log/error t "Error ending trace span on exception")))
+ (raise exception)))
+ (catch Throwable t
+ ;; Handler itself threw synchronously — end span and re-raise
+ (.setStatus span StatusCode/ERROR)
+ (.recordException span t)
+ (span/end-span! {:context span-ctx})
+ (raise t))
+ (finally
+ ;; Close scope on the middleware thread (restores previous context).
+ ;; For async handlers, the callback runs on a different thread — the scope
+ ;; closure here is still correct (cleans up this thread's context).
+ (.close scope)
+ (tracing/clear-pyroscope-context!)
+ (tracing/clear-forced-trace-id!)
+ (tracing/clear-trace-id-from-mdc!)))))
+ (handler request respond raise))))
diff --git a/src/metabase/session/task/session_cleanup.clj b/src/metabase/session/task/session_cleanup.clj
index f8b4d52cc03..52e5422db21 100644
--- a/src/metabase/session/task/session_cleanup.clj
+++ b/src/metabase/session/task/session_cleanup.clj
@@ -6,6 +6,7 @@
[metabase.app-db.core :as mdb]
[metabase.config.core :as config]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.honey-sql-2 :as h2x]
[toucan2.core :as t2]))
@@ -17,8 +18,11 @@
(let [oldest-allowed [:inline (h2x/add-interval-honeysql-form (mdb/db-type)
:%now
(- (config/config-int :max-session-age))
- :minute)]]
- (t2/delete! :model/Session :created_at [:< oldest-allowed])))
+ :minute)]
+ hsql {:delete-from [(t2/table-name :model/Session)]
+ :where [:< :created_at oldest-allowed]}]
+ (tracing/with-span :tasks "task.session-cleanup.delete" {:db/statement (tracing/best-effort-sanitize-sql hsql)}
+ (t2/query-one hsql))))
(def ^:private session-cleanup-job-key (jobs/key "metabase.task.session-cleanup.job"))
(def ^:private session-cleanup-trigger-key (triggers/key "metabase.task.session-cleanup.trigger"))
diff --git a/src/metabase/sync/analyze/classify.clj b/src/metabase/sync/analyze/classify.clj
index 431b7e2622b..9ee042991e1 100644
--- a/src/metabase/sync/analyze/classify.clj
+++ b/src/metabase/sync/analyze/classify.clj
@@ -27,6 +27,7 @@
[metabase.sync.analyze.fingerprint :as sync.fingerprint]
[metabase.sync.interface :as i]
[metabase.sync.util :as sync-util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -110,26 +111,27 @@
"Run various classifiers on the appropriate `fields` in a `table` that have not been previously analyzed. These do
things like inferring (and setting) the semantic types and preview display status for Fields belonging to `table`."
[table :- i/TableInstance]
- (let [table-id (:id table)]
- (when-let [fields (fields-to-classify table)]
- (let [existing-name-field (t2/count :model/Field
- :table_id table-id
- :active true
- :visibility_type [:not-in ["sensitive" "retired"]]
- :semantic_type :type/Name)
- {:keys [fields-failed]}
- (reduce (fn [state field]
- (let [result (classify! field state)]
- (cond-> state
- (instance? Exception result)
- (update :fields-failed inc)
-
- (= :type/Name (:semantic_type result))
- (assoc :exists-name true))))
- {:fields-failed 0 :exists-name (pos? existing-name-field)}
- fields)]
- {:fields-classified (count fields)
- :fields-failed fields-failed}))))
+ (tracing/with-span :sync "sync.classify.fields" {:sync/table (:name table)}
+ (let [table-id (:id table)]
+ (when-let [fields (fields-to-classify table)]
+ (let [existing-name-field (t2/count :model/Field
+ :table_id table-id
+ :active true
+ :visibility_type [:not-in ["sensitive" "retired"]]
+ :semantic_type :type/Name)
+ {:keys [fields-failed]}
+ (reduce (fn [state field]
+ (let [result (classify! field state)]
+ (cond-> state
+ (instance? Exception result)
+ (update :fields-failed inc)
+
+ (= :type/Name (:semantic_type result))
+ (assoc :exists-name true))))
+ {:fields-failed 0 :exists-name (pos? existing-name-field)}
+ fields)]
+ {:fields-classified (count fields)
+ :fields-failed fields-failed})))))
(mu/defn ^:always-validate classify-table!
"Run various classifiers on the `table`. These do things like inferring (and setting) entity type of `table`."
diff --git a/src/metabase/sync/analyze/fingerprint.clj b/src/metabase/sync/analyze/fingerprint.clj
index 2ea265369c7..62b3fdc17d5 100644
--- a/src/metabase/sync/analyze/fingerprint.clj
+++ b/src/metabase/sync/analyze/fingerprint.clj
@@ -11,6 +11,7 @@
[metabase.lib.schema.metadata.fingerprint :as lib.schema.metadata.fingerprint]
[metabase.sync.interface :as i]
[metabase.sync.util :as sync-util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -81,7 +82,8 @@
(map vector fields fingerprints)))))
driver (driver.u/database->driver (table/database table))
opts {:truncation-size *truncation-size*}]
- (driver/table-rows-sample driver table fields rff opts)))
+ (tracing/with-span :sync "sync.fingerprint.sample" {:sync/table (:name table) :sync/field-count (count fields)}
+ (driver/table-rows-sample driver table fields rff opts))))
;;; +----------------------------------------------------------------------------------------------------------------+
;;; | WHICH FIELDS NEED UPDATED FINGERPRINTS? |
@@ -193,16 +195,17 @@
(mu/defn fingerprint-table!
"Generate and save fingerprints for all the Fields in `table` that have not been previously analyzed."
[table :- i/TableInstance]
- (if-let [fields (fields-to-fingerprint table)]
- (do
- (log/infof "Fingerprinting %s fields in table %s" (count fields) (sync-util/name-for-logging table))
- (let [stats
- (sync-util/with-returning-throwable (format "Error fingerprinting %s" (sync-util/name-for-logging table))
- (fingerprint-fields! table fields))]
- (if (:throwable stats)
- (merge (empty-stats-map 0) stats)
- stats)))
- (empty-stats-map 0)))
+ (tracing/with-span :sync "sync.fingerprint.table" {:db/id (:db_id table) :sync/table (:name table)}
+ (if-let [fields (fields-to-fingerprint table)]
+ (do
+ (log/infof "Fingerprinting %s fields in table %s" (count fields) (sync-util/name-for-logging table))
+ (let [stats
+ (sync-util/with-returning-throwable (format "Error fingerprinting %s" (sync-util/name-for-logging table))
+ (fingerprint-fields! table fields))]
+ (if (:throwable stats)
+ (merge (empty-stats-map 0) stats)
+ stats)))
+ (empty-stats-map 0))))
(def ^:private LogProgressFn
[:=> [:cat :string [:schema i/TableInstance]] :any])
diff --git a/src/metabase/sync/field_values.clj b/src/metabase/sync/field_values.clj
index 74f532a83ec..61e8214b319 100644
--- a/src/metabase/sync/field_values.clj
+++ b/src/metabase/sync/field_values.clj
@@ -5,6 +5,7 @@
[metabase.app-db.core :as mdb]
[metabase.sync.interface :as i]
[metabase.sync.util :as sync-util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -67,8 +68,9 @@
(mu/defn- update-field-values-for-database!
[database :- i/DatabaseInstance]
- (let [tables (sync-util/reducible-sync-tables database)]
- (transduce (map update-field-values-for-table!) (partial merge-with +) tables)))
+ (tracing/with-span :sync "field-values.update" {:db/id (:id database)}
+ (let [tables (sync-util/reducible-sync-tables database)]
+ (transduce (map update-field-values-for-table!) (partial merge-with +) tables))))
(defn- update-field-values-summary [{:keys [created updated deleted errors]}]
(format "Updated %d field value sets, created %d, deleted %d with %d errors"
@@ -102,15 +104,16 @@
(mu/defn- delete-expired-advanced-field-values-for-database!
[database :- i/DatabaseInstance]
- (let [tables (sync-util/reducible-sync-tables database)]
- {:deleted (transduce (comp (map delete-expired-advanced-field-values-for-table!)
- (map (fn [result]
- (if (instance? Throwable result)
- (throw result)
- result))))
- +
- 0
- tables)}))
+ (tracing/with-span :sync "field-values.delete-expired-advanced" {:db/id (:id database)}
+ (let [tables (sync-util/reducible-sync-tables database)]
+ {:deleted (transduce (comp (map delete-expired-advanced-field-values-for-table!)
+ (map (fn [result]
+ (if (instance? Throwable result)
+ (throw result)
+ result))))
+ +
+ 0
+ tables)})))
(def ^:private sync-field-values-steps
[(sync-util/create-sync-step "delete-expired-advanced-field-values"
diff --git a/src/metabase/sync/sync.clj b/src/metabase/sync/sync.clj
index f8cef571ef3..a45761ade57 100644
--- a/src/metabase/sync/sync.clj
+++ b/src/metabase/sync/sync.clj
@@ -17,6 +17,7 @@
[metabase.sync.interface :as i]
[metabase.sync.sync-metadata :as sync-metadata]
[metabase.sync.util :as sync-util]
+ [metabase.tracing.core :as tracing]
[metabase.util.malli :as mu]
[metabase.util.malli.schema :as ms]
[metabase.warehouse-schema.models.field :as field]
@@ -45,14 +46,16 @@
[:metadata :analyze :field-values]))
(defn- do-phase! [database phase]
- (let [f (phase->fn phase)
- result (f database)]
- (if (instance? Throwable result)
- ;; do nothing if we're configured to just move on.
- (when-not sync-util/*log-exceptions-and-continue?*
- ;; but if we didn't expect any suppressed exceptions, rethrow it
- (throw result))
- (assoc result :name (name phase)))))
+ (tracing/with-span :sync (str "sync." (name phase)) {:db/id (:id database)
+ :sync/phase (name phase)}
+ (let [f (phase->fn phase)
+ result (f database)]
+ (if (instance? Throwable result)
+ ;; do nothing if we're configured to just move on.
+ (when-not sync-util/*log-exceptions-and-continue?*
+ ;; but if we didn't expect any suppressed exceptions, rethrow it
+ (throw result))
+ (assoc result :name (name phase))))))
(mu/defn sync-database! :- SyncDatabaseResults
"Perform all the different sync operations synchronously for `database`.
@@ -68,10 +71,11 @@
([database :- i/DatabaseInstance
{:keys [scan], :or {scan :full}} :- [:maybe [:map
[:scan {:optional true} [:maybe [:enum :schema :full]]]]]]
- (sync-util/sync-operation :sync database (format "Sync %s" (sync-util/name-for-logging database))
- (->> (scan-phases scan)
- (keep (partial do-phase! database))
- (doall)))))
+ (tracing/with-span :sync "sync.database" {:db/id (:id database)}
+ (sync-util/sync-operation :sync database (format "Sync %s" (sync-util/name-for-logging database))
+ (->> (scan-phases scan)
+ (keep (partial do-phase! database))
+ (doall))))))
(mu/defn sync-table!
"Perform all the different sync operations synchronously for a given `table`. Since often called on a sequence of
diff --git a/src/metabase/sync/sync_metadata.clj b/src/metabase/sync/sync_metadata.clj
index 8e09a93e786..5dfb4c4a175 100644
--- a/src/metabase/sync/sync_metadata.clj
+++ b/src/metabase/sync/sync_metadata.clj
@@ -17,6 +17,7 @@
[metabase.sync.sync-metadata.sync-timezone :as sync-tz]
[metabase.sync.sync-metadata.tables :as sync-tables]
[metabase.sync.util :as sync-util]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.malli :as mu]
[metabase.warehouse-schema.models.table :as table]))
@@ -63,7 +64,8 @@
"Sync the metadata for a Metabase `database`. This makes sure child Table & Field objects are synchronized."
[database :- i/DatabaseInstance]
(sync-util/sync-operation :sync-metadata database (format "Sync metadata for %s" (sync-util/name-for-logging database))
- (let [db-metadata (fetch-metadata/db-metadata database)]
+ (let [db-metadata (tracing/with-span :sync "sync.metadata.fetch-metadata" {:db/id (:id database)}
+ (fetch-metadata/db-metadata database))]
(u/prog1 (sync-util/run-sync-operation "sync" database (make-sync-steps db-metadata))
(if (some sync-util/abandon-sync? (map second (:steps <>)))
(sync-util/set-initial-database-sync-aborted! database)
diff --git a/src/metabase/sync/task/sync_databases.clj b/src/metabase/sync/task/sync_databases.clj
index ec8d46cb206..2790b2fa7f9 100644
--- a/src/metabase/sync/task/sync_databases.clj
+++ b/src/metabase/sync/task/sync_databases.clj
@@ -21,6 +21,7 @@
[metabase.sync.schedules :as sync.schedules]
[metabase.sync.sync-metadata :as sync-metadata]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.cron :as u.cron]
[metabase.util.log :as log]
@@ -88,12 +89,16 @@
e))]
(log/warnf ex "Cannot sync Database %s: %s" (:name database) (ex-message ex))
(database-routing/with-database-routing-off
- (let [metadata-results (sync-metadata/sync-db-metadata! database)
- analyze-results (when (:is_full_sync database)
- (analyze/analyze-db! database))
+ (let [db-id (:id database)
+ metadata-results (tracing/with-span :sync "sync.metadata" {:db/id db-id}
+ (sync-metadata/sync-db-metadata! database))
+ analyze-results (when (:is_full_sync database)
+ (tracing/with-span :sync "sync.analyze" {:db/id db-id}
+ (analyze/analyze-db! database)))
refingerprint-results (when (and (:refingerprint database)
(should-refingerprint-fields? analyze-results))
- (analyze/refingerprint-db! database))]
+ (tracing/with-span :sync "sync.refingerprint" {:db/id db-id}
+ (analyze/refingerprint-db! database)))]
(cond-> {:metadata-results metadata-results}
analyze-results (assoc :analyze-results analyze-results)
refingerprint-results (assoc :refingerprint-results refingerprint-results)))))))
diff --git a/src/metabase/sync/util.clj b/src/metabase/sync/util.clj
index 8e8c0aaffcf..ca51d85beb7 100644
--- a/src/metabase/sync/util.clj
+++ b/src/metabase/sync/util.clj
@@ -14,6 +14,7 @@
[metabase.query-processor.interface :as qp.i]
[metabase.sync.interface :as i]
[metabase.task-history.core :as task-history]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
[metabase.util.log :as log]
@@ -548,20 +549,21 @@
[database :- i/DatabaseInstance
{:keys [step-name sync-fn log-summary-fn] :as _step} :- StepDefinition]
(let [start-time (t/zoned-date-time)
- results (do-with-start-and-finish-debug-logging
- (format "step ''%s'' for %s"
- step-name
- (name-for-logging database))
- (fn [& args]
- (with-returning-throwable (format "Error running step ''%s'' for %s" step-name (name-for-logging database))
- (task-history/with-task-history
- {:task step-name
- :db_id (u/the-id database)
- :on-success-info (fn [update-map result]
- (if (instance? Throwable result)
- (throw result)
- (assoc update-map :task_details (dissoc result :start-time :end-time :log-summary-fn))))}
- (apply sync-fn database args)))))
+ results (tracing/with-span :sync (str "sync.step." step-name) {:db/id (u/the-id database) :sync/step step-name}
+ (do-with-start-and-finish-debug-logging
+ (format "step ''%s'' for %s"
+ step-name
+ (name-for-logging database))
+ (fn [& args]
+ (with-returning-throwable (format "Error running step ''%s'' for %s" step-name (name-for-logging database))
+ (task-history/with-task-history
+ {:task step-name
+ :db_id (u/the-id database)
+ :on-success-info (fn [update-map result]
+ (if (instance? Throwable result)
+ (throw result)
+ (assoc update-map :task_details (dissoc result :start-time :end-time :log-summary-fn))))}
+ (apply sync-fn database args))))))
end-time (t/zoned-date-time)]
[step-name (assoc results
:start-time start-time
diff --git a/src/metabase/task/bootstrap.clj b/src/metabase/task/bootstrap.clj
index ae0b9fb5d20..6e8ebd426f3 100644
--- a/src/metabase/task/bootstrap.clj
+++ b/src/metabase/task/bootstrap.clj
@@ -9,6 +9,16 @@
(defn- app-db ^javax.sql.DataSource []
((requiring-resolve 'metabase.app-db.core/app-db)))
+;; Optional interceptor for wrapping JDBC connections before Quartz uses them.
+;; Set by task.tracing to add SQL-level tracing. nil means no interception.
+(defonce ^:private connection-interceptor (atom nil))
+
+(defn set-connection-interceptor!
+ "Set an optional function to wrap JDBC connections before Quartz uses them.
+ Called by task.tracing to add SQL-level tracing. Pass nil to remove."
+ [f]
+ (reset! connection-interceptor f))
+
(defrecord ^:private ConnectionProvider []
org.quartz.utils.ConnectionProvider
(initialize [_])
@@ -19,11 +29,10 @@
;; very important! Fetch a new connection from the connection pool rather than using currently bound Connection if
;; one already exists -- because Quartz will close this connection when done, we don't want to screw up the
;; calling block
- ;;
- ;; in a perfect world we could just check whether we're creating a new Connection or not, and if using an existing
- ;; Connection, wrap it in a delegating proxy wrapper that makes `.close()` a no-op but forwards all other methods.
- ;; Now that would be a useful macro!
- (.getConnection (app-db)))
+ (let [conn (.getConnection (app-db))]
+ (if-let [interceptor @connection-interceptor]
+ (interceptor conn)
+ conn)))
(shutdown [_]))
(when-not *compile-files*
diff --git a/src/metabase/task/impl.clj b/src/metabase/task/impl.clj
index 3394b72e0ba..d2ba3f4a835 100644
--- a/src/metabase/task/impl.clj
+++ b/src/metabase/task/impl.clj
@@ -26,6 +26,7 @@
[metabase.classloader.core :as classloader]
[metabase.task.bootstrap]
[metabase.task.job-factory :as job-factory]
+ [metabase.tracing.core :as tracing]
[metabase.util :as u]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
@@ -352,11 +353,12 @@
#_{:clj-kondo/ignore [:discouraged-var]}
(defmacro defjob
- "Like `clojurewerkz.quartzite.task/defjob` but with a log context."
+ "Like `clojurewerkz.quartzite.task/defjob` but with a log context and an OpenTelemetry tracing span."
[jtype args & body]
`(jobs/defjob ~jtype ~args
(log/with-context {:quartz-job-type (quote ~jtype)}
- ~@body)))
+ (tracing/with-span :tasks (str "task." (quote ~jtype)) {:task/name (str (quote ~jtype))}
+ ~@body))))
(defn add-job-listener!
"Add a [Quartz Joblistener](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/tutorial-lesson-07.html). That will
diff --git a/src/metabase/task_history/task/task_history_cleanup.clj b/src/metabase/task_history/task/task_history_cleanup.clj
index cdb3ad19ce4..6b5b3c78a08 100644
--- a/src/metabase/task_history/task/task_history_cleanup.clj
+++ b/src/metabase/task_history/task/task_history_cleanup.clj
@@ -5,6 +5,7 @@
[clojurewerkz.quartzite.triggers :as triggers]
[metabase.task-history.models.task-history :as task-history]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.log :as log]))
(set! *warn-on-reflection* true)
@@ -17,12 +18,13 @@
"Delete older TaskHistory rows -- see docstring of `task-history/cleanup-task-history!` for more details."
[]
(log/debug "Cleaning up task history")
- (task-history/with-task-history {:task "task-history-cleanup"}
- (let [deleted-rows? (task-history/cleanup-task-history! history-rows-to-keep)]
- (log/debug
- (if deleted-rows?
- "Task history cleanup successful, rows were deleted"
- "Task history cleanup successful, no rows were deleted")))))
+ (tracing/with-span :tasks "task.history-cleanup.delete" {:task-history/rows-to-keep history-rows-to-keep}
+ (task-history/with-task-history {:task "task-history-cleanup"}
+ (let [deleted-rows? (task-history/cleanup-task-history! history-rows-to-keep)]
+ (log/debug
+ (if deleted-rows?
+ "Task history cleanup successful, rows were deleted"
+ "Task history cleanup successful, no rows were deleted"))))))
(task/defjob
^{:doc "Delete older TaskHistory rows -- see docstring of `task-history/cleanup-task-history!` for more details."}
diff --git a/src/metabase/task_history/task/task_run_heartbeat.clj b/src/metabase/task_history/task/task_run_heartbeat.clj
index 65fa4da5c93..7489be7aa44 100644
--- a/src/metabase/task_history/task/task_run_heartbeat.clj
+++ b/src/metabase/task_history/task/task_run_heartbeat.clj
@@ -11,6 +11,7 @@
[metabase.config.core :as config]
[metabase.models.interface :as mi]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.util.honey-sql-2 :as h2x]
[metabase.util.log :as log]
[toucan2.core :as t2]))
@@ -28,13 +29,14 @@
(defn send-heartbeat!
"Update updated_at for all :started runs belonging to this process."
[]
- (let [updated (t2/update! :model/TaskRun
- {:status :started
- :process_uuid config/local-process-uuid}
- {:updated_at (mi/now)})]
- (when (pos? updated)
- (log/debugf "Sent heartbeat for %d running task runs" updated))
- updated))
+ (tracing/with-span :tasks "task.heartbeat.update" {}
+ (let [updated (t2/update! :model/TaskRun
+ {:status :started
+ :process_uuid config/local-process-uuid}
+ {:updated_at (mi/now)})]
+ (when (pos? updated)
+ (log/debugf "Sent heartbeat for %d running task runs" updated))
+ updated)))
(defn mark-orphaned-runs!
"Mark runs as :abandoned if:
@@ -44,12 +46,14 @@
[]
(let [heartbeat-cutoff (h2x/add-interval-honeysql-form (mdb/db-type) :%now (- orphan-threshold-hours) :hour)
duration-cutoff (h2x/add-interval-honeysql-form (mdb/db-type) :%now (- max-run-duration-hours) :hour)
- orphaned-run-ids (t2/select-fn-set :id :model/TaskRun
- {:where [:and
- [:= :status "started"]
- [:or
- [:< :updated_at heartbeat-cutoff]
- [:< :started_at duration-cutoff]]]})]
+ where-clause {:where [:and
+ [:= :status "started"]
+ [:or
+ [:< :updated_at heartbeat-cutoff]
+ [:< :started_at duration-cutoff]]]}
+ orphaned-run-ids (tracing/with-span :tasks "task.heartbeat.mark-orphaned-runs"
+ {:db/statement (tracing/best-effort-sanitize-sql (assoc where-clause :select [:id] :from [:task_run]))}
+ (t2/select-fn-set :id :model/TaskRun where-clause))]
(when (seq orphaned-run-ids)
(t2/update! :model/TaskRun {:id [:in orphaned-run-ids]}
{:status :abandoned
@@ -61,14 +65,15 @@
"Mark tasks as :unknown if they belong to the given orphaned runs."
[orphaned-run-ids]
(when (seq orphaned-run-ids)
- (let [orphaned (t2/update! :model/TaskHistory
- {:status :started
- :run_id [:in orphaned-run-ids]}
- {:status :unknown
- :ended_at (mi/now)})]
- (when (pos? orphaned)
- (log/infof "Marked %d orphaned tasks as :unknown" orphaned))
- orphaned)))
+ (tracing/with-span :tasks "task.heartbeat.mark-orphaned-tasks" {:heartbeat/orphaned-run-count (count orphaned-run-ids)}
+ (let [orphaned (t2/update! :model/TaskHistory
+ {:status :started
+ :run_id [:in orphaned-run-ids]}
+ {:status :unknown
+ :ended_at (mi/now)})]
+ (when (pos? orphaned)
+ (log/infof "Marked %d orphaned tasks as :unknown" orphaned))
+ orphaned))))
(defn- task-run-heartbeat!
"Send heartbeat for running tasks and mark orphaned runs/tasks."
diff --git a/src/metabase/tracing/attributes.clj b/src/metabase/tracing/attributes.clj
new file mode 100644
index 00000000000..d14edb4751d
--- /dev/null
+++ b/src/metabase/tracing/attributes.clj
@@ -0,0 +1,15 @@
+(ns metabase.tracing.attributes
+ "Attribute helpers for OpenTelemetry spans."
+ (:require
+ [honey.sql :as sql]))
+
+(set! *warn-on-reflection* true)
+
+(defn best-effort-sanitize-sql
+ "Convert a HoneySQL map to a parameterized SQL string for trace attributes.
+ Values become ? placeholders. That is a best-effort sanitizing solution."
+ [hsql-map]
+ (try
+ (first (sql/format hsql-map {:quoted false}))
+ (catch Exception _
+ (pr-str hsql-map))))
diff --git a/src/metabase/tracing/core.clj b/src/metabase/tracing/core.clj
new file mode 100644
index 00000000000..f1b48805a3a
--- /dev/null
+++ b/src/metabase/tracing/core.clj
@@ -0,0 +1,356 @@
+(ns metabase.tracing.core
+ "Tracing module API. Add OpenTelemetry traces to your code.
+
+ ## Quick Start
+ (require '[metabase.tracing.core :as tracing])
+ (tracing/with-span :qp \"qp.preprocess\" {:db/id 42}
+ (preprocess query))
+
+ ## New Group
+ (tracing/register-group! :my-feature \"Description\")
+ (tracing/with-span :my-feature \"my-op\" {} (do-stuff))
+
+ ## Built-in Groups
+ :qp, :sync, :tasks, :search, :api, :db-user, :db-app, :events, :quartz
+
+ ## Configuration
+ MB_TRACING_ENABLED=true MB_TRACING_ENDPOINT=host:4317
+ MB_TRACING_GROUPS=qp,api MB_TRACING_SERVICE_NAME=metabase-prod-1"
+ (:require
+ [clojure.string :as str]
+ [metabase.tracing.attributes :as trace-attrs]
+ [metabase.tracing.settings :as tracing.settings]
+ [metabase.util.log :as log]
+ [potemkin :as p]
+ [steffan-westcott.clj-otel.api.otel :as otel]
+ [steffan-westcott.clj-otel.api.trace.span :as span]
+ [steffan-westcott.clj-otel.exporter.otlp.http.trace :as otlp-http]
+ [steffan-westcott.clj-otel.sdk.otel-sdk :as otel-sdk])
+ (:import
+ (io.opentelemetry.api.trace Span SpanContext)
+ (io.opentelemetry.sdk.trace IdGenerator)
+ (java.time Duration)
+ (org.apache.logging.log4j ThreadContext)))
+
+(set! *warn-on-reflection* true)
+
+(p/import-vars
+ [trace-attrs
+ best-effort-sanitize-sql])
+
+;;; ------------------------------------------------ Group Registry ------------------------------------------------
+
+(defonce ^:private group-registry
+ (atom {}))
+
+(defn register-group!
+ "Register a trace group. Groups can be selectively enabled via MB_TRACING_GROUPS."
+ [group-key description]
+ (swap! group-registry assoc group-key {:description description})
+ nil)
+
+;; Built-in groups
+(register-group! :qp "Query processor: preprocess, compile, execute, cache")
+(register-group! :sync "Database sync: metadata, analysis, fingerprinting, field values")
+(register-group! :tasks "Scheduled background tasks (Quartz jobs)")
+(register-group! :search "Search: full-text, semantic, indexing, ingestion")
+(register-group! :api "HTTP request/response lifecycle")
+(register-group! :db-user "Customer/user database operations: SQL execution, connection pool")
+(register-group! :db-app "Application/system database operations: sessions, settings, QE writes")
+(register-group! :events "Event system: view logging, audit, notifications")
+(register-group! :quartz "Quartz scheduler internals: trigger acquisition, locks, heartbeats, JDBC")
+
+(defn registered-groups
+ "Return a map of all registered trace groups."
+ []
+ @group-registry)
+
+;;; ----------------------------------------------- Runtime State --------------------------------------------------
+
+;; Cached set of enabled groups, or `:all`. nil means tracing is disabled.
+(defonce ^:private enabled-groups (atom nil))
+
+;; Cached trace log level string (e.g. "DEBUG"). Used by DynamicThresholdFilter in log4j2.xml
+;; to lower the effective log threshold for traced threads. nil means tracing is disabled.
+(defonce ^:private trace-log-level (atom nil))
+
+(defn init-enabled-groups!
+ "Parse the tracing-groups and log-level settings and cache the results.
+ `log-level-str` is the MB_TRACING_LOG_LEVEL value (e.g. \"DEBUG\", \"INFO\")."
+ [groups-str log-level-str]
+ (let [groups-str (or groups-str "all")]
+ (reset! enabled-groups
+ (if (= "all" (str/trim groups-str))
+ :all
+ (->> (str/split groups-str #",")
+ (map (comp keyword str/trim))
+ (set)))))
+ (reset! trace-log-level (or log-level-str "INFO")))
+
+(defn shutdown-groups!
+ "Clear cached enabled groups and trace log level."
+ []
+ (reset! enabled-groups nil)
+ (reset! trace-log-level nil))
+
+(defn group-enabled?
+ "Check if a trace group is currently enabled. Returns false when tracing is disabled."
+ [group]
+ (when-let [groups @enabled-groups]
+ (or (= groups :all)
+ (contains? groups group))))
+
+;;; ------------------------------------------ Forced Trace ID (Frontend) ----------------------------------------
+
+;; ThreadLocal for per-request trace ID override from frontend traceparent header.
+;; The custom IdGenerator reads (and clears) this value when creating a root span,
+;; so the span gets the frontend's trace ID without a parent-child link
+;; (which would cause "root span not yet received" in Tempo).
+(def ^:private ^ThreadLocal forced-trace-id-holder (ThreadLocal.))
+
+(defn force-trace-id!
+ "Set a trace ID to be used by the next root span created on this thread.
+ Consumed (and cleared) by the custom IdGenerator."
+ [^String trace-id]
+ (.set forced-trace-id-holder trace-id))
+
+(defn get-and-clear-forced-trace-id!
+ "Get and atomically clear the forced trace ID for this thread. Returns nil if none set.
+ Called by the custom IdGenerator — should not be called directly."
+ ^String []
+ (let [tid (.get forced-trace-id-holder)]
+ (when tid
+ (.remove forced-trace-id-holder))
+ tid))
+
+(defn clear-forced-trace-id!
+ "Clear any forced trace ID without consuming it. Safety cleanup."
+ []
+ (.remove forced-trace-id-holder))
+
+;;; -------------------------------------------- MDC Injection (Log<->Trace) -----------------------------------------
+
+(defn inject-trace-id-into-mdc!
+ "Inject current span's trace_id, span_id, and trace_level into Log4j2 MDC.
+ - trace_id/span_id: for log-to-trace correlation (Loki <-> Tempo).
+ - trace_level: consumed by DynamicThresholdFilter in log4j2.xml to lower
+ the effective log threshold for this thread during the span."
+ []
+ (let [span (Span/current)
+ ctx (.getSpanContext span)]
+ (ThreadContext/put "trace_id" (.getTraceId ctx))
+ (ThreadContext/put "span_id" (.getSpanId ctx)))
+ (when-let [level @trace-log-level]
+ (ThreadContext/put "trace_level" level)))
+
+(defn clear-trace-id-from-mdc!
+ "Remove trace_id, span_id, and trace_level from Log4j2 MDC."
+ []
+ (ThreadContext/remove "trace_id")
+ (ThreadContext/remove "span_id")
+ (ThreadContext/remove "trace_level"))
+
+;;; ------------------------------------------------- Tracer Access ------------------------------------------------
+
+(defn get-tracer
+ "Return an OTel `Tracer` for the given instrumentation library name.
+ Uses the clj-otel default OTel instance (set during SDK init), NOT
+ `GlobalOpenTelemetry` (which may be no-op if `:set-as-global` was false)."
+ ^io.opentelemetry.api.trace.Tracer [^String library-name]
+ (let [^io.opentelemetry.api.OpenTelemetry otel-instance (otel/get-default-otel!)]
+ (.getTracer otel-instance library-name)))
+
+;;; --------------------------------------------- Pyroscope Integration --------------------------------------------
+;;; Optional integration with Pyroscope continuous profiling. When pyroscope.jar is on the
+;;; classpath, tags profiling samples with span_id for trace-to-profile linking in Grafana.
+;;; All functions are no-ops when Pyroscope is not available. Uses runtime reflection only.
+
+;; PyroscopeAsyncProfiler.getAsyncProfiler() -> AsyncProfiler instance
+(defonce ^:private async-profiler-instance
+ (delay
+ (try
+ (let [factory-cls (Class/forName "io.pyroscope.PyroscopeAsyncProfiler")
+ ^java.lang.reflect.Method getter (.getMethod factory-cls "getAsyncProfiler" (make-array Class 0))]
+ (.invoke getter nil (make-array Object 0)))
+ (catch ClassNotFoundException _
+ nil)
+ (catch Exception e
+ (log/debug e "Could not resolve PyroscopeAsyncProfiler")
+ nil))))
+
+;; AsyncProfiler.setTracingContext(long spanId, long spanNameId)
+(defonce ^:private set-tracing-ctx-method
+ (delay
+ (when-let [profiler @async-profiler-instance]
+ (try
+ (.getMethod (class profiler) "setTracingContext"
+ (into-array Class [Long/TYPE Long/TYPE]))
+ (catch NoSuchMethodException _
+ (log/debug "AsyncProfiler.setTracingContext not available")
+ nil)
+ (catch Exception _ nil)))))
+
+;; Pyroscope.LabelsWrapper.registerConstant(String) -> long
+(defonce ^:private register-constant-method
+ (delay
+ (try
+ ;; v2.x package path
+ (let [cls (Class/forName "io.pyroscope.labels.v2.Pyroscope$LabelsWrapper")]
+ (.getMethod cls "registerConstant" (into-array Class [String])))
+ (catch ClassNotFoundException _
+ (try
+ ;; v1.x package path
+ (let [cls (Class/forName "io.pyroscope.labels.Pyroscope$LabelsWrapper")]
+ (.getMethod cls "registerConstant" (into-array Class [String])))
+ (catch ClassNotFoundException _ nil)
+ (catch Exception _ nil)))
+ (catch Exception _ nil))))
+
+(defn pyroscope-available?
+ "True if Pyroscope trace-to-profile integration is available.
+ Returns true when pyroscope.jar is on the classpath and the async-profiler
+ supports setTracingContext."
+ []
+ (boolean (and @async-profiler-instance @set-tracing-ctx-method)))
+
+(defn set-pyroscope-context!
+ "Tag current thread's profiling samples with span ID and name, and set the
+ `pyroscope.profile.id` attribute on the span (for Grafana trace-to-profile linking).
+ No-op if Pyroscope is not available."
+ [^Span span ^String span-id-hex ^String span-name]
+ (when-let [^java.lang.reflect.Method method @set-tracing-ctx-method]
+ (try
+ (let [span-id-long (Long/parseUnsignedLong span-id-hex 16)
+ span-name-id (if-let [^java.lang.reflect.Method rc @register-constant-method]
+ (long (.invoke rc nil (into-array Object [span-name])))
+ 0)]
+ (.invoke method @async-profiler-instance
+ (into-array Object [(Long/valueOf span-id-long)
+ (Long/valueOf span-name-id)]))
+ (.setAttribute span "pyroscope.profile.id" span-id-hex))
+ (catch Exception e
+ (log/debug e "Error setting Pyroscope profiling context")))))
+
+(defn clear-pyroscope-context!
+ "Clear profiling context for current thread. No-op if Pyroscope is not available."
+ []
+ (when-let [^java.lang.reflect.Method method @set-tracing-ctx-method]
+ (try
+ (.invoke method @async-profiler-instance
+ (into-array Object [(Long/valueOf 0) (Long/valueOf 0)]))
+ (catch Exception _))))
+
+;;; ------------------------------------------------ Primary Macro -------------------------------------------------
+
+(defmacro with-span
+ "Create an OTel span if tracing is enabled for `group`.
+
+ When disabled: zero overhead — single atom deref + boolean check, body runs directly.
+ When enabled: creates span AND injects trace_id/span_id into Log4j2 MDC
+ for log-to-trace correlation (Loki <-> Tempo in Grafana).
+
+ Saves and restores previous MDC values so that nested spans don't wipe the
+ parent span's trace_id/span_id from MDC when they exit.
+
+ For root spans (no parent in MDC), also sets Pyroscope profiling context so
+ that CPU/alloc/lock samples are tagged with the span ID. This covers task spans
+ and other entry points that don't go through the API trace middleware.
+ API requests get profiling context from wrap-trace instead.
+
+ Usage:
+ (with-span :qp \"qp.process-query\" {:db/id db-id}
+ (process-query query))"
+ [group span-name attrs & body]
+ `(if (group-enabled? ~group)
+ (let [prev-trace-id# (ThreadContext/get "trace_id")
+ prev-span-id# (ThreadContext/get "span_id")
+ root-span?# (nil? prev-span-id#)]
+ (span/with-span! {:name ~span-name :attributes ~attrs}
+ (inject-trace-id-into-mdc!)
+ ;; For root spans (no parent in MDC), set Pyroscope profiling context.
+ ;; Nested child spans skip this — the root's context covers all samples.
+ (when root-span?#
+ (let [^Span span# (Span/current)
+ ^SpanContext ctx# (.getSpanContext span#)
+ span-id# (.getSpanId ctx#)]
+ (set-pyroscope-context! span# span-id# ~span-name)))
+ (try
+ ~@body
+ (finally
+ (when root-span?#
+ (clear-pyroscope-context!))
+ (if prev-trace-id#
+ (do (ThreadContext/put "trace_id" prev-trace-id#)
+ (ThreadContext/put "span_id" prev-span-id#))
+ (clear-trace-id-from-mdc!))))))
+ (do ~@body)))
+
+;;; -------------------------------------------- SDK Lifecycle -------------------------------------------------
+;;; OpenTelemetry SDK initialization and shutdown. Initializes the OTel SDK with an OTLP
+;;; span exporter for pushing traces to a collector. Has no database dependency — can be
+;;; initialized very early in the startup process.
+
+;; Holds the initialized OpenTelemetrySdk instance for manual shutdown.
+(defonce ^:private otel-sdk-instance (atom nil))
+
+(defn- make-id-generator
+ "Create an IdGenerator that respects forced trace IDs from frontend traceparent
+ headers (via `force-trace-id!`), falling back to random generation.
+ This allows frontend-originated requests to share a trace ID without creating
+ a parent-child link to a non-existent browser span."
+ ^IdGenerator []
+ (let [^IdGenerator default-gen (IdGenerator/random)]
+ (reify IdGenerator
+ (generateTraceId [_]
+ (or (get-and-clear-forced-trace-id!)
+ (.generateTraceId default-gen)))
+ (generateSpanId [_]
+ (.generateSpanId default-gen)))))
+
+(defn init!
+ "Initialize the OTel SDK with OTLP HTTP exporter. No-op when MB_TRACING_ENABLED=false.
+ Should be called as early as possible in startup — has no database dependency."
+ []
+ (if-not (tracing.settings/tracing-enabled)
+ (log/info "OpenTelemetry tracing is disabled (MB_TRACING_ENABLED=false)")
+ (try
+ (let [endpoint (tracing.settings/tracing-endpoint)
+ service-name (tracing.settings/tracing-service-name)
+ groups-str (tracing.settings/tracing-groups)
+ log-level-str (tracing.settings/tracing-log-level)
+ queue-size (tracing.settings/tracing-max-queue-size)
+ timeout-ms (tracing.settings/tracing-export-timeout-ms)
+ delay-ms (tracing.settings/tracing-schedule-delay-ms)
+ exporter (otlp-http/span-exporter {:endpoint endpoint})]
+ (log/infof "Initializing OpenTelemetry tracing: service=%s endpoint=%s groups=%s log-level=%s"
+ service-name endpoint groups-str log-level-str)
+ (log/infof "Batch span processor config: max-queue-size=%d export-timeout-ms=%d schedule-delay-ms=%d"
+ queue-size timeout-ms delay-ms)
+ (let [otel (otel-sdk/init-otel-sdk!
+ service-name
+ {:set-as-default true
+ :register-shutdown-hook false ;; we manage shutdown ourselves
+ :tracer-provider
+ {:id-generator (make-id-generator)
+ :span-processors [{:exporters [exporter]
+ :max-queue-size queue-size
+ :exporter-timeout (Duration/ofMillis timeout-ms)
+ :schedule-delay (Duration/ofMillis delay-ms)}]}})]
+ (reset! otel-sdk-instance otel)
+ (init-enabled-groups! groups-str log-level-str)
+ (log/info "OpenTelemetry tracing initialized successfully")))
+ (catch Exception e
+ (log/error e "Failed to initialize OpenTelemetry tracing — tracing will be disabled")))))
+
+(defn shutdown!
+ "Shutdown the OTel SDK, flushing any pending spans. Called during application shutdown."
+ []
+ (when-let [otel @otel-sdk-instance]
+ (log/info "Shutting down OpenTelemetry tracing...")
+ (try
+ (otel-sdk/close-otel-sdk! otel)
+ (catch Exception e
+ (log/warn e "Error shutting down OpenTelemetry SDK")))
+ (reset! otel-sdk-instance nil)
+ (shutdown-groups!)
+ (log/info "OpenTelemetry tracing shut down")))
diff --git a/src/metabase/tracing/events.clj b/src/metabase/tracing/events.clj
new file mode 100644
index 00000000000..e960000600b
--- /dev/null
+++ b/src/metabase/tracing/events.clj
@@ -0,0 +1,25 @@
+(ns metabase.tracing.events
+ "Registers tracing instrumentation for the event system.
+ Loaded by metabase.tracing.init to add span wrapping around event publishing
+ without creating a cyclic dependency from events.impl -> tracing.core."
+ (:require
+ [metabase.events.impl :as events.impl]
+ [metabase.tracing.core :as tracing]
+ [methodical.core :as methodical]))
+
+(set! *warn-on-reflection* true)
+
+(defn- traced-publish-event!
+ "Around method that wraps event publishing in a tracing span."
+ [next-method topic event]
+ (tracing/with-span :events "events.publish" {:event/topic (str topic)}
+ (next-method topic event)))
+
+;; Register the tracing around method on the publish-event! multimethod.
+;; This uses a unique key so it can be cleanly removed if needed.
+(methodical/add-aux-method-with-unique-key!
+ #'events.impl/publish-event!
+ :around
+ :default
+ traced-publish-event!
+ ::tracing)
diff --git a/src/metabase/tracing/init.clj b/src/metabase/tracing/init.clj
new file mode 100644
index 00000000000..7f34df970f9
--- /dev/null
+++ b/src/metabase/tracing/init.clj
@@ -0,0 +1,8 @@
+(ns metabase.tracing.init
+ "Side-effect loading for the tracing module.
+ Loads settings (for env var recognition), quartz tracing (for task/init! registration),
+ and event tracing (for publish-event! instrumentation)."
+ (:require
+ [metabase.tracing.events]
+ [metabase.tracing.quartz]
+ [metabase.tracing.settings]))
diff --git a/src/metabase/tracing/quartz.clj b/src/metabase/tracing/quartz.clj
new file mode 100644
index 00000000000..e59ec348ab3
--- /dev/null
+++ b/src/metabase/tracing/quartz.clj
@@ -0,0 +1,192 @@
+(ns metabase.tracing.quartz
+ "Quartz scheduler tracing instrumentation.
+
+ Provides two levels of observability for Quartz internals:
+
+ 1. **Lifecycle spans** — a JobListener wraps each job execution in a parent span,
+ making Quartz scheduling overhead (lock acquisition, trigger state transitions)
+ visible as the gap between the listener span and the defjob child span.
+
+ 2. **JDBC-level spans** — a ConnectionProvider interceptor wraps JDBC connections
+ so that every SQL query Quartz runs (against QRTZ_* tables) gets its own span
+ with the full SQL statement text.
+
+ Both are gated on the `:quartz` trace group — zero overhead when disabled."
+ (:require
+ [clojure.string :as str]
+ [metabase.task.bootstrap :as task.bootstrap]
+ [metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
+ [metabase.util :as u]
+ [metabase.util.log :as log])
+ (:import
+ (io.opentelemetry.api.trace Span StatusCode)
+ (io.opentelemetry.context Scope)
+ (java.lang.reflect InvocationHandler InvocationTargetException Method Proxy)
+ (java.sql Connection PreparedStatement Statement)
+ (org.quartz JobListener)))
+
+(set! *warn-on-reflection* true)
+
+;;; ----------------------------------------------- JDBC Proxy -------------------------------------------------------
+
+(defn- sql-operation
+ "Extract the SQL verb (SELECT, UPDATE, INSERT, DELETE) from a SQL string."
+ ^String [^String sql]
+ (when sql
+ (let [trimmed (str/trim sql)
+ space (.indexOf trimmed " ")]
+ (if (pos? space)
+ (u/upper-case-en (subs trimmed 0 space))
+ (u/upper-case-en trimmed)))))
+
+(defn- invoke-or-unwrap
+ "Invoke a method on target, unwrapping InvocationTargetException to preserve
+ checked exception semantics (e.g. SQLException)."
+ [^Method method target ^objects args]
+ (try
+ (.invoke method target args)
+ (catch InvocationTargetException e
+ (throw (.getCause e)))))
+
+(def ^:private execute-method-names
+ #{"executeQuery" "executeUpdate" "execute" "executeBatch"
+ "executeLargeUpdate" "executeLargeBatch"})
+
+(defn- traced-prepared-statement
+ "Wrap a PreparedStatement in a proxy that creates a :quartz span around execute* methods."
+ [^PreparedStatement stmt ^String sql]
+ (let [operation (sql-operation sql)]
+ (Proxy/newProxyInstance
+ (.getClassLoader PreparedStatement)
+ (into-array Class [PreparedStatement])
+ (reify InvocationHandler
+ (invoke [_ _ method args]
+ (if (execute-method-names (.getName ^Method method))
+ (tracing/with-span :quartz "quartz.db.execute"
+ {:db/statement sql
+ :db/operation (or operation "UNKNOWN")}
+ (invoke-or-unwrap method stmt args))
+ (invoke-or-unwrap method stmt args)))))))
+
+(def ^:private statement-execute-with-sql
+ "Statement execute methods that take SQL as the first argument."
+ #{"execute" "executeQuery" "executeUpdate"})
+
+(defn- traced-statement
+ "Wrap a Statement in a proxy that creates a :quartz span around execute* methods.
+ Unlike PreparedStatement, Statement receives SQL at execute time."
+ [^Statement stmt]
+ (Proxy/newProxyInstance
+ (.getClassLoader Statement)
+ (into-array Class [Statement])
+ (reify InvocationHandler
+ (invoke [_ _ method args]
+ (let [method-name (.getName ^Method method)]
+ (if (and (statement-execute-with-sql method-name)
+ args
+ (pos? (alength ^objects args))
+ (instance? String (aget ^objects args 0)))
+ (let [sql (aget ^objects args 0)
+ operation (sql-operation sql)]
+ (tracing/with-span :quartz "quartz.db.execute"
+ {:db/statement sql
+ :db/operation (or operation "UNKNOWN")}
+ (invoke-or-unwrap method stmt args)))
+ ;; executeBatch and other non-SQL methods — pass through
+ (if (execute-method-names method-name)
+ (tracing/with-span :quartz "quartz.db.execute"
+ {:db/operation "BATCH"}
+ (invoke-or-unwrap method stmt args))
+ (invoke-or-unwrap method stmt args))))))))
+
+(defn- traced-connection
+ "Wrap a Connection in a proxy that intercepts prepareStatement, createStatement,
+ rollback, and commit to provide full JDBC-level tracing."
+ [^Connection conn]
+ (Proxy/newProxyInstance
+ (.getClassLoader Connection)
+ (into-array Class [Connection])
+ (reify InvocationHandler
+ (invoke [_ _ method args]
+ (let [method-name (.getName ^Method method)]
+ (case method-name
+ "prepareStatement"
+ (if (and args
+ (pos? (alength ^objects args))
+ (instance? String (aget ^objects args 0)))
+ (let [sql (aget ^objects args 0)
+ stmt (invoke-or-unwrap method conn args)]
+ (traced-prepared-statement stmt sql))
+ (invoke-or-unwrap method conn args))
+
+ "createStatement"
+ (let [stmt (invoke-or-unwrap method conn args)]
+ (traced-statement stmt))
+
+ "rollback"
+ (tracing/with-span :quartz "quartz.db.rollback" {}
+ (invoke-or-unwrap method conn args))
+
+ "commit"
+ (tracing/with-span :quartz "quartz.db.commit" {}
+ (invoke-or-unwrap method conn args))
+
+ ;; default — pass through
+ (invoke-or-unwrap method conn args)))))))
+
+(defn- connection-interceptor
+ "Connection wrapping function installed into bootstrap's ConnectionProvider.
+ Checks `:quartz` group at call time — wraps only when enabled."
+ [^Connection conn]
+ (if (tracing/group-enabled? :quartz)
+ (traced-connection conn)
+ conn))
+
+;;; ---------------------------------------------- Job Listener ------------------------------------------------------
+
+;; ThreadLocal storage for span state between jobToBeExecuted and jobWasExecuted callbacks.
+;; All callbacks for the same job execution run on the same Quartz worker thread.
+(def ^:private ^ThreadLocal listener-state (ThreadLocal.))
+
+(defn- create-tracing-job-listener
+ "Create a JobListener that wraps each job execution in a :quartz span.
+ The span becomes the parent of the defjob span (via OTel context propagation)."
+ ^JobListener []
+ (reify JobListener
+ (getName [_]
+ "metabase.tracing.quartz/quartz-tracing-listener")
+
+ (jobToBeExecuted [_ ctx]
+ (when (tracing/group-enabled? :quartz)
+ (try
+ (let [job-name (.. ctx getJobDetail getKey getName)
+ ^Span span (-> (tracing/get-tracer "metabase.quartz")
+ (.spanBuilder "quartz.job.execute")
+ (.setAttribute "quartz.job/name" job-name)
+ (.startSpan))
+ ^Scope scope (.makeCurrent span)]
+ (.set listener-state {:span span :scope scope}))
+ (catch Throwable t
+ (log/error t "Error starting quartz tracing span")))))
+
+ (jobExecutionVetoed [_ _])
+
+ (jobWasExecuted [_ _ job-exception]
+ (when-let [{:keys [^Span span ^Scope scope]} (.get listener-state)]
+ (.remove listener-state)
+ (try
+ (when job-exception
+ (.setStatus span StatusCode/ERROR (.getMessage ^Exception job-exception))
+ (.recordException span ^Throwable job-exception))
+ (.close scope)
+ (.end span)
+ (catch Throwable t
+ (log/error t "Error ending quartz tracing span")))))))
+
+;;; ------------------------------------------- Initialization -------------------------------------------------------
+
+(defmethod task/init! ::QuartzTracing [_]
+ (log/info "Initializing Quartz tracing instrumentation")
+ (task/add-job-listener! (create-tracing-job-listener))
+ (task.bootstrap/set-connection-interceptor! connection-interceptor))
diff --git a/src/metabase/tracing/settings.clj b/src/metabase/tracing/settings.clj
new file mode 100644
index 00000000000..cbbf9ddf9d6
--- /dev/null
+++ b/src/metabase/tracing/settings.clj
@@ -0,0 +1,85 @@
+(ns metabase.tracing.settings
+ "Env-var-only settings for the OpenTelemetry tracing module.
+ All settings are configured exclusively via environment variables (`:setter :none`)."
+ (:require
+ [metabase.settings.core :as setting :refer [defsetting]]
+ [metabase.util.i18n :refer [deferred-tru]]
+ [metabase.util.log :as log]))
+
+(set! *warn-on-reflection* true)
+
+(defsetting tracing-enabled
+ (deferred-tru "Enable OpenTelemetry tracing. When true, spans are exported to the configured OTLP collector.")
+ :type :boolean
+ :default false
+ :visibility :public
+ :export? false
+ :setter :none)
+
+(defsetting tracing-endpoint
+ (deferred-tru "OTLP HTTP collector endpoint for trace export (e.g., http://localhost:4318/v1/traces).")
+ :type :string
+ :default "http://localhost:4318/v1/traces"
+ :visibility :internal
+ :export? false
+ :setter :none)
+
+(defsetting tracing-groups
+ (deferred-tru "Comma-separated list of trace groups to enable, or \"all\" for everything.")
+ :type :string
+ :default "all"
+ :visibility :internal
+ :export? false
+ :setter :none)
+
+(defsetting tracing-log-level
+ (deferred-tru "Log level threshold during traced spans. When a span is active, log events at this level and above bypass per-logger level gates via Log4j2 DynamicThresholdFilter. Requires the filter in log4j2.xml. Valid values: \"INFO\", \"DEBUG\", \"TRACE\".")
+ :type :string
+ :default "INFO"
+ :visibility :internal
+ :export? false
+ :setter :none
+ :getter (fn reading-tracing-log-level []
+ (let [v (or (setting/get-raw-value :tracing-log-level string? identity)
+ "INFO")]
+ (if (#{"TRACE" "DEBUG" "INFO"} v)
+ v
+ (do
+ (log/warnf "MB_TRACING_LOG_LEVEL value '%s' is not valid, expected 'TRACE', 'DEBUG', or 'INFO'. Defaulting to 'INFO'." v)
+ "INFO")))))
+
+(defsetting tracing-max-queue-size
+ (deferred-tru "Maximum number of spans queued for export. When the queue is full, new spans are dropped silently. Controls memory usage when the collector is slow or unavailable.")
+ :type :integer
+ :default 2048
+ :visibility :internal
+ :export? false
+ :setter :none)
+
+(defsetting tracing-export-timeout-ms
+ (deferred-tru "Maximum time in milliseconds to wait for a batch export to complete. If the collector is slow or unreachable, the batch is dropped after this timeout.")
+ :type :integer
+ :default 10000
+ :visibility :internal
+ :export? false
+ :setter :none)
+
+(defsetting tracing-schedule-delay-ms
+ (deferred-tru "Delay in milliseconds between consecutive batch span exports to the collector.")
+ :type :integer
+ :default 5000
+ :visibility :internal
+ :export? false
+ :setter :none)
+
+(defsetting tracing-service-name
+ (deferred-tru "Service name reported in traces. Defaults to the hostname.")
+ :type :string
+ :visibility :internal
+ :export? false
+ :setter :none
+ :getter (fn reading-tracing-service-name []
+ (or (setting/get-raw-value :tracing-service-name string? identity)
+ (try (.getHostName (java.net.InetAddress/getLocalHost))
+ (catch Exception _
+ "metabase")))))
diff --git a/src/metabase/transforms/jobs.clj b/src/metabase/transforms/jobs.clj
index d733db93600..f01a132e6b7 100644
--- a/src/metabase/transforms/jobs.clj
+++ b/src/metabase/transforms/jobs.clj
@@ -11,6 +11,7 @@
[metabase.models.transforms.transform-run :as transform-run]
[metabase.revisions.core :as revisions]
[metabase.task.core :as task]
+ [metabase.tracing.core :as tracing]
[metabase.transforms-base.ordering :as transforms-base.ordering]
[metabase.transforms.execute :as transforms.execute]
[metabase.transforms.instrumentation :as transforms.instrumentation]
@@ -51,25 +52,26 @@
ordering)))
(defn- get-plan [transform-ids]
- (let [all-transforms (t2/select :model/Transform)
- global-ordering (transforms-base.ordering/transform-ordering all-transforms)
- relevant-ids (get-deps global-ordering transform-ids)
- transforms-by-id (into {}
- (keep (fn [{:keys [id] :as transform}]
- (when (relevant-ids id)
- [id transform])))
- all-transforms)
- ordering (sorted-ordering (select-keys global-ordering relevant-ids) transforms-by-id)]
- (when-let [cycle (transforms-base.ordering/find-cycle ordering)]
- (let [id->name (into {} (map (juxt :id :name)) all-transforms)]
- (throw (ex-info (str "Cyclic transform definitions detected: "
- (str/join " → " (map id->name cycle)))
- {:cycle cycle}))))
- (loop [complete (ordered-set/ordered-set)]
- (if-let [current-transform (next-transform ordering transforms-by-id complete)]
- (recur (conj complete (:id current-transform)))
- {:order (map transforms-by-id complete)
- :deps global-ordering}))))
+ (tracing/with-span :tasks "task.transform.plan" {:transform/count (count transform-ids)}
+ (let [all-transforms (t2/select :model/Transform)
+ global-ordering (transforms-base.ordering/transform-ordering all-transforms)
+ relevant-ids (get-deps global-ordering transform-ids)
+ transforms-by-id (into {}
+ (keep (fn [{:keys [id] :as transform}]
+ (when (relevant-ids id)
+ [id transform])))
+ all-transforms)
+ ordering (sorted-ordering (select-keys global-ordering relevant-ids) transforms-by-id)]
+ (when-let [cycle (transforms-base.ordering/find-cycle ordering)]
+ (let [id->name (into {} (map (juxt :id :name)) all-transforms)]
+ (throw (ex-info (str "Cyclic transform definitions detected: "
+ (str/join " → " (map id->name cycle)))
+ {:cycle cycle}))))
+ (loop [complete (ordered-set/ordered-set)]
+ (if-let [current-transform (next-transform ordering transforms-by-id complete)]
+ (recur (conj complete (:id current-transform)))
+ {:order (map transforms-by-id complete)
+ :deps global-ordering})))))
(defn- block-until-not-already-running [transform-id]
(when-let [active-run (transform-run/running-run-for-transform-id transform-id)]
@@ -80,7 +82,8 @@
(defn- run-transform! [run-id run-method user-id {transform-id :id :as transform}]
(if-not (transforms.u/check-feature-enabled transform)
(log/warnf "Skip running transform %d due to lacking premium features" transform-id)
- (do
+ (tracing/with-span :tasks "task.transform.execute" {:transform/id transform-id
+ :transform/name (:name transform)}
(block-until-not-already-running transform-id)
(let [try-exec
(fn []
@@ -94,7 +97,12 @@
(throw e)))))]
(loop []
(when (= :already-running (try-exec))
- (block-until-not-already-running transform-id)
+ (when (transform-run/running-run-for-transform-id transform-id)
+ (log/warn "Transform" (pr-str transform-id) "already running, waiting")
+ (loop []
+ (Thread/sleep 2000)
+ (when (transform-run/running-run-for-transform-id transform-id)
+ (recur))))
(recur))))
(transforms.job-run/add-run-activity! run-id))))
@@ -232,29 +240,32 @@
(let [transforms (job-transform-ids job-id)]
(log/info "Executing transform job" (pr-str job-id) "with transforms" (pr-str transforms))
(let [{run-id :id} (transforms.job-run/start-run! job-id run-method)]
- (transforms.instrumentation/with-job-timing [job-id run-method]
- (try ;; catch any catastrophic problems
- (let [result (run-transforms! run-id transforms opts)]
- (case (::status result)
- :succeeded (transforms.job-run/succeed-started-run! run-id)
- :failed (try
- (transforms.job-run/fail-started-run! run-id {:message (compile-transform-failure-messages (::failures result))})
- (when (= :cron run-method)
- (notify-transform-failures job-id (::failures result)))
- (catch Exception e
- (log/error e "Error when failing a transform run.")))))
- (catch Throwable t
- ;; We don't expect a catastrophic failure, but neither did the Titanic.
- ;; We should clean up in this case and notify the admin users.
- (try
- (transforms.job-run/fail-started-run! run-id {:message (.getMessage t)})
- (when (= :cron run-method)
- (if (::transform-failure (ex-data t))
- (notify-transform-failures job-id (::failures (ex-data t)))
- (notify-job-failure job-id (.getMessage t))))
- (catch Exception e
- (log/error e "Error when failing a transform job run.")))
- (throw t))))
+ (tracing/with-span :tasks "task.transform.run-job" {:transform.job/id job-id
+ :transform.job/run-method (name run-method)
+ :transform.job/count (count transforms)}
+ (transforms.instrumentation/with-job-timing [job-id run-method]
+ (try ;; catch any catastrophic problems
+ (let [result (run-transforms! run-id transforms opts)]
+ (case (::status result)
+ :succeeded (transforms.job-run/succeed-started-run! run-id)
+ :failed (try
+ (transforms.job-run/fail-started-run! run-id {:message (compile-transform-failure-messages (::failures result))})
+ (when (= :cron run-method)
+ (notify-transform-failures job-id (::failures result)))
+ (catch Exception e
+ (log/error e "Error when failing a transform run.")))))
+ (catch Throwable t
+ ;; We don't expect a catastrophic failure, but neither did the Titanic.
+ ;; We should clean up in this case and notify the admin users.
+ (try
+ (transforms.job-run/fail-started-run! run-id {:message (.getMessage t)})
+ (when (= :cron run-method)
+ (if (::transform-failure (ex-data t))
+ (notify-transform-failures job-id (::failures (ex-data t)))
+ (notify-job-failure job-id (.getMessage t))))
+ (catch Exception e
+ (log/error e "Error when failing a transform job run.")))
+ (throw t)))))
run-id))))
(def ^:private job-key "metabase.transforms.jobs.timeout-job")
diff --git a/src/metabase/transforms/query_impl.clj b/src/metabase/transforms/query_impl.clj
index ee8c2751454..de3ed6bb1aa 100644
--- a/src/metabase/transforms/query_impl.clj
+++ b/src/metabase/transforms/query_impl.clj
@@ -3,6 +3,7 @@
[clojure.core.async :as a]
[metabase.driver :as driver]
[metabase.driver.connection :as driver.conn]
+ [metabase.tracing.core :as tracing]
[metabase.transforms-base.interface :as transforms-base.i]
[metabase.transforms-base.util :as transforms-base.u]
[metabase.transforms.instrumentation :as transforms.instrumentation]
@@ -29,29 +30,33 @@
(driver.conn/with-write-connection
(log/info "Executing transform" id "with target" (pr-str target)
(when (driver.conn/write-connection-requested?) " using write connection"))
- (let [conn-spec (driver/connection-spec driver db)
- transform-details {:db-id (:id db) :conn-spec conn-spec :output-schema (:schema target)}]
- (transforms.instrumentation/with-stage-timing [run-id [:computation :mbql-query]]
- (transforms.u/run-cancelable-transform!
- run-id driver transform-details
- (fn [cancel-chan]
- (let [result (transforms-base.i/execute-base!
- transform
- {:cancelled? #(boolean (a/poll! cancel-chan))
- :run-id run-id
- :with-stage-timing-fn (fn [rid stage thunk]
- (transforms.instrumentation/with-stage-timing [rid stage]
- (thunk)))})]
- ;; Bridge result-map to exception-based flow for run-cancelable-transform!
- (when-not (= :succeeded (:status result))
- (throw (or (:error result) (ex-info "Transform failed" {:status (:status result)}))))
- result))))
- ;; Post-processing: sync, transform_id, events, secondary indexes (after succeed-started-run!)
- (transforms-base.u/complete-execution! transform
- {:run-id run-id
- :with-stage-timing-fn (fn [rid stage thunk]
- (transforms.instrumentation/with-stage-timing [rid stage]
- (thunk)))}))))
+ (tracing/with-span :tasks "task.transform.query" {:transform/id id
+ :transform/target-type (name (keyword (:type target)))
+ :db/id (:id db)
+ :db/engine (name driver)}
+ (let [conn-spec (driver/connection-spec driver db)
+ transform-details {:db-id (:id db) :conn-spec conn-spec :output-schema (:schema target)}]
+ (transforms.instrumentation/with-stage-timing [run-id [:computation :mbql-query]]
+ (transforms.u/run-cancelable-transform!
+ run-id driver transform-details
+ (fn [cancel-chan]
+ (let [result (transforms-base.i/execute-base!
+ transform
+ {:cancelled? #(boolean (a/poll! cancel-chan))
+ :run-id run-id
+ :with-stage-timing-fn (fn [rid stage thunk]
+ (transforms.instrumentation/with-stage-timing [rid stage]
+ (thunk)))})]
+ ;; Bridge result-map to exception-based flow for run-cancelable-transform!
+ (when-not (= :succeeded (:status result))
+ (throw (or (:error result) (ex-info "Transform failed" {:status (:status result)}))))
+ result))))
+ ;; Post-processing: sync, transform_id, events, secondary indexes (after succeed-started-run!)
+ (transforms-base.u/complete-execution! transform
+ {:run-id run-id
+ :with-stage-timing-fn (fn [rid stage thunk]
+ (transforms.instrumentation/with-stage-timing [rid stage]
+ (thunk)))})))))
(catch Throwable t
(if (= :already-running (:error (ex-data t)))
(log/warnf "Transform %d is already running" id)
diff --git a/test/metabase/server/middleware/trace_test.clj b/test/metabase/server/middleware/trace_test.clj
new file mode 100644
index 00000000000..2869c334b79
--- /dev/null
+++ b/test/metabase/server/middleware/trace_test.clj
@@ -0,0 +1,98 @@
+(ns metabase.server.middleware.trace-test
+ (:require
+ [clojure.test :refer [deftest is testing]]
+ [metabase.server.middleware.trace :as mw.trace]
+ [metabase.tracing.core :as tracing]))
+
+(set! *warn-on-reflection* true)
+
+(defn- call-handler
+ "Call wrap-trace middleware with a mock handler, returning the response or exception.
+ The mock handler immediately calls respond with {:status 200}."
+ [request]
+ (let [handler (fn [_req respond _raise] (respond {:status 200}))
+ wrapped (mw.trace/wrap-trace handler)
+ result (promise)]
+ (wrapped request #(deliver result %) #(deliver result %))
+ (deref result 1000 :timeout)))
+
+(deftest non-api-requests-skipped-test
+ (testing "non-API requests are passed through without tracing"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ ;; Static files, root page, and other non-API paths should be passed through
+ (is (= 200 (:status (call-handler {:uri "/app/fonts/lato.woff2" :request-method :get :headers {}}))))
+ (is (= 200 (:status (call-handler {:uri "/app/dist/app-main.js" :request-method :get :headers {}}))))
+ (is (= 200 (:status (call-handler {:uri "/favicon.ico" :request-method :get :headers {}}))))
+ (is (= 200 (:status (call-handler {:uri "/" :request-method :get :headers {}}))))
+ ;; API requests should also work (they go through tracing)
+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get :headers {}}))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest tracing-disabled-passthrough-test
+ (testing "when tracing is disabled, requests are passed through directly"
+ (tracing/shutdown-groups!)
+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get :headers {}}))))))
+
+(deftest traceparent-parsing-test
+ (testing "valid traceparent header is parsed and trace ID is forced"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (let [trace-id "abcdef1234567890abcdef1234567890"
+ request {:uri "/api/health"
+ :request-method :get
+ :headers {"traceparent" (str "00-" trace-id "-1234567890abcdef-01")}}]
+ ;; Ensure forced trace ID is clean before test
+ (tracing/clear-forced-trace-id!)
+ (is (= 200 (:status (call-handler request))))
+ ;; After the request, the forced trace ID should be consumed (cleared)
+ (is (nil? (tracing/get-and-clear-forced-trace-id!))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest invalid-traceparent-ignored-test
+ (testing "invalid traceparent headers are silently ignored"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ ;; Malformed headers should not cause errors
+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get
+ :headers {"traceparent" "garbage"}}))))
+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get
+ :headers {"traceparent" ""}}))))
+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get
+ :headers {"traceparent" "00-short-id-01"}}))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest exception-handling-test
+ (testing "exceptions in the handler are propagated via raise"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (let [handler (fn [_req _respond raise] (raise (ex-info "boom" {})))
+ wrapped (mw.trace/wrap-trace handler)
+ result (promise)]
+ (wrapped {:uri "/api/test" :request-method :get :headers {}}
+ #(deliver result %)
+ #(deliver result %))
+ (let [r (deref result 1000 :timeout)]
+ (is (instance? clojure.lang.ExceptionInfo r))
+ (is (= "boom" (.getMessage ^Exception r)))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest sync-exception-handling-test
+ (testing "synchronous exceptions from handler are caught and raised"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (let [handler (fn [_req _respond _raise] (throw (ex-info "sync-boom" {})))
+ wrapped (mw.trace/wrap-trace handler)
+ result (promise)]
+ (wrapped {:uri "/api/test" :request-method :get :headers {}}
+ #(deliver result %)
+ #(deliver result %))
+ (let [r (deref result 1000 :timeout)]
+ (is (instance? clojure.lang.ExceptionInfo r))
+ (is (= "sync-boom" (.getMessage ^Exception r)))))
+ (finally
+ (tracing/shutdown-groups!)))))
diff --git a/test/metabase/tracing/core_test.clj b/test/metabase/tracing/core_test.clj
new file mode 100644
index 00000000000..755d5ff9091
--- /dev/null
+++ b/test/metabase/tracing/core_test.clj
@@ -0,0 +1,258 @@
+(ns metabase.tracing.core-test
+ (:require
+ [clojure.test :refer [deftest is testing]]
+ [metabase.tracing.core :as tracing])
+ (:import
+ (org.apache.logging.log4j ThreadContext)))
+
+(set! *warn-on-reflection* true)
+
+(deftest group-enabled?-disabled-by-default-test
+ (testing "group-enabled? returns false when tracing is not initialized"
+ ;; By default, enabled-groups atom is nil (tracing not initialized)
+ (is (not (tracing/group-enabled? :qp)))
+ (is (not (tracing/group-enabled? :sync)))
+ (is (not (tracing/group-enabled? :api)))))
+
+(deftest init-enabled-groups!-all-test
+ (testing "init-enabled-groups! with 'all' enables all groups"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (is (tracing/group-enabled? :qp))
+ (is (tracing/group-enabled? :sync))
+ (is (tracing/group-enabled? :api))
+ (is (tracing/group-enabled? :db-user))
+ (is (tracing/group-enabled? :db-app))
+ (is (tracing/group-enabled? :tasks))
+ (is (tracing/group-enabled? :events))
+ ;; even non-registered groups should be enabled with :all
+ (is (tracing/group-enabled? :nonexistent-group))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest init-enabled-groups!-selective-test
+ (testing "init-enabled-groups! with comma-separated list enables only those groups"
+ (try
+ (tracing/init-enabled-groups! "qp,api,db-user" "INFO")
+ (is (tracing/group-enabled? :qp))
+ (is (tracing/group-enabled? :api))
+ (is (tracing/group-enabled? :db-user))
+ (is (not (tracing/group-enabled? :sync)))
+ (is (not (tracing/group-enabled? :tasks)))
+ (is (not (tracing/group-enabled? :events)))
+ (is (not (tracing/group-enabled? :db-app)))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest init-enabled-groups!-whitespace-test
+ (testing "init-enabled-groups! handles whitespace in group names"
+ (try
+ (tracing/init-enabled-groups! " qp , sync " "INFO")
+ (is (tracing/group-enabled? :qp))
+ (is (tracing/group-enabled? :sync))
+ (is (not (tracing/group-enabled? :api)))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest shutdown-groups!-test
+ (testing "shutdown-groups! disables all groups"
+ (tracing/init-enabled-groups! "all" "INFO")
+ (is (tracing/group-enabled? :qp))
+ (tracing/shutdown-groups!)
+ (is (not (tracing/group-enabled? :qp)))))
+
+(deftest register-group!-test
+ (testing "register-group! adds groups to the registry"
+ (tracing/register-group! :test-group "A test group")
+ (is (contains? (tracing/registered-groups) :test-group))
+ (is (= "A test group" (:description (:test-group (tracing/registered-groups)))))))
+
+(deftest built-in-groups-registered-test
+ (testing "all built-in groups are registered"
+ (let [groups (tracing/registered-groups)]
+ (is (contains? groups :qp))
+ (is (contains? groups :sync))
+ (is (contains? groups :tasks))
+ (is (contains? groups :api))
+ (is (contains? groups :db-user))
+ (is (contains? groups :db-app))
+ (is (contains? groups :events)))))
+
+(deftest with-span-disabled-executes-body-test
+ (testing "with-span executes body when tracing is disabled (no-op wrapper)"
+ ;; Tracing not initialized, so group-enabled? returns false
+ (tracing/shutdown-groups!)
+ (is (= 42 (tracing/with-span :qp "test-span" {} 42)))
+ (is (= "hello" (tracing/with-span :qp "test-span" {} "hello")))
+ (let [side-effect (atom false)]
+ (tracing/with-span :qp "test-span" {}
+ (reset! side-effect true))
+ (is @side-effect))))
+
+(deftest inject-trace-level-into-mdc-test
+ (testing "inject-trace-id-into-mdc! sets trace_level in MDC from cached setting"
+ (try
+ (tracing/init-enabled-groups! "all" "DEBUG")
+ (tracing/inject-trace-id-into-mdc!)
+ (is (= "DEBUG" (ThreadContext/get "trace_level")))
+ (tracing/clear-trace-id-from-mdc!)
+ (is (nil? (ThreadContext/get "trace_level")))
+ (finally
+ (tracing/shutdown-groups!))))
+ (testing "inject-trace-id-into-mdc! does not set trace_level when tracing is not initialized"
+ (tracing/shutdown-groups!)
+ (tracing/inject-trace-id-into-mdc!)
+ (is (nil? (ThreadContext/get "trace_level")))
+ (tracing/clear-trace-id-from-mdc!)))
+
+(deftest trace-log-level-values-test
+ (testing "each valid log level is correctly cached and injected into MDC"
+ (doseq [level ["INFO" "DEBUG" "TRACE"]]
+ (testing (str "level=" level)
+ (try
+ (tracing/init-enabled-groups! "all" level)
+ (tracing/inject-trace-id-into-mdc!)
+ (is (= level (ThreadContext/get "trace_level")))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!)))))))
+
+(deftest trace-log-level-nil-defaults-to-info-test
+ (testing "nil log-level-str defaults to INFO"
+ (try
+ (tracing/init-enabled-groups! "all" nil)
+ (tracing/inject-trace-id-into-mdc!)
+ (is (= "INFO" (ThreadContext/get "trace_level")))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!)))))
+
+(deftest shutdown-clears-trace-log-level-test
+ (testing "shutdown-groups! clears trace_level so MDC injection becomes a no-op"
+ (tracing/init-enabled-groups! "all" "DEBUG")
+ (tracing/shutdown-groups!)
+ (tracing/inject-trace-id-into-mdc!)
+ (is (nil? (ThreadContext/get "trace_level")))
+ (tracing/clear-trace-id-from-mdc!)))
+
+(deftest reinit-changes-trace-log-level-test
+ (testing "re-initializing with a different log level updates the cached value"
+ (try
+ (tracing/init-enabled-groups! "all" "DEBUG")
+ (tracing/inject-trace-id-into-mdc!)
+ (is (= "DEBUG" (ThreadContext/get "trace_level")))
+ (tracing/clear-trace-id-from-mdc!)
+ ;; re-init with different level
+ (tracing/init-enabled-groups! "all" "TRACE")
+ (tracing/inject-trace-id-into-mdc!)
+ (is (= "TRACE" (ThreadContext/get "trace_level")))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!)))))
+
+;;; ------------------------------------------ Forced Trace ID Tests -------------------------------------------
+
+(deftest force-trace-id-basic-test
+ (testing "force-trace-id! sets value, get-and-clear-forced-trace-id! returns and clears it"
+ (tracing/force-trace-id! "abcdef1234567890abcdef1234567890")
+ (is (= "abcdef1234567890abcdef1234567890" (tracing/get-and-clear-forced-trace-id!)))
+ ;; second call returns nil (already consumed)
+ (is (nil? (tracing/get-and-clear-forced-trace-id!)))))
+
+(deftest force-trace-id-nil-when-not-set-test
+ (testing "get-and-clear-forced-trace-id! returns nil when nothing is forced"
+ (tracing/clear-forced-trace-id!)
+ (is (nil? (tracing/get-and-clear-forced-trace-id!)))))
+
+(deftest clear-forced-trace-id-test
+ (testing "clear-forced-trace-id! removes the value without consuming it"
+ (tracing/force-trace-id! "1111111111111111aaaaaaaaaaaaaaaa")
+ (tracing/clear-forced-trace-id!)
+ (is (nil? (tracing/get-and-clear-forced-trace-id!)))))
+
+(deftest force-trace-id-thread-isolation-test
+ (testing "forced trace ID is thread-local — other threads don't see it"
+ (tracing/force-trace-id! "aaaa0000bbbb1111cccc2222dddd3333")
+ (let [other-thread-result (promise)]
+ (.start (Thread. (fn []
+ (deliver other-thread-result (tracing/get-and-clear-forced-trace-id!)))))
+ (is (nil? (deref other-thread-result 1000 :timeout)))
+ ;; still available on this thread
+ (is (= "aaaa0000bbbb1111cccc2222dddd3333" (tracing/get-and-clear-forced-trace-id!))))))
+
+(deftest nested-with-span-restores-mdc-test
+ (testing "nested with-span restores parent MDC values instead of clearing them"
+ (try
+ (tracing/init-enabled-groups! "all" "DEBUG")
+ ;; Simulate what wrap-trace does: inject MDC for the root span
+ (tracing/inject-trace-id-into-mdc!)
+ (let [root-trace-id (ThreadContext/get "trace_id")
+ root-span-id (ThreadContext/get "span_id")]
+ (is (some? root-trace-id) "root trace_id should be set")
+ (is (some? root-span-id) "root span_id should be set")
+ ;; Run a child with-span (simulates db-app.read-check, etc.)
+ (tracing/with-span :db-app "db-app.test-child" {}
+ ;; Inside child span, MDC should have values
+ (is (some? (ThreadContext/get "trace_id")) "child trace_id should be set")
+ (is (some? (ThreadContext/get "span_id")) "child span_id should be set"))
+ ;; After child exits, parent MDC should be RESTORED, not cleared
+ (is (= root-trace-id (ThreadContext/get "trace_id"))
+ "trace_id should be restored to parent value after child span exits")
+ (is (= root-span-id (ThreadContext/get "span_id"))
+ "span_id should be restored to parent value after child span exits")
+ (is (= "DEBUG" (ThreadContext/get "trace_level"))
+ "trace_level should still be set after child span exits"))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!))))
+
+ (testing "outermost with-span still clears MDC when no parent values exist"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ ;; No prior MDC set — this with-span is the outermost
+ (ThreadContext/remove "trace_id")
+ (ThreadContext/remove "span_id")
+ (tracing/with-span :qp "qp.test" {}
+ (is (some? (ThreadContext/get "trace_id"))))
+ ;; After outermost exits, MDC should be cleared
+ (is (nil? (ThreadContext/get "trace_id")) "trace_id should be cleared after outermost span")
+ (is (nil? (ThreadContext/get "span_id")) "span_id should be cleared after outermost span")
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!)))))
+
+(deftest with-span-pyroscope-root-span-noop-test
+ (testing "with-span calls pyroscope set/clear for root spans (no-op without agent)"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (ThreadContext/remove "trace_id")
+ (ThreadContext/remove "span_id")
+ ;; Root span — no prior MDC context. Pyroscope calls are no-ops but must not throw.
+ (is (= :ok (tracing/with-span :tasks "task.test-job" {} :ok)))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!))))
+
+ (testing "with-span skips pyroscope for nested spans (parent MDC already set)"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (ThreadContext/remove "trace_id")
+ (ThreadContext/remove "span_id")
+ ;; Outer with-span sets MDC → inner with-span sees prev-span-id → skips pyroscope
+ (tracing/with-span :tasks "task.outer" {}
+ (is (some? (ThreadContext/get "span_id")) "outer span should have MDC")
+ (is (= :inner-ok (tracing/with-span :qp "qp.inner" {} :inner-ok))))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!))))
+
+ (testing "rapid root span cycles don't throw or leak pyroscope state"
+ (try
+ (tracing/init-enabled-groups! "all" "INFO")
+ (dotimes [_ 50]
+ (ThreadContext/remove "trace_id")
+ (ThreadContext/remove "span_id")
+ (tracing/with-span :tasks "task.rapid" {} :ok))
+ (finally
+ (tracing/clear-trace-id-from-mdc!)
+ (tracing/shutdown-groups!)))))
diff --git a/test/metabase/tracing/pyroscope_test.clj b/test/metabase/tracing/pyroscope_test.clj
new file mode 100644
index 00000000000..217071c44aa
--- /dev/null
+++ b/test/metabase/tracing/pyroscope_test.clj
@@ -0,0 +1,32 @@
+(ns metabase.tracing.pyroscope-test
+ (:require
+ [clojure.test :refer [deftest is testing]]
+ [metabase.tracing.core :as tracing]))
+
+(set! *warn-on-reflection* true)
+
+(deftest pyroscope-available?-without-agent-test
+ (testing "pyroscope-available? returns false when pyroscope.jar is not on the classpath"
+ (is (false? (tracing/pyroscope-available?)))))
+
+(deftest set-pyroscope-context!-noop-without-agent-test
+ (testing "set-pyroscope-context! is a safe no-op when Pyroscope is not available"
+ ;; nil span is safe because the when-let guard prevents any execution
+ (is (nil? (tracing/set-pyroscope-context! nil "abcdef0123456789" "api.request")))))
+
+(deftest clear-pyroscope-context!-noop-without-agent-test
+ (testing "clear-pyroscope-context! is a safe no-op when Pyroscope is not available"
+ (is (nil? (tracing/clear-pyroscope-context!)))))
+
+(deftest set-pyroscope-context!-invalid-span-id-test
+ (testing "set-pyroscope-context! handles invalid span IDs gracefully"
+ ;; Even if Pyroscope were available, these should not throw
+ (is (nil? (tracing/set-pyroscope-context! nil "" "api.request")))
+ (is (nil? (tracing/set-pyroscope-context! nil "not-hex" "api.request")))
+ (is (nil? (tracing/set-pyroscope-context! nil nil "api.request")))))
+
+(deftest repeated-calls-are-safe-test
+ (testing "rapid set/clear cycles don't throw or leak state"
+ (dotimes [_ 100]
+ (tracing/set-pyroscope-context! nil "abcdef0123456789" "test.span")
+ (tracing/clear-pyroscope-context!))))
diff --git a/test/metabase/tracing/quartz_test.clj b/test/metabase/tracing/quartz_test.clj
new file mode 100644
index 00000000000..615288cfb1a
--- /dev/null
+++ b/test/metabase/tracing/quartz_test.clj
@@ -0,0 +1,258 @@
+(ns metabase.tracing.quartz-test
+ (:require
+ [clojure.test :refer [deftest is testing]]
+ [metabase.task.bootstrap :as task.bootstrap]
+ [metabase.tracing.core :as tracing]
+ [metabase.tracing.quartz :as tracing.quartz])
+ (:import
+ (java.sql Connection PreparedStatement ResultSet Statement)
+ (org.quartz JobDetail JobExecutionContext JobExecutionException JobKey JobListener)))
+
+(set! *warn-on-reflection* true)
+
+;;; ------------------------------------------------ Helpers ----------------------------------------------------------
+
+(defn- mock-prepared-statement
+ "Create a mock PreparedStatement that records which methods were called."
+ [calls-atom]
+ (reify PreparedStatement
+ (executeQuery [_]
+ (swap! calls-atom conj :executeQuery)
+ (reify ResultSet
+ (close [_])))
+ (executeUpdate [_]
+ (swap! calls-atom conj :executeUpdate)
+ 1)
+ (execute [_]
+ (swap! calls-atom conj :execute)
+ true)
+ (close [_]
+ (swap! calls-atom conj :close))))
+
+(defn- mock-statement
+ "Create a mock Statement that records which methods were called."
+ [calls-atom]
+ (reify Statement
+ (execute [_ sql]
+ (swap! calls-atom conj [:execute sql])
+ true)
+ (executeQuery [_ sql]
+ (swap! calls-atom conj [:executeQuery sql])
+ (reify ResultSet
+ (close [_])))
+ (executeUpdate [_ sql]
+ (swap! calls-atom conj [:executeUpdate sql])
+ 1)
+ (close [_]
+ (swap! calls-atom conj [:close-stmt]))))
+
+(defn- mock-connection
+ "Create a mock Connection that returns mock PreparedStatements and Statements."
+ [stmt-calls-atom]
+ (reify Connection
+ (prepareStatement [_ _sql]
+ (mock-prepared-statement stmt-calls-atom))
+ (createStatement [_]
+ (mock-statement stmt-calls-atom))
+ (rollback [_]
+ (swap! stmt-calls-atom conj :rollback))
+ (commit [_]
+ (swap! stmt-calls-atom conj :commit))
+ (close [_])
+ (isClosed [_] false)))
+
+(defn- mock-job-execution-context
+ "Create a mock JobExecutionContext for testing the listener."
+ ^JobExecutionContext [^String job-name]
+ (let [job-key (JobKey. job-name)
+ job-detail (reify JobDetail (getKey [_] job-key))]
+ (reify JobExecutionContext
+ (getJobDetail [_] job-detail))))
+
+;;; ---------------------------------------- sql-operation tests ------------------------------------------------------
+
+(deftest sql-operation-test
+ (testing "extracts SQL verb from various SQL statements"
+ (is (= "SELECT" (#'tracing.quartz/sql-operation "SELECT * FROM QRTZ_TRIGGERS")))
+ (is (= "UPDATE" (#'tracing.quartz/sql-operation "UPDATE QRTZ_TRIGGERS SET state = ?")))
+ (is (= "INSERT" (#'tracing.quartz/sql-operation "INSERT INTO QRTZ_FIRED_TRIGGERS VALUES (?)")))
+ (is (= "DELETE" (#'tracing.quartz/sql-operation "DELETE FROM QRTZ_FIRED_TRIGGERS WHERE id = ?"))))
+ (testing "handles edge cases"
+ (is (= "SELECT" (#'tracing.quartz/sql-operation " SELECT * FROM t ")))
+ (is (= "COMMIT" (#'tracing.quartz/sql-operation "COMMIT")))
+ (is (nil? (#'tracing.quartz/sql-operation nil)))))
+
+;;; ----------------------------------------- JDBC proxy tests --------------------------------------------------------
+
+(deftest traced-connection-intercepts-prepareStatement-test
+ (testing "traced-connection wraps prepareStatement results"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ traced (#'tracing.quartz/traced-connection conn)]
+ (testing "returned connection implements Connection"
+ (is (instance? Connection traced)))
+ (testing "prepareStatement returns a PreparedStatement proxy"
+ (let [^PreparedStatement stmt (.prepareStatement ^Connection traced "SELECT * FROM QRTZ_TRIGGERS")]
+ (is (instance? PreparedStatement stmt))
+ (testing "execute methods delegate to the underlying statement"
+ (.executeQuery stmt)
+ (is (= [:executeQuery] @calls)))
+ (testing "executeUpdate also delegates"
+ (.executeUpdate stmt)
+ (is (= [:executeQuery :executeUpdate] @calls))))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest traced-connection-delegates-non-intercepted-methods-test
+ (testing "methods other than prepareStatement pass through unchanged"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ traced (#'tracing.quartz/traced-connection conn)]
+ (testing "close delegates to original"
+ (.close ^Connection traced)
+ ;; no exception means the method was delegated
+ (is true))
+ (testing "isClosed delegates to original"
+ (is (false? (.isClosed ^Connection traced)))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest traced-connection-intercepts-createStatement-test
+ (testing "traced-connection wraps createStatement results"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ traced (#'tracing.quartz/traced-connection conn)]
+ (testing "createStatement returns a Statement proxy"
+ (let [^Statement stmt (.createStatement ^Connection traced)]
+ (is (instance? Statement stmt))
+ (testing "execute with SQL delegates and records SQL"
+ (.execute stmt "DISCARD ALL;")
+ (is (= [[:execute "DISCARD ALL;"]] @calls)))
+ (testing "executeUpdate with SQL delegates"
+ (.executeUpdate stmt "UPDATE QRTZ_TRIGGERS SET state = 'WAITING'")
+ (is (= [[:execute "DISCARD ALL;"]
+ [:executeUpdate "UPDATE QRTZ_TRIGGERS SET state = 'WAITING'"]]
+ @calls))))))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest traced-connection-intercepts-rollback-test
+ (testing "traced-connection wraps rollback in a span"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ traced (#'tracing.quartz/traced-connection conn)]
+ (.rollback ^Connection traced)
+ (is (= [:rollback] @calls)
+ "rollback should delegate to the underlying connection"))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest traced-connection-intercepts-commit-test
+ (testing "traced-connection wraps commit in a span"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ traced (#'tracing.quartz/traced-connection conn)]
+ (.commit ^Connection traced)
+ (is (= [:commit] @calls)
+ "commit should delegate to the underlying connection"))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+(deftest connection-interceptor-respects-group-enabled-test
+ (testing "when :quartz group is disabled, returns unwrapped connection"
+ (tracing/shutdown-groups!)
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ result (#'tracing.quartz/connection-interceptor conn)]
+ (is (identical? conn result)
+ "should return the exact same connection object when tracing disabled")))
+ (testing "when :quartz group is enabled, returns wrapped connection"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [calls (atom [])
+ conn (mock-connection calls)
+ result (#'tracing.quartz/connection-interceptor conn)]
+ (is (not (identical? conn result))
+ "should return a different (wrapped) connection when tracing enabled")
+ (is (instance? Connection result)))
+ (finally
+ (tracing/shutdown-groups!)))))
+
+;;; ---------------------------------------- JobListener tests --------------------------------------------------------
+
+(deftest job-listener-name-test
+ (testing "listener has a descriptive name"
+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)]
+ (is (= "metabase.tracing.quartz/quartz-tracing-listener" (.getName listener))))))
+
+(deftest job-listener-disabled-is-noop-test
+ (testing "when :quartz group is disabled, jobToBeExecuted is a no-op"
+ (tracing/shutdown-groups!)
+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)
+ ctx (mock-job-execution-context "TestJob")]
+ (.jobToBeExecuted listener ctx)
+ ;; No span should be in ThreadLocal
+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state)))
+ ;; jobWasExecuted should also be safe to call (no-op since no state)
+ (.jobWasExecuted listener ctx nil)
+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state))))))
+
+(deftest job-listener-enabled-creates-and-ends-span-test
+ (testing "when :quartz group is enabled, jobToBeExecuted creates a span"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)
+ ctx (mock-job-execution-context "TestSessionCleanup")]
+ (.jobToBeExecuted listener ctx)
+ (testing "span state is stored in ThreadLocal"
+ (let [state (.get ^ThreadLocal @#'tracing.quartz/listener-state)]
+ (is (some? state))
+ (is (contains? state :span))
+ (is (contains? state :scope))))
+ (testing "jobWasExecuted cleans up ThreadLocal"
+ (.jobWasExecuted listener ctx nil)
+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state)))))
+ (finally
+ ;; Safety: clean up any lingering state
+ (.remove ^ThreadLocal @#'tracing.quartz/listener-state)
+ (tracing/shutdown-groups!)))))
+
+(deftest job-listener-records-exception-test
+ (testing "when job fails, exception is recorded on the span"
+ (try
+ (tracing/init-enabled-groups! "quartz" "INFO")
+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)
+ ctx (mock-job-execution-context "FailingJob")
+ exception (JobExecutionException. "something broke")]
+ (.jobToBeExecuted listener ctx)
+ (is (some? (.get ^ThreadLocal @#'tracing.quartz/listener-state)))
+ ;; Should not throw — exception is recorded on span, not re-thrown
+ (.jobWasExecuted listener ctx exception)
+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state))))
+ (finally
+ (.remove ^ThreadLocal @#'tracing.quartz/listener-state)
+ (tracing/shutdown-groups!)))))
+
+;;; ----------------------------------- bootstrap interceptor tests ---------------------------------------------------
+
+(deftest set-connection-interceptor-test
+ (testing "set-connection-interceptor! installs and removes interceptors"
+ (let [the-atom @#'task.bootstrap/connection-interceptor
+ original @the-atom]
+ (try
+ (task.bootstrap/set-connection-interceptor! identity)
+ (is (some? @the-atom))
+ (task.bootstrap/set-connection-interceptor! nil)
+ (is (nil? @the-atom))
+ (finally
+ (reset! the-atom original))))))