The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Add OpenTelemetry Tracing module (#69295)
Anton Kostenko5765b0df156c
diff --git a/.claude/skills/add-tracing/SKILL.md b/.claude/skills/add-tracing/SKILL.mdnew file mode 100644index 00000000000..ffa661f21d8--- /dev/null+++ b/.claude/skills/add-tracing/SKILL.md@@ -0,0 +1,357 @@+--b-+name: add-tracing+description: Add OpenTelemetry tracing spans to Clojure code following Metabase tracing conventions. Use when instrumenting backend code with trace coverage.+---++# Add Tracing Spans to Clojure Code++This skill helps you add OpenTelemetry (OTel) tracing spans to the Metabase backend codebase using the custom `tracing/with-span` macro.++## Reference Files++- `src/metabase/tracing/core.clj` - `with-span` macro, group registry, SDK lifecycle, `best-effort-sanitize-sql`, Pyroscope integration+- `src/metabase/task/impl.clj` - `defjob` macro that wraps Quartz jobs with root spans+- `.clj-kondo/config/modules/config.edn` - Module boundary configuration++## Module Architecture++The tracing module has a deliberately minimal API surface. **Only 2 namespaces are public** (listed in `:api` in the module config):++| Namespace | Role | Status |+|---|---|---|+| `tracing.core` | Primary API: `with-span`, groups, SDK lifecycle, Pyroscope, MDC, `best-effort-sanitize-sql` | **Public API** |+| `tracing.init` | Side-effect loader for `quartz` and `settings` | **Public API** (init convention) |+| `tracing.attributes` | `best-effort-sanitize-sql` implementation (re-exported via `tracing.core`) | Internal |+| `tracing.settings` | Setting definitions (`MB_TRACING_*` env vars) | Internal |+| `tracing.quartz` | Quartz JDBC proxy + JobListener | Internal |++**Rules:**+- Only require `[metabase.tracing.core :as tracing]` from outside the module. `tracing/best-effort-sanitize-sql` and all other public functions are available from this single namespace.+- Do not add new API namespaces. Add new public functions to `tracing.core` instead.+- Do not require internal namespaces (`tracing.attributes`, `tracing.settings`, `tracing.quartz`) from outside the module.+- `:uses :any` on the `core` module does NOT bypass the target module's `:api` check — internal namespaces are still enforced.++### Cyclic Dependency Avoidance++`tracing/core.clj` is required by many modules across the codebase. It **must NOT** compile-time require `tracing.settings`, as this creates transitive cyclic load dependencies (e.g., `settings/core -> tracing/settings -> tracing/core -> events/impl -> events/core`).++Instead, `tracing/core.clj` uses `requiring-resolve` for settings access:++```clojure+;; CORRECT — lazy runtime resolution, no compile-time dependency+((requiring-resolve 'metabase.tracing.settings/tracing-enabled))++;; WRONG — creates cyclic load dependency+(require '[metabase.tracing.settings :as settings])+(settings/tracing-enabled)+```++External library namespaces (clj-otel API, SDK, exporters) are safe to require normally — they don't participate in Metabase namespace cycles.++**Important:** `requiring-resolve` must use **literal quoted symbols**. Kondo hooks validate that `required-namespaces` are all simple symbols, so dynamic construction fails:++```clojure+;; CORRECT — literal quoted symbol+(requiring-resolve 'metabase.tracing.settings/tracing-endpoint)++;; WRONG — kondo hook rejects this: "Assert failed: (every? simple-symbol? required-namespaces)"+(requiring-resolve (symbol "metabase.tracing.settings" "tracing-endpoint"))+```++## Quick Checklist++When adding tracing spans:++- [ ] Module has `tracing` in its `:uses` set in `.clj-kondo/config/modules/config.edn`+- [ ] Added `[metabase.tracing.core :as tracing]` to ns requires (alphabetically sorted)+- [ ] Span wraps a meaningful I/O boundary (not pure computation)+- [ ] Group matches the domain (check `src/metabase/tracing/core.clj` for registered groups; add a new one if none fit)+- [ ] Span name follows dot-notation convention (`"domain.subsystem.operation"`)+- [ ] Attributes use namespaced keywords (`:search/query-length`, `:db/id`)+- [ ] No sensitive data in attributes (use `best-effort-sanitize-sql` for HoneySQL, never raw SQL)+- [ ] No new tracing namespaces created (add to `tracing.core` instead)+- [ ] No `DO_NOT_ADD_NEW_FILES_HERE.txt` violations in the target directory+- [ ] Run `clj-kondo --lint <files>` to verify 0 errors, 0 warnings+- [ ] Add or update tests in the corresponding `test/` path (see Testing section below)+- [ ] Run tests: `clojure -X:dev:test :only <test-ns>`++## The `with-span` Macro++```clojure+(tracing/with-span group span-name attrs & body)+```++- **group** - A keyword selecting which trace group this span belongs to (e.g., `:tasks`, `:sync`)+- **span-name** - A string identifying the span in traces (e.g., `"search.execute"`)+- **attrs** - A map of span attributes (e.g., `{:db/id 42}`)+- **body** - The code to execute inside the span++**When disabled:** zero overhead -- single atom deref + boolean check, body runs directly.+**When enabled:** creates OTel span AND injects `trace_id`/`span_id` into Log4j2 MDC for log-to-trace correlation.++## Trace Groups++Groups are registered in `src/metabase/tracing/core.clj`. Check that file for the current list. The general rule: **match the group to the domain, not the call site.** If code runs inside a Quartz job but is logically search work, use `:search`, not `:tasks`.++To add a new group:++```clojure+;; In src/metabase/tracing/core.clj+(register-group! :my-domain "Description of what this covers")+```++Users enable groups via `MB_TRACING_GROUPS=tasks,search,sync` (comma-separated, or `"all"`).++## Naming Conventions++### Span Names++Use dot-separated hierarchical names: `"domain.subsystem.operation"`. The domain prefix should match the group name:++```+search.execute -- `:search` group+sync.fingerprint.table -- `:sync` group+task.session-cleanup.delete -- `:tasks` group+db-app.collection-items -- `:db-app` group+```++### Attributes++Use namespaced keywords. The namespace groups related attributes:++```clojure+:db/id -- Database ID (integer)+:db/engine -- Database engine name (string)+:db/statement -- Sanitized SQL (string, via best-effort-sanitize-sql)+:search/engine -- Search engine name (string)+:search/query-length -- Query string length (integer)+:sync/table -- Table name (string)+:sync/step -- Sync step name (string)+:task/name -- Task name (string)+:http/method -- HTTP method (string)+:http/url -- Request URL (string)+```++Invent new namespaced attributes as needed (e.g., `:pulse/id`, `:transform/count`). Keep values as primitives (strings, numbers, booleans) -- no maps or collections.++## Step-by-Step: Adding a Span++### 1. Check module boundaries++Look up the module for your namespace in `.clj-kondo/config/modules/config.edn`. If `tracing` is not in the module's `:uses` set, add it (keep alphabetically sorted):++```edn+my-module+{:team "MyTeam"+ :uses #{analytics config tracing util}}+```++### 2. Add the require++```clojure+(ns metabase.my-module.thing+ (:require+ [metabase.tracing.core :as tracing]+ [metabase.util :as u]))+```++`best-effort-sanitize-sql` is available from `tracing.core` — no additional require needed.++### 3. Identify the I/O boundary++Only wrap code at **meaningful I/O boundaries**:++**DO trace:**+- External API calls (embedding APIs, metabot, webhooks)+- Database queries (both app DB and user DB)+- Network requests (HTTP calls to external services)+- Heavy batch processing (batch indexing, batch embedding)+- Top-level orchestration functions that coordinate multiple sub-operations++**DO NOT trace:**+- Pure computation (sorting, filtering, mapping)+- Simple single-row lookups (`t2/select-one :model/Setting :key k`)+- Every function in a call chain (only boundaries matter)+- Trivial operations (string formatting, hash calculations)++### 4. Wrap with `with-span`++```clojure+;; Simple span (no attributes needed)+(tracing/with-span :search "search.init-index" {}+ (do-expensive-thing))++;; Span with static attributes+(tracing/with-span :sync "sync.fingerprint.table"+ {:db/id (:db_id table)+ :sync/table (:name table)}+ (fingerprint-fields! table fields))++;; Span with computed attributes+(tracing/with-span :search "search.execute"+ {:search/engine (name (:search-engine ctx))+ :search/query-length (count (:search-string ctx))}+ (search.engine/results ctx))++;; Span with sanitized SQL (for dynamic HoneySQL queries)+(let [hsql {:delete-from [(t2/table-name :model/Session)]+ :where [:< :created_at oldest-allowed]}]+ (tracing/with-span :tasks "task.session-cleanup.delete"+ {:db/statement (tracing/best-effort-sanitize-sql hsql)}+ (t2/query-one hsql)))++;; Sub-spans breaking a function into I/O phases+(let [embedding (tracing/with-span :search "search.semantic.embedding"+ {:search.semantic/provider (:provider model)}+ (get-embedding model search-string))+ results (tracing/with-span :search "search.semantic.db-query" {}+ (into [] xform reducible))]+ (process results))++;; Per-item iteration spans+(doseq [e (search.engine/active-engines)]+ (tracing/with-span :search "search.ingestion.update" {:search/engine (name e)}+ (search.engine/update! e batch)))+```++### 5. Add tests++Create or update tests in the corresponding `test/` path. Follow the patterns in existing tracing tests:++- **Reference tests:** `test/metabase/tracing/quartz_test.clj`, `test/metabase/server/middleware/trace_test.clj`+- Use `tracing/init-enabled-groups!` / `tracing/shutdown-groups!` with `try`/`finally` to manage group lifecycle+- Test both enabled and disabled paths (verify zero overhead when group is off)+- Use `reify` mocks for Java interfaces (Connection, PreparedStatement, JobListener, etc.)+- Add `(set! *warn-on-reflection* true)` and type-hint proxy/reify calls to avoid reflection warnings++```clojure+(deftest my-span-enabled-test+ (testing "when group is enabled, span is created"+ (try+ (tracing/init-enabled-groups! "my-group" "INFO")+ ;; ... test that span behavior occurs ...+ (finally+ (tracing/shutdown-groups!)))))++(deftest my-span-disabled-test+ (testing "when group is disabled, code runs without tracing"+ (tracing/shutdown-groups!)+ ;; ... test that code still works, no wrapping applied ...+ ))+```++### 6. Lint and run tests++```bash+# Lint modified source and test files — expect 0 errors, 0 warnings+clj-kondo --lint path/to/modified/file.clj path/to/test/file.clj++# Run tests (requires Java 21+)+ clojure -X:dev:test :only my-ns.test-ns+```++Expect: all tests pass, 0 failures, 0 errors, no reflection warnings from your files.++## Sanitizing SQL for Attributes++When including SQL in span attributes, **always** use `tracing/best-effort-sanitize-sql`. This converts HoneySQL maps to parameterized SQL strings where values become `?` placeholders -- no data leaks.++```clojure+(let [hsql {:delete-from [:core_session]+ :where [:< :created_at some-timestamp]}]+ (tracing/with-span :tasks "task.cleanup.delete"+ {:db/statement (tracing/best-effort-sanitize-sql hsql)}+ (t2/query-one hsql)))+;; Trace attribute: db/statement = "DELETE FROM core_session WHERE created_at < ?"+```++**Rules:**+- Never put raw SQL strings or user-provided values in attributes+- Use `best-effort-sanitize-sql` only for app DB (HoneySQL) queries+- For external/user DB queries, trace only timing and counts, not SQL content++## Defjob and Root Spans++The `defjob` macro in `metabase.task.impl` automatically wraps every Quartz job with a `:tasks` root span:++```clojure+(task/defjob ^{DisallowConcurrentExecution true} SessionCleanup [_]+ (cleanup-sessions!))+;; Automatically creates span: "task.SessionCleanup" {:task/name "SessionCleanup"}+```++You do NOT need a root span inside `defjob` bodies. Add **child spans** for I/O inside the job.++For code on plain `Thread`s (not Quartz), add the root span manually:++```clojure+(defn init! []+ (tracing/with-span :search "search.task.init" {}+ (search/init-index!)))+```++## What NOT to Do++### Span Usage Mistakes++```clojure+;; WRONG - pure computation, no I/O+(tracing/with-span :search "search.format-results" {}+ (map format-result results))++;; WRONG - trivial single-row lookup+(tracing/with-span :db-app "db-app.get-setting" {}+ (t2/select-one :model/Setting :key "my-setting"))++;; WRONG - raw SQL in attributes (data leak)+(tracing/with-span :tasks "task.cleanup" {:db/statement raw-sql-string}+ (execute! raw-sql-string))++;; WRONG - wrong group (search work should use :search, not :tasks)+(tracing/with-span :tasks "search.execute" {} ...)++;; WRONG - redundant nesting (do-search already has a span)+(tracing/with-span :search "search.process" {}+ (let [results (do-search ctx)]+ (tracing/with-span :search "search.format" {}+ (format-results results))))+```++### Architecture Mistakes++```clojure+;; WRONG - creating a new tracing namespace+(ns metabase.tracing.my-feature ...)++;; WRONG - requiring internal tracing namespaces from outside the module+(ns metabase.my-module.thing+ (:require [metabase.tracing.attributes :as trace-attrs] ;; internal!+ [metabase.tracing.settings :as tracing.settings] ;; internal!+ [metabase.tracing.quartz :as tracing.quartz])) ;; internal!++;; WRONG - adding compile-time requires to tracing/core.clj for settings or SDK+;; This creates cyclic load dependencies because tracing/core is widely required+(ns metabase.tracing.core+ (:require [metabase.tracing.settings :as settings])) ;; causes cycle!++;; WRONG - dynamic symbol construction with requiring-resolve (kondo rejects it)+(requiring-resolve (symbol "metabase.tracing.settings" "tracing-enabled"))+```++## Configuration++All settings are env-var-only (defined in `src/metabase/tracing/settings.clj`):++```bash+# Core+MB_TRACING_ENABLED=true # Enable tracing (default: false)+MB_TRACING_ENDPOINT=host:4317 # OTLP collector endpoint (default: http://localhost:4317)+MB_TRACING_GROUPS=tasks,search,sync # Comma-separated groups or "all" (default: all)+MB_TRACING_SERVICE_NAME=metabase # Service name in traces (default: hostname)+MB_TRACING_LOG_LEVEL=DEBUG # Log threshold for traced threads: TRACE/DEBUG/INFO (default: INFO)++# Batch span processor tuning+MB_TRACING_MAX_QUEUE_SIZE=2048 # Max spans queued for export; drops when full (default: 2048)+MB_TRACING_EXPORT_TIMEOUT_MS=10000 # Max wait for batch export to complete (default: 10000)+MB_TRACING_SCHEDULE_DELAY_MS=5000 # Delay between consecutive batch exports (default: 5000)+```diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.ednindex 3ef710719b6..a7c939bd67d 100644--- a/.clj-kondo/config/modules/config.edn+++ b/.clj-kondo/config/modules/config.edn@@ -215,6 +215,7 @@settingstasktask-history+ tracingutil}}auth-identity@@ -397,6 +398,7 @@requestrevisionssettings+ tracingtransformsuploadutil}}@@ -589,6 +591,7 @@settingssql-toolssystem+ tracingutilwarehouse-schemawarehouses}}@@ -667,7 +670,8 @@:api #{metabase.events.coremetabase.events.init}:uses #{models- util}}+ util}+ :friends #{tracing}}formatter{:team "UX West"@@ -891,6 +895,7 @@systemtasktask-history+ tracingutil}}models@@ -964,6 +969,7 @@systemtasktask-history+ tracingusersutil}}@@ -1007,6 +1013,7 @@premium-featuresremote-syncsettings+ tracingutilenterprise/advanced-permissions}}@@ -1052,6 +1059,7 @@settingsstartuptask+ tracingutil}}product-feedback@@ -1119,6 +1127,7 @@requesttasktask-history+ tracingutilenterprise/advanced-permissionsenterprise/sandbox}}@@ -1262,6 +1271,7 @@serversettingssystem+ tracingusersutil}}@@ -1343,6 +1353,7 @@settingsstartuptask+ tracingtransformsutil}}@@ -1389,6 +1400,7 @@sessionsettingssystem+ tracingusersutilenterprise/sso}}@@ -1410,6 +1422,7 @@ssosystemtask+ tracingutil}}settings@@ -1532,6 +1545,7 @@settingstasktask-history+ tracingutilwarehouse-schemawarehouses}}@@ -1548,6 +1562,7 @@metabase.task.core}:uses #{app-dbclassloader+ tracingutil}}task-history@@ -1564,6 +1579,7 @@query-processorrequesttask+ tracingutil}}tenants@@ -1611,6 +1627,15 @@modelsutil}}+ tracing+ {:team "DevEx"+ :api #{metabase.tracing.core+ metabase.tracing.init}+ :uses #{events+ settings+ task+ util}}+transforms{:team "Querying":api #{metabase.transforms.core@@ -1633,6 +1658,7 @@settingstasktask-history+ tracingtransforms-baseutil}:friends #{models@@ -2434,6 +2460,7 @@searchsettingstask+ tracingutilenterprise/llmenterprise/metabot-v3}}diff --git a/deps.edn b/deps.ednindex 9067a65462e..c79389008c0 100644--- a/deps.edn+++ b/deps.edn@@ -47,6 +47,8 @@com.github.seancorfield/honeysql {:mvn/version "2.7.1350"} ; Honey SQL 2. SQL generation from Clojure data mapscom.github.seancorfield/next.jdbc {:mvn/version "1.3.1070"} ; Talk to JDBC DBscom.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.8"} ; Telemetry library+ com.github.steffan-westcott/clj-otel-exporter-otlp {:mvn/version "0.2.8"} ; OTLP span exporter (gRPC and HTTP)+ com.github.steffan-westcott/clj-otel-sdk {:mvn/version "0.2.8"} ; OpenTelemetry SDK for programmatic configurationcom.github.vertical-blank/sql-formatter {:mvn/version "2.0.5"} ; Java SQL formatting library https://github.com/vertical-blank/sql-formattercom.google.guava/guava {:mvn/version "33.5.0-jre"} ; dep for BigQuery, Spark, and GA. Require here rather than letting different dep versions stomp on each other — see comments on #9697com.h2database/h2 ^:antq/exclude ; https://metaboat.slack.com/archives/CKZEMT1MJ/p1727191010259979diff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/core.cljindex 0e989215cae..d9683ca4fbe 100644--- a/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/core.clj@@ -12,6 +12,7 @@[metabase.analytics.core :as analytics][metabase.premium-features.core :refer [defenterprise]][metabase.search.engine :as search.engine]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][toucan2.realize :as t2.realize]))@@ -38,53 +39,54 @@when semantic search returns too few results and some results were filtered out (e.g. due to permission checks).":feature :semantic-search[search-ctx]- (try- (let [{:keys [results raw-count]}- (semantic.pgvector-api/query (semantic.env/get-pgvector-datasource!)- (semantic.env/get-index-metadata)- search-ctx)- final-count (count results)- threshold (semantic.settings/semantic-search-min-results-threshold)]- (if (or (>= final-count threshold)- (and (zero? raw-count)- ;; :search-string is nil when using search to populate the list of tables for a given database in- ;; the native query editor. Semantic search doesn't support this, so fallback in this case.- (not (str/blank? (:search-string search-ctx)))))- results- ;; Fallback: semantic search found results but some were filtered out (e.g. due to permission checks), so try to- ;; supplement with appdb search.- (let [fallback (fallback-engine)]- (log/debugf "Semantic search returned %d final results (< %d) from %d raw results, supplementing with %s search"- final-count threshold raw-count fallback)- (analytics/inc! :metabase-search/semantic-fallback-triggered {:fallback-engine fallback})- (analytics/observe! :metabase-search/semantic-results-before-fallback final-count)+ (tracing/with-span :search "search.semantic.execute" {:search/query-length (count (:search-string search-ctx))}+ (try+ (let [{:keys [results raw-count]}+ (semantic.pgvector-api/query (semantic.env/get-pgvector-datasource!)+ (semantic.env/get-index-metadata)+ search-ctx)+ final-count (count results)+ threshold (semantic.settings/semantic-search-min-results-threshold)]+ (if (or (>= final-count threshold)+ (and (zero? raw-count)+ ;; :search-string is nil when using search to populate the list of tables for a given database in+ ;; the native query editor. Semantic search doesn't support this, so fallback in this case.+ (not (str/blank? (:search-string search-ctx)))))+ results+ ;; Fallback: semantic search found results but some were filtered out (e.g. due to permission checks), so try to+ ;; supplement with appdb search.+ (let [fallback (fallback-engine)]+ (log/debugf "Semantic search returned %d final results (< %d) from %d raw results, supplementing with %s search"+ final-count threshold raw-count fallback)+ (analytics/inc! :metabase-search/semantic-fallback-triggered {:fallback-engine fallback})+ (analytics/observe! :metabase-search/semantic-results-before-fallback final-count)- (when (some-> (:offset-int search-ctx) pos?)- (log/warn "Using an offset with semantic search will produce strange results, e.g. missing expected results, or duplicating them across pages"))+ (when (some-> (:offset-int search-ctx) pos?)+ (log/warn "Using an offset with semantic search will produce strange results, e.g. missing expected results, or duplicating them across pages"))- (let [total-limit (semantic.settings/semantic-search-results-limit)- fallback-results (try- (cond->> (search.engine/results (assoc search-ctx :search-engine fallback))- ;; The in-place engine returns a reducible (but not seqable) result that needs to- ;; be realized before we concat and dedup with the semantic engine results.- (= :search.engine/in-place fallback)- (into [] (comp (map t2.realize/realize)- (take total-limit))))- (catch Throwable t- (log/warn t "Semantic search fallback errored, ignoring")- []))- fallback-results (take total-limit fallback-results)- _ (analytics/observe! :metabase-search/semantic-fallback-results-usage (count fallback-results))- combined-results (concat results fallback-results)- deduped-results (m/distinct-by (juxt :model :id) combined-results)]- (take total-limit deduped-results)))))- (catch Exception e- (log/error e "Error executing semantic search, falling back to appdb")- (let [fallback (fallback-engine)]- (analytics/inc! :metabase-search/semantic-error-fallback {:fallback-engine fallback})- (if fallback- (search.engine/results (assoc search-ctx :search-engine fallback))- (throw (ex-info "Error executing semantic search" {:type :semantic-search-error} e)))))))+ (let [total-limit (semantic.settings/semantic-search-results-limit)+ fallback-results (try+ (cond->> (search.engine/results (assoc search-ctx :search-engine fallback))+ ;; The in-place engine returns a reducible (but not seqable) result that needs to+ ;; be realized before we concat and dedup with the semantic engine results.+ (= :search.engine/in-place fallback)+ (into [] (comp (map t2.realize/realize)+ (take total-limit))))+ (catch Throwable t+ (log/warn t "Semantic search fallback errored, ignoring")+ []))+ fallback-results (take total-limit fallback-results)+ _ (analytics/observe! :metabase-search/semantic-fallback-results-usage (count fallback-results))+ combined-results (concat results fallback-results)+ deduped-results (m/distinct-by (juxt :model :id) combined-results)]+ (take total-limit deduped-results)))))+ (catch Exception e+ (log/error e "Error executing semantic search, falling back to appdb")+ (let [fallback (fallback-engine)]+ (analytics/inc! :metabase-search/semantic-error-fallback {:fallback-engine fallback})+ (if fallback+ (search.engine/results (assoc search-ctx :search-engine fallback))+ (throw (ex-info "Error executing semantic search" {:type :semantic-search-error} e))))))))(defenterprise update-index!"Enterprise implementation of semantic index updating."diff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.cljindex db03fb74511..8a4eac2acba 100644--- a/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/embedding.clj@@ -6,6 +6,7 @@[metabase-enterprise.semantic-search.models.token-tracking :as semantic.models.token-tracking][metabase-enterprise.semantic-search.settings :as semantic-settings][metabase.analytics.core :as analytics]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.json :as json][metabase.util.log :as log])@@ -279,26 +280,30 @@[embedding-model texts process-fn & {:as opts}](when (seq texts)(let [{:keys [model-name provider vector-dimensions]} embedding-model]- (u/profile (str "Generating embeddings " {:model model-name- :dimensions vector-dimensions- :texts (calc-token-metrics texts)})- (if (= "openai" provider)- (let [max-tokens-per-batch (semantic-settings/openai-max-tokens-per-batch)- batches (create-batches max-tokens-per-batch count-tokens texts)-- process-batch- (fn [batch-idx batch-texts]- (let [embeddings (u/profile (format "Embedding batch %d/%d %s"- (inc batch-idx) (count batches) (str (calc-token-metrics batch-texts)))- (openai-get-embeddings-batch embedding-model batch-texts opts))- text-embedding-map (zipmap batch-texts embeddings)]- (process-fn text-embedding-map)))]-- (transduce (map-indexed process-batch) (partial merge-with +) batches))-- (let [embeddings (get-embeddings-batch embedding-model texts opts)- text-embedding-map (zipmap texts embeddings)]- (process-fn text-embedding-map)))))))+ (tracing/with-span :search "search.semantic.embeddings-batch"+ {:search.semantic/provider provider+ :search.semantic/model-name model-name+ :search.semantic/text-count (count texts)}+ (u/profile (str "Generating embeddings " {:model model-name+ :dimensions vector-dimensions+ :texts (calc-token-metrics texts)})+ (if (= "openai" provider)+ (let [max-tokens-per-batch (semantic-settings/openai-max-tokens-per-batch)+ batches (create-batches max-tokens-per-batch count-tokens texts)++ process-batch+ (fn [batch-idx batch-texts]+ (let [embeddings (u/profile (format "Embedding batch %d/%d %s"+ (inc batch-idx) (count batches) (str (calc-token-metrics batch-texts)))+ (openai-get-embeddings-batch embedding-model batch-texts opts))+ text-embedding-map (zipmap batch-texts embeddings)]+ (process-fn text-embedding-map)))]++ (transduce (map-indexed process-batch) (partial merge-with +) batches))++ (let [embeddings (get-embeddings-batch embedding-model texts opts)+ text-embedding-map (zipmap texts embeddings)]+ (process-fn text-embedding-map))))))))(comment;; Configuration:diff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/index.cljindex 94cf79185fe..c01613f4a41 100644--- a/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/index.clj@@ -16,6 +16,7 @@[metabase.models.interface :as mi][metabase.search.config :as search.config][metabase.search.core :as search]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.json :as json][metabase.util.log :as log]@@ -389,20 +390,21 @@model + model_id already exists, it will be replaced. Parallelizes batch insertionusing a shared thread pool with a configurable thread count (default: 2)."([connectable index documents-reducible & {:keys [serial?] :or {serial? false}}]- (not-empty- (let [pool @index-update-executor- results (transduce- (comp (partition-all *batch-size*)- (map (if serial?- #(upsert-index-batch! connectable index % {:type :index})- #(upsert-index-pooled! pool connectable index % {:type :index}))))- conj- documents-reducible)]- (reduce (fn [update-counts result]- (let [value (if (future? result) @result result)]- (merge-with + update-counts (when value value))))- {}- results)))))+ (tracing/with-span :search "search.semantic.index-upsert" {}+ (not-empty+ (let [pool @index-update-executor+ results (transduce+ (comp (partition-all *batch-size*)+ (map (if serial?+ #(upsert-index-batch! connectable index % {:type :index})+ #(upsert-index-pooled! pool connectable index % {:type :index}))))+ conj+ documents-reducible)]+ (reduce (fn [update-counts result]+ (let [value (if (future? result) @result result)]+ (merge-with + update-counts (when value value))))+ {}+ results))))))(defn- drop-index-table-sql[{:keys [table-name]}]@@ -803,7 +805,10 @@{:results [] :raw-count 0}(let [timer (u/start-timer)- embedding (embedding/get-embedding embedding-model search-string {:type :query})+ embedding (tracing/with-span :search "search.semantic.embedding"+ {:search.semantic/provider (:provider embedding-model)+ :search.semantic/model-name (:model-name embedding-model)}+ (embedding/get-embedding embedding-model search-string {:type :query}))embedding-time-ms (u/since-ms timer)db-timer (u/start-timer)@@ -813,14 +818,18 @@xform (comp (map decode-metadata)(map (partial legacy-input-with-score weights (keys scorers))))reducible (reducible-search-query db query)- raw-results (into [] xform reducible)+ raw-results (tracing/with-span :search "search.semantic.db-query"+ {:search/query-length (count search-string)}+ (into [] xform reducible))db-query-time-ms (u/since-ms db-timer)filter-timer (u/start-timer)- filtered-results (->> raw-results- filter-read-permitted- (apply-collection-id-filter search-context)- (mapv search/collapse-id))+ filtered-results (tracing/with-span :search "search.semantic.permission-filter"+ {:search.semantic/raw-count (count raw-results)}+ (->> raw-results+ filter-read-permitted+ (apply-collection-id-filter search-context)+ (mapv search/collapse-id)))filter-time-ms (u/since-ms filter-timer)appdb-scorers (scoring/appdb-scorers search-context)diff --git a/frontend/src/metabase-types/api/mocks/settings.ts b/frontend/src/metabase-types/api/mocks/settings.tsindex c3fbf4fbb73..d370f40d252 100644--- a/frontend/src/metabase-types/api/mocks/settings.ts+++ b/frontend/src/metabase-types/api/mocks/settings.ts@@ -385,6 +385,7 @@ export const createMockSettings = ("subscription-allowed-domains": null,"token-features": createMockTokenFeatures(),"token-status": null,+ "tracing-enabled": false,"transforms-enabled": false,version: createMockVersion(),"version-info": createMockVersionInfo(),diff --git a/frontend/src/metabase-types/api/settings.ts b/frontend/src/metabase-types/api/settings.tsindex 848989f6e20..acc25fd40ef 100644--- a/frontend/src/metabase-types/api/settings.ts+++ b/frontend/src/metabase-types/api/settings.ts@@ -599,6 +599,7 @@ interface PublicSettings {"snowplow-url": string;"start-of-week": DayOfWeekId;"token-features": TokenFeatures;+ "tracing-enabled": boolean;"transforms-enabled": boolean;version: Version;"version-info-last-checked": string | null;diff --git a/frontend/src/metabase/app.js b/frontend/src/metabase/app.jsindex 01db6526077..0013ed7d05e 100644--- a/frontend/src/metabase/app.js+++ b/frontend/src/metabase/app.js@@ -39,6 +39,7 @@ import { createTracker } from "metabase/lib/analytics";import api from "metabase/lib/api";import { initializeEmbedding } from "metabase/lib/embed";import { captureConsoleErrors } from "metabase/lib/errors";+import { initTracing, rotateTraceId } from "metabase/lib/otel";import { MetabaseReduxProvider } from "metabase/lib/redux/custom-context";import MetabaseSettings from "metabase/lib/settings";import { PLUGIN_APP_INIT_FUNCTIONS, PLUGIN_METABOT } from "metabase/plugins";@@ -72,6 +73,15 @@ function _init(reducers, getRoutes, callback) {createTracker(store);+ // Initialize distributed tracing if enabled via MB_TRACING_ENABLED.+ // Uses bootstrap data so it's available before the first API call.+ if (window.MetabaseBootstrap?.["tracing-enabled"]) {+ initTracing();+ // Rotate trace ID on route changes so all API calls within+ // a single page view share one trace.+ syncedHistory.listen(() => rotateTraceId());+ }+initializeEmbedding(store);const root = createRoot(document.getElementById("root"));diff --git a/frontend/src/metabase/lib/api.js b/frontend/src/metabase/lib/api.jsindex acdb86284ab..3f6ba628ec5 100644--- a/frontend/src/metabase/lib/api.js+++ b/frontend/src/metabase/lib/api.js@@ -6,6 +6,7 @@ import { isEmbeddingSdk } from "metabase/embedding-sdk/config";import { isTest } from "metabase/env";import { isWithinIframe } from "metabase/lib/dom";import { IFRAMED_IN_SELF } from "metabase/lib/iframe";+import { getTraceparentHeader } from "metabase/lib/otel";import { delay } from "metabase/lib/promise";import { PLUGIN_API, PLUGIN_EMBEDDING_SDK } from "metabase/plugins";@@ -103,6 +104,11 @@ export class Api extends EventEmitter {DEFAULT_OPTIONS.headers["X-Metabase-Locale"];}+ const traceparent = getTraceparentHeader();+ if (traceparent) {+ headers["traceparent"] = traceparent;+ }+return headers;}diff --git a/frontend/src/metabase/lib/otel.ts b/frontend/src/metabase/lib/otel.tsnew file mode 100644index 00000000000..0849e4b42de--- /dev/null+++ b/frontend/src/metabase/lib/otel.ts@@ -0,0 +1,106 @@+/**+ * Lightweight W3C Trace Context propagation for distributed tracing.+ *+ * Generates `traceparent` headers (https://www.w3.org/TR/trace-context/)+ * without requiring the full OpenTelemetry browser SDK.+ *+ * How it works:+ * - A new trace ID is generated for each "activity burst" — a group of+ * API requests that happen close together (e.g., a dashboard load,+ * a filter application, a query run).+ * - After a short idle period (no API calls), the next request starts+ * a new trace. This naturally groups related requests together.+ * - Route changes always start a new trace immediately.+ * - Each individual API request gets a unique span ID (parent-id).+ * - The backend extracts the `traceparent` header and creates child spans.+ *+ * traceparent format: {version}-{trace-id}-{parent-id}-{trace-flags}+ * version: "00"+ * trace-id: 32 hex chars (128-bit)+ * parent-id: 16 hex chars (64-bit)+ * trace-flags: "01" (sampled)+ */++let currentTraceId: string | null = null;+let enabled = false;+let idleTimer: ReturnType<typeof setTimeout> | null = null;+let traceExpired = false;++/** Idle time in ms before the current trace expires and the next request starts a new one. */+const IDLE_TIMEOUT_MS = 5000;++function generateHexId(byteLength: number): string {+ const buf = new Uint8Array(byteLength);+ crypto.getRandomValues(buf);+ return Array.from(buf, (b) => b.toString(16).padStart(2, "0")).join("");+}++function newTraceId(): void {+ currentTraceId = generateHexId(16); // 128-bit = 32 hex chars+ traceExpired = false;+}++function resetIdleTimer(): void {+ if (idleTimer !== null) {+ clearTimeout(idleTimer);+ }+ idleTimer = setTimeout(() => {+ traceExpired = true;+ }, IDLE_TIMEOUT_MS);+}++/**+ * Enable tracing and generate the first trace ID.+ * Called once at app startup when `tracing-enabled` is true.+ */+export function initTracing(): void {+ enabled = true;+ newTraceId();+}++/**+ * Force a new trace ID immediately. Called on route changes+ * so navigating between pages always starts a fresh trace.+ */+export function rotateTraceId(): void {+ if (enabled) {+ newTraceId();+ }+}++/**+ * Get a W3C `traceparent` header value for the current request.+ * Returns null when tracing is disabled.+ *+ * If the idle timeout has elapsed since the last API call, a new+ * trace ID is generated automatically — so each "burst" of related+ * requests (dashboard load, filter click, etc.) gets its own trace.+ *+ * Each call generates a unique span ID (parent-id) so every+ * API request is a distinct span within the shared trace.+ */+/** Reset all state. Exported only for tests. */+export function _resetForTesting(): void {+ currentTraceId = null;+ enabled = false;+ traceExpired = false;+ if (idleTimer !== null) {+ clearTimeout(idleTimer);+ idleTimer = null;+ }+}++export function getTraceparentHeader(): string | null {+ if (!enabled || !currentTraceId) {+ return null;+ }++ if (traceExpired) {+ newTraceId();+ }++ resetIdleTimer();++ const spanId = generateHexId(8); // 64-bit = 16 hex chars+ return `00-${currentTraceId}-${spanId}-01`;+}diff --git a/frontend/src/metabase/lib/otel.unit.spec.ts b/frontend/src/metabase/lib/otel.unit.spec.tsnew file mode 100644index 00000000000..c7f785f87b0--- /dev/null+++ b/frontend/src/metabase/lib/otel.unit.spec.ts@@ -0,0 +1,133 @@+import {+ _resetForTesting,+ getTraceparentHeader,+ initTracing,+ rotateTraceId,+} from "metabase/lib/otel";++// W3C traceparent: 00-{32 hex}-{16 hex}-01+const TRACEPARENT_REGEX = /^00-[0-9a-f]{32}-[0-9a-f]{16}-01$/;++describe("otel", () => {+ beforeEach(() => {+ _resetForTesting();+ jest.useFakeTimers();+ });++ afterEach(() => {+ jest.useRealTimers();+ });++ describe("getTraceparentHeader", () => {+ it("returns null before initialization", () => {+ expect(getTraceparentHeader()).toBeNull();+ });++ it("returns a valid W3C traceparent after init", () => {+ initTracing();+ const header = getTraceparentHeader();+ expect(header).toMatch(TRACEPARENT_REGEX);+ });++ it("returns the same trace ID for consecutive calls within idle timeout", () => {+ initTracing();+ const header1 = getTraceparentHeader()!;+ const header2 = getTraceparentHeader()!;++ const traceId1 = header1.split("-")[1];+ const traceId2 = header2.split("-")[1];+ expect(traceId1).toBe(traceId2);+ });++ it("generates unique span IDs for each call", () => {+ initTracing();+ const header1 = getTraceparentHeader()!;+ const header2 = getTraceparentHeader()!;++ const spanId1 = header1.split("-")[2];+ const spanId2 = header2.split("-")[2];+ expect(spanId1).not.toBe(spanId2);+ });+ });++ describe("rotateTraceId", () => {+ it("changes the trace ID immediately", () => {+ initTracing();+ const before = getTraceparentHeader()!.split("-")[1];+ rotateTraceId();+ const after = getTraceparentHeader()!.split("-")[1];++ expect(before).not.toBe(after);+ });++ it("is a no-op when tracing is not initialized", () => {+ rotateTraceId();+ expect(getTraceparentHeader()).toBeNull();+ });+ });++ describe("idle timeout", () => {+ it("rotates trace ID after idle period expires", () => {+ initTracing();+ const first = getTraceparentHeader()!.split("-")[1];++ // Advance past the 5s idle timeout+ jest.advanceTimersByTime(6000);++ const second = getTraceparentHeader()!.split("-")[1];+ expect(first).not.toBe(second);+ });++ it("keeps the same trace ID if requests happen within idle window", () => {+ initTracing();+ const first = getTraceparentHeader()!.split("-")[1];++ // Make a request at 3s (resets the timer)+ jest.advanceTimersByTime(3000);+ const second = getTraceparentHeader()!.split("-")[1];+ expect(first).toBe(second);++ // Make another at 3s later (6s total, but only 3s since last call)+ jest.advanceTimersByTime(3000);+ const third = getTraceparentHeader()!.split("-")[1];+ expect(first).toBe(third);+ });++ it("rotates after idle even without route change", () => {+ initTracing();+ const first = getTraceparentHeader()!.split("-")[1];++ // Simulate user inactivity then a new action+ jest.advanceTimersByTime(6000);+ const second = getTraceparentHeader()!.split("-")[1];++ jest.advanceTimersByTime(6000);+ const third = getTraceparentHeader()!.split("-")[1];++ expect(first).not.toBe(second);+ expect(second).not.toBe(third);+ });+ });++ describe("traceparent format", () => {+ it("has version 00", () => {+ initTracing();+ const header = getTraceparentHeader()!;+ expect(header.startsWith("00-")).toBe(true);+ });++ it("has trace-flags 01 (sampled)", () => {+ initTracing();+ const header = getTraceparentHeader()!;+ expect(header.endsWith("-01")).toBe(true);+ });++ it("has a 32-char trace ID and 16-char span ID", () => {+ initTracing();+ const parts = getTraceparentHeader()!.split("-");+ expect(parts).toHaveLength(4);+ expect(parts[1]).toHaveLength(32);+ expect(parts[2]).toHaveLength(16);+ });+ });+});diff --git a/mage/src/mage/modules.clj b/mage/src/mage/modules.cljindex 3af63fdab1c..9c22f5cd006 100644--- a/mage/src/mage/modules.clj+++ b/mage/src/mage/modules.clj@@ -132,6 +132,7 @@tasktask-historytimeline+ tracingtypesusersutildiff --git a/src/metabase/audit_app/task/truncate_audit_tables.clj b/src/metabase/audit_app/task/truncate_audit_tables.cljindex 88df8678282..b24607a193e 100644--- a/src/metabase/audit_app/task/truncate_audit_tables.clj+++ b/src/metabase/audit_app/task/truncate_audit_tables.clj@@ -11,6 +11,7 @@[metabase.premium-features.core :refer [defenterprise]][metabase.task-history.core :as task-history][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.log :as log][toucan2.core :as t2]))@@ -44,18 +45,21 @@[model time-column](when-not (infinite? (audit-app.settings/audit-max-retention-days))(let [table-name (name (t2/table-name model))]- (try- (log/infof "Cleaning up %s table" table-name)- (loop [total-rows-deleted 0]- (let [batch-rows-deleted (truncate-table-batched! table-name time-column)]- ;; Only try to delete another batch if the last batch was full- (if (= batch-rows-deleted (audit-app.settings/audit-table-truncation-batch-size))- (recur (+ total-rows-deleted (long batch-rows-deleted)))- (if (not= total-rows-deleted 0)- (log/infof "%s cleanup successful, %d rows were deleted" table-name total-rows-deleted)- (log/infof "%s cleanup successful, no rows were deleted" table-name)))))- (catch Throwable e- (log/errorf e "%s cleanup failed" table-name))))))+ (tracing/with-span :tasks "task.audit-cleanup.table" {:audit/table table-name+ :audit/time-column (name time-column)+ :db/type (name (mdb/db-type))}+ (try+ (log/infof "Cleaning up %s table" table-name)+ (loop [total-rows-deleted 0]+ (let [batch-rows-deleted (truncate-table-batched! table-name time-column)]+ ;; Only try to delete another batch if the last batch was full+ (if (= batch-rows-deleted (audit-app.settings/audit-table-truncation-batch-size))+ (recur (+ total-rows-deleted (long batch-rows-deleted)))+ (if (not= total-rows-deleted 0)+ (log/infof "%s cleanup successful, %d rows were deleted" table-name total-rows-deleted)+ (log/infof "%s cleanup successful, no rows were deleted" table-name)))))+ (catch Throwable e+ (log/errorf e "%s cleanup failed" table-name)))))))(defenterprise audit-models-to-truncate"List of models to truncate. OSS implementation only truncates `query_execution` table."diff --git a/src/metabase/collections_rest/api.clj b/src/metabase/collections_rest/api.cljindex f0ee92bc1bd..7b943b574da 100644--- a/src/metabase/collections_rest/api.clj+++ b/src/metabase/collections_rest/api.clj@@ -27,6 +27,7 @@[metabase.queries.core :as queries][metabase.request.core :as request][metabase.revisions.core :as revisions]+ [metabase.tracing.core :as tracing][metabase.transforms.feature-gating :as transforms.gating][metabase.transforms.util :as transforms.u][metabase.upload.core :as upload]@@ -1102,11 +1103,13 @@;; :total_count:limit (if (zero? limit) 1 limit):offset offset))- rows (mdb/query limit-query)+ rows (tracing/with-span :db-app "db-app.collection-items-query" {:collection/id (:id collection)}+ (mdb/query limit-query))res {:total (->> rows first :total_count):data (if (= limit 0)[]- (post-process-rows options collection rows))+ (tracing/with-span :db-app "db-app.collection-items-post-process" {:collection/id (:id collection)}+ (post-process-rows options collection rows))):models models}limit-res (assoc res:limit (request/limit)diff --git a/src/metabase/core/core.clj b/src/metabase/core/core.cljindex 14499530da1..3050b0e1ce4 100644--- a/src/metabase/core/core.clj+++ b/src/metabase/core/core.clj@@ -32,6 +32,7 @@[metabase.startup.core :as startup][metabase.system.core :as system][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.queue :as queue]@@ -106,6 +107,7 @@(queue/stop-listeners!)(task/stop-scheduler!)(server/stop-web-server!)+ (tracing/shutdown!)(analytics/shutdown!)(notification/shutdown!);; This timeout was chosen based on a 30s default termination grace period in Kubernetes.@@ -167,6 +169,8 @@(init-status/set-progress! 0.2);; Ensure the classloader is installed as soon as possible.(classloader/the-classloader)+ ;; Initialize OpenTelemetry tracing early (before plugins, no DB dependency)+ (tracing/init!);; load any plugins as needed(plugins/load-plugins!)(init-status/set-progress! 0.3)diff --git a/src/metabase/core/init.clj b/src/metabase/core/init.cljindex 3e546ff3f53..64cd03d38be 100644--- a/src/metabase/core/init.clj+++ b/src/metabase/core/init.clj@@ -65,6 +65,7 @@[metabase.task-history.init][metabase.testing-api.init][metabase.tiles.init]+ [metabase.tracing.init][metabase.transforms-base.init][metabase.transforms.init][metabase.types.init]diff --git a/src/metabase/driver/sql_jdbc/execute.clj b/src/metabase/driver/sql_jdbc/execute.cljindex d535e7fd2d9..c4233dd9ebb 100644--- a/src/metabase/driver/sql_jdbc/execute.clj+++ b/src/metabase/driver/sql_jdbc/execute.clj@@ -22,6 +22,7 @@[metabase.driver.sql-jdbc.sync.interface :as sql-jdbc.sync.interface][metabase.lib.schema.info :as lib.schema.info][metabase.premium-features.core :refer [defenterprise]]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.i18n :refer [tru]][metabase.util.log :as log]@@ -773,48 +774,49 @@{:added "0.35.0", :arglists '([driver query context respond])}[driver {{sql :query, params :params} :native, :as outer-query} _context respond]{:pre [(string? sql) (seq sql)]}- (let [database (driver-api/database (driver-api/metadata-provider))- sql (if (get-in (driver.conn/effective-details database) [:include-user-id-and-hash] true)- (->> (driver-api/query->remark driver outer-query)- (inject-remark driver sql))- sql)- max-rows (driver-api/determine-query-max-rows outer-query)]- (do-with-connection-with-options- driver- (driver-api/database (driver-api/metadata-provider))- {:session-timezone (driver-api/report-timezone-id-if-supported driver (driver-api/database (driver-api/metadata-provider)))- :download? (download? (-> outer-query :info :context))}- (fn [^Connection conn]- (with-open [stmt (statement-or-prepared-statement driver conn sql params (driver-api/canceled-chan))- ^ResultSet rs (try- (execute-statement-or-prepared-statement! driver stmt max-rows params sql)- (catch Throwable e- (throw (ex-info (tru "Error executing query: {0}" (ex-message e))- (cond-> {:driver driver- :sql (str/split-lines (driver/prettify-native-form driver sql))- :params params- :type driver-api/qp.error-type.invalid-query}- (driver/query-canceled? driver e)- (assoc :query/query-canceled? true))- e))))]- (let [rsmeta (.getMetaData rs)- results-metadata {:cols (column-metadata driver rsmeta)}]- (try (respond results-metadata (reducible-rows driver rs rsmeta (driver-api/canceled-chan)))+ (tracing/with-span :db-user "db-user.execute-query" {:db/engine (name driver)}+ (let [database (driver-api/database (driver-api/metadata-provider))+ sql (if (get-in (driver.conn/effective-details database) [:include-user-id-and-hash] true)+ (->> (driver-api/query->remark driver outer-query)+ (inject-remark driver sql))+ sql)+ max-rows (driver-api/determine-query-max-rows outer-query)]+ (do-with-connection-with-options+ driver+ (driver-api/database (driver-api/metadata-provider))+ {:session-timezone (driver-api/report-timezone-id-if-supported driver (driver-api/database (driver-api/metadata-provider)))+ :download? (download? (-> outer-query :info :context))}+ (fn [^Connection conn]+ (with-open [stmt (statement-or-prepared-statement driver conn sql params (driver-api/canceled-chan))+ ^ResultSet rs (try+ (execute-statement-or-prepared-statement! driver stmt max-rows params sql)+ (catch Throwable e+ (throw (ex-info (tru "Error executing query: {0}" (ex-message e))+ (cond-> {:driver driver+ :sql (str/split-lines (driver/prettify-native-form driver sql))+ :params params+ :type driver-api/qp.error-type.invalid-query}+ (driver/query-canceled? driver e)+ (assoc :query/query-canceled? true))+ e))))]+ (let [rsmeta (.getMetaData rs)+ results-metadata {:cols (column-metadata driver rsmeta)}]+ (try (respond results-metadata (reducible-rows driver rs rsmeta (driver-api/canceled-chan)));; Following cancels the statement on the dbms side.;; It avoids blocking `.close` call, in case we reduced the results subset eg. by means of;; [[metabase.query-processor.middleware.limit/limit-xform]] middleware, while statement is still;; in progress. This problem was encountered on Redshift. For details see the issue #39018.;; It also handles situation where query is canceled through [[driver-api/canceled-chan]] (#41448).- (finally+ (finally;; TODO: Following `when` is in place just to find out if vertica is flaking because of cancelations.;; It should be removed afterwards!- (when-not (= :vertica driver)- (try (.cancel stmt)- (catch SQLFeatureNotSupportedException _- (log/warnf "Statemet's `.cancel` method is not supported by the `%s` driver."- (name driver)))- (catch Throwable _- (log/warn "Statement cancelation failed."))))))))))))+ (when-not (= :vertica driver)+ (try (.cancel stmt)+ (catch SQLFeatureNotSupportedException _+ (log/warnf "Statemet's `.cancel` method is not supported by the `%s` driver."+ (name driver)))+ (catch Throwable _+ (log/warn "Statement cancelation failed.")))))))))))))(defn reducible-query"Returns a reducible collection of rows as maps from `db` and a given SQL query. This is similar to [[jdbc/reducible-query]] but reuses thediff --git a/src/metabase/model_persistence/task/persist_refresh.clj b/src/metabase/model_persistence/task/persist_refresh.cljindex a66b3d5da18..8345b5d9553 100644--- a/src/metabase/model_persistence/task/persist_refresh.clj+++ b/src/metabase/model_persistence/task/persist_refresh.clj@@ -17,6 +17,7 @@[metabase.query-processor.timezone :as qp.timezone][metabase.task-history.core :as task-history][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.honey-sql-2 :as h2x][metabase.util.log :as log]@@ -56,36 +57,39 @@;; Since this could be long running, double check state just before refreshing(driver.conn/with-write-connection(when (contains? (persisted-info/refreshable-states) (t2/select-one-fn :state :model/PersistedInfo :id (:id persisted-info)))- (log/infof "Attempting to refresh persisted model %s." (:card_id persisted-info))- (let [card (t2/select-one :model/Card :id (:card_id persisted-info))- definition (persisted-info/metadata->definition (:result_metadata card)- (:table_name persisted-info))- _ (t2/update! :model/PersistedInfo (u/the-id persisted-info)- {:definition definition,- :query_hash (persisted-info/query-hash (:dataset_query card))- :active false,- :refresh_begin :%now,- :refresh_end nil,- :state "refreshing"- :state_change_at :%now})- {:keys [state error]} (try- (refresh! refresher database definition card)- (catch Exception e- (log/infof e "Error refreshing persisting model with card-id %s"- (:card_id persisted-info))- {:state :error :error (ex-message e)}))]- (t2/update! :model/PersistedInfo (u/the-id persisted-info)- {:active (= state :success),- :refresh_end :%now,- :state (if (= state :success) "persisted" "error")- :state_change_at :%now- :error (when (= state :error) error)})- (if (= :success state)- (update stats :success inc)- (-> stats- (update :error-details conj {:persisted-info-id (:id persisted-info)- :error error})- (update :error inc)))))))+ (tracing/with-span :tasks "task.persist.refresh-model" {:db/id (u/the-id database)+ :persist/card-id (:card_id persisted-info)+ :persist/table (:table_name persisted-info)}+ (log/infof "Attempting to refresh persisted model %s." (:card_id persisted-info))+ (let [card (t2/select-one :model/Card :id (:card_id persisted-info))+ definition (persisted-info/metadata->definition (:result_metadata card)+ (:table_name persisted-info))+ _ (t2/update! :model/PersistedInfo (u/the-id persisted-info)+ {:definition definition,+ :query_hash (persisted-info/query-hash (:dataset_query card))+ :active false,+ :refresh_begin :%now,+ :refresh_end nil,+ :state "refreshing"+ :state_change_at :%now})+ {:keys [state error]} (try+ (refresh! refresher database definition card)+ (catch Exception e+ (log/infof e "Error refreshing persisting model with card-id %s"+ (:card_id persisted-info))+ {:state :error :error (ex-message e)}))]+ (t2/update! :model/PersistedInfo (u/the-id persisted-info)+ {:active (= state :success),+ :refresh_end :%now,+ :state (if (= state :success) "persisted" "error")+ :state_change_at :%now+ :error (when (= state :error) error)})+ (if (= :success state)+ (update stats :success inc)+ (-> stats+ (update :error-details conj {:persisted-info-id (:id persisted-info)+ :error error})+ (update :error inc))))))))(defn- error-details[results]@@ -139,14 +143,16 @@(not= (:type card-info) :model))(let [database (-> persisted-info :database_id db-id->db)](log/infof "Unpersisting model with card-id %s" (:card_id persisted-info))- (try- (unpersist! refresher database persisted-info)- (when-not (= "off" current-state)- (t2/delete! :model/PersistedInfo :id (:id persisted-info)))- (update stats :success inc)- (catch Exception e- (log/infof e "Error unpersisting model with card-id %s" (:card_id persisted-info))- (update stats :error inc))))+ (tracing/with-span :tasks "task.persist.unpersist-model" {:db/id (:database_id persisted-info)+ :persist/card-id (:card_id persisted-info)}+ (try+ (unpersist! refresher database persisted-info)+ (when-not (= "off" current-state)+ (t2/delete! :model/PersistedInfo :id (:id persisted-info)))+ (update stats :success inc)+ (catch Exception e+ (log/infof e "Error unpersisting model with card-id %s" (:card_id persisted-info))+ (update stats :error inc)))))(update stats :skipped inc)))){:success 0, :error 0, :skipped 0}deletables))]@@ -158,8 +164,7 @@persisted info records pointing to cards that are no longer models, archived cards/models, and all records where the correspondingcard or database has been permanently deleted."[]- (t2/select :model/PersistedInfo- {:select [:p.*]+ (let [hsql {:select [:p.*]:from [[:persisted_info :p]]:left-join [[:report_card :c] [:= :c.id :p.card_id]]:where [:or@@ -174,20 +179,23 @@[:= :c.type "question"][:= :c.archived true];; card_id is set to null when the corresponding card is deleted- [:= :p.card_id nil]]}))+ [:= :p.card_id nil]]}]+ (tracing/with-span :tasks "task.persist.find-deletable" {:db/statement (tracing/best-effort-sanitize-sql hsql)}+ (t2/select :model/PersistedInfo hsql))))(defn- refreshable-models"Returns refreshable models for a database id. Must still be models and not archived."[database-id]- (t2/select :model/PersistedInfo- {:select [:p.* :c.type :c.archived :c.name]+ (let [hsql {:select [:p.* :c.type :c.archived :c.name]:from [[:persisted_info :p]]:left-join [[:report_card :c] [:= :c.id :p.card_id]]:where [:and[:= :p.database_id database-id][:in :p.state (persisted-info/refreshable-states)][:= :c.archived false]- [:= :c.type "model"]]}))+ [:= :c.type "model"]]}]+ (tracing/with-span :tasks "task.persist.find-refreshable" {:db/id database-id :db/statement (tracing/best-effort-sanitize-sql hsql)}+ (t2/select :model/PersistedInfo hsql))))(defn- prune-all-deletable!"Prunes all deletable PersistInfos, should not be called from tests asdiff --git a/src/metabase/notification/task/send.clj b/src/metabase/notification/task/send.cljindex 5670afaf0a5..9a9103847c2 100644--- a/src/metabase/notification/task/send.clj+++ b/src/metabase/notification/task/send.clj@@ -10,6 +10,7 @@[metabase.query-processor.timezone :as qp.timezone][metabase.task-history.core :as task-history][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.log :as log][toucan2.core :as t2])(:import@@ -107,19 +108,21 @@(cond(:active notification)- (task-history/with-task-run (some-> (notification.send/notification->task-run-info notification) (assoc :auto-complete false))- (try- (log/info "Submitting to the notification queue")- (task-history/with-task-history {:task "notification-trigger"- :task_details {:trigger_type :notification-subscription/cron- :notification_subscription_id subscription-id- :cron_schedule (:cron_schedule subscription)- :notification_ids [notification-id]}}- (notification.send/send-notification! (assoc notification :triggering_subscription subscription)))- (log/info "Submitted to the notification queue")- (catch Exception e- (log/error e "Failed to submit to the notification queue")- (throw e))))+ (tracing/with-span :tasks "task.notification.send" {:notification/subscription-id subscription-id+ :notification/id notification-id}+ (task-history/with-task-run (some-> (notification.send/notification->task-run-info notification) (assoc :auto-complete false))+ (try+ (log/info "Submitting to the notification queue")+ (task-history/with-task-history {:task "notification-trigger"+ :task_details {:trigger_type :notification-subscription/cron+ :notification_subscription_id subscription-id+ :cron_schedule (:cron_schedule subscription)+ :notification_ids [notification-id]}}+ (notification.send/send-notification! (assoc notification :triggering_subscription subscription)))+ (log/info "Submitted to the notification queue")+ (catch Exception e+ (log/error e "Failed to submit to the notification queue")+ (throw e)))))(nil? notification)(dodiff --git a/src/metabase/permissions/user.clj b/src/metabase/permissions/user.cljindex e1117a396a7..f46264e9a43 100644--- a/src/metabase/permissions/user.clj+++ b/src/metabase/permissions/user.clj@@ -4,6 +4,7 @@[metabase.permissions.models.data-permissions :as data-perms][metabase.permissions.path :as permissions.path][metabase.premium-features.core :refer [defenterprise]]+ [metabase.tracing.core :as tracing][metabase.util :as u]))(defenterprise user->tenant-collection-and-descendant-ids@@ -15,30 +16,31 @@(defn user-permissions-set"Return a set of all permissions object paths that `user-or-id` has been granted access to. (2 DB Calls)"[user-or-id]- (let [s- (set (when-let [user-id (u/the-id user-or-id)]- (concat+ (tracing/with-span :db-app "db-app.permissions-load" {}+ (let [s+ (set (when-let [user-id (u/the-id user-or-id)]+ (concat;; Current User always gets readwrite perms for their Personal Collection and for its descendants! (1 DB;; Call)- (map permissions.path/collection-readwrite-path- ((requiring-resolve 'metabase.collections.models.collection/user->personal-collection-and-descendant-ids)- user-or-id))+ (map permissions.path/collection-readwrite-path+ ((requiring-resolve 'metabase.collections.models.collection/user->personal-collection-and-descendant-ids)+ user-or-id));; Current User always gets readwrite perms for their Tenant Collection and for its descendants! (3 DB Calls)- (map permissions.path/collection-readwrite-path- (user->tenant-collection-and-descendant-ids user-or-id))+ (map permissions.path/collection-readwrite-path+ (user->tenant-collection-and-descendant-ids user-or-id));; Current User always gets read perms for Transforms if they are an analyst (1 DB Call)- (when (or (data-perms/is-data-analyst? user-id) (data-perms/is-superuser? user-id))- (concat ["/collection/namespace/transforms/root/"]- (map permissions.path/collection-readwrite-path ((requiring-resolve 'metabase.collections.models.collection/collections-in-namespace)- :transforms))))+ (when (or (data-perms/is-data-analyst? user-id) (data-perms/is-superuser? user-id))+ (concat ["/collection/namespace/transforms/root/"]+ (map permissions.path/collection-readwrite-path ((requiring-resolve 'metabase.collections.models.collection/collections-in-namespace)+ :transforms))));; include the other Perms entries for any Group this User is in (1 DB Call)- (map :object (app-db/query {:select [:p.object]- :from [[:permissions_group_membership :pgm]]- :join [[:permissions_group :pg] [:= :pgm.group_id :pg.id]- [:permissions :p] [:= :p.group_id :pg.id]]- :where [:= :pgm.user_id user-id]})))))]- ;; Append permissions as a vector for more efficient iteration in checks that go over each permission linearly.- (with-meta s {:as-vec (vec s)})))+ (map :object (app-db/query {:select [:p.object]+ :from [[:permissions_group_membership :pgm]]+ :join [[:permissions_group :pg] [:= :pgm.group_id :pg.id]+ [:permissions :p] [:= :p.group_id :pg.id]]+ :where [:= :pgm.user_id user-id]})))))]+ ;; Append permissions as a vector for more efficient iteration in checks that go over each permission linearly.+ (with-meta s {:as-vec (vec s)}))))diff --git a/src/metabase/premium_features/token_check.clj b/src/metabase/premium_features/token_check.cljindex 372286b9af8..c5ab6ec4c66 100644--- a/src/metabase/premium_features/token_check.clj+++ b/src/metabase/premium_features/token_check.clj@@ -23,6 +23,7 @@[metabase.premium-features.defenterprise :refer [defenterprise]][metabase.premium-features.settings :as premium-features.settings][metabase.settings.core :as setting]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.i18n :refer [trs tru]][metabase.util.json :as json]@@ -218,19 +219,20 @@[](when-let [token (premium-features.settings/premium-embedding-token)](when (mr/validate [:re RemoteCheckedToken] token)- (let [site-uuid (premium-features.settings/site-uuid-for-premium-features-token-checks)- stats (-> (metering-stats)- ;; for backwards compatibility, we send values as strings- (update-vals str))]- (try- (http/post (metering-url token token-check-url)- {:body (json/encode (merge stats- {:site-uuid site-uuid- :mb-version (:tag config/mb-version-info)}))- :content-type :json- :throw-exceptions false})- (catch Throwable e- (log/error e "Error sending metering events")))))))+ (tracing/with-span :tasks "metering.send-events" {}+ (let [site-uuid (premium-features.settings/site-uuid-for-premium-features-token-checks)+ stats (-> (metering-stats)+ ;; for backwards compatibility, we send values as strings+ (update-vals str))]+ (try+ (http/post (metering-url token token-check-url)+ {:body (json/encode (merge stats+ {:site-uuid site-uuid+ :mb-version (:tag config/mb-version-info)}))+ :content-type :json+ :throw-exceptions false})+ (catch Throwable e+ (log/error e "Error sending metering events"))))))));;;;;;;;;;;;;;;;;;;; Airgap Tokens ;;;;;;;;;;;;;;;;;;;;diff --git a/src/metabase/pulse/task/send_pulses.clj b/src/metabase/pulse/task/send_pulses.cljindex 765c6852a7a..0f236d6411c 100644--- a/src/metabase/pulse/task/send_pulses.clj+++ b/src/metabase/pulse/task/send_pulses.clj@@ -17,6 +17,7 @@[metabase.query-processor.timezone :as qp.timezone][metabase.task-history.core :as task-history][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.cron :as u.cron][metabase.util.log :as log][metabase.util.malli :as mu]@@ -48,22 +49,24 @@(defn- send-pulse![pulse-id channel-ids]- (try- (if-let [pulse (models.pulse/retrieve-notification pulse-id- :archived false- ;; alerts should all be migrated to notifications by now- :alert_condition nil)]- (task-history/with-task-run (some-> (pulse.send/pulse->task-run-info pulse) (assoc :auto-complete false))- (task-history/with-task-history {:task "send-pulse"- :task_details {:pulse-id pulse-id- :channel-ids (seq channel-ids)}}- (log/debugf "Starting Pulse Execution: %d" pulse-id)- (pulse.send/send-pulse! pulse :channel-ids channel-ids :async? true)- (log/debugf "Finished Pulse Execution: %d" pulse-id)- :done))- (log/debugf "Pulse %d not found, Skipping." pulse-id))- (catch Throwable e- (log/errorf e "Error sending Pulse %d to channel ids: %s" pulse-id (str/join ", " channel-ids)))))+ (tracing/with-span :tasks "task.pulse.send" {:pulse/id pulse-id+ :pulse/channel-count (count channel-ids)}+ (try+ (if-let [pulse (models.pulse/retrieve-notification pulse-id+ :archived false+ ;; alerts should all be migrated to notifications by now+ :alert_condition nil)]+ (task-history/with-task-run (some-> (pulse.send/pulse->task-run-info pulse) (assoc :auto-complete false))+ (task-history/with-task-history {:task "send-pulse"+ :task_details {:pulse-id pulse-id+ :channel-ids (seq channel-ids)}}+ (log/debugf "Starting Pulse Execution: %d" pulse-id)+ (pulse.send/send-pulse! pulse :channel-ids channel-ids :async? true)+ (log/debugf "Finished Pulse Execution: %d" pulse-id)+ :done))+ (log/debugf "Pulse %d not found, Skipping." pulse-id))+ (catch Throwable e+ (log/errorf e "Error sending Pulse %d to channel ids: %s" pulse-id (str/join ", " channel-ids))))))(defn- send-trigger-timezone[]@@ -104,24 +107,25 @@(defn- clear-pulse-channels-no-recipients!"Delete PulseChannels that have no recipients and no channel set for a pulse, returns the channel ids that were deleted."[pulse-id]- (when-let [ids-to-delete (seq- (for [channel (t2/select [:model/PulseChannel :id :details :channel_id :channel_type]- :pulse_id pulse-id- :id [:not-in {:select [[:pulse_channel_id :id]]- :from :pulse_channel_recipient- :group-by [:pulse_channel_id]- :having [:>= :%count.* [:raw 1]]}])- :when (case (:channel_type channel)- (empty? (get-in channel [:details :emails]))- :slack- (empty? (get-in channel [:details :channel]))- :http- (nil? (:channel_id channel)))]- (:id channel)))]- (log/infof "Deleting %d PulseChannels with id: %s due to having no recipients" (count ids-to-delete) (str/join ", " ids-to-delete))- (t2/delete! :model/PulseChannel :id [:in ids-to-delete])- (set ids-to-delete)))+ (tracing/with-span :tasks "task.pulse.clear-orphan-channels" {:pulse/id pulse-id}+ (when-let [ids-to-delete (seq+ (for [channel (t2/select [:model/PulseChannel :id :details :channel_id :channel_type]+ :pulse_id pulse-id+ :id [:not-in {:select [[:pulse_channel_id :id]]+ :from :pulse_channel_recipient+ :group-by [:pulse_channel_id]+ :having [:>= :%count.* [:raw 1]]}])+ :when (case (:channel_type channel)+ (empty? (get-in channel [:details :emails]))+ :slack+ (empty? (get-in channel [:details :channel]))+ :http+ (nil? (:channel_id channel)))]+ (:id channel)))]+ (log/infof "Deleting %d PulseChannels with id: %s due to having no recipients" (count ids-to-delete) (str/join ", " ids-to-delete))+ (t2/delete! :model/PulseChannel :id [:in ids-to-delete])+ (set ids-to-delete))))(defn- send-pulse!*"Do several things:diff --git a/src/metabase/query_processor.clj b/src/metabase/query_processor.cljindex fad0c7688a5..35bb3b0fa78 100644--- a/src/metabase/query_processor.clj+++ b/src/metabase/query_processor.clj@@ -19,6 +19,7 @@[metabase.query-processor.reducible :as qp.reducible][metabase.query-processor.schema :as qp.schema][metabase.query-processor.setup :as qp.setup]+ [metabase.tracing.core :as tracing][metabase.util.log :as log][metabase.util.malli :as mu]))@@ -44,10 +45,14 @@(defn- process-query** [query rff](qp.debug/debug> (list `process-query query))- (let [preprocessed (qp.preprocess/preprocess query)- compiled (qp.compile/attach-compiled-query preprocessed)- rff (qp.postprocess/post-processing-rff preprocessed rff)]- (qp.execute/execute compiled rff)))+ (tracing/with-span :qp "qp.pipeline" {:query/type (some-> (:type query) name)+ :db/id (:database query)}+ (let [preprocessed (tracing/with-span :qp "qp.preprocess" {}+ (qp.preprocess/preprocess query))+ compiled (qp.compile/attach-compiled-query preprocessed)+ rff (qp.postprocess/post-processing-rff preprocessed rff)]+ (tracing/with-span :qp "qp.execute" {}+ (qp.execute/execute compiled rff)))))(def ^:private ^{:arglists '([query rff])} process-query* nil)@@ -76,9 +81,11 @@([query :- ::qp.schema/any-queryrff :- [:maybe ::qp.schema/rff]]- (qp.setup/with-qp-setup [query query]- (let [rff (or rff qp.reducible/default-rff)]- (process-query* query rff)))))+ (tracing/with-span :qp "qp.process-query" {:query/type (some-> (:type query) name)+ :db/id (:database query)}+ (qp.setup/with-qp-setup [query query]+ (let [rff (or rff qp.reducible/default-rff)]+ (process-query* query rff))))))(mu/defn userland-query :- ::qp.schema/any-query"Add middleware options and `:info` to a `query` so it is ran as a 'userland' query, which slightly changes the QPdiff --git a/src/metabase/query_processor/middleware/cache.clj b/src/metabase/query_processor/middleware/cache.cljindex c89d52e5715..50e90cfe759 100644--- a/src/metabase/query_processor/middleware/cache.clj+++ b/src/metabase/query_processor/middleware/cache.clj@@ -21,6 +21,7 @@[metabase.query-processor.pipeline :as qp.pipeline][metabase.query-processor.schema :as qp.schema][metabase.query-processor.util :as qp.util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.malli :as mu]@@ -266,5 +267,6 @@(let [cacheable? (is-cacheable? query)](log/tracef "Query is %scacheable: %s" (if-not cacheable? "not " "") (get-cache-eligibility-description query))(if cacheable?- (run-query-with-cache qp query rff)+ (tracing/with-span :qp "qp.cache" {:cache/eligible true}+ (run-query-with-cache qp query rff))(qp query rff)))))diff --git a/src/metabase/query_processor/middleware/process_userland_query.clj b/src/metabase/query_processor/middleware/process_userland_query.cljindex 94672a35f6c..71a20d400b2 100644--- a/src/metabase/query_processor/middleware/process_userland_query.clj+++ b/src/metabase/query_processor/middleware/process_userland_query.clj@@ -14,6 +14,7 @@[metabase.queries.models.query :as query][metabase.query-processor.schema :as qp.schema][metabase.query-processor.util :as qp.util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.malli :as mu]@@ -42,11 +43,12 @@(defn- save-execution-metadata!*"Save a `QueryExecution` and update the average execution time for the corresponding `Query`."[{query :json_query, query-hash :hash, running-time :running_time, context :context :as query-execution}]- (when-not (:cache_hit query-execution)- (query/save-query-and-update-average-execution-time! query query-hash running-time))- (if-not context- (log/warn "Cannot save QueryExecution, missing :context")- (t2/insert-returning-pk! :model/QueryExecution (dissoc query-execution :json_query))))+ (tracing/with-span :db-app "db-app.save-query-execution" {}+ (when-not (:cache_hit query-execution)+ (query/save-query-and-update-average-execution-time! query query-hash running-time))+ (if-not context+ (log/warn "Cannot save QueryExecution, missing :context")+ (t2/insert-returning-pk! :model/QueryExecution (dissoc query-execution :json_query)))))(defn- save-execution-metadata!"Save a `QueryExecution` row containing `execution-info`. Done asynchronously when a query is finished."diff --git a/src/metabase/search/appdb/core.clj b/src/metabase/search/appdb/core.cljindex 7fdf12e9e91..899ba2c4dd6 100644--- a/src/metabase/search/appdb/core.clj+++ b/src/metabase/search/appdb/core.clj@@ -19,6 +19,7 @@[metabase.search.spec :as search.spec][metabase.search.util :as search.util][metabase.settings.core :as setting]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.i18n :as i18n][metabase.util.json :as json]@@ -145,33 +146,34 @@:index-state-after @@#'search.index/*indexes*:index-metadata (t2/select :model/SearchIndexMetadata :engine :appdb)}))))- (try- (when (setting/string->boolean (:mb-experimental-search-block-on-queue env/env))- ;; wait for a bit for the queue to be drained- (let [pending-updates #(.size ^Queue @#'search.ingestion/queue)]- (when-not (u/poll {:thunk pending-updates- :done? zero?- :timeout-ms 2000- :interval-ms 100})- (log/warn "Returning search results even though they may be stale. Queue size:" (pending-updates)))))-- (let [weights (search.config/weights search-ctx)- scorers (search.scoring/scorers search-ctx)- query (->> (search.index/search-query search-string search-ctx [:legacy_input])- (add-collection-join-and-where-clauses search-ctx)- (add-table-where-clauses search-ctx)- (#(sql.helpers/where % (search.filter/transform-source-type-where-clause- search-ctx- :search_index.model- :search_index.source_type)))- (search.scoring/with-scores search-ctx scorers)- (search.filter/with-filters search-ctx))]- (->> (t2/query query)- (map (partial rehydrate weights (keys scorers)))))- (catch Exception e- ;; Rule out the error coming from stale index metadata.- (#'search.index/sync-tracking-atoms!)- (throw e))))+ (tracing/with-span :search "search.appdb.query" {:search/query-length (count search-string)}+ (try+ (when (setting/string->boolean (:mb-experimental-search-block-on-queue env/env))+ ;; wait for a bit for the queue to be drained+ (let [pending-updates #(.size ^Queue @#'search.ingestion/queue)]+ (when-not (u/poll {:thunk pending-updates+ :done? zero?+ :timeout-ms 2000+ :interval-ms 100})+ (log/warn "Returning search results even though they may be stale. Queue size:" (pending-updates)))))++ (let [weights (search.config/weights search-ctx)+ scorers (search.scoring/scorers search-ctx)+ query (->> (search.index/search-query search-string search-ctx [:legacy_input])+ (add-collection-join-and-where-clauses search-ctx)+ (add-table-where-clauses search-ctx)+ (#(sql.helpers/where % (search.filter/transform-source-type-where-clause+ search-ctx+ :search_index.model+ :search_index.source_type)))+ (search.scoring/with-scores search-ctx scorers)+ (search.filter/with-filters search-ctx))]+ (->> (t2/query query)+ (map (partial rehydrate weights (keys scorers)))))+ (catch Exception e+ ;; Rule out the error coming from stale index metadata.+ (#'search.index/sync-tracking-atoms!)+ (throw e)))))(defmethod search.engine/results :search.engine/appdb[search-ctx]diff --git a/src/metabase/search/appdb/index.clj b/src/metabase/search/appdb/index.cljindex 377ee08fc30..8ac64def192 100644--- a/src/metabase/search/appdb/index.clj+++ b/src/metabase/search/appdb/index.clj@@ -15,6 +15,7 @@[metabase.search.ingestion :as search.ingestion][metabase.search.models.search-index-metadata :as search-index-metadata][metabase.search.spec :as search.spec]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.i18n :as i18n][metabase.util.json :as json]@@ -316,10 +317,11 @@"Indexes the documents. The context should be :search/updating or :search/reindexing.Context should be :search/updating or :search/reindexing to help control how to manage the updates"[context document-reducible]- (transduce (comp (partition-all insert-batch-size)- (map (partial batch-update! context)))- (partial merge-with +)- document-reducible))+ (tracing/with-span :search "search.appdb.index-docs" {:search/context (name context)}+ (transduce (comp (partition-all insert-batch-size)+ (map (partial batch-update! context)))+ (partial merge-with +)+ document-reducible)))(defmethod search.engine/update! :search.engine/appdb [_engine document-reducible](index-docs! :search/updating document-reducible))diff --git a/src/metabase/search/core.clj b/src/metabase/search/core.cljindex 2623ebd8a55..7f7518ea36a 100644--- a/src/metabase/search/core.clj+++ b/src/metabase/search/core.clj@@ -10,6 +10,7 @@[metabase.search.ingestion :as search.ingestion][metabase.search.spec :as search.spec][metabase.search.util :as search.util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][potemkin :as p]))@@ -87,48 +88,50 @@[& {:as opts}](when (supports-index?)(log/info "Initializing search indexes")- (lib-be/with-metadata-provider-cache- ;; If there are multiple indexes, return the peak inserted for each type. In practice, they should all be the same.- (try- (let [timer (u/start-timer)- report (reduce (partial merge-with max)- nil- (for [e (search.engine/active-engines)]- (search.engine/init! e opts)))- duration (u/since-ms timer)]- (if (seq report)- (do- (analytics/inc! :metabase-search/index-reindex-ms duration)- (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)- (doseq [[model cnt] report]- (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))- (log/infof "Index initialized in %.0fms %s" duration (sort-by (comp - val) report))- report)- (log/info "Found existing search index, and using it.")))- (catch Exception e- (analytics/inc! :metabase-search/index-error)- (throw e))))))+ (tracing/with-span :search "search.init-index" {}+ (lib-be/with-metadata-provider-cache+ ;; If there are multiple indexes, return the peak inserted for each type. In practice, they should all be the same.+ (try+ (let [timer (u/start-timer)+ report (reduce (partial merge-with max)+ nil+ (for [e (search.engine/active-engines)]+ (search.engine/init! e opts)))+ duration (u/since-ms timer)]+ (if (seq report)+ (do+ (analytics/inc! :metabase-search/index-reindex-ms duration)+ (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)+ (doseq [[model cnt] report]+ (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))+ (log/infof "Index initialized in %.0fms %s" duration (sort-by (comp - val) report))+ report)+ (log/info "Found existing search index, and using it.")))+ (catch Exception e+ (analytics/inc! :metabase-search/index-error)+ (throw e)))))))(defn- reindex-logic! [opts](when (supports-index?)- (lib-be/with-metadata-provider-cache- (try- (log/info "Reindexing searchable entities")- (let [timer (u/start-timer)- report (reduce (partial merge-with max)- nil- (for [e (search.engine/active-engines)]- (search.engine/reindex! e opts)))- duration (u/since-ms timer)]- (analytics/inc! :metabase-search/index-reindex-ms duration)- (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)- (doseq [[model cnt] report]- (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))- (log/infof "Done reindexing in %.0fms %s" duration (sort-by (comp - val) report))- report)- (catch Exception e- (analytics/inc! :metabase-search/index-error)- (throw e))))))+ (tracing/with-span :search "search.reindex" {}+ (lib-be/with-metadata-provider-cache+ (try+ (log/info "Reindexing searchable entities")+ (let [timer (u/start-timer)+ report (reduce (partial merge-with max)+ nil+ (for [e (search.engine/active-engines)]+ (search.engine/reindex! e opts)))+ duration (u/since-ms timer)]+ (analytics/inc! :metabase-search/index-reindex-ms duration)+ (prometheus/observe! :metabase-search/index-reindex-duration-ms duration)+ (doseq [[model cnt] report]+ (analytics/inc! :metabase-search/index-reindexes {:model model} cnt))+ (log/infof "Done reindexing in %.0fms %s" duration (sort-by (comp - val) report))+ report)+ (catch Exception e+ (analytics/inc! :metabase-search/index-error)+ (throw e)))))))(defn reindex!"Populate a new index, and make it active. Simultaneously updates the current index.diff --git a/src/metabase/search/impl.clj b/src/metabase/search/impl.cljindex a89c0831685..1d43fefa3c9 100644--- a/src/metabase/search/impl.clj+++ b/src/metabase/search/impl.clj@@ -11,6 +11,7 @@[metabase.search.filter :as search.filter][metabase.search.in-place.filter :as search.in-place.filter][metabase.search.in-place.scoring :as scoring]+ [metabase.tracing.core :as tracing][metabase.transforms.feature-gating :as transforms.gating][metabase.util :as u][metabase.util.i18n :refer [deferred-tru tru]]@@ -440,19 +441,22 @@(mu/defn search"Builds a search query that includes all the searchable entities, and runs it."[search-ctx :- search.config/SearchContext]- (let [reducible-results (search.engine/results search-ctx)- scoring-ctx (select-keys search-ctx [:search-engine :search-string :search-native-query])- xf (comp- (take search.config/*db-max-results*)- (map normalize-result)- (filter (partial check-permissions-for-model search-ctx))- (map (partial normalize-result-more search-ctx))- (keep #(search.engine/score scoring-ctx %)))- total-results (cond->> (scoring/top-results reducible-results search.config/max-filtered-results xf)- true hydrate-dashboards- true hydrate-user-metadata- (:include-metadata? search-ctx) (add-metadata)- (:model-ancestors? search-ctx) (add-dataset-collection-hierarchy)- true (add-collection-effective-location)- true (map serialize))]- (search-results search-ctx search.engine/model-set total-results)))+ (tracing/with-span :search "search.execute" {:search/engine (name (:search-engine search-ctx))+ :search/query-length (count (:search-string search-ctx))+ :search/model-count (count (:models search-ctx))}+ (let [reducible-results (search.engine/results search-ctx)+ scoring-ctx (select-keys search-ctx [:search-engine :search-string :search-native-query])+ xf (comp+ (take search.config/*db-max-results*)+ (map normalize-result)+ (filter (partial check-permissions-for-model search-ctx))+ (map (partial normalize-result-more search-ctx))+ (keep #(search.engine/score scoring-ctx %)))+ total-results (cond->> (scoring/top-results reducible-results search.config/max-filtered-results xf)+ true hydrate-dashboards+ true hydrate-user-metadata+ (:include-metadata? search-ctx) (add-metadata)+ (:model-ancestors? search-ctx) (add-dataset-collection-hierarchy)+ true (add-collection-effective-location)+ true (map serialize))]+ (search-results search-ctx search.engine/model-set total-results))))diff --git a/src/metabase/search/ingestion.clj b/src/metabase/search/ingestion.cljindex 355fd7a1b16..5494df087d9 100644--- a/src/metabase/search/ingestion.clj+++ b/src/metabase/search/ingestion.clj@@ -9,6 +9,7 @@[metabase.lib-be.core :as lib-be][metabase.search.engine :as search.engine][metabase.search.spec :as search.spec]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.performance :as perf]@@ -231,22 +232,23 @@;; We are partitioning the documents into batches at this level and sending each batch to all the engines;; to avoid having to retain the head of the sequences as we work through all the documents.;; Individual engines may also partition the documents further if they prefer- (let [timer (u/start-timer)- update-report (reduce (fn [_ batch] (search.engine/update! e batch)) nil- (eduction (partition-all 150) documents-reducible))- delete-report (reduce (fn [acc batch]- (->> batch- (remove nil?)- (u/group-by first second)- (map (fn [[group ids]] (search.engine/delete! e group ids)))- (apply merge-with + acc))) {}- (eduction (partition-all 1000) removed-models-reducible))- duration (u/since-ms timer)]- (log/debugf "Updated search entries in %.0fms Updated: %s Deleted: %s" duration (sort-by (comp - val) update-report) (sort-by (comp - val) delete-report))- (analytics/inc! :metabase-search/index-update-ms duration)- (prometheus/observe! :metabase-search/index-update-duration-ms duration)- (doseq [[model cnt] (merge-with + update-report delete-report)]- (analytics/inc! :metabase-search/index-updates {:model model} cnt)))))+ (tracing/with-span :search "search.ingestion.update" {:search/engine (name e)}+ (let [timer (u/start-timer)+ update-report (reduce (fn [_ batch] (search.engine/update! e batch)) nil+ (eduction (partition-all 150) documents-reducible))+ delete-report (reduce (fn [acc batch]+ (->> batch+ (remove nil?)+ (u/group-by first second)+ (map (fn [[group ids]] (search.engine/delete! e group ids)))+ (apply merge-with + acc))) {}+ (eduction (partition-all 1000) removed-models-reducible))+ duration (u/since-ms timer)]+ (log/debugf "Updated search entries in %.0fms Updated: %s Deleted: %s" duration (sort-by (comp - val) update-report) (sort-by (comp - val) delete-report))+ (analytics/inc! :metabase-search/index-update-ms duration)+ (prometheus/observe! :metabase-search/index-update-duration-ms duration)+ (doseq [[model cnt] (merge-with + update-report delete-report)]+ (analytics/inc! :metabase-search/index-updates {:model model} cnt))))))(comment(u/group-by first second [["metric" 124] ["dataset" 124] ["metric" 124] ["other" 5]]))@@ -271,24 +273,25 @@(defn bulk-ingest!"Process the given search model updates."[updates]- (lib-be/with-metadata-provider-cache- (if (seq (search.engine/active-engines))- (let [documents (->> (for [[search-model where-clauses] (u/group-by first second updates)]- (spec-index-reducible search-model (into [:or] (distinct where-clauses))))- ;; init collection is only for clj-kondo, as we know that the list is non-empty- (reduce u/rconcat [])- query->documents)- passed-documents (map extract-model-and-id updates)- indexed-documents (map (juxt :model (comp str :id)) (into [] documents))- ;; TODO: The list of documents to delete is not completely accurate.- ;; We are attempting to figure it out based on the ids that are passed in to be indexed vs. the ids of the rows that were actually indexed.- ;; This will not work for cases like indexed-entries with compound PKs,- ;; but it's fine for now because that model doesn't have a where clause so never needs to be purged during an update.- ;; Long-term, we should find a better approach to knowing what to purge.- to-delete (remove (set indexed-documents) passed-documents)]-- (update! documents to-delete))- {})))+ (tracing/with-span :search "search.ingestion.bulk-ingest" {:search/update-count (count updates)}+ (lib-be/with-metadata-provider-cache+ (if (seq (search.engine/active-engines))+ (let [documents (->> (for [[search-model where-clauses] (u/group-by first second updates)]+ (spec-index-reducible search-model (into [:or] (distinct where-clauses))))+ ;; init collection is only for clj-kondo, as we know that the list is non-empty+ (reduce u/rconcat [])+ query->documents)+ passed-documents (map extract-model-and-id updates)+ indexed-documents (map (juxt :model (comp str :id)) (into [] documents))+ ;; TODO: The list of documents to delete is not completely accurate.+ ;; We are attempting to figure it out based on the ids that are passed in to be indexed vs. the ids of the rows that were actually indexed.+ ;; This will not work for cases like indexed-entries with compound PKs,+ ;; but it's fine for now because that model doesn't have a where clause so never needs to be purged during an update.+ ;; Long-term, we should find a better approach to knowing what to purge.+ to-delete (remove (set indexed-documents) passed-documents)]++ (update! documents to-delete))+ {}))))(defn- track-queue-size! [](analytics/set! :metabase-search/queue-size (.size queue)))diff --git a/src/metabase/search/task/search_index.clj b/src/metabase/search/task/search_index.cljindex 5ed31f890e1..c6826673097 100644--- a/src/metabase/search/task/search_index.clj+++ b/src/metabase/search/task/search_index.clj@@ -8,6 +8,7 @@[metabase.search.ingestion :as ingestion][metabase.startup.core :as startup][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.queue :as queue])(:import(java.time Instant)@@ -34,8 +35,9 @@"Create a new index, if necessary"[](when (search/supports-index?)- (cluster-lock/with-cluster-lock cluster-lock-name- (search/init-index! {:force-reset? false, :re-populate? false}))))+ (tracing/with-span :search "search.task.init" {}+ (cluster-lock/with-cluster-lock cluster-lock-name+ (search/init-index! {:force-reset? false, :re-populate? false})))))(task/defjob ^{DisallowConcurrentExecution true:doc "Populate a new Search Index"}diff --git a/src/metabase/server/handler.clj b/src/metabase/server/handler.cljindex 798d6a57e15..18b090e4473 100644--- a/src/metabase/server/handler.clj+++ b/src/metabase/server/handler.clj@@ -18,6 +18,7 @@[metabase.server.middleware.session :as mw.session][metabase.server.middleware.settings-cache :as mw.settings-cache][metabase.server.middleware.ssl :as mw.ssl]+ [metabase.server.middleware.trace :as mw.trace][metabase.util.log :as log][metabase.util.malli :as mu][ring.core.protocols :as ring.protocols]@@ -94,6 +95,7 @@#'mw.misc/add-content-type ; Adds a Content-Type header for any response that doesn't already have one#'mw.misc/disable-streaming-buffering ; Add header to streaming (async) responses so nginx doesn't buffer keepalive bytes#'wrap-gzip ; GZIP response if client can handle it+ #'mw.trace/wrap-trace ; Create root OpenTelemetry span per request (after request-id is available)#'mw.request-id/wrap-request-id ; Add a unique request ID to the request#'mw.misc/bind-request ; bind `metabase.middleware.misc/*request*` for the duration of the request#'mw.ssl/redirect-to-https-middlewarediff --git a/src/metabase/server/middleware/log.clj b/src/metabase/server/middleware/log.cljindex 2fa57b13a73..3a44d2f3da6 100644--- a/src/metabase/server/middleware/log.clj+++ b/src/metabase/server/middleware/log.clj@@ -19,6 +19,7 @@(clojure.core.async.impl.channels ManyToManyChannel)(com.mchange.v2.c3p0 PoolBackedDataSource)(metabase.server.streaming_response StreamingResponse)+ (org.apache.logging.log4j ThreadContext)(org.eclipse.jetty.util.thread QueuedThreadPool)))(set! *warn-on-reflection* true)@@ -159,29 +160,42 @@(defn- log-core-async-response"For async responses that return a `core.async` channel, wait for the channel to return a response before logging the- API request info."- [{{chan :body, :as _response} :response, :as info}]+ API request info. `mdc-ctx` is the MDC snapshot from the request thread — restored before logging+ so trace_id/span_id appear in async log lines."+ [{{chan :body, :as _response} :response, :as info} mdc-ctx]{:pre [(async.u/promise-chan? chan)]};; [async] wait for the pipe to close the canceled/finished channel and log the API response(a/go(let [result (a/<! chan)]- (log-info (assoc info :async-status (if (nil? result) "canceled" "completed"))))))--(defn- log-streaming-response [{{streaming-response :body, :as _response} :response, :as info}]+ (try+ (ThreadContext/putAll mdc-ctx)+ (log-info (assoc info :async-status (if (nil? result) "canceled" "completed")))+ (finally+ (ThreadContext/removeAll (keys mdc-ctx)))))))++(defn- log-streaming-response+ "Log a streaming response after it completes. `mdc-ctx` is the MDC snapshot from the request thread."+ [{{streaming-response :body, :as _response} :response, :as info} mdc-ctx];; [async] wait for the streaming response to be canceled/finished channel and log the API response(let [finished-chan (streaming-response/finished-chan streaming-response)](a/go(let [result (a/<! finished-chan)]- (log-info (assoc info :async-status (name result)))))))+ (try+ (ThreadContext/putAll mdc-ctx)+ (log-info (assoc info :async-status (name result)))+ (finally+ (ThreadContext/removeAll (keys mdc-ctx))))))))(defn- logged-response"Log an API response. Returns response, possibly modified (i.e., core.async channels will be wrapped); this value- should be passed to the normal `respond` function."+ should be passed to the normal `respond` function.+ Captures MDC context from the request thread so that async log lines include trace_id/span_id."[{{:keys [body], :as response} :response, :as info}]- (condp instance? body- ManyToManyChannel (log-core-async-response info)- StreamingResponse (log-streaming-response info)- (log-info info))+ (let [mdc-ctx (ThreadContext/getImmutableContext)]+ (condp instance? body+ ManyToManyChannel (log-core-async-response info mdc-ctx)+ StreamingResponse (log-streaming-response info mdc-ctx)+ (log-info info)))response);;; +----------------------------------------------------------------------------------------------------------------+diff --git a/src/metabase/server/middleware/session.clj b/src/metabase/server/middleware/session.cljindex 9bcb15aaa89..94b4fb28482 100644--- a/src/metabase/server/middleware/session.clj+++ b/src/metabase/server/middleware/session.clj@@ -29,6 +29,7 @@[metabase.request.schema :as request.schema][metabase.session.core :as session][metabase.settings.core :as setting]+ [metabase.tracing.core :as tracing][metabase.util.honey-sql-2 :as h2x][metabase.util.i18n :as i18n][metabase.util.log :as log]@@ -246,7 +247,9 @@token OR a valid API key was passed."[handler](fn [request respond raise]- (handler (merge-current-user-info request) respond raise)))+ (let [request' (tracing/with-span :db-app "db-app.session-lookup" {}+ (merge-current-user-info request))]+ (handler request' respond raise))));;; +----------------------------------------------------------------------------------------------------------------+;;; | bind-current-user |diff --git a/src/metabase/server/middleware/trace.clj b/src/metabase/server/middleware/trace.cljnew file mode 100644index 00000000000..fd73044b54d--- /dev/null+++ b/src/metabase/server/middleware/trace.clj@@ -0,0 +1,116 @@+(ns metabase.server.middleware.trace+ "Ring middleware that creates a root OpenTelemetry span per HTTP request.++ When a frontend `traceparent` header is present, the trace ID is extracted+ and forced onto the root span via a custom IdGenerator (in tracing.core).+ This correlates frontend and backend spans under the same trace WITHOUT+ creating a parent-child link to a non-existent browser span — avoiding+ the 'root span not yet received' message in Tempo.++ Uses clj-otel's `new-span!`/`end-span!` (manual span lifecycle) instead of+ `with-span!` (auto-close) because Metabase Ring handlers are async — the handler+ returns immediately and calls `respond` later on a different thread. The span+ must stay open until `respond` or `raise` is called."+ (:require+ [metabase.tracing.core :as tracing]+ [metabase.util.log :as log]+ [steffan-westcott.clj-otel.api.trace.span :as span])+ (:import+ (io.opentelemetry.api.trace Span StatusCode)+ (io.opentelemetry.context Context Scope)))++(set! *warn-on-reflection* true)++(defn- parse-frontend-trace-id+ "Extract the trace ID from a W3C traceparent header.+ Returns nil if the header is missing or malformed.+ Format: 00-{32-hex-trace-id}-{16-hex-parent-id}-{2-hex-flags}"+ ^String [request]+ (when-let [tp (get-in request [:headers "traceparent"])]+ (second (re-matches #"00-([0-9a-f]{32})-[0-9a-f]{16}-[0-9a-f]{2}" tp))))++(defn- end-span-with-status!+ "End a span from its context, setting HTTP status code and OTel status.+ Sets OK for 1xx-4xx, ERROR for 5xx (per OTel HTTP server span conventions)."+ [^Span span ^Context span-ctx status-code]+ (when status-code+ (.setAttribute span "http.status_code" (long status-code))+ (if (>= (long status-code) 500)+ (.setStatus span StatusCode/ERROR)+ (.setStatus span StatusCode/OK)))+ (span/end-span! {:context span-ctx}))++(defn- api-request?+ "Returns true for API requests (/api/*) which are the only requests worth tracing.+ Excludes static files (/app/*, /favicon*), the root page (/), and any other+ non-API routes that would create noise as independent traces in the collector."+ [request]+ (.startsWith (str (:uri request)) "/api/"))++(defn wrap-trace+ "Middleware that wraps each HTTP request in an OpenTelemetry span (`:api` group).+ Extracts trace ID from the frontend's traceparent header for correlation.+ Includes HTTP method, URI, and request ID as span attributes.+ Skips static file requests (/app/*, /favicon*) to avoid trace noise.++ Uses clj-otel's `new-span!`/`end-span!` for manual span lifecycle to correctly+ handle async Ring handlers — the span stays open until `respond` or `raise` is called."+ [handler]+ (fn [request respond raise]+ (if (and (tracing/group-enabled? :api)+ (api-request? request))+ (do+ ;; If frontend sent a traceparent, force its trace ID for this thread.+ ;; The custom IdGenerator in tracing.core will return it instead of a random one.+ (when-let [tid (parse-frontend-trace-id request)]+ (tracing/force-trace-id! tid))+ (let [;; Create as root span — no parent context means no parent-child link+ ;; to the browser's phantom span. If a frontend trace ID was forced,+ ;; the IdGenerator returns it; otherwise a random trace ID is generated.+ span-ctx (span/new-span! {:name "api.request"+ :parent nil+ :span-kind :server+ :attributes (cond-> {:http/method (some-> (:request-method request) name)+ :http/url (:uri request)+ :http/request-id (:metabase-request-id request)+ :http/referer (get-in request [:headers "referer"])}+ (:query-string request)+ (assoc :http/query-string (:query-string request)))})+ ^Span span (span/get-span span-ctx)+ ^Scope scope (.makeCurrent ^Context span-ctx)+ span-id (.getSpanId (.getSpanContext span))]+ (tracing/inject-trace-id-into-mdc!)+ ;; Tag profiling samples with this span's ID and set pyroscope.profile.id+ ;; attribute on the span (no-op without Pyroscope agent).+ (tracing/set-pyroscope-context! span span-id "api.request")+ (try+ (handler request+ (fn trace-respond [response]+ (try+ (end-span-with-status! span span-ctx (:status response))+ (catch Throwable t+ (log/error t "Error ending trace span")))+ (respond response))+ (fn trace-raise [exception]+ (try+ (.setStatus span StatusCode/ERROR)+ (.recordException span exception)+ (span/end-span! {:context span-ctx})+ (catch Throwable t+ (log/error t "Error ending trace span on exception")))+ (raise exception)))+ (catch Throwable t+ ;; Handler itself threw synchronously — end span and re-raise+ (.setStatus span StatusCode/ERROR)+ (.recordException span t)+ (span/end-span! {:context span-ctx})+ (raise t))+ (finally+ ;; Close scope on the middleware thread (restores previous context).+ ;; For async handlers, the callback runs on a different thread — the scope+ ;; closure here is still correct (cleans up this thread's context).+ (.close scope)+ (tracing/clear-pyroscope-context!)+ (tracing/clear-forced-trace-id!)+ (tracing/clear-trace-id-from-mdc!)))))+ (handler request respond raise))))diff --git a/src/metabase/session/task/session_cleanup.clj b/src/metabase/session/task/session_cleanup.cljindex f8b4d52cc03..52e5422db21 100644--- a/src/metabase/session/task/session_cleanup.clj+++ b/src/metabase/session/task/session_cleanup.clj@@ -6,6 +6,7 @@[metabase.app-db.core :as mdb][metabase.config.core :as config][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.honey-sql-2 :as h2x][toucan2.core :as t2]))@@ -17,8 +18,11 @@(let [oldest-allowed [:inline (h2x/add-interval-honeysql-form (mdb/db-type):%now(- (config/config-int :max-session-age))- :minute)]]- (t2/delete! :model/Session :created_at [:< oldest-allowed])))+ :minute)]+ hsql {:delete-from [(t2/table-name :model/Session)]+ :where [:< :created_at oldest-allowed]}]+ (tracing/with-span :tasks "task.session-cleanup.delete" {:db/statement (tracing/best-effort-sanitize-sql hsql)}+ (t2/query-one hsql))))(def ^:private session-cleanup-job-key (jobs/key "metabase.task.session-cleanup.job"))(def ^:private session-cleanup-trigger-key (triggers/key "metabase.task.session-cleanup.trigger"))diff --git a/src/metabase/sync/analyze/classify.clj b/src/metabase/sync/analyze/classify.cljindex 431b7e2622b..9ee042991e1 100644--- a/src/metabase/sync/analyze/classify.clj+++ b/src/metabase/sync/analyze/classify.clj@@ -27,6 +27,7 @@[metabase.sync.analyze.fingerprint :as sync.fingerprint][metabase.sync.interface :as i][metabase.sync.util :as sync-util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.malli :as mu]@@ -110,26 +111,27 @@"Run various classifiers on the appropriate `fields` in a `table` that have not been previously analyzed. These dothings like inferring (and setting) the semantic types and preview display status for Fields belonging to `table`."[table :- i/TableInstance]- (let [table-id (:id table)]- (when-let [fields (fields-to-classify table)]- (let [existing-name-field (t2/count :model/Field- :table_id table-id- :active true- :visibility_type [:not-in ["sensitive" "retired"]]- :semantic_type :type/Name)- {:keys [fields-failed]}- (reduce (fn [state field]- (let [result (classify! field state)]- (cond-> state- (instance? Exception result)- (update :fields-failed inc)-- (= :type/Name (:semantic_type result))- (assoc :exists-name true))))- {:fields-failed 0 :exists-name (pos? existing-name-field)}- fields)]- {:fields-classified (count fields)- :fields-failed fields-failed}))))+ (tracing/with-span :sync "sync.classify.fields" {:sync/table (:name table)}+ (let [table-id (:id table)]+ (when-let [fields (fields-to-classify table)]+ (let [existing-name-field (t2/count :model/Field+ :table_id table-id+ :active true+ :visibility_type [:not-in ["sensitive" "retired"]]+ :semantic_type :type/Name)+ {:keys [fields-failed]}+ (reduce (fn [state field]+ (let [result (classify! field state)]+ (cond-> state+ (instance? Exception result)+ (update :fields-failed inc)++ (= :type/Name (:semantic_type result))+ (assoc :exists-name true))))+ {:fields-failed 0 :exists-name (pos? existing-name-field)}+ fields)]+ {:fields-classified (count fields)+ :fields-failed fields-failed})))))(mu/defn ^:always-validate classify-table!"Run various classifiers on the `table`. These do things like inferring (and setting) entity type of `table`."diff --git a/src/metabase/sync/analyze/fingerprint.clj b/src/metabase/sync/analyze/fingerprint.cljindex 2ea265369c7..62b3fdc17d5 100644--- a/src/metabase/sync/analyze/fingerprint.clj+++ b/src/metabase/sync/analyze/fingerprint.clj@@ -11,6 +11,7 @@[metabase.lib.schema.metadata.fingerprint :as lib.schema.metadata.fingerprint][metabase.sync.interface :as i][metabase.sync.util :as sync-util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.malli :as mu]@@ -81,7 +82,8 @@(map vector fields fingerprints)))))driver (driver.u/database->driver (table/database table))opts {:truncation-size *truncation-size*}]- (driver/table-rows-sample driver table fields rff opts)))+ (tracing/with-span :sync "sync.fingerprint.sample" {:sync/table (:name table) :sync/field-count (count fields)}+ (driver/table-rows-sample driver table fields rff opts))));;; +----------------------------------------------------------------------------------------------------------------+;;; | WHICH FIELDS NEED UPDATED FINGERPRINTS? |@@ -193,16 +195,17 @@(mu/defn fingerprint-table!"Generate and save fingerprints for all the Fields in `table` that have not been previously analyzed."[table :- i/TableInstance]- (if-let [fields (fields-to-fingerprint table)]- (do- (log/infof "Fingerprinting %s fields in table %s" (count fields) (sync-util/name-for-logging table))- (let [stats- (sync-util/with-returning-throwable (format "Error fingerprinting %s" (sync-util/name-for-logging table))- (fingerprint-fields! table fields))]- (if (:throwable stats)- (merge (empty-stats-map 0) stats)- stats)))- (empty-stats-map 0)))+ (tracing/with-span :sync "sync.fingerprint.table" {:db/id (:db_id table) :sync/table (:name table)}+ (if-let [fields (fields-to-fingerprint table)]+ (do+ (log/infof "Fingerprinting %s fields in table %s" (count fields) (sync-util/name-for-logging table))+ (let [stats+ (sync-util/with-returning-throwable (format "Error fingerprinting %s" (sync-util/name-for-logging table))+ (fingerprint-fields! table fields))]+ (if (:throwable stats)+ (merge (empty-stats-map 0) stats)+ stats)))+ (empty-stats-map 0))))(def ^:private LogProgressFn[:=> [:cat :string [:schema i/TableInstance]] :any])diff --git a/src/metabase/sync/field_values.clj b/src/metabase/sync/field_values.cljindex 74f532a83ec..61e8214b319 100644--- a/src/metabase/sync/field_values.clj+++ b/src/metabase/sync/field_values.clj@@ -5,6 +5,7 @@[metabase.app-db.core :as mdb][metabase.sync.interface :as i][metabase.sync.util :as sync-util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.malli :as mu]@@ -67,8 +68,9 @@(mu/defn- update-field-values-for-database![database :- i/DatabaseInstance]- (let [tables (sync-util/reducible-sync-tables database)]- (transduce (map update-field-values-for-table!) (partial merge-with +) tables)))+ (tracing/with-span :sync "field-values.update" {:db/id (:id database)}+ (let [tables (sync-util/reducible-sync-tables database)]+ (transduce (map update-field-values-for-table!) (partial merge-with +) tables))))(defn- update-field-values-summary [{:keys [created updated deleted errors]}](format "Updated %d field value sets, created %d, deleted %d with %d errors"@@ -102,15 +104,16 @@(mu/defn- delete-expired-advanced-field-values-for-database![database :- i/DatabaseInstance]- (let [tables (sync-util/reducible-sync-tables database)]- {:deleted (transduce (comp (map delete-expired-advanced-field-values-for-table!)- (map (fn [result]- (if (instance? Throwable result)- (throw result)- result))))- +- 0- tables)}))+ (tracing/with-span :sync "field-values.delete-expired-advanced" {:db/id (:id database)}+ (let [tables (sync-util/reducible-sync-tables database)]+ {:deleted (transduce (comp (map delete-expired-advanced-field-values-for-table!)+ (map (fn [result]+ (if (instance? Throwable result)+ (throw result)+ result))))+ ++ 0+ tables)})))(def ^:private sync-field-values-steps[(sync-util/create-sync-step "delete-expired-advanced-field-values"diff --git a/src/metabase/sync/sync.clj b/src/metabase/sync/sync.cljindex f8cef571ef3..a45761ade57 100644--- a/src/metabase/sync/sync.clj+++ b/src/metabase/sync/sync.clj@@ -17,6 +17,7 @@[metabase.sync.interface :as i][metabase.sync.sync-metadata :as sync-metadata][metabase.sync.util :as sync-util]+ [metabase.tracing.core :as tracing][metabase.util.malli :as mu][metabase.util.malli.schema :as ms][metabase.warehouse-schema.models.field :as field]@@ -45,14 +46,16 @@[:metadata :analyze :field-values]))(defn- do-phase! [database phase]- (let [f (phase->fn phase)- result (f database)]- (if (instance? Throwable result)- ;; do nothing if we're configured to just move on.- (when-not sync-util/*log-exceptions-and-continue?*- ;; but if we didn't expect any suppressed exceptions, rethrow it- (throw result))- (assoc result :name (name phase)))))+ (tracing/with-span :sync (str "sync." (name phase)) {:db/id (:id database)+ :sync/phase (name phase)}+ (let [f (phase->fn phase)+ result (f database)]+ (if (instance? Throwable result)+ ;; do nothing if we're configured to just move on.+ (when-not sync-util/*log-exceptions-and-continue?*+ ;; but if we didn't expect any suppressed exceptions, rethrow it+ (throw result))+ (assoc result :name (name phase))))))(mu/defn sync-database! :- SyncDatabaseResults"Perform all the different sync operations synchronously for `database`.@@ -68,10 +71,11 @@([database :- i/DatabaseInstance{:keys [scan], :or {scan :full}} :- [:maybe [:map[:scan {:optional true} [:maybe [:enum :schema :full]]]]]]- (sync-util/sync-operation :sync database (format "Sync %s" (sync-util/name-for-logging database))- (->> (scan-phases scan)- (keep (partial do-phase! database))- (doall)))))+ (tracing/with-span :sync "sync.database" {:db/id (:id database)}+ (sync-util/sync-operation :sync database (format "Sync %s" (sync-util/name-for-logging database))+ (->> (scan-phases scan)+ (keep (partial do-phase! database))+ (doall))))))(mu/defn sync-table!"Perform all the different sync operations synchronously for a given `table`. Since often called on a sequence ofdiff --git a/src/metabase/sync/sync_metadata.clj b/src/metabase/sync/sync_metadata.cljindex 8e09a93e786..5dfb4c4a175 100644--- a/src/metabase/sync/sync_metadata.clj+++ b/src/metabase/sync/sync_metadata.clj@@ -17,6 +17,7 @@[metabase.sync.sync-metadata.sync-timezone :as sync-tz][metabase.sync.sync-metadata.tables :as sync-tables][metabase.sync.util :as sync-util]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.malli :as mu][metabase.warehouse-schema.models.table :as table]))@@ -63,7 +64,8 @@"Sync the metadata for a Metabase `database`. This makes sure child Table & Field objects are synchronized."[database :- i/DatabaseInstance](sync-util/sync-operation :sync-metadata database (format "Sync metadata for %s" (sync-util/name-for-logging database))- (let [db-metadata (fetch-metadata/db-metadata database)]+ (let [db-metadata (tracing/with-span :sync "sync.metadata.fetch-metadata" {:db/id (:id database)}+ (fetch-metadata/db-metadata database))](u/prog1 (sync-util/run-sync-operation "sync" database (make-sync-steps db-metadata))(if (some sync-util/abandon-sync? (map second (:steps <>)))(sync-util/set-initial-database-sync-aborted! database)diff --git a/src/metabase/sync/task/sync_databases.clj b/src/metabase/sync/task/sync_databases.cljindex ec8d46cb206..2790b2fa7f9 100644--- a/src/metabase/sync/task/sync_databases.clj+++ b/src/metabase/sync/task/sync_databases.clj@@ -21,6 +21,7 @@[metabase.sync.schedules :as sync.schedules][metabase.sync.sync-metadata :as sync-metadata][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.cron :as u.cron][metabase.util.log :as log]@@ -88,12 +89,16 @@e))](log/warnf ex "Cannot sync Database %s: %s" (:name database) (ex-message ex))(database-routing/with-database-routing-off- (let [metadata-results (sync-metadata/sync-db-metadata! database)- analyze-results (when (:is_full_sync database)- (analyze/analyze-db! database))+ (let [db-id (:id database)+ metadata-results (tracing/with-span :sync "sync.metadata" {:db/id db-id}+ (sync-metadata/sync-db-metadata! database))+ analyze-results (when (:is_full_sync database)+ (tracing/with-span :sync "sync.analyze" {:db/id db-id}+ (analyze/analyze-db! database)))refingerprint-results (when (and (:refingerprint database)(should-refingerprint-fields? analyze-results))- (analyze/refingerprint-db! database))]+ (tracing/with-span :sync "sync.refingerprint" {:db/id db-id}+ (analyze/refingerprint-db! database)))](cond-> {:metadata-results metadata-results}analyze-results (assoc :analyze-results analyze-results)refingerprint-results (assoc :refingerprint-results refingerprint-results)))))))diff --git a/src/metabase/sync/util.clj b/src/metabase/sync/util.cljindex 8e8c0aaffcf..ca51d85beb7 100644--- a/src/metabase/sync/util.clj+++ b/src/metabase/sync/util.clj@@ -14,6 +14,7 @@[metabase.query-processor.interface :as qp.i][metabase.sync.interface :as i][metabase.task-history.core :as task-history]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.date-2 :as u.date][metabase.util.log :as log]@@ -548,20 +549,21 @@[database :- i/DatabaseInstance{:keys [step-name sync-fn log-summary-fn] :as _step} :- StepDefinition](let [start-time (t/zoned-date-time)- results (do-with-start-and-finish-debug-logging- (format "step ''%s'' for %s"- step-name- (name-for-logging database))- (fn [& args]- (with-returning-throwable (format "Error running step ''%s'' for %s" step-name (name-for-logging database))- (task-history/with-task-history- {:task step-name- :db_id (u/the-id database)- :on-success-info (fn [update-map result]- (if (instance? Throwable result)- (throw result)- (assoc update-map :task_details (dissoc result :start-time :end-time :log-summary-fn))))}- (apply sync-fn database args)))))+ results (tracing/with-span :sync (str "sync.step." step-name) {:db/id (u/the-id database) :sync/step step-name}+ (do-with-start-and-finish-debug-logging+ (format "step ''%s'' for %s"+ step-name+ (name-for-logging database))+ (fn [& args]+ (with-returning-throwable (format "Error running step ''%s'' for %s" step-name (name-for-logging database))+ (task-history/with-task-history+ {:task step-name+ :db_id (u/the-id database)+ :on-success-info (fn [update-map result]+ (if (instance? Throwable result)+ (throw result)+ (assoc update-map :task_details (dissoc result :start-time :end-time :log-summary-fn))))}+ (apply sync-fn database args))))))end-time (t/zoned-date-time)][step-name (assoc results:start-time start-timediff --git a/src/metabase/task/bootstrap.clj b/src/metabase/task/bootstrap.cljindex ae0b9fb5d20..6e8ebd426f3 100644--- a/src/metabase/task/bootstrap.clj+++ b/src/metabase/task/bootstrap.clj@@ -9,6 +9,16 @@(defn- app-db ^javax.sql.DataSource []((requiring-resolve 'metabase.app-db.core/app-db)))+;; Optional interceptor for wrapping JDBC connections before Quartz uses them.+;; Set by task.tracing to add SQL-level tracing. nil means no interception.+(defonce ^:private connection-interceptor (atom nil))++(defn set-connection-interceptor!+ "Set an optional function to wrap JDBC connections before Quartz uses them.+ Called by task.tracing to add SQL-level tracing. Pass nil to remove."+ [f]+ (reset! connection-interceptor f))+(defrecord ^:private ConnectionProvider []org.quartz.utils.ConnectionProvider(initialize [_])@@ -19,11 +29,10 @@;; very important! Fetch a new connection from the connection pool rather than using currently bound Connection if;; one already exists -- because Quartz will close this connection when done, we don't want to screw up the;; calling block- ;;- ;; in a perfect world we could just check whether we're creating a new Connection or not, and if using an existing- ;; Connection, wrap it in a delegating proxy wrapper that makes `.close()` a no-op but forwards all other methods.- ;; Now that would be a useful macro!- (.getConnection (app-db)))+ (let [conn (.getConnection (app-db))]+ (if-let [interceptor @connection-interceptor]+ (interceptor conn)+ conn)))(shutdown [_]))(when-not *compile-files*diff --git a/src/metabase/task/impl.clj b/src/metabase/task/impl.cljindex 3394b72e0ba..d2ba3f4a835 100644--- a/src/metabase/task/impl.clj+++ b/src/metabase/task/impl.clj@@ -26,6 +26,7 @@[metabase.classloader.core :as classloader][metabase.task.bootstrap][metabase.task.job-factory :as job-factory]+ [metabase.tracing.core :as tracing][metabase.util :as u][metabase.util.log :as log][metabase.util.malli :as mu]@@ -352,11 +353,12 @@#_{:clj-kondo/ignore [:discouraged-var]}(defmacro defjob- "Like `clojurewerkz.quartzite.task/defjob` but with a log context."+ "Like `clojurewerkz.quartzite.task/defjob` but with a log context and an OpenTelemetry tracing span."[jtype args & body]`(jobs/defjob ~jtype ~args(log/with-context {:quartz-job-type (quote ~jtype)}- ~@body)))+ (tracing/with-span :tasks (str "task." (quote ~jtype)) {:task/name (str (quote ~jtype))}+ ~@body))))(defn add-job-listener!"Add a [Quartz Joblistener](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/tutorial-lesson-07.html). That willdiff --git a/src/metabase/task_history/task/task_history_cleanup.clj b/src/metabase/task_history/task/task_history_cleanup.cljindex cdb3ad19ce4..6b5b3c78a08 100644--- a/src/metabase/task_history/task/task_history_cleanup.clj+++ b/src/metabase/task_history/task/task_history_cleanup.clj@@ -5,6 +5,7 @@[clojurewerkz.quartzite.triggers :as triggers][metabase.task-history.models.task-history :as task-history][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.log :as log]))(set! *warn-on-reflection* true)@@ -17,12 +18,13 @@"Delete older TaskHistory rows -- see docstring of `task-history/cleanup-task-history!` for more details."[](log/debug "Cleaning up task history")- (task-history/with-task-history {:task "task-history-cleanup"}- (let [deleted-rows? (task-history/cleanup-task-history! history-rows-to-keep)]- (log/debug- (if deleted-rows?- "Task history cleanup successful, rows were deleted"- "Task history cleanup successful, no rows were deleted")))))+ (tracing/with-span :tasks "task.history-cleanup.delete" {:task-history/rows-to-keep history-rows-to-keep}+ (task-history/with-task-history {:task "task-history-cleanup"}+ (let [deleted-rows? (task-history/cleanup-task-history! history-rows-to-keep)]+ (log/debug+ (if deleted-rows?+ "Task history cleanup successful, rows were deleted"+ "Task history cleanup successful, no rows were deleted"))))))(task/defjob^{:doc "Delete older TaskHistory rows -- see docstring of `task-history/cleanup-task-history!` for more details."}diff --git a/src/metabase/task_history/task/task_run_heartbeat.clj b/src/metabase/task_history/task/task_run_heartbeat.cljindex 65fa4da5c93..7489be7aa44 100644--- a/src/metabase/task_history/task/task_run_heartbeat.clj+++ b/src/metabase/task_history/task/task_run_heartbeat.clj@@ -11,6 +11,7 @@[metabase.config.core :as config][metabase.models.interface :as mi][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.util.honey-sql-2 :as h2x][metabase.util.log :as log][toucan2.core :as t2]))@@ -28,13 +29,14 @@(defn send-heartbeat!"Update updated_at for all :started runs belonging to this process."[]- (let [updated (t2/update! :model/TaskRun- {:status :started- :process_uuid config/local-process-uuid}- {:updated_at (mi/now)})]- (when (pos? updated)- (log/debugf "Sent heartbeat for %d running task runs" updated))- updated))+ (tracing/with-span :tasks "task.heartbeat.update" {}+ (let [updated (t2/update! :model/TaskRun+ {:status :started+ :process_uuid config/local-process-uuid}+ {:updated_at (mi/now)})]+ (when (pos? updated)+ (log/debugf "Sent heartbeat for %d running task runs" updated))+ updated)))(defn mark-orphaned-runs!"Mark runs as :abandoned if:@@ -44,12 +46,14 @@[](let [heartbeat-cutoff (h2x/add-interval-honeysql-form (mdb/db-type) :%now (- orphan-threshold-hours) :hour)duration-cutoff (h2x/add-interval-honeysql-form (mdb/db-type) :%now (- max-run-duration-hours) :hour)- orphaned-run-ids (t2/select-fn-set :id :model/TaskRun- {:where [:and- [:= :status "started"]- [:or- [:< :updated_at heartbeat-cutoff]- [:< :started_at duration-cutoff]]]})]+ where-clause {:where [:and+ [:= :status "started"]+ [:or+ [:< :updated_at heartbeat-cutoff]+ [:< :started_at duration-cutoff]]]}+ orphaned-run-ids (tracing/with-span :tasks "task.heartbeat.mark-orphaned-runs"+ {:db/statement (tracing/best-effort-sanitize-sql (assoc where-clause :select [:id] :from [:task_run]))}+ (t2/select-fn-set :id :model/TaskRun where-clause))](when (seq orphaned-run-ids)(t2/update! :model/TaskRun {:id [:in orphaned-run-ids]}{:status :abandoned@@ -61,14 +65,15 @@"Mark tasks as :unknown if they belong to the given orphaned runs."[orphaned-run-ids](when (seq orphaned-run-ids)- (let [orphaned (t2/update! :model/TaskHistory- {:status :started- :run_id [:in orphaned-run-ids]}- {:status :unknown- :ended_at (mi/now)})]- (when (pos? orphaned)- (log/infof "Marked %d orphaned tasks as :unknown" orphaned))- orphaned)))+ (tracing/with-span :tasks "task.heartbeat.mark-orphaned-tasks" {:heartbeat/orphaned-run-count (count orphaned-run-ids)}+ (let [orphaned (t2/update! :model/TaskHistory+ {:status :started+ :run_id [:in orphaned-run-ids]}+ {:status :unknown+ :ended_at (mi/now)})]+ (when (pos? orphaned)+ (log/infof "Marked %d orphaned tasks as :unknown" orphaned))+ orphaned))))(defn- task-run-heartbeat!"Send heartbeat for running tasks and mark orphaned runs/tasks."diff --git a/src/metabase/tracing/attributes.clj b/src/metabase/tracing/attributes.cljnew file mode 100644index 00000000000..d14edb4751d--- /dev/null+++ b/src/metabase/tracing/attributes.clj@@ -0,0 +1,15 @@+(ns metabase.tracing.attributes+ "Attribute helpers for OpenTelemetry spans."+ (:require+ [honey.sql :as sql]))++(set! *warn-on-reflection* true)++(defn best-effort-sanitize-sql+ "Convert a HoneySQL map to a parameterized SQL string for trace attributes.+ Values become ? placeholders. That is a best-effort sanitizing solution."+ [hsql-map]+ (try+ (first (sql/format hsql-map {:quoted false}))+ (catch Exception _+ (pr-str hsql-map))))diff --git a/src/metabase/tracing/core.clj b/src/metabase/tracing/core.cljnew file mode 100644index 00000000000..f1b48805a3a--- /dev/null+++ b/src/metabase/tracing/core.clj@@ -0,0 +1,356 @@+(ns metabase.tracing.core+ "Tracing module API. Add OpenTelemetry traces to your code.++ ## Quick Start+ (require '[metabase.tracing.core :as tracing])+ (tracing/with-span :qp \"qp.preprocess\" {:db/id 42}+ (preprocess query))++ ## New Group+ (tracing/register-group! :my-feature \"Description\")+ (tracing/with-span :my-feature \"my-op\" {} (do-stuff))++ ## Built-in Groups+ :qp, :sync, :tasks, :search, :api, :db-user, :db-app, :events, :quartz++ ## Configuration+ MB_TRACING_ENABLED=true MB_TRACING_ENDPOINT=host:4317+ MB_TRACING_GROUPS=qp,api MB_TRACING_SERVICE_NAME=metabase-prod-1"+ (:require+ [clojure.string :as str]+ [metabase.tracing.attributes :as trace-attrs]+ [metabase.tracing.settings :as tracing.settings]+ [metabase.util.log :as log]+ [potemkin :as p]+ [steffan-westcott.clj-otel.api.otel :as otel]+ [steffan-westcott.clj-otel.api.trace.span :as span]+ [steffan-westcott.clj-otel.exporter.otlp.http.trace :as otlp-http]+ [steffan-westcott.clj-otel.sdk.otel-sdk :as otel-sdk])+ (:import+ (io.opentelemetry.api.trace Span SpanContext)+ (io.opentelemetry.sdk.trace IdGenerator)+ (java.time Duration)+ (org.apache.logging.log4j ThreadContext)))++(set! *warn-on-reflection* true)++(p/import-vars+ [trace-attrs+ best-effort-sanitize-sql])++;;; ------------------------------------------------ Group Registry ------------------------------------------------++(defonce ^:private group-registry+ (atom {}))++(defn register-group!+ "Register a trace group. Groups can be selectively enabled via MB_TRACING_GROUPS."+ [group-key description]+ (swap! group-registry assoc group-key {:description description})+ nil)++;; Built-in groups+(register-group! :qp "Query processor: preprocess, compile, execute, cache")+(register-group! :sync "Database sync: metadata, analysis, fingerprinting, field values")+(register-group! :tasks "Scheduled background tasks (Quartz jobs)")+(register-group! :search "Search: full-text, semantic, indexing, ingestion")+(register-group! :api "HTTP request/response lifecycle")+(register-group! :db-user "Customer/user database operations: SQL execution, connection pool")+(register-group! :db-app "Application/system database operations: sessions, settings, QE writes")+(register-group! :events "Event system: view logging, audit, notifications")+(register-group! :quartz "Quartz scheduler internals: trigger acquisition, locks, heartbeats, JDBC")++(defn registered-groups+ "Return a map of all registered trace groups."+ []+ @group-registry)++;;; ----------------------------------------------- Runtime State --------------------------------------------------++;; Cached set of enabled groups, or `:all`. nil means tracing is disabled.+(defonce ^:private enabled-groups (atom nil))++;; Cached trace log level string (e.g. "DEBUG"). Used by DynamicThresholdFilter in log4j2.xml+;; to lower the effective log threshold for traced threads. nil means tracing is disabled.+(defonce ^:private trace-log-level (atom nil))++(defn init-enabled-groups!+ "Parse the tracing-groups and log-level settings and cache the results.+ `log-level-str` is the MB_TRACING_LOG_LEVEL value (e.g. \"DEBUG\", \"INFO\")."+ [groups-str log-level-str]+ (let [groups-str (or groups-str "all")]+ (reset! enabled-groups+ (if (= "all" (str/trim groups-str))+ :all+ (->> (str/split groups-str #",")+ (map (comp keyword str/trim))+ (set)))))+ (reset! trace-log-level (or log-level-str "INFO")))++(defn shutdown-groups!+ "Clear cached enabled groups and trace log level."+ []+ (reset! enabled-groups nil)+ (reset! trace-log-level nil))++(defn group-enabled?+ "Check if a trace group is currently enabled. Returns false when tracing is disabled."+ [group]+ (when-let [groups @enabled-groups]+ (or (= groups :all)+ (contains? groups group))))++;;; ------------------------------------------ Forced Trace ID (Frontend) ----------------------------------------++;; ThreadLocal for per-request trace ID override from frontend traceparent header.+;; The custom IdGenerator reads (and clears) this value when creating a root span,+;; so the span gets the frontend's trace ID without a parent-child link+;; (which would cause "root span not yet received" in Tempo).+(def ^:private ^ThreadLocal forced-trace-id-holder (ThreadLocal.))++(defn force-trace-id!+ "Set a trace ID to be used by the next root span created on this thread.+ Consumed (and cleared) by the custom IdGenerator."+ [^String trace-id]+ (.set forced-trace-id-holder trace-id))++(defn get-and-clear-forced-trace-id!+ "Get and atomically clear the forced trace ID for this thread. Returns nil if none set.+ Called by the custom IdGenerator — should not be called directly."+ ^String []+ (let [tid (.get forced-trace-id-holder)]+ (when tid+ (.remove forced-trace-id-holder))+ tid))++(defn clear-forced-trace-id!+ "Clear any forced trace ID without consuming it. Safety cleanup."+ []+ (.remove forced-trace-id-holder))++;;; -------------------------------------------- MDC Injection (Log<->Trace) -----------------------------------------++(defn inject-trace-id-into-mdc!+ "Inject current span's trace_id, span_id, and trace_level into Log4j2 MDC.+ - trace_id/span_id: for log-to-trace correlation (Loki <-> Tempo).+ - trace_level: consumed by DynamicThresholdFilter in log4j2.xml to lower+ the effective log threshold for this thread during the span."+ []+ (let [span (Span/current)+ ctx (.getSpanContext span)]+ (ThreadContext/put "trace_id" (.getTraceId ctx))+ (ThreadContext/put "span_id" (.getSpanId ctx)))+ (when-let [level @trace-log-level]+ (ThreadContext/put "trace_level" level)))++(defn clear-trace-id-from-mdc!+ "Remove trace_id, span_id, and trace_level from Log4j2 MDC."+ []+ (ThreadContext/remove "trace_id")+ (ThreadContext/remove "span_id")+ (ThreadContext/remove "trace_level"))++;;; ------------------------------------------------- Tracer Access ------------------------------------------------++(defn get-tracer+ "Return an OTel `Tracer` for the given instrumentation library name.+ Uses the clj-otel default OTel instance (set during SDK init), NOT+ `GlobalOpenTelemetry` (which may be no-op if `:set-as-global` was false)."+ ^io.opentelemetry.api.trace.Tracer [^String library-name]+ (let [^io.opentelemetry.api.OpenTelemetry otel-instance (otel/get-default-otel!)]+ (.getTracer otel-instance library-name)))++;;; --------------------------------------------- Pyroscope Integration --------------------------------------------+;;; Optional integration with Pyroscope continuous profiling. When pyroscope.jar is on the+;;; classpath, tags profiling samples with span_id for trace-to-profile linking in Grafana.+;;; All functions are no-ops when Pyroscope is not available. Uses runtime reflection only.++;; PyroscopeAsyncProfiler.getAsyncProfiler() -> AsyncProfiler instance+(defonce ^:private async-profiler-instance+ (delay+ (try+ (let [factory-cls (Class/forName "io.pyroscope.PyroscopeAsyncProfiler")+ ^java.lang.reflect.Method getter (.getMethod factory-cls "getAsyncProfiler" (make-array Class 0))]+ (.invoke getter nil (make-array Object 0)))+ (catch ClassNotFoundException _+ nil)+ (catch Exception e+ (log/debug e "Could not resolve PyroscopeAsyncProfiler")+ nil))))++;; AsyncProfiler.setTracingContext(long spanId, long spanNameId)+(defonce ^:private set-tracing-ctx-method+ (delay+ (when-let [profiler @async-profiler-instance]+ (try+ (.getMethod (class profiler) "setTracingContext"+ (into-array Class [Long/TYPE Long/TYPE]))+ (catch NoSuchMethodException _+ (log/debug "AsyncProfiler.setTracingContext not available")+ nil)+ (catch Exception _ nil)))))++;; Pyroscope.LabelsWrapper.registerConstant(String) -> long+(defonce ^:private register-constant-method+ (delay+ (try+ ;; v2.x package path+ (let [cls (Class/forName "io.pyroscope.labels.v2.Pyroscope$LabelsWrapper")]+ (.getMethod cls "registerConstant" (into-array Class [String])))+ (catch ClassNotFoundException _+ (try+ ;; v1.x package path+ (let [cls (Class/forName "io.pyroscope.labels.Pyroscope$LabelsWrapper")]+ (.getMethod cls "registerConstant" (into-array Class [String])))+ (catch ClassNotFoundException _ nil)+ (catch Exception _ nil)))+ (catch Exception _ nil))))++(defn pyroscope-available?+ "True if Pyroscope trace-to-profile integration is available.+ Returns true when pyroscope.jar is on the classpath and the async-profiler+ supports setTracingContext."+ []+ (boolean (and @async-profiler-instance @set-tracing-ctx-method)))++(defn set-pyroscope-context!+ "Tag current thread's profiling samples with span ID and name, and set the+ `pyroscope.profile.id` attribute on the span (for Grafana trace-to-profile linking).+ No-op if Pyroscope is not available."+ [^Span span ^String span-id-hex ^String span-name]+ (when-let [^java.lang.reflect.Method method @set-tracing-ctx-method]+ (try+ (let [span-id-long (Long/parseUnsignedLong span-id-hex 16)+ span-name-id (if-let [^java.lang.reflect.Method rc @register-constant-method]+ (long (.invoke rc nil (into-array Object [span-name])))+ 0)]+ (.invoke method @async-profiler-instance+ (into-array Object [(Long/valueOf span-id-long)+ (Long/valueOf span-name-id)]))+ (.setAttribute span "pyroscope.profile.id" span-id-hex))+ (catch Exception e+ (log/debug e "Error setting Pyroscope profiling context")))))++(defn clear-pyroscope-context!+ "Clear profiling context for current thread. No-op if Pyroscope is not available."+ []+ (when-let [^java.lang.reflect.Method method @set-tracing-ctx-method]+ (try+ (.invoke method @async-profiler-instance+ (into-array Object [(Long/valueOf 0) (Long/valueOf 0)]))+ (catch Exception _))))++;;; ------------------------------------------------ Primary Macro -------------------------------------------------++(defmacro with-span+ "Create an OTel span if tracing is enabled for `group`.++ When disabled: zero overhead — single atom deref + boolean check, body runs directly.+ When enabled: creates span AND injects trace_id/span_id into Log4j2 MDC+ for log-to-trace correlation (Loki <-> Tempo in Grafana).++ Saves and restores previous MDC values so that nested spans don't wipe the+ parent span's trace_id/span_id from MDC when they exit.++ For root spans (no parent in MDC), also sets Pyroscope profiling context so+ that CPU/alloc/lock samples are tagged with the span ID. This covers task spans+ and other entry points that don't go through the API trace middleware.+ API requests get profiling context from wrap-trace instead.++ Usage:+ (with-span :qp \"qp.process-query\" {:db/id db-id}+ (process-query query))"+ [group span-name attrs & body]+ `(if (group-enabled? ~group)+ (let [prev-trace-id# (ThreadContext/get "trace_id")+ prev-span-id# (ThreadContext/get "span_id")+ root-span?# (nil? prev-span-id#)]+ (span/with-span! {:name ~span-name :attributes ~attrs}+ (inject-trace-id-into-mdc!)+ ;; For root spans (no parent in MDC), set Pyroscope profiling context.+ ;; Nested child spans skip this — the root's context covers all samples.+ (when root-span?#+ (let [^Span span# (Span/current)+ ^SpanContext ctx# (.getSpanContext span#)+ span-id# (.getSpanId ctx#)]+ (set-pyroscope-context! span# span-id# ~span-name)))+ (try+ ~@body+ (finally+ (when root-span?#+ (clear-pyroscope-context!))+ (if prev-trace-id#+ (do (ThreadContext/put "trace_id" prev-trace-id#)+ (ThreadContext/put "span_id" prev-span-id#))+ (clear-trace-id-from-mdc!))))))+ (do ~@body)))++;;; -------------------------------------------- SDK Lifecycle -------------------------------------------------+;;; OpenTelemetry SDK initialization and shutdown. Initializes the OTel SDK with an OTLP+;;; span exporter for pushing traces to a collector. Has no database dependency — can be+;;; initialized very early in the startup process.++;; Holds the initialized OpenTelemetrySdk instance for manual shutdown.+(defonce ^:private otel-sdk-instance (atom nil))++(defn- make-id-generator+ "Create an IdGenerator that respects forced trace IDs from frontend traceparent+ headers (via `force-trace-id!`), falling back to random generation.+ This allows frontend-originated requests to share a trace ID without creating+ a parent-child link to a non-existent browser span."+ ^IdGenerator []+ (let [^IdGenerator default-gen (IdGenerator/random)]+ (reify IdGenerator+ (generateTraceId [_]+ (or (get-and-clear-forced-trace-id!)+ (.generateTraceId default-gen)))+ (generateSpanId [_]+ (.generateSpanId default-gen)))))++(defn init!+ "Initialize the OTel SDK with OTLP HTTP exporter. No-op when MB_TRACING_ENABLED=false.+ Should be called as early as possible in startup — has no database dependency."+ []+ (if-not (tracing.settings/tracing-enabled)+ (log/info "OpenTelemetry tracing is disabled (MB_TRACING_ENABLED=false)")+ (try+ (let [endpoint (tracing.settings/tracing-endpoint)+ service-name (tracing.settings/tracing-service-name)+ groups-str (tracing.settings/tracing-groups)+ log-level-str (tracing.settings/tracing-log-level)+ queue-size (tracing.settings/tracing-max-queue-size)+ timeout-ms (tracing.settings/tracing-export-timeout-ms)+ delay-ms (tracing.settings/tracing-schedule-delay-ms)+ exporter (otlp-http/span-exporter {:endpoint endpoint})]+ (log/infof "Initializing OpenTelemetry tracing: service=%s endpoint=%s groups=%s log-level=%s"+ service-name endpoint groups-str log-level-str)+ (log/infof "Batch span processor config: max-queue-size=%d export-timeout-ms=%d schedule-delay-ms=%d"+ queue-size timeout-ms delay-ms)+ (let [otel (otel-sdk/init-otel-sdk!+ service-name+ {:set-as-default true+ :register-shutdown-hook false ;; we manage shutdown ourselves+ :tracer-provider+ {:id-generator (make-id-generator)+ :span-processors [{:exporters [exporter]+ :max-queue-size queue-size+ :exporter-timeout (Duration/ofMillis timeout-ms)+ :schedule-delay (Duration/ofMillis delay-ms)}]}})]+ (reset! otel-sdk-instance otel)+ (init-enabled-groups! groups-str log-level-str)+ (log/info "OpenTelemetry tracing initialized successfully")))+ (catch Exception e+ (log/error e "Failed to initialize OpenTelemetry tracing — tracing will be disabled")))))++(defn shutdown!+ "Shutdown the OTel SDK, flushing any pending spans. Called during application shutdown."+ []+ (when-let [otel @otel-sdk-instance]+ (log/info "Shutting down OpenTelemetry tracing...")+ (try+ (otel-sdk/close-otel-sdk! otel)+ (catch Exception e+ (log/warn e "Error shutting down OpenTelemetry SDK")))+ (reset! otel-sdk-instance nil)+ (shutdown-groups!)+ (log/info "OpenTelemetry tracing shut down")))diff --git a/src/metabase/tracing/events.clj b/src/metabase/tracing/events.cljnew file mode 100644index 00000000000..e960000600b--- /dev/null+++ b/src/metabase/tracing/events.clj@@ -0,0 +1,25 @@+(ns metabase.tracing.events+ "Registers tracing instrumentation for the event system.+ Loaded by metabase.tracing.init to add span wrapping around event publishing+ without creating a cyclic dependency from events.impl -> tracing.core."+ (:require+ [metabase.events.impl :as events.impl]+ [metabase.tracing.core :as tracing]+ [methodical.core :as methodical]))++(set! *warn-on-reflection* true)++(defn- traced-publish-event!+ "Around method that wraps event publishing in a tracing span."+ [next-method topic event]+ (tracing/with-span :events "events.publish" {:event/topic (str topic)}+ (next-method topic event)))++;; Register the tracing around method on the publish-event! multimethod.+;; This uses a unique key so it can be cleanly removed if needed.+(methodical/add-aux-method-with-unique-key!+ #'events.impl/publish-event!+ :around+ :default+ traced-publish-event!+ ::tracing)diff --git a/src/metabase/tracing/init.clj b/src/metabase/tracing/init.cljnew file mode 100644index 00000000000..7f34df970f9--- /dev/null+++ b/src/metabase/tracing/init.clj@@ -0,0 +1,8 @@+(ns metabase.tracing.init+ "Side-effect loading for the tracing module.+ Loads settings (for env var recognition), quartz tracing (for task/init! registration),+ and event tracing (for publish-event! instrumentation)."+ (:require+ [metabase.tracing.events]+ [metabase.tracing.quartz]+ [metabase.tracing.settings]))diff --git a/src/metabase/tracing/quartz.clj b/src/metabase/tracing/quartz.cljnew file mode 100644index 00000000000..e59ec348ab3--- /dev/null+++ b/src/metabase/tracing/quartz.clj@@ -0,0 +1,192 @@+(ns metabase.tracing.quartz+ "Quartz scheduler tracing instrumentation.++ Provides two levels of observability for Quartz internals:++ 1. **Lifecycle spans** — a JobListener wraps each job execution in a parent span,+ making Quartz scheduling overhead (lock acquisition, trigger state transitions)+ visible as the gap between the listener span and the defjob child span.++ 2. **JDBC-level spans** — a ConnectionProvider interceptor wraps JDBC connections+ so that every SQL query Quartz runs (against QRTZ_* tables) gets its own span+ with the full SQL statement text.++ Both are gated on the `:quartz` trace group — zero overhead when disabled."+ (:require+ [clojure.string :as str]+ [metabase.task.bootstrap :as task.bootstrap]+ [metabase.task.core :as task]+ [metabase.tracing.core :as tracing]+ [metabase.util :as u]+ [metabase.util.log :as log])+ (:import+ (io.opentelemetry.api.trace Span StatusCode)+ (io.opentelemetry.context Scope)+ (java.lang.reflect InvocationHandler InvocationTargetException Method Proxy)+ (java.sql Connection PreparedStatement Statement)+ (org.quartz JobListener)))++(set! *warn-on-reflection* true)++;;; ----------------------------------------------- JDBC Proxy -------------------------------------------------------++(defn- sql-operation+ "Extract the SQL verb (SELECT, UPDATE, INSERT, DELETE) from a SQL string."+ ^String [^String sql]+ (when sql+ (let [trimmed (str/trim sql)+ space (.indexOf trimmed " ")]+ (if (pos? space)+ (u/upper-case-en (subs trimmed 0 space))+ (u/upper-case-en trimmed)))))++(defn- invoke-or-unwrap+ "Invoke a method on target, unwrapping InvocationTargetException to preserve+ checked exception semantics (e.g. SQLException)."+ [^Method method target ^objects args]+ (try+ (.invoke method target args)+ (catch InvocationTargetException e+ (throw (.getCause e)))))++(def ^:private execute-method-names+ #{"executeQuery" "executeUpdate" "execute" "executeBatch"+ "executeLargeUpdate" "executeLargeBatch"})++(defn- traced-prepared-statement+ "Wrap a PreparedStatement in a proxy that creates a :quartz span around execute* methods."+ [^PreparedStatement stmt ^String sql]+ (let [operation (sql-operation sql)]+ (Proxy/newProxyInstance+ (.getClassLoader PreparedStatement)+ (into-array Class [PreparedStatement])+ (reify InvocationHandler+ (invoke [_ _ method args]+ (if (execute-method-names (.getName ^Method method))+ (tracing/with-span :quartz "quartz.db.execute"+ {:db/statement sql+ :db/operation (or operation "UNKNOWN")}+ (invoke-or-unwrap method stmt args))+ (invoke-or-unwrap method stmt args)))))))++(def ^:private statement-execute-with-sql+ "Statement execute methods that take SQL as the first argument."+ #{"execute" "executeQuery" "executeUpdate"})++(defn- traced-statement+ "Wrap a Statement in a proxy that creates a :quartz span around execute* methods.+ Unlike PreparedStatement, Statement receives SQL at execute time."+ [^Statement stmt]+ (Proxy/newProxyInstance+ (.getClassLoader Statement)+ (into-array Class [Statement])+ (reify InvocationHandler+ (invoke [_ _ method args]+ (let [method-name (.getName ^Method method)]+ (if (and (statement-execute-with-sql method-name)+ args+ (pos? (alength ^objects args))+ (instance? String (aget ^objects args 0)))+ (let [sql (aget ^objects args 0)+ operation (sql-operation sql)]+ (tracing/with-span :quartz "quartz.db.execute"+ {:db/statement sql+ :db/operation (or operation "UNKNOWN")}+ (invoke-or-unwrap method stmt args)))+ ;; executeBatch and other non-SQL methods — pass through+ (if (execute-method-names method-name)+ (tracing/with-span :quartz "quartz.db.execute"+ {:db/operation "BATCH"}+ (invoke-or-unwrap method stmt args))+ (invoke-or-unwrap method stmt args))))))))++(defn- traced-connection+ "Wrap a Connection in a proxy that intercepts prepareStatement, createStatement,+ rollback, and commit to provide full JDBC-level tracing."+ [^Connection conn]+ (Proxy/newProxyInstance+ (.getClassLoader Connection)+ (into-array Class [Connection])+ (reify InvocationHandler+ (invoke [_ _ method args]+ (let [method-name (.getName ^Method method)]+ (case method-name+ "prepareStatement"+ (if (and args+ (pos? (alength ^objects args))+ (instance? String (aget ^objects args 0)))+ (let [sql (aget ^objects args 0)+ stmt (invoke-or-unwrap method conn args)]+ (traced-prepared-statement stmt sql))+ (invoke-or-unwrap method conn args))++ "createStatement"+ (let [stmt (invoke-or-unwrap method conn args)]+ (traced-statement stmt))++ "rollback"+ (tracing/with-span :quartz "quartz.db.rollback" {}+ (invoke-or-unwrap method conn args))++ "commit"+ (tracing/with-span :quartz "quartz.db.commit" {}+ (invoke-or-unwrap method conn args))++ ;; default — pass through+ (invoke-or-unwrap method conn args)))))))++(defn- connection-interceptor+ "Connection wrapping function installed into bootstrap's ConnectionProvider.+ Checks `:quartz` group at call time — wraps only when enabled."+ [^Connection conn]+ (if (tracing/group-enabled? :quartz)+ (traced-connection conn)+ conn))++;;; ---------------------------------------------- Job Listener ------------------------------------------------------++;; ThreadLocal storage for span state between jobToBeExecuted and jobWasExecuted callbacks.+;; All callbacks for the same job execution run on the same Quartz worker thread.+(def ^:private ^ThreadLocal listener-state (ThreadLocal.))++(defn- create-tracing-job-listener+ "Create a JobListener that wraps each job execution in a :quartz span.+ The span becomes the parent of the defjob span (via OTel context propagation)."+ ^JobListener []+ (reify JobListener+ (getName [_]+ "metabase.tracing.quartz/quartz-tracing-listener")++ (jobToBeExecuted [_ ctx]+ (when (tracing/group-enabled? :quartz)+ (try+ (let [job-name (.. ctx getJobDetail getKey getName)+ ^Span span (-> (tracing/get-tracer "metabase.quartz")+ (.spanBuilder "quartz.job.execute")+ (.setAttribute "quartz.job/name" job-name)+ (.startSpan))+ ^Scope scope (.makeCurrent span)]+ (.set listener-state {:span span :scope scope}))+ (catch Throwable t+ (log/error t "Error starting quartz tracing span")))))++ (jobExecutionVetoed [_ _])++ (jobWasExecuted [_ _ job-exception]+ (when-let [{:keys [^Span span ^Scope scope]} (.get listener-state)]+ (.remove listener-state)+ (try+ (when job-exception+ (.setStatus span StatusCode/ERROR (.getMessage ^Exception job-exception))+ (.recordException span ^Throwable job-exception))+ (.close scope)+ (.end span)+ (catch Throwable t+ (log/error t "Error ending quartz tracing span")))))))++;;; ------------------------------------------- Initialization -------------------------------------------------------++(defmethod task/init! ::QuartzTracing [_]+ (log/info "Initializing Quartz tracing instrumentation")+ (task/add-job-listener! (create-tracing-job-listener))+ (task.bootstrap/set-connection-interceptor! connection-interceptor))diff --git a/src/metabase/tracing/settings.clj b/src/metabase/tracing/settings.cljnew file mode 100644index 00000000000..cbbf9ddf9d6--- /dev/null+++ b/src/metabase/tracing/settings.clj@@ -0,0 +1,85 @@+(ns metabase.tracing.settings+ "Env-var-only settings for the OpenTelemetry tracing module.+ All settings are configured exclusively via environment variables (`:setter :none`)."+ (:require+ [metabase.settings.core :as setting :refer [defsetting]]+ [metabase.util.i18n :refer [deferred-tru]]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++(defsetting tracing-enabled+ (deferred-tru "Enable OpenTelemetry tracing. When true, spans are exported to the configured OTLP collector.")+ :type :boolean+ :default false+ :visibility :public+ :export? false+ :setter :none)++(defsetting tracing-endpoint+ (deferred-tru "OTLP HTTP collector endpoint for trace export (e.g., http://localhost:4318/v1/traces).")+ :type :string+ :default "http://localhost:4318/v1/traces"+ :visibility :internal+ :export? false+ :setter :none)++(defsetting tracing-groups+ (deferred-tru "Comma-separated list of trace groups to enable, or \"all\" for everything.")+ :type :string+ :default "all"+ :visibility :internal+ :export? false+ :setter :none)++(defsetting tracing-log-level+ (deferred-tru "Log level threshold during traced spans. When a span is active, log events at this level and above bypass per-logger level gates via Log4j2 DynamicThresholdFilter. Requires the filter in log4j2.xml. Valid values: \"INFO\", \"DEBUG\", \"TRACE\".")+ :type :string+ :default "INFO"+ :visibility :internal+ :export? false+ :setter :none+ :getter (fn reading-tracing-log-level []+ (let [v (or (setting/get-raw-value :tracing-log-level string? identity)+ "INFO")]+ (if (#{"TRACE" "DEBUG" "INFO"} v)+ v+ (do+ (log/warnf "MB_TRACING_LOG_LEVEL value '%s' is not valid, expected 'TRACE', 'DEBUG', or 'INFO'. Defaulting to 'INFO'." v)+ "INFO")))))++(defsetting tracing-max-queue-size+ (deferred-tru "Maximum number of spans queued for export. When the queue is full, new spans are dropped silently. Controls memory usage when the collector is slow or unavailable.")+ :type :integer+ :default 2048+ :visibility :internal+ :export? false+ :setter :none)++(defsetting tracing-export-timeout-ms+ (deferred-tru "Maximum time in milliseconds to wait for a batch export to complete. If the collector is slow or unreachable, the batch is dropped after this timeout.")+ :type :integer+ :default 10000+ :visibility :internal+ :export? false+ :setter :none)++(defsetting tracing-schedule-delay-ms+ (deferred-tru "Delay in milliseconds between consecutive batch span exports to the collector.")+ :type :integer+ :default 5000+ :visibility :internal+ :export? false+ :setter :none)++(defsetting tracing-service-name+ (deferred-tru "Service name reported in traces. Defaults to the hostname.")+ :type :string+ :visibility :internal+ :export? false+ :setter :none+ :getter (fn reading-tracing-service-name []+ (or (setting/get-raw-value :tracing-service-name string? identity)+ (try (.getHostName (java.net.InetAddress/getLocalHost))+ (catch Exception _+ "metabase")))))diff --git a/src/metabase/transforms/jobs.clj b/src/metabase/transforms/jobs.cljindex d733db93600..f01a132e6b7 100644--- a/src/metabase/transforms/jobs.clj+++ b/src/metabase/transforms/jobs.clj@@ -11,6 +11,7 @@[metabase.models.transforms.transform-run :as transform-run][metabase.revisions.core :as revisions][metabase.task.core :as task]+ [metabase.tracing.core :as tracing][metabase.transforms-base.ordering :as transforms-base.ordering][metabase.transforms.execute :as transforms.execute][metabase.transforms.instrumentation :as transforms.instrumentation]@@ -51,25 +52,26 @@ordering)))(defn- get-plan [transform-ids]- (let [all-transforms (t2/select :model/Transform)- global-ordering (transforms-base.ordering/transform-ordering all-transforms)- relevant-ids (get-deps global-ordering transform-ids)- transforms-by-id (into {}- (keep (fn [{:keys [id] :as transform}]- (when (relevant-ids id)- [id transform])))- all-transforms)- ordering (sorted-ordering (select-keys global-ordering relevant-ids) transforms-by-id)]- (when-let [cycle (transforms-base.ordering/find-cycle ordering)]- (let [id->name (into {} (map (juxt :id :name)) all-transforms)]- (throw (ex-info (str "Cyclic transform definitions detected: "- (str/join " → " (map id->name cycle)))- {:cycle cycle}))))- (loop [complete (ordered-set/ordered-set)]- (if-let [current-transform (next-transform ordering transforms-by-id complete)]- (recur (conj complete (:id current-transform)))- {:order (map transforms-by-id complete)- :deps global-ordering}))))+ (tracing/with-span :tasks "task.transform.plan" {:transform/count (count transform-ids)}+ (let [all-transforms (t2/select :model/Transform)+ global-ordering (transforms-base.ordering/transform-ordering all-transforms)+ relevant-ids (get-deps global-ordering transform-ids)+ transforms-by-id (into {}+ (keep (fn [{:keys [id] :as transform}]+ (when (relevant-ids id)+ [id transform])))+ all-transforms)+ ordering (sorted-ordering (select-keys global-ordering relevant-ids) transforms-by-id)]+ (when-let [cycle (transforms-base.ordering/find-cycle ordering)]+ (let [id->name (into {} (map (juxt :id :name)) all-transforms)]+ (throw (ex-info (str "Cyclic transform definitions detected: "+ (str/join " → " (map id->name cycle)))+ {:cycle cycle}))))+ (loop [complete (ordered-set/ordered-set)]+ (if-let [current-transform (next-transform ordering transforms-by-id complete)]+ (recur (conj complete (:id current-transform)))+ {:order (map transforms-by-id complete)+ :deps global-ordering})))))(defn- block-until-not-already-running [transform-id](when-let [active-run (transform-run/running-run-for-transform-id transform-id)]@@ -80,7 +82,8 @@(defn- run-transform! [run-id run-method user-id {transform-id :id :as transform}](if-not (transforms.u/check-feature-enabled transform)(log/warnf "Skip running transform %d due to lacking premium features" transform-id)- (do+ (tracing/with-span :tasks "task.transform.execute" {:transform/id transform-id+ :transform/name (:name transform)}(block-until-not-already-running transform-id)(let [try-exec(fn []@@ -94,7 +97,12 @@(throw e)))))](loop [](when (= :already-running (try-exec))- (block-until-not-already-running transform-id)+ (when (transform-run/running-run-for-transform-id transform-id)+ (log/warn "Transform" (pr-str transform-id) "already running, waiting")+ (loop []+ (Thread/sleep 2000)+ (when (transform-run/running-run-for-transform-id transform-id)+ (recur))))(recur))))(transforms.job-run/add-run-activity! run-id))))@@ -232,29 +240,32 @@(let [transforms (job-transform-ids job-id)](log/info "Executing transform job" (pr-str job-id) "with transforms" (pr-str transforms))(let [{run-id :id} (transforms.job-run/start-run! job-id run-method)]- (transforms.instrumentation/with-job-timing [job-id run-method]- (try ;; catch any catastrophic problems- (let [result (run-transforms! run-id transforms opts)]- (case (::status result)- :succeeded (transforms.job-run/succeed-started-run! run-id)- :failed (try- (transforms.job-run/fail-started-run! run-id {:message (compile-transform-failure-messages (::failures result))})- (when (= :cron run-method)- (notify-transform-failures job-id (::failures result)))- (catch Exception e- (log/error e "Error when failing a transform run.")))))- (catch Throwable t- ;; We don't expect a catastrophic failure, but neither did the Titanic.- ;; We should clean up in this case and notify the admin users.- (try- (transforms.job-run/fail-started-run! run-id {:message (.getMessage t)})- (when (= :cron run-method)- (if (::transform-failure (ex-data t))- (notify-transform-failures job-id (::failures (ex-data t)))- (notify-job-failure job-id (.getMessage t))))- (catch Exception e- (log/error e "Error when failing a transform job run.")))- (throw t))))+ (tracing/with-span :tasks "task.transform.run-job" {:transform.job/id job-id+ :transform.job/run-method (name run-method)+ :transform.job/count (count transforms)}+ (transforms.instrumentation/with-job-timing [job-id run-method]+ (try ;; catch any catastrophic problems+ (let [result (run-transforms! run-id transforms opts)]+ (case (::status result)+ :succeeded (transforms.job-run/succeed-started-run! run-id)+ :failed (try+ (transforms.job-run/fail-started-run! run-id {:message (compile-transform-failure-messages (::failures result))})+ (when (= :cron run-method)+ (notify-transform-failures job-id (::failures result)))+ (catch Exception e+ (log/error e "Error when failing a transform run.")))))+ (catch Throwable t+ ;; We don't expect a catastrophic failure, but neither did the Titanic.+ ;; We should clean up in this case and notify the admin users.+ (try+ (transforms.job-run/fail-started-run! run-id {:message (.getMessage t)})+ (when (= :cron run-method)+ (if (::transform-failure (ex-data t))+ (notify-transform-failures job-id (::failures (ex-data t)))+ (notify-job-failure job-id (.getMessage t))))+ (catch Exception e+ (log/error e "Error when failing a transform job run.")))+ (throw t)))))run-id))))(def ^:private job-key "metabase.transforms.jobs.timeout-job")diff --git a/src/metabase/transforms/query_impl.clj b/src/metabase/transforms/query_impl.cljindex ee8c2751454..de3ed6bb1aa 100644--- a/src/metabase/transforms/query_impl.clj+++ b/src/metabase/transforms/query_impl.clj@@ -3,6 +3,7 @@[clojure.core.async :as a][metabase.driver :as driver][metabase.driver.connection :as driver.conn]+ [metabase.tracing.core :as tracing][metabase.transforms-base.interface :as transforms-base.i][metabase.transforms-base.util :as transforms-base.u][metabase.transforms.instrumentation :as transforms.instrumentation]@@ -29,29 +30,33 @@(driver.conn/with-write-connection(log/info "Executing transform" id "with target" (pr-str target)(when (driver.conn/write-connection-requested?) " using write connection"))- (let [conn-spec (driver/connection-spec driver db)- transform-details {:db-id (:id db) :conn-spec conn-spec :output-schema (:schema target)}]- (transforms.instrumentation/with-stage-timing [run-id [:computation :mbql-query]]- (transforms.u/run-cancelable-transform!- run-id driver transform-details- (fn [cancel-chan]- (let [result (transforms-base.i/execute-base!- transform- {:cancelled? #(boolean (a/poll! cancel-chan))- :run-id run-id- :with-stage-timing-fn (fn [rid stage thunk]- (transforms.instrumentation/with-stage-timing [rid stage]- (thunk)))})]- ;; Bridge result-map to exception-based flow for run-cancelable-transform!- (when-not (= :succeeded (:status result))- (throw (or (:error result) (ex-info "Transform failed" {:status (:status result)}))))- result))))- ;; Post-processing: sync, transform_id, events, secondary indexes (after succeed-started-run!)- (transforms-base.u/complete-execution! transform- {:run-id run-id- :with-stage-timing-fn (fn [rid stage thunk]- (transforms.instrumentation/with-stage-timing [rid stage]- (thunk)))}))))+ (tracing/with-span :tasks "task.transform.query" {:transform/id id+ :transform/target-type (name (keyword (:type target)))+ :db/id (:id db)+ :db/engine (name driver)}+ (let [conn-spec (driver/connection-spec driver db)+ transform-details {:db-id (:id db) :conn-spec conn-spec :output-schema (:schema target)}]+ (transforms.instrumentation/with-stage-timing [run-id [:computation :mbql-query]]+ (transforms.u/run-cancelable-transform!+ run-id driver transform-details+ (fn [cancel-chan]+ (let [result (transforms-base.i/execute-base!+ transform+ {:cancelled? #(boolean (a/poll! cancel-chan))+ :run-id run-id+ :with-stage-timing-fn (fn [rid stage thunk]+ (transforms.instrumentation/with-stage-timing [rid stage]+ (thunk)))})]+ ;; Bridge result-map to exception-based flow for run-cancelable-transform!+ (when-not (= :succeeded (:status result))+ (throw (or (:error result) (ex-info "Transform failed" {:status (:status result)}))))+ result))))+ ;; Post-processing: sync, transform_id, events, secondary indexes (after succeed-started-run!)+ (transforms-base.u/complete-execution! transform+ {:run-id run-id+ :with-stage-timing-fn (fn [rid stage thunk]+ (transforms.instrumentation/with-stage-timing [rid stage]+ (thunk)))})))))(catch Throwable t(if (= :already-running (:error (ex-data t)))(log/warnf "Transform %d is already running" id)diff --git a/test/metabase/server/middleware/trace_test.clj b/test/metabase/server/middleware/trace_test.cljnew file mode 100644index 00000000000..2869c334b79--- /dev/null+++ b/test/metabase/server/middleware/trace_test.clj@@ -0,0 +1,98 @@+(ns metabase.server.middleware.trace-test+ (:require+ [clojure.test :refer [deftest is testing]]+ [metabase.server.middleware.trace :as mw.trace]+ [metabase.tracing.core :as tracing]))++(set! *warn-on-reflection* true)++(defn- call-handler+ "Call wrap-trace middleware with a mock handler, returning the response or exception.+ The mock handler immediately calls respond with {:status 200}."+ [request]+ (let [handler (fn [_req respond _raise] (respond {:status 200}))+ wrapped (mw.trace/wrap-trace handler)+ result (promise)]+ (wrapped request #(deliver result %) #(deliver result %))+ (deref result 1000 :timeout)))++(deftest non-api-requests-skipped-test+ (testing "non-API requests are passed through without tracing"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ ;; Static files, root page, and other non-API paths should be passed through+ (is (= 200 (:status (call-handler {:uri "/app/fonts/lato.woff2" :request-method :get :headers {}}))))+ (is (= 200 (:status (call-handler {:uri "/app/dist/app-main.js" :request-method :get :headers {}}))))+ (is (= 200 (:status (call-handler {:uri "/favicon.ico" :request-method :get :headers {}}))))+ (is (= 200 (:status (call-handler {:uri "/" :request-method :get :headers {}}))))+ ;; API requests should also work (they go through tracing)+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get :headers {}}))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest tracing-disabled-passthrough-test+ (testing "when tracing is disabled, requests are passed through directly"+ (tracing/shutdown-groups!)+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get :headers {}}))))))++(deftest traceparent-parsing-test+ (testing "valid traceparent header is parsed and trace ID is forced"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (let [trace-id "abcdef1234567890abcdef1234567890"+ request {:uri "/api/health"+ :request-method :get+ :headers {"traceparent" (str "00-" trace-id "-1234567890abcdef-01")}}]+ ;; Ensure forced trace ID is clean before test+ (tracing/clear-forced-trace-id!)+ (is (= 200 (:status (call-handler request))))+ ;; After the request, the forced trace ID should be consumed (cleared)+ (is (nil? (tracing/get-and-clear-forced-trace-id!))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest invalid-traceparent-ignored-test+ (testing "invalid traceparent headers are silently ignored"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ ;; Malformed headers should not cause errors+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get+ :headers {"traceparent" "garbage"}}))))+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get+ :headers {"traceparent" ""}}))))+ (is (= 200 (:status (call-handler {:uri "/api/health" :request-method :get+ :headers {"traceparent" "00-short-id-01"}}))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest exception-handling-test+ (testing "exceptions in the handler are propagated via raise"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (let [handler (fn [_req _respond raise] (raise (ex-info "boom" {})))+ wrapped (mw.trace/wrap-trace handler)+ result (promise)]+ (wrapped {:uri "/api/test" :request-method :get :headers {}}+ #(deliver result %)+ #(deliver result %))+ (let [r (deref result 1000 :timeout)]+ (is (instance? clojure.lang.ExceptionInfo r))+ (is (= "boom" (.getMessage ^Exception r)))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest sync-exception-handling-test+ (testing "synchronous exceptions from handler are caught and raised"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (let [handler (fn [_req _respond _raise] (throw (ex-info "sync-boom" {})))+ wrapped (mw.trace/wrap-trace handler)+ result (promise)]+ (wrapped {:uri "/api/test" :request-method :get :headers {}}+ #(deliver result %)+ #(deliver result %))+ (let [r (deref result 1000 :timeout)]+ (is (instance? clojure.lang.ExceptionInfo r))+ (is (= "sync-boom" (.getMessage ^Exception r)))))+ (finally+ (tracing/shutdown-groups!)))))diff --git a/test/metabase/tracing/core_test.clj b/test/metabase/tracing/core_test.cljnew file mode 100644index 00000000000..755d5ff9091--- /dev/null+++ b/test/metabase/tracing/core_test.clj@@ -0,0 +1,258 @@+(ns metabase.tracing.core-test+ (:require+ [clojure.test :refer [deftest is testing]]+ [metabase.tracing.core :as tracing])+ (:import+ (org.apache.logging.log4j ThreadContext)))++(set! *warn-on-reflection* true)++(deftest group-enabled?-disabled-by-default-test+ (testing "group-enabled? returns false when tracing is not initialized"+ ;; By default, enabled-groups atom is nil (tracing not initialized)+ (is (not (tracing/group-enabled? :qp)))+ (is (not (tracing/group-enabled? :sync)))+ (is (not (tracing/group-enabled? :api)))))++(deftest init-enabled-groups!-all-test+ (testing "init-enabled-groups! with 'all' enables all groups"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (is (tracing/group-enabled? :qp))+ (is (tracing/group-enabled? :sync))+ (is (tracing/group-enabled? :api))+ (is (tracing/group-enabled? :db-user))+ (is (tracing/group-enabled? :db-app))+ (is (tracing/group-enabled? :tasks))+ (is (tracing/group-enabled? :events))+ ;; even non-registered groups should be enabled with :all+ (is (tracing/group-enabled? :nonexistent-group))+ (finally+ (tracing/shutdown-groups!)))))++(deftest init-enabled-groups!-selective-test+ (testing "init-enabled-groups! with comma-separated list enables only those groups"+ (try+ (tracing/init-enabled-groups! "qp,api,db-user" "INFO")+ (is (tracing/group-enabled? :qp))+ (is (tracing/group-enabled? :api))+ (is (tracing/group-enabled? :db-user))+ (is (not (tracing/group-enabled? :sync)))+ (is (not (tracing/group-enabled? :tasks)))+ (is (not (tracing/group-enabled? :events)))+ (is (not (tracing/group-enabled? :db-app)))+ (finally+ (tracing/shutdown-groups!)))))++(deftest init-enabled-groups!-whitespace-test+ (testing "init-enabled-groups! handles whitespace in group names"+ (try+ (tracing/init-enabled-groups! " qp , sync " "INFO")+ (is (tracing/group-enabled? :qp))+ (is (tracing/group-enabled? :sync))+ (is (not (tracing/group-enabled? :api)))+ (finally+ (tracing/shutdown-groups!)))))++(deftest shutdown-groups!-test+ (testing "shutdown-groups! disables all groups"+ (tracing/init-enabled-groups! "all" "INFO")+ (is (tracing/group-enabled? :qp))+ (tracing/shutdown-groups!)+ (is (not (tracing/group-enabled? :qp)))))++(deftest register-group!-test+ (testing "register-group! adds groups to the registry"+ (tracing/register-group! :test-group "A test group")+ (is (contains? (tracing/registered-groups) :test-group))+ (is (= "A test group" (:description (:test-group (tracing/registered-groups)))))))++(deftest built-in-groups-registered-test+ (testing "all built-in groups are registered"+ (let [groups (tracing/registered-groups)]+ (is (contains? groups :qp))+ (is (contains? groups :sync))+ (is (contains? groups :tasks))+ (is (contains? groups :api))+ (is (contains? groups :db-user))+ (is (contains? groups :db-app))+ (is (contains? groups :events)))))++(deftest with-span-disabled-executes-body-test+ (testing "with-span executes body when tracing is disabled (no-op wrapper)"+ ;; Tracing not initialized, so group-enabled? returns false+ (tracing/shutdown-groups!)+ (is (= 42 (tracing/with-span :qp "test-span" {} 42)))+ (is (= "hello" (tracing/with-span :qp "test-span" {} "hello")))+ (let [side-effect (atom false)]+ (tracing/with-span :qp "test-span" {}+ (reset! side-effect true))+ (is @side-effect))))++(deftest inject-trace-level-into-mdc-test+ (testing "inject-trace-id-into-mdc! sets trace_level in MDC from cached setting"+ (try+ (tracing/init-enabled-groups! "all" "DEBUG")+ (tracing/inject-trace-id-into-mdc!)+ (is (= "DEBUG" (ThreadContext/get "trace_level")))+ (tracing/clear-trace-id-from-mdc!)+ (is (nil? (ThreadContext/get "trace_level")))+ (finally+ (tracing/shutdown-groups!))))+ (testing "inject-trace-id-into-mdc! does not set trace_level when tracing is not initialized"+ (tracing/shutdown-groups!)+ (tracing/inject-trace-id-into-mdc!)+ (is (nil? (ThreadContext/get "trace_level")))+ (tracing/clear-trace-id-from-mdc!)))++(deftest trace-log-level-values-test+ (testing "each valid log level is correctly cached and injected into MDC"+ (doseq [level ["INFO" "DEBUG" "TRACE"]]+ (testing (str "level=" level)+ (try+ (tracing/init-enabled-groups! "all" level)+ (tracing/inject-trace-id-into-mdc!)+ (is (= level (ThreadContext/get "trace_level")))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!)))))))++(deftest trace-log-level-nil-defaults-to-info-test+ (testing "nil log-level-str defaults to INFO"+ (try+ (tracing/init-enabled-groups! "all" nil)+ (tracing/inject-trace-id-into-mdc!)+ (is (= "INFO" (ThreadContext/get "trace_level")))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!)))))++(deftest shutdown-clears-trace-log-level-test+ (testing "shutdown-groups! clears trace_level so MDC injection becomes a no-op"+ (tracing/init-enabled-groups! "all" "DEBUG")+ (tracing/shutdown-groups!)+ (tracing/inject-trace-id-into-mdc!)+ (is (nil? (ThreadContext/get "trace_level")))+ (tracing/clear-trace-id-from-mdc!)))++(deftest reinit-changes-trace-log-level-test+ (testing "re-initializing with a different log level updates the cached value"+ (try+ (tracing/init-enabled-groups! "all" "DEBUG")+ (tracing/inject-trace-id-into-mdc!)+ (is (= "DEBUG" (ThreadContext/get "trace_level")))+ (tracing/clear-trace-id-from-mdc!)+ ;; re-init with different level+ (tracing/init-enabled-groups! "all" "TRACE")+ (tracing/inject-trace-id-into-mdc!)+ (is (= "TRACE" (ThreadContext/get "trace_level")))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!)))))++;;; ------------------------------------------ Forced Trace ID Tests -------------------------------------------++(deftest force-trace-id-basic-test+ (testing "force-trace-id! sets value, get-and-clear-forced-trace-id! returns and clears it"+ (tracing/force-trace-id! "abcdef1234567890abcdef1234567890")+ (is (= "abcdef1234567890abcdef1234567890" (tracing/get-and-clear-forced-trace-id!)))+ ;; second call returns nil (already consumed)+ (is (nil? (tracing/get-and-clear-forced-trace-id!)))))++(deftest force-trace-id-nil-when-not-set-test+ (testing "get-and-clear-forced-trace-id! returns nil when nothing is forced"+ (tracing/clear-forced-trace-id!)+ (is (nil? (tracing/get-and-clear-forced-trace-id!)))))++(deftest clear-forced-trace-id-test+ (testing "clear-forced-trace-id! removes the value without consuming it"+ (tracing/force-trace-id! "1111111111111111aaaaaaaaaaaaaaaa")+ (tracing/clear-forced-trace-id!)+ (is (nil? (tracing/get-and-clear-forced-trace-id!)))))++(deftest force-trace-id-thread-isolation-test+ (testing "forced trace ID is thread-local — other threads don't see it"+ (tracing/force-trace-id! "aaaa0000bbbb1111cccc2222dddd3333")+ (let [other-thread-result (promise)]+ (.start (Thread. (fn []+ (deliver other-thread-result (tracing/get-and-clear-forced-trace-id!)))))+ (is (nil? (deref other-thread-result 1000 :timeout)))+ ;; still available on this thread+ (is (= "aaaa0000bbbb1111cccc2222dddd3333" (tracing/get-and-clear-forced-trace-id!))))))++(deftest nested-with-span-restores-mdc-test+ (testing "nested with-span restores parent MDC values instead of clearing them"+ (try+ (tracing/init-enabled-groups! "all" "DEBUG")+ ;; Simulate what wrap-trace does: inject MDC for the root span+ (tracing/inject-trace-id-into-mdc!)+ (let [root-trace-id (ThreadContext/get "trace_id")+ root-span-id (ThreadContext/get "span_id")]+ (is (some? root-trace-id) "root trace_id should be set")+ (is (some? root-span-id) "root span_id should be set")+ ;; Run a child with-span (simulates db-app.read-check, etc.)+ (tracing/with-span :db-app "db-app.test-child" {}+ ;; Inside child span, MDC should have values+ (is (some? (ThreadContext/get "trace_id")) "child trace_id should be set")+ (is (some? (ThreadContext/get "span_id")) "child span_id should be set"))+ ;; After child exits, parent MDC should be RESTORED, not cleared+ (is (= root-trace-id (ThreadContext/get "trace_id"))+ "trace_id should be restored to parent value after child span exits")+ (is (= root-span-id (ThreadContext/get "span_id"))+ "span_id should be restored to parent value after child span exits")+ (is (= "DEBUG" (ThreadContext/get "trace_level"))+ "trace_level should still be set after child span exits"))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!))))++ (testing "outermost with-span still clears MDC when no parent values exist"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ ;; No prior MDC set — this with-span is the outermost+ (ThreadContext/remove "trace_id")+ (ThreadContext/remove "span_id")+ (tracing/with-span :qp "qp.test" {}+ (is (some? (ThreadContext/get "trace_id"))))+ ;; After outermost exits, MDC should be cleared+ (is (nil? (ThreadContext/get "trace_id")) "trace_id should be cleared after outermost span")+ (is (nil? (ThreadContext/get "span_id")) "span_id should be cleared after outermost span")+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!)))))++(deftest with-span-pyroscope-root-span-noop-test+ (testing "with-span calls pyroscope set/clear for root spans (no-op without agent)"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (ThreadContext/remove "trace_id")+ (ThreadContext/remove "span_id")+ ;; Root span — no prior MDC context. Pyroscope calls are no-ops but must not throw.+ (is (= :ok (tracing/with-span :tasks "task.test-job" {} :ok)))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!))))++ (testing "with-span skips pyroscope for nested spans (parent MDC already set)"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (ThreadContext/remove "trace_id")+ (ThreadContext/remove "span_id")+ ;; Outer with-span sets MDC → inner with-span sees prev-span-id → skips pyroscope+ (tracing/with-span :tasks "task.outer" {}+ (is (some? (ThreadContext/get "span_id")) "outer span should have MDC")+ (is (= :inner-ok (tracing/with-span :qp "qp.inner" {} :inner-ok))))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!))))++ (testing "rapid root span cycles don't throw or leak pyroscope state"+ (try+ (tracing/init-enabled-groups! "all" "INFO")+ (dotimes [_ 50]+ (ThreadContext/remove "trace_id")+ (ThreadContext/remove "span_id")+ (tracing/with-span :tasks "task.rapid" {} :ok))+ (finally+ (tracing/clear-trace-id-from-mdc!)+ (tracing/shutdown-groups!)))))diff --git a/test/metabase/tracing/pyroscope_test.clj b/test/metabase/tracing/pyroscope_test.cljnew file mode 100644index 00000000000..217071c44aa--- /dev/null+++ b/test/metabase/tracing/pyroscope_test.clj@@ -0,0 +1,32 @@+(ns metabase.tracing.pyroscope-test+ (:require+ [clojure.test :refer [deftest is testing]]+ [metabase.tracing.core :as tracing]))++(set! *warn-on-reflection* true)++(deftest pyroscope-available?-without-agent-test+ (testing "pyroscope-available? returns false when pyroscope.jar is not on the classpath"+ (is (false? (tracing/pyroscope-available?)))))++(deftest set-pyroscope-context!-noop-without-agent-test+ (testing "set-pyroscope-context! is a safe no-op when Pyroscope is not available"+ ;; nil span is safe because the when-let guard prevents any execution+ (is (nil? (tracing/set-pyroscope-context! nil "abcdef0123456789" "api.request")))))++(deftest clear-pyroscope-context!-noop-without-agent-test+ (testing "clear-pyroscope-context! is a safe no-op when Pyroscope is not available"+ (is (nil? (tracing/clear-pyroscope-context!)))))++(deftest set-pyroscope-context!-invalid-span-id-test+ (testing "set-pyroscope-context! handles invalid span IDs gracefully"+ ;; Even if Pyroscope were available, these should not throw+ (is (nil? (tracing/set-pyroscope-context! nil "" "api.request")))+ (is (nil? (tracing/set-pyroscope-context! nil "not-hex" "api.request")))+ (is (nil? (tracing/set-pyroscope-context! nil nil "api.request")))))++(deftest repeated-calls-are-safe-test+ (testing "rapid set/clear cycles don't throw or leak state"+ (dotimes [_ 100]+ (tracing/set-pyroscope-context! nil "abcdef0123456789" "test.span")+ (tracing/clear-pyroscope-context!))))diff --git a/test/metabase/tracing/quartz_test.clj b/test/metabase/tracing/quartz_test.cljnew file mode 100644index 00000000000..615288cfb1a--- /dev/null+++ b/test/metabase/tracing/quartz_test.clj@@ -0,0 +1,258 @@+(ns metabase.tracing.quartz-test+ (:require+ [clojure.test :refer [deftest is testing]]+ [metabase.task.bootstrap :as task.bootstrap]+ [metabase.tracing.core :as tracing]+ [metabase.tracing.quartz :as tracing.quartz])+ (:import+ (java.sql Connection PreparedStatement ResultSet Statement)+ (org.quartz JobDetail JobExecutionContext JobExecutionException JobKey JobListener)))++(set! *warn-on-reflection* true)++;;; ------------------------------------------------ Helpers ----------------------------------------------------------++(defn- mock-prepared-statement+ "Create a mock PreparedStatement that records which methods were called."+ [calls-atom]+ (reify PreparedStatement+ (executeQuery [_]+ (swap! calls-atom conj :executeQuery)+ (reify ResultSet+ (close [_])))+ (executeUpdate [_]+ (swap! calls-atom conj :executeUpdate)+ 1)+ (execute [_]+ (swap! calls-atom conj :execute)+ true)+ (close [_]+ (swap! calls-atom conj :close))))++(defn- mock-statement+ "Create a mock Statement that records which methods were called."+ [calls-atom]+ (reify Statement+ (execute [_ sql]+ (swap! calls-atom conj [:execute sql])+ true)+ (executeQuery [_ sql]+ (swap! calls-atom conj [:executeQuery sql])+ (reify ResultSet+ (close [_])))+ (executeUpdate [_ sql]+ (swap! calls-atom conj [:executeUpdate sql])+ 1)+ (close [_]+ (swap! calls-atom conj [:close-stmt]))))++(defn- mock-connection+ "Create a mock Connection that returns mock PreparedStatements and Statements."+ [stmt-calls-atom]+ (reify Connection+ (prepareStatement [_ _sql]+ (mock-prepared-statement stmt-calls-atom))+ (createStatement [_]+ (mock-statement stmt-calls-atom))+ (rollback [_]+ (swap! stmt-calls-atom conj :rollback))+ (commit [_]+ (swap! stmt-calls-atom conj :commit))+ (close [_])+ (isClosed [_] false)))++(defn- mock-job-execution-context+ "Create a mock JobExecutionContext for testing the listener."+ ^JobExecutionContext [^String job-name]+ (let [job-key (JobKey. job-name)+ job-detail (reify JobDetail (getKey [_] job-key))]+ (reify JobExecutionContext+ (getJobDetail [_] job-detail))))++;;; ---------------------------------------- sql-operation tests ------------------------------------------------------++(deftest sql-operation-test+ (testing "extracts SQL verb from various SQL statements"+ (is (= "SELECT" (#'tracing.quartz/sql-operation "SELECT * FROM QRTZ_TRIGGERS")))+ (is (= "UPDATE" (#'tracing.quartz/sql-operation "UPDATE QRTZ_TRIGGERS SET state = ?")))+ (is (= "INSERT" (#'tracing.quartz/sql-operation "INSERT INTO QRTZ_FIRED_TRIGGERS VALUES (?)")))+ (is (= "DELETE" (#'tracing.quartz/sql-operation "DELETE FROM QRTZ_FIRED_TRIGGERS WHERE id = ?"))))+ (testing "handles edge cases"+ (is (= "SELECT" (#'tracing.quartz/sql-operation " SELECT * FROM t ")))+ (is (= "COMMIT" (#'tracing.quartz/sql-operation "COMMIT")))+ (is (nil? (#'tracing.quartz/sql-operation nil)))))++;;; ----------------------------------------- JDBC proxy tests --------------------------------------------------------++(deftest traced-connection-intercepts-prepareStatement-test+ (testing "traced-connection wraps prepareStatement results"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [calls (atom [])+ conn (mock-connection calls)+ traced (#'tracing.quartz/traced-connection conn)]+ (testing "returned connection implements Connection"+ (is (instance? Connection traced)))+ (testing "prepareStatement returns a PreparedStatement proxy"+ (let [^PreparedStatement stmt (.prepareStatement ^Connection traced "SELECT * FROM QRTZ_TRIGGERS")]+ (is (instance? PreparedStatement stmt))+ (testing "execute methods delegate to the underlying statement"+ (.executeQuery stmt)+ (is (= [:executeQuery] @calls)))+ (testing "executeUpdate also delegates"+ (.executeUpdate stmt)+ (is (= [:executeQuery :executeUpdate] @calls))))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest traced-connection-delegates-non-intercepted-methods-test+ (testing "methods other than prepareStatement pass through unchanged"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [calls (atom [])+ conn (mock-connection calls)+ traced (#'tracing.quartz/traced-connection conn)]+ (testing "close delegates to original"+ (.close ^Connection traced)+ ;; no exception means the method was delegated+ (is true))+ (testing "isClosed delegates to original"+ (is (false? (.isClosed ^Connection traced)))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest traced-connection-intercepts-createStatement-test+ (testing "traced-connection wraps createStatement results"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [calls (atom [])+ conn (mock-connection calls)+ traced (#'tracing.quartz/traced-connection conn)]+ (testing "createStatement returns a Statement proxy"+ (let [^Statement stmt (.createStatement ^Connection traced)]+ (is (instance? Statement stmt))+ (testing "execute with SQL delegates and records SQL"+ (.execute stmt "DISCARD ALL;")+ (is (= [[:execute "DISCARD ALL;"]] @calls)))+ (testing "executeUpdate with SQL delegates"+ (.executeUpdate stmt "UPDATE QRTZ_TRIGGERS SET state = 'WAITING'")+ (is (= [[:execute "DISCARD ALL;"]+ [:executeUpdate "UPDATE QRTZ_TRIGGERS SET state = 'WAITING'"]]+ @calls))))))+ (finally+ (tracing/shutdown-groups!)))))++(deftest traced-connection-intercepts-rollback-test+ (testing "traced-connection wraps rollback in a span"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [calls (atom [])+ conn (mock-connection calls)+ traced (#'tracing.quartz/traced-connection conn)]+ (.rollback ^Connection traced)+ (is (= [:rollback] @calls)+ "rollback should delegate to the underlying connection"))+ (finally+ (tracing/shutdown-groups!)))))++(deftest traced-connection-intercepts-commit-test+ (testing "traced-connection wraps commit in a span"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [calls (atom [])+ conn (mock-connection calls)+ traced (#'tracing.quartz/traced-connection conn)]+ (.commit ^Connection traced)+ (is (= [:commit] @calls)+ "commit should delegate to the underlying connection"))+ (finally+ (tracing/shutdown-groups!)))))++(deftest connection-interceptor-respects-group-enabled-test+ (testing "when :quartz group is disabled, returns unwrapped connection"+ (tracing/shutdown-groups!)+ (let [calls (atom [])+ conn (mock-connection calls)+ result (#'tracing.quartz/connection-interceptor conn)]+ (is (identical? conn result)+ "should return the exact same connection object when tracing disabled")))+ (testing "when :quartz group is enabled, returns wrapped connection"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [calls (atom [])+ conn (mock-connection calls)+ result (#'tracing.quartz/connection-interceptor conn)]+ (is (not (identical? conn result))+ "should return a different (wrapped) connection when tracing enabled")+ (is (instance? Connection result)))+ (finally+ (tracing/shutdown-groups!)))))++;;; ---------------------------------------- JobListener tests --------------------------------------------------------++(deftest job-listener-name-test+ (testing "listener has a descriptive name"+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)]+ (is (= "metabase.tracing.quartz/quartz-tracing-listener" (.getName listener))))))++(deftest job-listener-disabled-is-noop-test+ (testing "when :quartz group is disabled, jobToBeExecuted is a no-op"+ (tracing/shutdown-groups!)+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)+ ctx (mock-job-execution-context "TestJob")]+ (.jobToBeExecuted listener ctx)+ ;; No span should be in ThreadLocal+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state)))+ ;; jobWasExecuted should also be safe to call (no-op since no state)+ (.jobWasExecuted listener ctx nil)+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state))))))++(deftest job-listener-enabled-creates-and-ends-span-test+ (testing "when :quartz group is enabled, jobToBeExecuted creates a span"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)+ ctx (mock-job-execution-context "TestSessionCleanup")]+ (.jobToBeExecuted listener ctx)+ (testing "span state is stored in ThreadLocal"+ (let [state (.get ^ThreadLocal @#'tracing.quartz/listener-state)]+ (is (some? state))+ (is (contains? state :span))+ (is (contains? state :scope))))+ (testing "jobWasExecuted cleans up ThreadLocal"+ (.jobWasExecuted listener ctx nil)+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state)))))+ (finally+ ;; Safety: clean up any lingering state+ (.remove ^ThreadLocal @#'tracing.quartz/listener-state)+ (tracing/shutdown-groups!)))))++(deftest job-listener-records-exception-test+ (testing "when job fails, exception is recorded on the span"+ (try+ (tracing/init-enabled-groups! "quartz" "INFO")+ (let [^JobListener listener (#'tracing.quartz/create-tracing-job-listener)+ ctx (mock-job-execution-context "FailingJob")+ exception (JobExecutionException. "something broke")]+ (.jobToBeExecuted listener ctx)+ (is (some? (.get ^ThreadLocal @#'tracing.quartz/listener-state)))+ ;; Should not throw — exception is recorded on span, not re-thrown+ (.jobWasExecuted listener ctx exception)+ (is (nil? (.get ^ThreadLocal @#'tracing.quartz/listener-state))))+ (finally+ (.remove ^ThreadLocal @#'tracing.quartz/listener-state)+ (tracing/shutdown-groups!)))))++;;; ----------------------------------- bootstrap interceptor tests ---------------------------------------------------++(deftest set-connection-interceptor-test+ (testing "set-connection-interceptor! installs and removes interceptors"+ (let [the-atom @#'task.bootstrap/connection-interceptor+ original @the-atom]+ (try+ (task.bootstrap/set-connection-interceptor! identity)+ (is (some? @the-atom))+ (task.bootstrap/set-connection-interceptor! nil)+ (is (nil? @the-atom))+ (finally+ (reset! the-atom original))))))