Skip to content

SigNoz is an open-source, OpenTelemetry-native observability platform for your team and their AI agents. Get logs, metrics, and traces in one tool with features like APM, distributed tracing, log management, infra monitoring, etc. Combined with SigNoz MCP and a native AI teammate (in SigNoz Cloud) it helps you build more resilient apps.

feat(prometheus): add clickhouseprometheusv2 native read path (#12323)

Srikanth Chekuri052255abf51e
diff --git a/docs/contributing/prometheus.md b/docs/contributing/prometheus.md
new file mode 100644
index 0000000000..ffaf59ef69
--- /dev/null
+++ b/docs/contributing/prometheus.md
@@ -0,0 +1,123 @@
+# PromQL Serving — clickhouseprometheusv2
+
+This document is the subsystem context for `pkg/prometheus/clickhouseprometheusv2`,
+the second-generation ClickHouse-backed Prometheus provider. It explains why the
+package exists, the correctness constraints that shaped it, and how each fetch
+reduction is proven not to change results. Any change to the provider must keep
+these invariants; if a change would violate one, it must be flagged and
+discussed.
+
+---
+
+## Why a second provider
+
+The v1 provider (`pkg/prometheus/clickhouseprometheus`) serves the promql engine
+through the remote-read protobuf adapter: every raw sample of a query's union
+window is fetched, serialized, and handed to the engine. The cost is a function
+of ingested data, not of the question asked — which is how a dashboard of PromQL
+panels can take an instance down.
+
+In v2 the stock promql engine evaluates over a native `storage.Querier`: no
+translation layer, per-selector fetch windows, and fetch reductions that are
+provably invisible to the engine.
+
+**The core constraint: every reduction either preserves engine semantics exactly
+or is not performed.** A PromQL result that differs from upstream Prometheus is
+a lost user. The conformance suite
+(`tests/integration/tests/promqlconformance/`) replays Prometheus' own test
+corpus against both providers and is the arbiter.
+
+---
+
+## Series lookup
+
+Matchers resolve to series once per selector (`selectSeries`) against the series
+tables, which hold one row per (fingerprint, bucket) at 1h/6h/1d/1w
+granularities. Table selection and window rounding delegate to the shared
+metrics schema package (`pkg/telemetryschema/metricstelemetryschema`); the
+window start rounds down to the bucket boundary so a window beginning mid-bucket
+still matches the bucket's row.
+
+How matchers become SQL is documented at `applySeriesConditions`. The rules that
+carry semantics:
+
+- `__name__` matchers (all four types) translate to the `metric_name` column.
+- Every other matcher becomes a `JSONExtractString` condition on the labels
+ column. An equality matcher against `""` matches series *without* the label,
+ mirroring PromQL, because `JSONExtractString` returns `""` for missing keys.
+- Regexes are anchored (`^(?:...)$`) before they reach `match()`: PromQL
+ matchers match the whole value, ClickHouse `match()` searches for a
+ substring.
+- The series-lookup upper bound is inclusive (`unix_milli <= end`) because the
+ exporter floors registration rows to the bucket start: a series first
+ registered in the bucket beginning exactly at `end` would otherwise be
+ invisible while its samples are in range.
+
+Empty-valued labels come off at this boundary: an empty value means "label
+absent" in Prometheus, but stored attribute JSON can carry them.
+
+---
+
+## Sample fetch
+
+Samples are fetched per selector using the engine's per-selector hints, not the
+query-wide union window — `foo / foo offset 1d` reads two narrow windows
+instead of the widest one twice.
+
+**Last-sample-per-step reduction.** Instant selectors of subquery-free queries
+fetch only the last sample per step bucket. The engine resolves an instant
+selector at each grid timestamp `t` to the latest sample in the left-open
+lookback window `(t − lookback, t]`. Buckets anchor at the selector's first
+evaluation timestamp — recovered from the hints as
+`hints.Start + lookback − 1ms`, the inverse of how the engine derives
+`hints.Start` — so bucket boundaries coincide with evaluation timestamps, and a
+non-final sample of a bucket can never be the latest sample in
+`(t − lookback, t]` for any grid `t`. Real timestamps are preserved, so the
+engine's own lookback and staleness handling stay exact.
+
+Range selectors always fetch raw — every sample feeds the range function. The
+subquery-free proof travels in the context as `prometheus.QueryTraits`, because
+subquery selectors evaluate at the subquery's step while the hints carry the
+top-level step; call sites that do not attach traits get the conservative raw
+fetch.
+
+**Row assembly** maps stale flags to the engine's `StaleNaN` and merges series
+with identical label sets (`sortAndMerge`) — the engine assumes storages never
+emit duplicates. Duplicate timestamps pass through as stored: uniqueness is
+ingest's job, and v1 feeds them to the engine as-is over the same data.
+
+**The fingerprint filter is a shard-local semi-join.** The samples query
+restricts to the matched series by re-running the series predicates as an
+`IN (SELECT fingerprint FROM <local series table> ...)` subquery, not a GLOBAL
+broadcast of the matched set. ClickHouse materializes the subquery's set per
+shard before the scan, so it still engages the fingerprint primary-key column.
+Because the subquery re-executes the predicates after the lookup ran, it can
+match series registered in between; sample rows whose fingerprint the lookup
+never saw are skipped — the lookup is the read snapshot.
+
+---
+
+## Sharding
+
+`samples_v4` and `time_series_v4` (and all their rollups) shard on the same key
+— `cityHash64(env, temporality, metric_name, fingerprint)` — so a series'
+samples and catalog rows live on the same shard. The semi-join above exploits
+that: each shard filters by its own series rows, which are exactly the series
+of that shard's samples.
+
+The temporality filter on every samples statement
+(`temporality IN ['Cumulative', 'Unspecified']`) is a semantic no-op — the
+matched fingerprints already come from those temporalities — that engages the
+leading samples primary-key column.
+
+Delta-temporality series stay invisible to PromQL exactly as they are in v1:
+the rollout gate is parity with v1, and a Delta stream fed to `rate()`
+as-if-cumulative would be wrong, not just new.
+
+---
+
+## Observability
+
+Every statement carries a `log_comment` with
+`code.namespace=clickhouse-prometheus-v2` and `code.function.name` naming the
+call site, so this provider's work is attributable in `system.query_log`.
diff --git a/pkg/prometheus/clickhouseprometheusv2/capture.go b/pkg/prometheus/clickhouseprometheusv2/capture.go
new file mode 100644
index 0000000000..118716841b
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/capture.go
@@ -0,0 +1,85 @@
+package clickhouseprometheusv2
+
+import (
+ "context"
+ "sync"
+
+ "github.com/SigNoz/signoz/pkg/prometheus"
+ "github.com/prometheus/prometheus/model/labels"
+ "github.com/prometheus/prometheus/storage"
+ "github.com/prometheus/prometheus/util/annotations"
+)
+
+// statementRecorder collects the statements a PromQL evaluation would run.
+// Safe for concurrent use: the engine may Select selectors concurrently.
+type statementRecorder struct {
+ mu sync.Mutex
+ statements []prometheus.CapturedStatement
+}
+
+func (r *statementRecorder) record(query string, args []any) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ r.statements = append(r.statements, prometheus.CapturedStatement{Query: query, Args: args})
+}
+
+func (r *statementRecorder) Statements() []prometheus.CapturedStatement {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ out := make([]prometheus.CapturedStatement, len(r.statements))
+ copy(out, r.statements)
+ return out
+}
+
+type captureQueryable struct {
+ client *client
+ recorder *statementRecorder
+}
+
+func (c *captureQueryable) Querier(mint, maxt int64) (storage.Querier, error) {
+ return &captureQuerier{
+ querier: querier{mint: mint, maxt: maxt, client: c.client},
+ recorder: c.recorder,
+ }, nil
+}
+
+// captureQuerier builds the same SQL as the live querier but records it and
+// returns no data. The fingerprint filter always takes the subquery form:
+// without executing the series lookup, the inline literal set is unknown.
+type captureQuerier struct {
+ querier
+ recorder *statementRecorder
+}
+
+func (c *captureQuerier) Select(ctx context.Context, _ bool, hints *storage.SelectHints, matchers ...*labels.Matcher) storage.SeriesSet {
+
+ start, end := c.window(hints)
+
+ samplesQuery, args, err := buildSamplesQuery(start, end, metricNamesFromMatchers(matchers), matchers, c.lastSamplePerStepFor(ctx, hints))
+ if err != nil {
+ return storage.ErrSeriesSet(err)
+ }
+ c.recorder.record(samplesQuery, args)
+
+ return storage.EmptySeriesSet()
+}
+
+func (c *captureQuerier) LabelValues(context.Context, string, *storage.LabelHints, ...*labels.Matcher) ([]string, annotations.Annotations, error) {
+ return nil, nil, nil
+}
+
+func (c *captureQuerier) LabelNames(context.Context, *storage.LabelHints, ...*labels.Matcher) ([]string, annotations.Annotations, error) {
+ return nil, nil, nil
+}
+
+// metricNamesFromMatchers extracts the statically known metric name, if any.
+// The live path derives names from the matched series; the capture path has
+// no execution results, so only a __name__ equality contributes.
+func metricNamesFromMatchers(matchers []*labels.Matcher) []string {
+ for _, m := range matchers {
+ if m.Name == metricNameLabel && m.Type == labels.MatchEqual && m.Value != "" {
+ return []string{m.Value}
+ }
+ }
+ return nil
+}
diff --git a/pkg/prometheus/clickhouseprometheusv2/client.go b/pkg/prometheus/clickhouseprometheusv2/client.go
new file mode 100644
index 0000000000..b1db85ed57
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/client.go
@@ -0,0 +1,180 @@
+package clickhouseprometheusv2
+
+import (
+ "context"
+ "encoding/json"
+ "log/slog"
+ "math"
+ "slices"
+
+ "github.com/SigNoz/signoz/pkg/factory"
+ "github.com/SigNoz/signoz/pkg/prometheus"
+ "github.com/SigNoz/signoz/pkg/telemetrystore"
+ "github.com/SigNoz/signoz/pkg/types/ctxtypes"
+ "github.com/SigNoz/signoz/pkg/types/instrumentationtypes"
+ "github.com/SigNoz/signoz/pkg/types/telemetrytypes"
+ "github.com/prometheus/prometheus/model/labels"
+ promValue "github.com/prometheus/prometheus/model/value"
+)
+
+// seriesLookup holds a series lookup's result: matched fingerprints with
+// their labels, and the distinct metric names seen on them.
+type seriesLookup struct {
+ fingerprints map[uint64]labels.Labels
+ metricNames []string
+}
+
+// client executes the series, samples and raw queries against ClickHouse.
+type client struct {
+ settings factory.ScopedProviderSettings
+ telemetryStore telemetrystore.TelemetryStore
+ lookbackMs int64
+}
+
+func newClient(settings factory.ScopedProviderSettings, telemetryStore telemetrystore.TelemetryStore, cfg prometheus.Config) *client {
+ lookback := cfg.LookbackDelta
+ if lookback <= 0 {
+ // Mirror the engine: promql defaults an unset lookback to 5m.
+ lookback = defaultLookbackDelta
+ }
+ return &client{
+ settings: settings,
+ telemetryStore: telemetryStore,
+ lookbackMs: lookback.Milliseconds(),
+ }
+}
+
+func (c *client) withContext(ctx context.Context, functionName string) context.Context {
+ return ctxtypes.NewContextWithCommentVals(ctx, map[string]string{
+ instrumentationtypes.TelemetrySignal: telemetrytypes.SignalMetrics.StringValue(),
+ instrumentationtypes.CodeNamespace: "clickhouse-prometheus-v2",
+ instrumentationtypes.CodeFunctionName: functionName,
+ })
+}
+
+func (c *client) selectSeries(ctx context.Context, query string, args []any) (*seriesLookup, error) {
+ ctx = c.withContext(ctx, "selectSeries")
+ rows, err := c.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+
+ lookup := &seriesLookup{fingerprints: make(map[uint64]labels.Labels)}
+ names := make(map[string]struct{})
+
+ var fingerprint uint64
+ var labelsJSON string
+ for rows.Next() {
+ if err := rows.Scan(&fingerprint, &labelsJSON); err != nil {
+ return nil, err
+ }
+ lset, err := unmarshalLabels(labelsJSON)
+ if err != nil {
+ return nil, err
+ }
+ lookup.fingerprints[fingerprint] = lset
+ if name := lset.Get(metricNameLabel); name != "" {
+ names[name] = struct{}{}
+ }
+ }
+ if err := rows.Err(); err != nil {
+ return nil, err
+ }
+
+ for name := range names {
+ lookup.metricNames = append(lookup.metricNames, name)
+ }
+ slices.Sort(lookup.metricNames)
+
+ return lookup, nil
+}
+
+// unmarshalLabels parses the labels JSON column, dropping empty-valued
+// labels: empty means "absent" in Prometheus, but stored attribute JSON can
+// carry them.
+func unmarshalLabels(s string) (labels.Labels, error) {
+ m := make(map[string]string)
+ if err := json.Unmarshal([]byte(s), &m); err != nil {
+ return labels.EmptyLabels(), err
+ }
+ builder := labels.NewScratchBuilder(len(m))
+ for k, v := range m {
+ if v == "" {
+ continue
+ }
+ builder.Add(k, v)
+ }
+ builder.Sort()
+ return builder.Labels(), nil
+}
+
+// selectSamples assembles per-series sample slices from a samples query (raw
+// or last-sample-per-step; same column shape), whose rows arrive ordered by
+// (fingerprint, unix_milli). Fingerprints missing from the lookup are skipped:
+// the samples query's semi-join re-runs the series predicates and can match
+// series registered after the lookup ran — the lookup is the read snapshot.
+// Stale flags map to the engine's StaleNaN. Duplicate
+// timestamps pass through: uniqueness is ingest's job, and v1 feeds them to
+// the engine as-is over the same dirty data.
+func (c *client) selectSamples(ctx context.Context, query string, args []any, lookup *seriesLookup) ([]*series, error) {
+ ctx = c.withContext(ctx, "selectSamples")
+ rows, err := c.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+
+ var (
+ result []*series
+ current *series
+ fingerprint uint64
+ prevFp uint64
+ timestampMs int64
+ val float64
+ flags uint32
+ first = true
+ haveCurrent bool
+ staleMarker = math.Float64frombits(promValue.StaleNaN)
+ unknownCount int
+ )
+
+ for rows.Next() {
+ if err := rows.Scan(&fingerprint, &timestampMs, &val, &flags); err != nil {
+ return nil, err
+ }
+
+ if first || fingerprint != prevFp {
+ first = false
+ prevFp = fingerprint
+ lset, ok := lookup.fingerprints[fingerprint]
+ if !ok {
+ unknownCount++
+ haveCurrent = false
+ continue
+ }
+ current = &series{lset: lset}
+ result = append(result, current)
+ haveCurrent = true
+ }
+ if !haveCurrent {
+ continue
+ }
+
+ if flags&1 == 1 {
+ val = staleMarker
+ }
+ current.ts = append(current.ts, timestampMs)
+ current.vs = append(current.vs, val)
+ }
+ if err := rows.Err(); err != nil {
+ return nil, err
+ }
+
+ if unknownCount > 0 {
+ c.settings.Logger().DebugContext(ctx, "skipped samples of fingerprints missing from series lookup",
+ slog.Int("unknown_fingerprints", unknownCount))
+ }
+
+ return result, nil
+}
diff --git a/pkg/prometheus/clickhouseprometheusv2/provider.go b/pkg/prometheus/clickhouseprometheusv2/provider.go
new file mode 100644
index 0000000000..e1fbb42f40
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/provider.go
@@ -0,0 +1,71 @@
+package clickhouseprometheusv2
+
+import (
+ "context"
+
+ "github.com/SigNoz/signoz/pkg/factory"
+ "github.com/SigNoz/signoz/pkg/prometheus"
+ "github.com/SigNoz/signoz/pkg/telemetrystore"
+ "github.com/prometheus/prometheus/storage"
+)
+
+// provider ties the package together: its own engine and parser, and the
+// ClickHouse client behind the native storage.Querier. It stays unexported:
+// callers hold the prometheus.Prometheus interface, which is the boundary
+// between the two provider implementations.
+type provider struct {
+ settings factory.ScopedProviderSettings
+ engine *prometheus.Engine
+ parser prometheus.Parser
+ client *client
+}
+
+var (
+ _ prometheus.Prometheus = (*provider)(nil)
+ _ prometheus.StatementCapturer = (*provider)(nil)
+)
+
+func NewFactory(telemetryStore telemetrystore.TelemetryStore) factory.ProviderFactory[prometheus.Prometheus, prometheus.Config] {
+ return factory.NewProviderFactory(factory.MustNewName("clickhousev2"), func(ctx context.Context, providerSettings factory.ProviderSettings, config prometheus.Config) (prometheus.Prometheus, error) {
+ return New(ctx, providerSettings, config, telemetryStore)
+ })
+}
+
+func New(_ context.Context, providerSettings factory.ProviderSettings, config prometheus.Config, telemetryStore telemetrystore.TelemetryStore) (prometheus.Prometheus, error) {
+ settings := factory.NewScopedProviderSettings(providerSettings, "github.com/SigNoz/signoz/pkg/prometheus/clickhouseprometheusv2")
+
+ engine := prometheus.NewEngine(settings.Logger(), config)
+ parser := prometheus.NewParser()
+ client := newClient(settings, telemetryStore, config)
+
+ return &provider{
+ settings: settings,
+ engine: engine,
+ parser: parser,
+ client: client,
+ }, nil
+}
+
+func (p *provider) Engine() *prometheus.Engine {
+ return p.engine
+}
+
+func (p *provider) Parser() prometheus.Parser {
+ return p.parser
+}
+
+func (p *provider) Storage() storage.Queryable {
+ return p
+}
+
+func (p *provider) Querier(mint, maxt int64) (storage.Querier, error) {
+ return &querier{mint: mint, maxt: maxt, client: p.client}, nil
+}
+
+// CapturingStorage implements prometheus.StatementCapturer: a storage that
+// records each selector's SQL without executing it, for the preview path.
+// A fresh recorder per call keeps concurrent dry-runs isolated.
+func (p *provider) CapturingStorage() (storage.Queryable, prometheus.StatementRecorder) {
+ recorder := &statementRecorder{}
+ return &captureQueryable{client: p.client, recorder: recorder}, recorder
+}
diff --git a/pkg/prometheus/clickhouseprometheusv2/querier.go b/pkg/prometheus/clickhouseprometheusv2/querier.go
new file mode 100644
index 0000000000..358bf51ead
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/querier.go
@@ -0,0 +1,171 @@
+package clickhouseprometheusv2
+
+import (
+ "context"
+ "fmt"
+ "slices"
+ "time"
+
+ "github.com/SigNoz/signoz/pkg/prometheus"
+ "github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
+ "github.com/huandu/go-sqlbuilder"
+ "github.com/prometheus/prometheus/model/labels"
+ "github.com/prometheus/prometheus/storage"
+ "github.com/prometheus/prometheus/util/annotations"
+)
+
+// defaultLookbackDelta mirrors promql's default when the config leaves the
+// lookback unset; the engine and the storage must agree on it for
+// last-sample-per-step bucket anchoring.
+const defaultLookbackDelta = 5 * time.Minute
+
+// querier is a native storage.Querier over ClickHouse: Select builds SQL
+// directly from the matchers and hints, with no remote-read protobuf layer.
+type querier struct {
+ mint, maxt int64
+ client *client
+}
+
+var _ storage.Querier = (*querier)(nil)
+
+func (q *querier) Select(ctx context.Context, sortSeries bool, hints *storage.SelectHints, matchers ...*labels.Matcher) storage.SeriesSet {
+ start, end := q.window(hints)
+
+ seriesQuery, seriesArgs, err := buildSeriesQuery(start, end, matchers)
+ if err != nil {
+ return storage.ErrSeriesSet(err)
+ }
+ lookup, err := q.client.selectSeries(ctx, seriesQuery, seriesArgs)
+ if err != nil {
+ return storage.ErrSeriesSet(err)
+ }
+ if len(lookup.fingerprints) == 0 {
+ return storage.EmptySeriesSet()
+ }
+
+ list, err := q.fetchSamples(ctx, start, end, matchers, lookup, q.lastSamplePerStepFor(ctx, hints))
+ if err != nil {
+ return storage.ErrSeriesSet(err)
+ }
+
+ // The engine assumes storages never emit duplicate label sets.
+ list = sortAndMerge(list)
+ return newSeriesSet(list)
+}
+
+func (q *querier) LabelValues(ctx context.Context, name string, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
+ sb := sqlbuilder.NewSelectBuilder()
+ if name == metricNameLabel {
+ sb.Select("DISTINCT metric_name AS value")
+ } else {
+ sb.Select(fmt.Sprintf("DISTINCT JSONExtractString(labels, %s) AS value", sb.Var(name)))
+ }
+ adjustedStart, _, table, _ := metricstelemetryschema.WhichTSTableToUse(uint64(q.mint), uint64(q.maxt), false, nil)
+ sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, table))
+ if err := applySeriesConditions(sb, int64(adjustedStart), q.maxt, matchers); err != nil {
+ return nil, nil, err
+ }
+ sb.Where("value != ''")
+ if hints != nil && hints.Limit > 0 {
+ sb.Limit(hints.Limit)
+ }
+
+ query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
+ values, err := q.selectStrings(ctx, "LabelValues", query, args)
+ if err != nil {
+ return nil, nil, err
+ }
+ slices.Sort(values)
+ return values, nil, nil
+}
+
+func (q *querier) LabelNames(ctx context.Context, hints *storage.LabelHints, matchers ...*labels.Matcher) ([]string, annotations.Annotations, error) {
+ sb := sqlbuilder.NewSelectBuilder()
+ sb.Select("DISTINCT arrayJoin(JSONExtractKeys(labels)) AS name")
+ adjustedStart, _, table, _ := metricstelemetryschema.WhichTSTableToUse(uint64(q.mint), uint64(q.maxt), false, nil)
+ sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, table))
+ if err := applySeriesConditions(sb, int64(adjustedStart), q.maxt, matchers); err != nil {
+ return nil, nil, err
+ }
+ if hints != nil && hints.Limit > 0 {
+ sb.Limit(hints.Limit)
+ }
+
+ query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
+ names, err := q.selectStrings(ctx, "LabelNames", query, args)
+ if err != nil {
+ return nil, nil, err
+ }
+ slices.Sort(names)
+ return names, nil, nil
+}
+
+func (q *querier) Close() error {
+ return nil
+}
+
+// window returns the per-selector fetch window from the hints (already
+// adjusted for offset, @, range and lookback); mint/maxt span the union of
+// all selectors and are the fallback.
+func (q *querier) window(hints *storage.SelectHints) (int64, int64) {
+ if hints != nil && hints.Start != 0 && hints.End != 0 && hints.Start <= hints.End {
+ return hints.Start, hints.End
+ }
+ return q.mint, q.maxt
+}
+
+// lastSamplePerStepFor decides whether the fetch can keep only the last
+// sample per step bucket (see lastSamplePerStep). Only instant selectors
+// (hints.Range == 0) of subquery-free queries qualify: range selectors need
+// every raw sample, and subquery selectors evaluate at the subquery's own
+// step while hints carry the top-level step — the subquery-free proof
+// arrives as QueryTraits in the context. The first evaluation timestamp is
+// recovered as hints.Start + lookback - 1ms, inverting how the engine
+// derives hints.Start.
+func (q *querier) lastSamplePerStepFor(ctx context.Context, hints *storage.SelectHints) *lastSamplePerStep {
+ if hints == nil || hints.Range != 0 || hints.Start <= 0 {
+ return nil
+ }
+ traits, ok := prometheus.QueryTraitsFromContext(ctx)
+ if !ok || !traits.SubqueryFree {
+ return nil
+ }
+ firstEval := hints.Start + q.client.lookbackMs - 1
+ if firstEval > hints.End {
+ // Defensive: never anchor a bucket past the window.
+ firstEval = hints.End
+ }
+ return &lastSamplePerStep{firstEvalMs: firstEval, stepMs: hints.Step}
+}
+
+// fetchSamples runs the samples query for the matched series (see
+// buildSamplesQuery).
+func (q *querier) fetchSamples(ctx context.Context, start, end int64, matchers []*labels.Matcher, lookup *seriesLookup, lastPerStep *lastSamplePerStep) ([]*series, error) {
+ query, args, err := buildSamplesQuery(start, end, lookup.metricNames, matchers, lastPerStep)
+ if err != nil {
+ return nil, err
+ }
+ return q.client.selectSamples(ctx, query, args, lookup)
+}
+
+func (q *querier) selectStrings(ctx context.Context, fn, query string, args []any) ([]string, error) {
+ ctx = q.client.withContext(ctx, fn)
+ rows, err := q.client.telemetryStore.ClickhouseDB().Query(ctx, query, args...)
+ if err != nil {
+ return nil, err
+ }
+ defer rows.Close()
+
+ var out []string
+ var v string
+ for rows.Next() {
+ if err := rows.Scan(&v); err != nil {
+ return nil, err
+ }
+ out = append(out, v)
+ }
+ if err := rows.Err(); err != nil {
+ return nil, err
+ }
+ return out, nil
+}
diff --git a/pkg/prometheus/clickhouseprometheusv2/seriesset.go b/pkg/prometheus/clickhouseprometheusv2/seriesset.go
new file mode 100644
index 0000000000..04d3f1ec23
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/seriesset.go
@@ -0,0 +1,184 @@
+package clickhouseprometheusv2
+
+import (
+ "sort"
+
+ "github.com/prometheus/prometheus/model/histogram"
+ "github.com/prometheus/prometheus/model/labels"
+ "github.com/prometheus/prometheus/storage"
+ "github.com/prometheus/prometheus/tsdb/chunkenc"
+ "github.com/prometheus/prometheus/util/annotations"
+)
+
+// series is one time series with samples as parallel slices ordered by
+// timestamp. Deliberately not storage.NewListSeries: that boxes every sample
+// as an interface value, a per-sample allocation this fetch path exists to
+// avoid.
+type series struct {
+ lset labels.Labels
+ ts []int64
+ vs []float64
+}
+
+var _ storage.Series = (*series)(nil)
+
+func (s *series) Labels() labels.Labels {
+ return s.lset
+}
+
+func (s *series) Iterator(it chunkenc.Iterator) chunkenc.Iterator {
+ if fit, ok := it.(*floatIterator); ok {
+ fit.reset(s)
+ return fit
+ }
+ fit := &floatIterator{}
+ fit.reset(s)
+ return fit
+}
+
+// floatIterator implements chunkenc.Iterator over a series' sample slices.
+type floatIterator struct {
+ s *series
+ i int
+}
+
+var _ chunkenc.Iterator = (*floatIterator)(nil)
+
+func (it *floatIterator) reset(s *series) {
+ it.s = s
+ it.i = -1
+}
+
+func (it *floatIterator) Next() chunkenc.ValueType {
+ it.i++
+ if it.i >= len(it.s.ts) {
+ return chunkenc.ValNone
+ }
+ return chunkenc.ValFloat
+}
+
+func (it *floatIterator) Seek(t int64) chunkenc.ValueType { //nolint:govet // stdmethods flags io.Seeker; this is chunkenc.Iterator's Seek
+ if it.i < 0 {
+ it.i = 0
+ }
+ if it.i >= len(it.s.ts) {
+ return chunkenc.ValNone
+ }
+ // The current position, once valid, must not move backwards.
+ if it.s.ts[it.i] >= t {
+ return chunkenc.ValFloat
+ }
+ it.i += sort.Search(len(it.s.ts)-it.i, func(j int) bool {
+ return it.s.ts[it.i+j] >= t
+ })
+ if it.i >= len(it.s.ts) {
+ return chunkenc.ValNone
+ }
+ return chunkenc.ValFloat
+}
+
+func (it *floatIterator) At() (int64, float64) {
+ return it.s.ts[it.i], it.s.vs[it.i]
+}
+
+func (it *floatIterator) AtHistogram(*histogram.Histogram) (int64, *histogram.Histogram) {
+ return 0, nil
+}
+
+func (it *floatIterator) AtFloatHistogram(*histogram.FloatHistogram) (int64, *histogram.FloatHistogram) {
+ return 0, nil
+}
+
+func (it *floatIterator) AtT() int64 {
+ return it.s.ts[it.i]
+}
+
+// AtST returns the current start timestamp; not tracked by this storage.
+func (it *floatIterator) AtST() int64 {
+ return 0
+}
+
+func (it *floatIterator) Err() error {
+ return nil
+}
+
+// seriesSet iterates a fully materialized, label-sorted list of series.
+type seriesSet struct {
+ series []*series
+ i int
+}
+
+var _ storage.SeriesSet = (*seriesSet)(nil)
+
+func newSeriesSet(list []*series) *seriesSet {
+ return &seriesSet{series: list, i: -1}
+}
+
+func (s *seriesSet) Next() bool {
+ s.i++
+ return s.i < len(s.series)
+}
+
+func (s *seriesSet) At() storage.Series {
+ return s.series[s.i]
+}
+
+func (s *seriesSet) Err() error {
+ return nil
+}
+
+func (s *seriesSet) Warnings() annotations.Annotations {
+ return nil
+}
+
+// sortAndMerge orders series by label set and merges identical label sets
+// by timestamp (first sample wins ties): distinct fingerprints can carry
+// identical label sets, and the engine assumes storages never emit
+// duplicates.
+func sortAndMerge(list []*series) []*series {
+ if len(list) < 2 {
+ return list
+ }
+ sort.Slice(list, func(i, j int) bool {
+ return labels.Compare(list[i].lset, list[j].lset) < 0
+ })
+ out := list[:1]
+ for _, s := range list[1:] {
+ last := out[len(out)-1]
+ if labels.Compare(last.lset, s.lset) != 0 {
+ out = append(out, s)
+ continue
+ }
+ merged := mergeSamples(last, s)
+ out[len(out)-1] = merged
+ }
+ return out
+}
+
+func mergeSamples(a, b *series) *series {
+ ts := make([]int64, 0, len(a.ts)+len(b.ts))
+ vs := make([]float64, 0, len(a.ts)+len(b.ts))
+ i, j := 0, 0
+ for i < len(a.ts) && j < len(b.ts) {
+ switch {
+ case a.ts[i] < b.ts[j]:
+ ts = append(ts, a.ts[i])
+ vs = append(vs, a.vs[i])
+ i++
+ case a.ts[i] > b.ts[j]:
+ ts = append(ts, b.ts[j])
+ vs = append(vs, b.vs[j])
+ j++
+ default:
+ ts = append(ts, a.ts[i])
+ vs = append(vs, a.vs[i])
+ i++
+ j++
+ }
+ }
+ ts = append(ts, a.ts[i:]...)
+ vs = append(vs, a.vs[i:]...)
+ ts = append(ts, b.ts[j:]...)
+ vs = append(vs, b.vs[j:]...)
+ return &series{lset: a.lset, ts: ts, vs: vs}
+}
diff --git a/pkg/prometheus/clickhouseprometheusv2/sql.go b/pkg/prometheus/clickhouseprometheusv2/sql.go
new file mode 100644
index 0000000000..4e065cd335
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/sql.go
@@ -0,0 +1,172 @@
+package clickhouseprometheusv2
+
+import (
+ "fmt"
+
+ "github.com/SigNoz/signoz/pkg/errors"
+ "github.com/SigNoz/signoz/pkg/telemetryschema/metricstelemetryschema"
+ "github.com/huandu/go-sqlbuilder"
+ "github.com/prometheus/prometheus/model/labels"
+)
+
+// metricNameLabel is the reserved PromQL label holding the metric name.
+const metricNameLabel = "__name__"
+
+// buildSeriesQuery renders the series lookup: one row per matched fingerprint
+// with its labels.
+func buildSeriesQuery(start, end int64, matchers []*labels.Matcher) (string, []any, error) {
+ // The series tables hold one row per (fingerprint, bucket) at 1h/6h/1d/1w
+ // granularities; the schema package picks the table whose bucket fits the
+ // window and rounds the start down to the bucket boundary, so a window
+ // beginning mid-bucket still matches the bucket's row.
+ adjustedStart, _, table, _ := metricstelemetryschema.WhichTSTableToUse(uint64(start), uint64(end), false, nil)
+ sb := sqlbuilder.NewSelectBuilder()
+ sb.Select("fingerprint", "any(labels)")
+ sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, table))
+ if err := applySeriesConditions(sb, int64(adjustedStart), end, matchers); err != nil {
+ return "", nil, err
+ }
+ sb.GroupBy("fingerprint")
+ query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
+ return query, args, nil
+}
+
+// buildSamplesQuery renders the samples fetch for the matched series: a
+// semi-join re-runs the series predicates against the shard-local series
+// table (complete by fingerprint co-locality, see localTimeSeriesTable — a
+// GLOBAL broadcast would ship the matched set to every shard, and ClickHouse
+// materializes the subquery's set per shard before the scan, so it still
+// engages the fingerprint primary-key column). metricNames (observed on the
+// matched series when the selector had no __name__ equality) narrows the
+// primary-key scan. A non-nil lastPerStep groups to the last sample per
+// step bucket.
+func buildSamplesQuery(start, end int64, metricNames []string, matchers []*labels.Matcher, lastPerStep *lastSamplePerStep) (string, []any, error) {
+ sb := sqlbuilder.NewSelectBuilder()
+ if lastPerStep != nil {
+ // Aliases must not shadow source columns: ClickHouse resolves aliases
+ // in WHERE too, and "max(unix_milli) AS unix_milli" would put an
+ // aggregate into the WHERE clause (error 184).
+ sb.Select("fingerprint", "max(unix_milli) AS ts", "argMax(value, unix_milli) AS val", "argMax(flags, unix_milli) AS fl")
+ } else {
+ sb.Select("fingerprint", "unix_milli", "value", "flags")
+ }
+ sb.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, metricstelemetryschema.SamplesV4TableName))
+
+ switch len(metricNames) {
+ case 0:
+ // No name constraint derivable; the primary-key prefix goes unused.
+ case 1:
+ sb.Where(sb.EQ("metric_name", metricNames[0]))
+ default:
+ sb.Where(sb.In("metric_name", sqlbuilder.List(metricNames)))
+ }
+ // Semantically redundant (the fingerprints already come from these
+ // temporalities) but engages the leading primary-key column.
+ sb.Where("temporality IN ['Cumulative', 'Unspecified']")
+ sub := sqlbuilder.NewSelectBuilder()
+ sub.Select("fingerprint")
+ adjustedStart, _, _, localTable := metricstelemetryschema.WhichTSTableToUse(uint64(start), uint64(end), false, nil)
+ sub.From(fmt.Sprintf("%s.%s", metricstelemetryschema.DBName, localTable))
+ if err := applySeriesConditions(sub, int64(adjustedStart), end, matchers); err != nil {
+ return "", nil, err
+ }
+ sb.Where(sb.In("fingerprint", sub))
+ sb.Where(sb.GTE("unix_milli", start), sb.LTE("unix_milli", end))
+
+ if lastPerStep != nil {
+ sb.GroupBy("fingerprint")
+ if expr := lastPerStep.bucketExpr(); expr != "" {
+ sb.GroupBy(expr)
+ }
+ sb.OrderBy("fingerprint", "ts")
+ } else {
+ sb.OrderBy("fingerprint", "unix_milli")
+ }
+
+ query, args := sb.BuildWithFlavor(sqlbuilder.ClickHouse)
+ return query, args, nil
+}
+
+// applySeriesConditions adds the WHERE conditions of a series table scan:
+// __name__ matchers (all four types) translate to the metric_name column,
+// every other matcher to a JSONExtractString condition on the labels column.
+// An equality matcher against "" matches series without the label, mirroring
+// PromQL, because JSONExtractString returns "" for missing keys. Regexes are
+// anchored: PromQL matchers match the whole value, ClickHouse match()
+// searches for a substring.
+func applySeriesConditions(sb *sqlbuilder.SelectBuilder, start, end int64, matchers []*labels.Matcher) error {
+ for _, m := range matchers {
+ if m.Name != metricNameLabel {
+ continue
+ }
+ switch m.Type {
+ case labels.MatchEqual:
+ sb.Where(sb.EQ("metric_name", m.Value))
+ case labels.MatchNotEqual:
+ sb.Where(sb.NE("metric_name", m.Value))
+ case labels.MatchRegexp:
+ sb.Where(fmt.Sprintf("match(metric_name, %s)", sb.Var(anchorRegex(m.Value))))
+ case labels.MatchNotRegexp:
+ sb.Where(fmt.Sprintf("NOT match(metric_name, %s)", sb.Var(anchorRegex(m.Value))))
+ default:
+ return errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported matcher type %q for __name__", m.Type)
+ }
+ }
+
+ sb.Where("temporality IN ['Cumulative', 'Unspecified']")
+ // Inclusive upper bound: registration rows are hour-floored (and 6h/1d/1w
+ // for the rollup tables) by the exporter, so a series first registered in
+ // the bucket starting exactly at `end` would otherwise be invisible while
+ // its samples (<= end) are in range.
+ sb.Where(sb.GTE("unix_milli", start), sb.LTE("unix_milli", end))
+
+ for _, m := range matchers {
+ if m.Name == metricNameLabel {
+ continue
+ }
+ switch m.Type {
+ case labels.MatchEqual:
+ sb.Where(fmt.Sprintf("JSONExtractString(labels, %s) = %s", sb.Var(m.Name), sb.Var(m.Value)))
+ case labels.MatchNotEqual:
+ sb.Where(fmt.Sprintf("JSONExtractString(labels, %s) != %s", sb.Var(m.Name), sb.Var(m.Value)))
+ case labels.MatchRegexp:
+ sb.Where(fmt.Sprintf("match(JSONExtractString(labels, %s), %s)", sb.Var(m.Name), sb.Var(anchorRegex(m.Value))))
+ case labels.MatchNotRegexp:
+ sb.Where(fmt.Sprintf("NOT match(JSONExtractString(labels, %s), %s)", sb.Var(m.Name), sb.Var(anchorRegex(m.Value))))
+ default:
+ return errors.NewInvalidInputf(errors.CodeInvalidInput, "unsupported matcher type %q", m.Type)
+ }
+ }
+
+ return nil
+}
+
+// anchorRegex turns a PromQL regex into its fully-anchored form (see
+// applySeriesConditions).
+func anchorRegex(v string) string {
+ return "^(?:" + v + ")$"
+}
+
+// lastSamplePerStep reduces an instant-selector fetch to the last sample of
+// each step bucket: bucket 0 is (start, firstEval], bucket i is
+// (firstEval+(i-1)·step, firstEval+i·step]. Anchoring buckets at the first
+// evaluation timestamp makes every bucket boundary an evaluation timestamp,
+// so a non-final sample of a bucket can never be the latest sample in any
+// (t-lookback, t] the engine resolves — the reduction is lossless. Real
+// timestamps are preserved, so the engine's own lookback and staleness
+// handling remain exact.
+type lastSamplePerStep struct {
+ firstEvalMs int64
+ stepMs int64
+}
+
+func (t *lastSamplePerStep) bucketExpr() string {
+ if t.stepMs <= 0 {
+ // Instant query: a single evaluation at firstEval; one bucket.
+ return ""
+ }
+ return fmt.Sprintf(
+ "if(unix_milli <= %d, 0, intDiv(unix_milli - %d - 1, %d) + 1)",
+ t.firstEvalMs, t.firstEvalMs, t.stepMs,
+ )
+}
diff --git a/pkg/prometheus/clickhouseprometheusv2/sql_test.go b/pkg/prometheus/clickhouseprometheusv2/sql_test.go
new file mode 100644
index 0000000000..f95f6cbf46
--- /dev/null
+++ b/pkg/prometheus/clickhouseprometheusv2/sql_test.go
@@ -0,0 +1,115 @@
+package clickhouseprometheusv2
+
+import (
+ "testing"
+ "time"
+
+ "github.com/prometheus/prometheus/model/labels"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+func mustMatcher(t *testing.T, mt labels.MatchType, name, value string) *labels.Matcher {
+ t.Helper()
+ m, err := labels.NewMatcher(mt, name, value)
+ require.NoError(t, err)
+ return m
+}
+
+func TestBuildSeriesQuery(t *testing.T) {
+ start := int64(1_700_000_000_000)
+ end := start + time.Hour.Milliseconds()
+ // The series table window rounds down to the table's bucket boundary.
+ adjustedStart := start - (start % time.Hour.Milliseconds())
+
+ t.Run("equality name and label matchers", func(t *testing.T) {
+ query, args, err := buildSeriesQuery(start, end, []*labels.Matcher{
+ mustMatcher(t, labels.MatchEqual, "__name__", "http_requests_total"),
+ mustMatcher(t, labels.MatchEqual, "job", "api"),
+ })
+ require.NoError(t, err)
+ assert.Equal(t,
+ "SELECT fingerprint, any(labels) FROM signoz_metrics.distributed_time_series_v4 WHERE metric_name = ? AND temporality IN ['Cumulative', 'Unspecified'] AND unix_milli >= ? AND unix_milli <= ? AND JSONExtractString(labels, ?) = ? GROUP BY fingerprint",
+ query,
+ )
+ assert.Equal(t, []any{"http_requests_total", adjustedStart, end, "job", "api"}, args)
+ })
+
+ t.Run("regex matchers are anchored", func(t *testing.T) {
+ _, args, err := buildSeriesQuery(start, end, []*labels.Matcher{
+ mustMatcher(t, labels.MatchEqual, "__name__", "up"),
+ mustMatcher(t, labels.MatchRegexp, "instance", "prod.*"),
+ mustMatcher(t, labels.MatchNotRegexp, "env", "dev|test"),
+ })
+ require.NoError(t, err)
+ assert.Equal(t, []any{"up", adjustedStart, end, "instance", "^(?:prod.*)$", "env", "^(?:dev|test)$"}, args)
+ })
+
+ t.Run("regex name matcher uses metric_name column", func(t *testing.T) {
+ query, args, err := buildSeriesQuery(start, end, []*labels.Matcher{
+ mustMatcher(t, labels.MatchRegexp, "__name__", "node_cpu.*|node_memory.*"),
+ })
+ require.NoError(t, err)
+ assert.Contains(t, query, "match(metric_name, ?)")
+ assert.NotContains(t, query, "JSONExtractString")
+ assert.Equal(t, []any{"^(?:node_cpu.*|node_memory.*)$", adjustedStart, end}, args)
+ })
+
+ t.Run("no name matcher omits metric_name condition", func(t *testing.T) {
+ query, _, err := buildSeriesQuery(start, end, []*labels.Matcher{
+ mustMatcher(t, labels.MatchEqual, "job", "api"),
+ })
+ require.NoError(t, err)
+ assert.NotContains(t, query, "metric_name")
+ })
+}
+
+func TestBuildSamplesQuery(t *testing.T) {
+ start := int64(1_700_000_000_000)
+ end := start + time.Hour.Milliseconds()
+ adjustedStart := start - (start % time.Hour.Milliseconds())
+ matchers := []*labels.Matcher{
+ mustMatcher(t, labels.MatchEqual, "__name__", "up"),
+ mustMatcher(t, labels.MatchEqual, "job", "api"),
+ }
+
+ t.Run("raw fetch filters by a shard-local semi-join", func(t *testing.T) {
+ query, args, err := buildSamplesQuery(start, end, []string{"up"}, matchers, nil)
+ require.NoError(t, err)
+ assert.Contains(t, query, "fingerprint IN (SELECT fingerprint FROM signoz_metrics.time_series_v4 WHERE ")
+ assert.NotContains(t, query, "GLOBAL IN")
+ assert.Contains(t, query, "ORDER BY fingerprint, unix_milli")
+ // Args follow placeholder order: samples metric name, the semi-join's
+ // series predicates, then the samples window bounds.
+ assert.Equal(t, []any{"up", "up", adjustedStart, end, "job", "api", start, end}, args)
+ })
+
+ t.Run("last-sample-per-step groups by step bucket anchored at first eval", func(t *testing.T) {
+ lastPerStep := &lastSamplePerStep{firstEvalMs: start + 299_999, stepMs: 60_000}
+ query, _, err := buildSamplesQuery(start, end, []string{"up"}, matchers, lastPerStep)
+ require.NoError(t, err)
+ assert.Contains(t, query, "argMax(value, unix_milli) AS val")
+ assert.Contains(t, query, "argMax(flags, unix_milli) AS fl")
+ assert.Contains(t, query, "GROUP BY fingerprint, if(unix_milli <= 1700000299999, 0, intDiv(unix_milli - 1700000299999 - 1, 60000) + 1)")
+ assert.Contains(t, query, "ORDER BY fingerprint, ts")
+ // Aliases must not shadow the source columns referenced in WHERE.
+ assert.NotContains(t, query, "AS unix_milli")
+ assert.NotContains(t, query, "AS value")
+ assert.NotContains(t, query, "AS flags")
+ })
+
+ t.Run("instant query keeps one bucket", func(t *testing.T) {
+ lastPerStep := &lastSamplePerStep{firstEvalMs: end, stepMs: 0}
+ query, _, err := buildSamplesQuery(start, end, []string{"up"}, matchers, lastPerStep)
+ require.NoError(t, err)
+ assert.Contains(t, query, "GROUP BY fingerprint ORDER BY fingerprint, ts")
+ assert.NotContains(t, query, "intDiv")
+ })
+
+ t.Run("multiple metric names from regex selector", func(t *testing.T) {
+ query, args, err := buildSamplesQuery(start, end, []string{"node_cpu", "node_memory"}, matchers, nil)
+ require.NoError(t, err)
+ assert.Contains(t, query, "metric_name IN (?, ?)")
+ assert.Equal(t, []any{"node_cpu", "node_memory", "up", adjustedStart, end, "job", "api", start, end}, args)
+ })
+}
diff --git a/pkg/prometheus/traits.go b/pkg/prometheus/traits.go
new file mode 100644
index 0000000000..b1e6a326b6
--- /dev/null
+++ b/pkg/prometheus/traits.go
@@ -0,0 +1,49 @@
+package prometheus
+
+import (
+ "context"
+
+ "github.com/prometheus/prometheus/promql/parser"
+)
+
+type queryTraitsKey struct{}
+
+// QueryTraits carries per-query facts a storage implementation cannot derive
+// from SelectHints alone. Call sites that parse the PromQL expression attach
+// traits to the context before handing it to the engine; storages treat a
+// missing traits value as "unknown" and stay conservative.
+type QueryTraits struct {
+ // SubqueryFree is true when the query contains no subquery expression.
+ // Subquery selectors are evaluated at the subquery's own step, but
+ // SelectHints.Step always carries the top-level step, so step-aligned
+ // storage optimizations (e.g. keeping only the last sample per step
+ // bucket) are safe only when this is true.
+ SubqueryFree bool
+}
+
+// DetectQueryTraits derives QueryTraits from a parsed PromQL expression.
+func DetectQueryTraits(expr parser.Expr) QueryTraits {
+ subqueryFree := true
+ parser.Inspect(expr, func(node parser.Node, _ []parser.Node) error {
+ if _, ok := node.(*parser.SubqueryExpr); ok {
+ subqueryFree = false
+ }
+ return nil
+ })
+ return QueryTraits{SubqueryFree: subqueryFree}
+}
+
+// NewContextWithQueryTraits returns a context carrying the given traits.
+func NewContextWithQueryTraits(ctx context.Context, traits QueryTraits) context.Context {
+ return context.WithValue(ctx, queryTraitsKey{}, traits)
+}
+
+// QueryTraitsFromContext returns the traits attached to ctx, if any.
+//
+// Context is used here, unlike for backend selection, because traits must
+// cross the promql engine to reach storage.Querier.Select, and the engine's
+// interfaces offer no other channel; the alternative is a Prometheus fork.
+func QueryTraitsFromContext(ctx context.Context) (QueryTraits, bool) {
+ traits, ok := ctx.Value(queryTraitsKey{}).(QueryTraits)
+ return traits, ok
+}