Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(observability): unify observability context && support otel && etc. (#1666)
baojun-zhang17d2c5603e13
diff --git a/docs/design/metric-design.md b/docs/design/metric-design.mdindex 618a2b260..a5daaf8b5 100644--- a/docs/design/metric-design.md+++ b/docs/design/metric-design.md@@ -810,6 +810,7 @@ Openviking 指标体系采用如下兼容性策略:- 保持现有 retrieval / embedding / vlm / cache 指标名尽量不变;- `/metrics` 路由地址保持不变;- `server.observability.metrics.enabled` 配置保持不变;+- `server.observability.metrics.exporters.*` 作为 exporter 扩展点:默认 Prometheus exporter 可继续工作,OTLP exporter 可按需启用;- `/api/v1/observer/*` 与 `/api/v1/stats/*` 行为保持不变。## 6. 风险与缓解diff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.mdindex e4bf34275..689b6cc2c 100644--- a/docs/en/concepts/12-metrics.md+++ b/docs/en/concepts/12-metrics.md@@ -365,6 +365,48 @@ Recommended mental model:- `server.observability.metrics.enabled`: master switch for the metrics subsystem- `server.observability.metrics.account_dimension`: controls whether `account_id` labels are enabled and where they are allowed+### Exporters++By default, OpenViking exports metrics via Prometheus exposition format at `/metrics`.+You can also enable additional exporters under `server.observability.metrics.exporters`.++Key fields:++- `server.observability.metrics.exporters.prometheus.enabled`: enable the Prometheus exporter (serves `/metrics`)+- `server.observability.metrics.exporters.otel.enabled`: enable OTLP export from the same in-process registry+- `server.observability.metrics.exporters.otel.protocol`: `"grpc"` or `"http"`+- `server.observability.metrics.exporters.otel.tls.insecure`: OTLP/gRPC only; `true` means plaintext (no TLS)+- `server.observability.metrics.exporters.otel.endpoint`: OTLP endpoint (for gRPC, use `host:4317`; for HTTP, use a full URL)++Example:++```json+{+ "server": {+ "observability": {+ "metrics": {+ "enabled": true,+ "exporters": {+ "prometheus": {+ "enabled": true+ },+ "otel": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server",+ "export_interval_ms": 10000+ }+ }+ }+ }+ }+}+```+### Recommended `account_id` Usage- enabled by default, but only allowlisted metric families will receive tenant ids (empty allowlist still yields `__unknown__`)diff --git a/docs/en/guides/05-observability.md b/docs/en/guides/05-observability.mdindex e9f68587e..3b0c5bd7e 100644--- a/docs/en/guides/05-observability.md+++ b/docs/en/guides/05-observability.md@@ -270,6 +270,61 @@ Add the following to `~/.openviking/ov.conf` (or the path passed via `--config`)Restart OpenViking Server after editing the config.+### Observability config hierarchy++OpenViking groups signal-level observability configuration under `server.observability`:++- `server.observability.metrics`: metrics subsystem and exporters+- `server.observability.traces`: trace export configuration+- `server.observability.logs`: log export configuration++Example:++```json+{+ "server": {+ "observability": {+ "metrics": {+ "enabled": true,+ "exporters": {+ "prometheus": {+ "enabled": true+ },+ "otel": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server",+ "export_interval_ms": 10000+ }+ }+ },+ "traces": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server"+ },+ "logs": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server"+ }+ }+ }+}+```+For full fields, supported ranges, and more examples, see:- [Metrics](../concepts/12-metrics.md)diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.mdindex a4f1668f8..1e3c8360e 100644--- a/docs/zh/concepts/12-metrics.md+++ b/docs/zh/concepts/12-metrics.md@@ -368,6 +368,48 @@ scrape_configs:- `server.observability.metrics.enabled`:指标体系总开关- `server.observability.metrics.account_dimension`:控制 `account_id` 标签是否启用以及启用范围+### Exporters 配置++默认情况下,OpenViking 会通过 Prometheus exposition 格式在 `/metrics` 输出指标。+如果希望在保留 `/metrics` 的同时把同一份进程内指标导出到 OTLP 后端,可以在 `server.observability.metrics.exporters` 下启用 exporter。++关键字段:++- `server.observability.metrics.exporters.prometheus.enabled`:是否启用 Prometheus exporter(提供 `/metrics`)+- `server.observability.metrics.exporters.otel.enabled`:是否启用 OTLP 导出(复用同一份 registry)+- `server.observability.metrics.exporters.otel.protocol`:`"grpc"` 或 `"http"`+- `server.observability.metrics.exporters.otel.tls.insecure`:仅对 OTLP/gRPC 生效;`true` 表示明文连接(无 TLS)+- `server.observability.metrics.exporters.otel.endpoint`:OTLP 端点(gRPC 用 `host:4317`;HTTP 必须是完整 URL)++示例:++```json+{+ "server": {+ "observability": {+ "metrics": {+ "enabled": true,+ "exporters": {+ "prometheus": {+ "enabled": true+ },+ "otel": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server",+ "export_interval_ms": 10000+ }+ }+ }+ }+ }+}+```+### `account_id` 标签的使用建议- 默认开启,但仅对白名单指标启用(`metric_allowlist` 为空时仍会输出为 `__unknown__`)diff --git a/docs/zh/guides/05-observability.md b/docs/zh/guides/05-observability.mdindex 3ec1b1796..f5c0bcafc 100644--- a/docs/zh/guides/05-observability.md+++ b/docs/zh/guides/05-observability.md@@ -269,6 +269,61 @@ curl -X POST http://localhost:1933/api/v1/search/find \```改完配置后需要**重启 OpenViking Server** 才会生效。+### observability 配置层级++OpenViking 将信号级别的可观测性配置统一放在 `server.observability` 下:++- `server.observability.metrics`:metrics 子系统与 exporter 配置+- `server.observability.traces`:trace 导出配置+- `server.observability.logs`:log 导出配置++示例:++```json+{+ "server": {+ "observability": {+ "metrics": {+ "enabled": true,+ "exporters": {+ "prometheus": {+ "enabled": true+ },+ "otel": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server",+ "export_interval_ms": 10000+ }+ }+ },+ "traces": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server"+ },+ "logs": {+ "enabled": true,+ "protocol": "grpc",+ "tls": {+ "insecure": true+ },+ "endpoint": "otel-collector:4317",+ "service_name": "openviking-server"+ }+ }+ }+}+```+完整字段、支持范围和更多示例见:- [指标](../concepts/12-metrics.md)diff --git a/examples/ov.conf.example b/examples/ov.conf.exampleindex d0979655a..08ba4cf44 100644--- a/examples/ov.conf.example+++ b/examples/ov.conf.example@@ -5,16 +5,49 @@"root_api_key": null,"cors_origins": ["*"],"observability": {- "metrics": {- "enabled": true,- "account_dimension": {- "enabled": true,- "max_active_accounts": 100,- "metric_allowlist": [- "openviking_*"- ]+ "metrics": {+ "enabled": true,+ "account_dimension": {+ "enabled": true,+ "max_active_accounts": 100,+ "metric_allowlist": [+ "openviking_*"+ ]+ },+ "exporters": {+ "prometheus": {+ "enabled": true+ },+ "otel": {+ "enabled": false,+ "protocol": "grpc",+ "tls": {+ "insecure": false+ },+ "endpoint": "localhost:4317",+ "service_name": "openviking-server",+ "export_interval_ms": 10000+ }+ }+ },+ "traces": {+ "enabled": false,+ "protocol": "grpc",+ "tls": {+ "insecure": false+ },+ "endpoint": "localhost:4317",+ "service_name": "openviking-server"+ },+ "logs": {+ "enabled": false,+ "protocol": "grpc",+ "tls": {+ "insecure": false+ },+ "endpoint": "localhost:4317",+ "service_name": "openviking-server"}- }}},"storage": {diff --git a/openviking/metrics/account_context.py b/openviking/metrics/account_context.pydeleted file mode 100644index 81d0b2591..000000000--- a/openviking/metrics/account_context.py+++ /dev/null@@ -1,80 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0--"""-Request-scoped account context for metrics label injection.--This module keeps metrics-specific request context separate from business-layer request-objects so collectors can resolve `account_id` without depending on FastAPI internals.-"""--from __future__ import annotations--import contextvars-from dataclasses import dataclass---@dataclass(frozen=True, slots=True)-class MetricAccountContext:- """- Store request-scoped account information used during metrics label resolution.-- The context currently keeps the HTTP-derived account identifier that collectors and- account-dimension policy code can consult without reaching back into framework-specific- request objects.- """-- http_account_id: str | None = None---_CURRENT_METRIC_ACCOUNT_CONTEXT: contextvars.ContextVar[MetricAccountContext | None] = (- contextvars.ContextVar(- "openviking_metric_account_context",- default=None,- )-)---def get_metric_account_context() -> MetricAccountContext:- """- Return the metrics account context bound to the current execution flow.-- When no context has been bound yet, the function returns an empty `MetricAccountContext`- instance so callers can treat the result as a stable object instead of handling `None`.- """- context = _CURRENT_METRIC_ACCOUNT_CONTEXT.get()- if context is None:- return MetricAccountContext()- return context---def bind_metric_account_context(- *, account_id: str | None-) -> contextvars.Token[MetricAccountContext | None]:- """- Bind a new metrics account context and return the token needed to restore the prior one.-- This helper is intended for request or worker entry points that need scoped lifetime- management through `contextvars.Token`, typically via `try/finally`.- """- return _CURRENT_METRIC_ACCOUNT_CONTEXT.set(MetricAccountContext(http_account_id=account_id))---def set_metric_account_context(*, account_id: str | None) -> None:- """- Overwrite the current metrics account context for the active execution flow.-- Unlike `bind_metric_account_context`, this helper does not provide a restoration token and- is therefore best suited to one-way updates where the caller owns the full scope lifetime.- """- _CURRENT_METRIC_ACCOUNT_CONTEXT.set(MetricAccountContext(http_account_id=account_id))---def reset_metric_account_context(token: contextvars.Token[MetricAccountContext | None]) -> None:- """- Restore the previous metrics account context captured by `bind_metric_account_context`.-- Callers should use the exact token returned by the matching bind operation so nested context- updates unwind in the correct order.- """- _CURRENT_METRIC_ACCOUNT_CONTEXT.reset(token)diff --git a/openviking/metrics/account_dimension.py b/openviking/metrics/account_dimension.pyindex 96bb05166..96bc7d79b 100644--- a/openviking/metrics/account_dimension.py+++ b/openviking/metrics/account_dimension.py@@ -13,7 +13,7 @@ from __future__ import annotationsimport threadingfrom dataclasses import dataclass-from .account_context import get_metric_account_context+from openviking.observability.context import get_root_observability_contextUNKNOWN_ACCOUNT_ID = "__unknown__"OVERFLOW_ACCOUNT_ID = "__overflow__"@@ -96,7 +96,8 @@ class MetricAccountContextResolver:if explicit_account_id and str(explicit_account_id).strip():return str(explicit_account_id).strip()- http_account_id = get_metric_account_context().http_account_id+ root_context = get_root_observability_context()+ http_account_id = root_context.account_id if root_context is not None else Noneif http_account_id and str(http_account_id).strip():return str(http_account_id).strip()diff --git a/openviking/metrics/exporters/__init__.py b/openviking/metrics/exporters/__init__.pyindex 118902a47..f912459bd 100644--- a/openviking/metrics/exporters/__init__.py+++ b/openviking/metrics/exporters/__init__.py@@ -3,6 +3,7 @@"""Exporter entry points that render the shared metrics registry into external formats."""+from .otel import OTelMetricExporterfrom .prometheus import PrometheusExporter-__all__ = ["PrometheusExporter"]+__all__ = ["PrometheusExporter", "OTelMetricExporter"]diff --git a/openviking/metrics/exporters/otel.py b/openviking/metrics/exporters/otel.pynew file mode 100644index 000000000..de3af171a--- /dev/null+++ b/openviking/metrics/exporters/otel.py@@ -0,0 +1,713 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+OpenTelemetry OTLP metrics exporter implementation.++This exporter serializes the in-process MetricRegistry directly into OTLP protobuf+messages and sends them to an OTLP endpoint over HTTP or gRPC.++Key design goals:+- Export the same MetricRegistry snapshot semantics that PrometheusExporter renders+- Preserve histogram bucket/count/sum semantics without re-aggregation+- Keep exporter lifecycle self-contained through start()/shutdown()+- Refresh collectors before export so snapshot-based metrics stay aligned+"""++from __future__ import annotations++import asyncio+import logging+import time+from typing import Any, Iterable, Optional, Tuple++import requests++from openviking.metrics.core.base import MetricExporter++logger = logging.getLogger(__name__)++try:+ import grpc+except ImportError:+ grpc = None++try:+ from requests import RequestException+except Exception:+ RequestException = Exception++try:+ from opentelemetry.proto.collector.metrics.v1.metrics_service_pb2 import (+ ExportMetricsServiceRequest,+ )+ from opentelemetry.proto.collector.metrics.v1.metrics_service_pb2_grpc import (+ MetricsServiceStub,+ )+ from opentelemetry.proto.common.v1.common_pb2 import AnyValue, InstrumentationScope, KeyValue+ from opentelemetry.proto.metrics.v1.metrics_pb2 import (+ AGGREGATION_TEMPORALITY_CUMULATIVE,+ Gauge,+ Histogram,+ HistogramDataPoint,+ Metric,+ NumberDataPoint,+ Sum,+ )+ from opentelemetry.proto.resource.v1.resource_pb2 import Resource+except ImportError:+ ExportMetricsServiceRequest = None+ MetricsServiceStub = None+ AnyValue = None+ InstrumentationScope = None+ KeyValue = None+ Gauge = None+ Histogram = None+ HistogramDataPoint = None+ Metric = None+ NumberDataPoint = None+ Sum = None+ Resource = None+ AGGREGATION_TEMPORALITY_CUMULATIVE = None+++def _compute_next_tick_and_sleep(+ *,+ now: float,+ interval_s: float,+ next_tick: float,+) -> tuple[float, float]:+ """+ Compute next tick time and sleep duration for a fixed-interval loop.++ This is a pure function to keep scheduling behavior testable without adding+ production-only hooks.++ Rules:+ - If the loop overruns (now >= next_tick), advance next_tick by interval until it is in the future.+ - Sleep only until the next_tick.++ Args:+ now: Current monotonic time in seconds.+ interval_s: Target loop interval in seconds.+ next_tick: Previously scheduled tick time in seconds.++ Returns:+ (next_tick, sleep_seconds)+ """+ while next_tick <= now:+ next_tick += interval_s+ return next_tick, max(0.0, next_tick - now)+++class OTelMetricExporter(MetricExporter):+ """+ Export MetricRegistry snapshots to OTLP metrics.++ This exporter intentionally avoids OTel SDK metric aggregation. Instead, it+ periodically snapshots MetricRegistry, converts the snapshot into OTLP protobuf+ payloads, and pushes them to the configured collector endpoint.+ """++ def __init__(+ self,+ *,+ registry,+ collector_manager=None,+ refresh_deadline_seconds: float = 1.0,+ protocol: str = "grpc",+ insecure: bool = False,+ endpoint: str = "localhost:4317",+ service_name: str = "openviking-server",+ export_interval_ms: int = 10000,+ export_timeout_seconds: Optional[float] = None,+ enabled: bool = True,+ ) -> None:+ """+ Initialize the OTLP metrics exporter.++ Args:+ registry: The in-process metric registry backing all exporters.+ collector_manager: Optional collector refresh orchestrator.+ refresh_deadline_seconds: Best-effort refresh timeout before each export.+ protocol: Transport protocol, either "grpc" or "http".+ insecure: Whether gRPC transport should use plaintext.+ endpoint: OTLP collector endpoint.+ service_name: Service name written into OTLP resource attributes.+ export_interval_ms: Periodic export interval in milliseconds.+ export_timeout_seconds: Transport timeout for one OTLP export request.+ enabled: Whether the exporter is enabled.+ """+ self._registry = registry+ self._collector_manager = collector_manager+ self._refresh_deadline_seconds = float(refresh_deadline_seconds)+ self._protocol = protocol.lower()+ self._insecure = bool(insecure)+ self._endpoint = endpoint+ self._service_name = service_name+ self._export_interval_ms = max(1000, int(export_interval_ms))+ self._export_timeout_seconds = float(+ export_timeout_seconds+ if export_timeout_seconds is not None+ else max(5.0, self._refresh_deadline_seconds)+ )+ self._enabled = bool(enabled)+ self._export_task: Optional[asyncio.Task[Any]] = None+ self._http_session: Optional[requests.Session] = None+ self._grpc_channel: Optional[Any] = None+ self._grpc_stub: Optional[Any] = None+ self._start_time_ns = time.time_ns()++ if self._protocol not in {"grpc", "http"}:+ logger.warning(+ "[OTelMetricExporter] invalid protocol=%s, defaulting to grpc",+ self._protocol,+ )+ self._protocol = "grpc"++ if self._enabled:+ self._init_transport()++ def _init_transport(self) -> None:+ """+ Initialize the transport client for the configured OTLP protocol.++ The exporter keeps the transport client alive across export cycles so the+ periodic loop only pays the cost of serializing current registry snapshots.+ """+ if not self._protobuf_available():+ logger.warning(+ "[OTelMetricExporter] initialization failed: OTLP metrics protobuf dependencies are not available"+ )+ self._enabled = False+ return++ try:+ if self._protocol == "http":+ self._validate_http_endpoint(self._endpoint)+ self._http_session = requests.Session()+ else:+ if grpc is None or MetricsServiceStub is None:+ raise ImportError("gRPC OTLP metrics dependencies are not available")+ if self._insecure:+ self._grpc_channel = grpc.insecure_channel(self._endpoint)+ else:+ self._grpc_channel = grpc.secure_channel(+ self._endpoint,+ grpc.ssl_channel_credentials(),+ )+ self._grpc_stub = MetricsServiceStub(self._grpc_channel)++ logger.info(+ "[OTelMetricExporter] initialized: phase=init_transport protocol=%s endpoint=%s service_name=%s export_interval_ms=%s export_timeout_seconds=%s",+ self._protocol,+ self._endpoint,+ self._service_name,+ self._export_interval_ms,+ self._export_timeout_seconds,+ )+ except Exception as exc:+ logger.warning(+ "[OTelMetricExporter] initialization failed: phase=init_transport protocol=%s endpoint=%s service_name=%s export_timeout_seconds=%s error=%s",+ self._protocol,+ self._endpoint,+ self._service_name,+ self._export_timeout_seconds,+ exc,+ )+ self._enabled = False++ def _protobuf_available(self) -> bool:+ """Return whether all required OTLP protobuf symbols are available."""+ return all(+ symbol is not None+ for symbol in (+ ExportMetricsServiceRequest,+ AnyValue,+ InstrumentationScope,+ KeyValue,+ Metric,+ NumberDataPoint,+ HistogramDataPoint,+ Gauge,+ Sum,+ Histogram,+ Resource,+ AGGREGATION_TEMPORALITY_CUMULATIVE,+ )+ )++ def _validate_http_endpoint(self, endpoint: str) -> None:+ """+ Validate that an OTLP/HTTP endpoint includes a URL scheme and path.++ Args:+ endpoint: OTLP/HTTP metrics endpoint.+ """+ if not endpoint.startswith("http://") and not endpoint.startswith("https://"):+ raise ValueError(+ "OTLP/HTTP endpoint must include scheme, e.g. 'http://localhost:4318/v1/metrics'"+ )++ async def export(self) -> str:+ """+ Refresh collectors and export the current MetricRegistry snapshot.++ Returns:+ An empty string to preserve the MetricExporter interface contract.+ """+ if not self._enabled:+ return ""++ await self._refresh_collectors()+ request = self._build_export_request()+ if request is None:+ return ""++ await self._send_request(request)+ return ""++ async def _refresh_collectors(self) -> None:+ """+ Best-effort refresh of snapshot-driven collectors before serialization.++ Export must never break application flow, so refresh failures are swallowed.+ """+ if self._collector_manager is None:+ return+ started = time.monotonic()+ try:+ await asyncio.wait_for(+ self._collector_manager.refresh_all(+ self._registry,+ deadline_seconds=self._refresh_deadline_seconds,+ ),+ timeout=self._refresh_deadline_seconds,+ )+ except asyncio.TimeoutError as exc:+ elapsed_ms = int((time.monotonic() - started) * 1000)+ logger.warning(+ "[OTelMetricExporter] collector refresh failed: phase=refresh_collectors reason=timeout protocol=%s endpoint=%s deadline_seconds=%s elapsed_ms=%s error=%s",+ self._protocol,+ self._endpoint,+ self._refresh_deadline_seconds,+ elapsed_ms,+ exc,+ )+ except Exception as exc:+ elapsed_ms = int((time.monotonic() - started) * 1000)+ logger.warning(+ "[OTelMetricExporter] collector refresh failed: phase=refresh_collectors reason=exception protocol=%s endpoint=%s deadline_seconds=%s elapsed_ms=%s error=%s",+ self._protocol,+ self._endpoint,+ self._refresh_deadline_seconds,+ elapsed_ms,+ exc,+ )++ def _build_export_request(self) -> Optional[ExportMetricsServiceRequest]:+ """+ Build an OTLP ExportMetricsServiceRequest from the current registry snapshot.++ Returns:+ A populated OTLP export request, or None when there is no metric family to send.+ """+ if not self._protobuf_available():+ return None++ now_ns = time.time_ns()+ request = ExportMetricsServiceRequest()+ resource_metrics = request.resource_metrics.add()+ resource_metrics.resource.CopyFrom(self._build_resource())+ scope_metrics = resource_metrics.scope_metrics.add()+ scope_metrics.scope.CopyFrom(+ InstrumentationScope(name="openviking.metrics.exporter.otel", version="1")+ )++ self._append_counter_metrics(scope_metrics.metrics, now_ns)+ self._append_gauge_metrics(scope_metrics.metrics, now_ns)+ self._append_histogram_metrics(scope_metrics.metrics, now_ns)+ self._append_dropped_series_metrics(scope_metrics.metrics, now_ns)++ if not scope_metrics.metrics:+ return None+ return request++ def _build_resource(self) -> Resource:+ """+ Build the OTLP resource envelope for this exporter.++ Returns:+ A Resource message containing the service name.+ """+ resource = Resource()+ resource.attributes.append(+ KeyValue(key="service.name", value=AnyValue(string_value=self._service_name))+ )+ return resource++ def _append_counter_metrics(self, metrics, now_ns: int) -> None:+ """+ Append all counter families from MetricRegistry as cumulative OTLP sums.++ Empty unlabeled counters emit a single zero data point to match PrometheusExporter.+ """+ for name, counter_series in self._registry.iter_counters():+ metric = metrics.add()+ metric.name = name+ metric.description = "OpenViking metric."+ metric.sum.CopyFrom(self._build_sum(counter_series, now_ns, name=name))++ def _append_gauge_metrics(self, metrics, now_ns: int) -> None:+ """+ Append all gauge families from MetricRegistry as OTLP gauges.++ Empty unlabeled gauges emit a single zero data point to match PrometheusExporter.+ """+ for name, gauge_series in self._registry.iter_gauges():+ metric = metrics.add()+ metric.name = name+ metric.description = "OpenViking metric."+ metric.gauge.CopyFrom(self._build_gauge(gauge_series, now_ns, name=name))++ def _append_histogram_metrics(self, metrics, now_ns: int) -> None:+ """+ Append all histogram families from MetricRegistry as cumulative OTLP histograms.++ Bucket counts and explicit bounds are copied directly from the registry snapshot.+ """+ for name, label_names, bucket_bounds, series_iter in self._registry.iter_histograms():+ series_list = list(series_iter)+ metric = metrics.add()+ metric.name = name+ metric.description = "OpenViking metric."+ metric.histogram.CopyFrom(+ self._build_histogram(+ label_names=label_names,+ bucket_bounds=bucket_bounds,+ series_list=series_list,+ now_ns=now_ns,+ )+ )++ def _append_dropped_series_metrics(self, metrics, now_ns: int) -> None:+ """+ Append dropped-series diagnostics as a cumulative OTLP sum family.++ This mirrors the `openviking_metrics_dropped_series_total` metric emitted by+ PrometheusExporter.+ """+ dropped_points: list[tuple[tuple[tuple[str, str], ...], float]] = []+ for metric_name, dropped in self._registry.iter_dropped_series():+ dropped_points.append(((("metric", metric_name),), float(dropped)))++ if not dropped_points:+ return++ metric = metrics.add()+ metric.name = "openviking_metrics_dropped_series_total"+ metric.description = "OpenViking metric."+ metric.sum.CopyFrom(self._build_sum(dropped_points, now_ns, name=metric.name))++ def _build_sum(self, series, now_ns: int, *, name: str) -> Sum:+ """+ Build an OTLP Sum message from a counter-like registry family.++ Args:+ series: List of `(labels, value)` tuples from MetricRegistry.+ now_ns: Export timestamp in nanoseconds.+ name: Metric family name for zero-series handling.+ """+ sum_metric = Sum(+ aggregation_temporality=AGGREGATION_TEMPORALITY_CUMULATIVE,+ is_monotonic=True,+ )++ if not series and not self._registry.counter_label_names(name):+ point = NumberDataPoint(+ start_time_unix_nano=self._start_time_ns,+ time_unix_nano=now_ns,+ as_int=0,+ )+ sum_metric.data_points.append(point)+ return sum_metric++ for labels, value in series:+ point = NumberDataPoint(+ start_time_unix_nano=self._start_time_ns,+ time_unix_nano=now_ns,+ )+ point.attributes.extend(self._build_attributes(labels))+ if float(value).is_integer():+ point.as_int = int(value)+ else:+ point.as_double = float(value)+ sum_metric.data_points.append(point)+ return sum_metric++ def _build_gauge(self, series, now_ns: int, *, name: str) -> Gauge:+ """+ Build an OTLP Gauge message from a registry gauge family.++ Args:+ series: List of `(labels, value)` tuples from MetricRegistry.+ now_ns: Export timestamp in nanoseconds.+ name: Metric family name for zero-series handling.+ """+ gauge_metric = Gauge()++ if not series and not self._registry.gauge_label_names(name):+ point = NumberDataPoint(time_unix_nano=now_ns, as_int=0)+ gauge_metric.data_points.append(point)+ return gauge_metric++ for labels, value in series:+ point = NumberDataPoint(time_unix_nano=now_ns)+ point.attributes.extend(self._build_attributes(labels))+ if float(value).is_integer():+ point.as_int = int(value)+ else:+ point.as_double = float(value)+ gauge_metric.data_points.append(point)+ return gauge_metric++ def _build_histogram(+ self,+ *,+ label_names: tuple[str, ...],+ bucket_bounds: tuple[float, ...],+ series_list,+ now_ns: int,+ ) -> Histogram:+ """+ Build an OTLP Histogram message from a registry histogram family.++ Args:+ name: Histogram family name.+ label_names: Registered label keys for the family.+ bucket_bounds: Explicit bucket upper bounds from the registry.+ series_list: Histogram series snapshot from the registry.+ now_ns: Export timestamp in nanoseconds.+ """+ histogram_metric = Histogram(+ aggregation_temporality=AGGREGATION_TEMPORALITY_CUMULATIVE,+ )++ if not series_list and not label_names:+ point = HistogramDataPoint(+ start_time_unix_nano=self._start_time_ns,+ time_unix_nano=now_ns,+ count=0,+ sum=0.0,+ )+ point.bucket_counts.extend((0,) * (len(bucket_bounds) + 1))+ point.explicit_bounds.extend(bucket_bounds)+ histogram_metric.data_points.append(point)+ return histogram_metric++ for labels, bucket_counts, count, value_sum in series_list:+ point = HistogramDataPoint(+ start_time_unix_nano=self._start_time_ns,+ time_unix_nano=now_ns,+ count=int(count),+ sum=float(value_sum),+ )+ point.attributes.extend(self._build_attributes(labels))+ point.bucket_counts.extend(int(v) for v in bucket_counts)+ point.explicit_bounds.extend(float(v) for v in bucket_bounds)+ histogram_metric.data_points.append(point)+ return histogram_metric++ def _build_attributes(self, labels: Iterable[Tuple[str, str]]) -> list[KeyValue]:+ """+ Convert MetricRegistry labels into OTLP KeyValue attributes.++ Args:+ labels: Iterable of sorted `(key, value)` label pairs.+ """+ return [KeyValue(key=key, value=AnyValue(string_value=value)) for key, value in labels]++ async def _send_request(self, request: ExportMetricsServiceRequest) -> None:+ """+ Send one OTLP metrics export request using the configured transport.++ Args:+ request: The populated OTLP metrics export payload.+ """+ metric_count = self._count_otlp_metrics(request)+ request_bytes = request.ByteSize()+ try:+ if self._protocol == "http":+ await asyncio.to_thread(self._send_http_request, request)+ else:+ await asyncio.to_thread(self._send_grpc_request, request)+ except (RequestException, OSError, TimeoutError) as exc:+ logger.warning(+ "[OTelMetricExporter] export failed: phase=send_request protocol=%s endpoint=%s timeout_seconds=%s metric_count=%s request_bytes=%s error_type=%s error=%s",+ self._protocol,+ self._endpoint,+ self._export_timeout_seconds,+ metric_count,+ request_bytes,+ type(exc).__name__,+ exc,+ )+ except Exception as exc:+ # Keep exporter failure best-effort. Log unexpected failures at warning level.+ logger.warning(+ "[OTelMetricExporter] export failed: phase=send_request protocol=%s endpoint=%s timeout_seconds=%s metric_count=%s request_bytes=%s error_type=%s error=%s",+ self._protocol,+ self._endpoint,+ self._export_timeout_seconds,+ metric_count,+ request_bytes,+ type(exc).__name__,+ exc,+ )++ def _count_otlp_metrics(self, request: ExportMetricsServiceRequest) -> int:+ """+ Count OTLP metric families in one export request for logging purposes.++ Args:+ request: The populated OTLP metrics export payload.++ Returns:+ Number of OTLP Metric messages in the request.+ """+ count = 0+ for resource_metrics in request.resource_metrics:+ for scope_metrics in resource_metrics.scope_metrics:+ count += len(scope_metrics.metrics)+ return count++ def _send_http_request(self, request: ExportMetricsServiceRequest) -> None:+ """+ Send one OTLP/HTTP metrics request.++ Args:+ request: The populated OTLP metrics export payload.+ """+ if self._http_session is None:+ return+ payload = request.SerializeToString()+ self._http_session.post(+ self._endpoint,+ data=payload,+ headers={"Content-Type": "application/x-protobuf"},+ timeout=self._export_timeout_seconds,+ ).raise_for_status()++ def _send_grpc_request(self, request: ExportMetricsServiceRequest) -> None:+ """+ Send one OTLP/gRPC metrics request.++ Args:+ request: The populated OTLP metrics export payload.+ """+ if self._grpc_stub is None:+ return+ self._grpc_stub.Export(request, timeout=self._export_timeout_seconds)++ def start(self) -> None:+ """+ Start the exporter-owned periodic background loop.++ The exporter keeps its own lifecycle so application startup can create the+ exporter once and let it handle periodic exports internally.+ """+ if not self._enabled:+ return+ if self._export_task is not None and not self._export_task.done():+ return+ try:+ self._export_task = asyncio.create_task(+ self._run_export_loop(),+ name="openviking-otel-metrics-export-loop",+ )+ except RuntimeError:+ logger.warning(+ "[OTelMetricExporter] periodic export loop not started: phase=start protocol=%s endpoint=%s reason=no running event loop",+ self._protocol,+ self._endpoint,+ )+ self._export_task = None++ async def _run_export_loop(self) -> None:+ """+ Run the periodic export loop until shutdown is requested.+ """+ interval_s = float(self._export_interval_ms) / 1000.0+ next_tick = time.monotonic() + interval_s+ while self._enabled:+ try:+ await self.export()+ except asyncio.CancelledError:+ raise+ except Exception as exc:+ logger.warning(+ "[OTelMetricExporter] periodic export loop iteration failed: phase=run_export_loop protocol=%s endpoint=%s error=%s",+ self._protocol,+ self._endpoint,+ exc,+ )+ now = time.monotonic()+ next_tick, sleep_s = _compute_next_tick_and_sleep(+ now=now,+ interval_s=interval_s,+ next_tick=next_tick,+ )+ await asyncio.sleep(sleep_s)++ async def shutdown(self) -> None:+ """+ Stop the exporter loop and release transport resources.+ """+ self._enabled = False++ if self._export_task is not None:+ self._export_task.cancel()+ try:+ await self._export_task+ except asyncio.CancelledError:+ logger.info(+ "[OTelMetricExporter] periodic export loop cancelled: phase=shutdown protocol=%s endpoint=%s",+ self._protocol,+ self._endpoint,+ )+ except Exception as exc:+ logger.warning(+ "[OTelMetricExporter] periodic export loop shutdown failed: phase=shutdown protocol=%s endpoint=%s error=%s",+ self._protocol,+ self._endpoint,+ exc,+ )+ self._export_task = None++ if self._http_session is not None:+ try:+ self._http_session.close()+ except Exception as exc:+ logger.warning(+ "[OTelMetricExporter] http session close failed: phase=shutdown protocol=%s endpoint=%s error_type=%s error=%s",+ self._protocol,+ self._endpoint,+ type(exc).__name__,+ exc,+ )+ self._http_session = None++ if self._grpc_channel is not None:+ try:+ # grpc.Channel.close() is synchronous and returns None.+ self._grpc_channel.close()+ except Exception as exc:+ logger.warning(+ "[OTelMetricExporter] grpc channel close failed: phase=shutdown protocol=%s endpoint=%s error_type=%s error=%s",+ self._protocol,+ self._endpoint,+ type(exc).__name__,+ exc,+ )+ self._grpc_channel = None+ self._grpc_stub = Nonediff --git a/openviking/metrics/global_api.py b/openviking/metrics/global_api.pyindex d65ff03ff..1e35b98e3 100644--- a/openviking/metrics/global_api.py+++ b/openviking/metrics/global_api.py@@ -17,6 +17,9 @@ only layer allowed to write into MetricRegistry, keeping the architecture consisfrom __future__ import annotations+import asyncio+import inspect+import loggingimport threadingfrom typing import Optional@@ -41,14 +44,51 @@ from .collectors.telemetry_bridge import TelemetryBridgeCollectorfrom .collectors.vlm import VLMCollectorfrom .core.registry import MetricRegistryfrom .core.runtime import EventCollectorRouter+from .exporters.otel import OTelMetricExporterfrom .exporters.prometheus import PrometheusExporter+logger = logging.getLogger(__name__)+_lock = threading.Lock()_registry: Optional[MetricRegistry] = None-_exporter: Optional[PrometheusExporter] = None+_exporters: list = []_event_router: EventCollectorRouter | None = None+def _shutdown_exporters_best_effort(exporters: list) -> None:+ """Shutdown exporters without raising.++ This is used both for explicit shutdown and for re-initialization, to avoid leaking+ background tasks (e.g. OTel periodic export loops) across multiple init cycles.++ Note: This is a synchronous function. For proper async shutdown in async contexts,+ use `shutdown_metrics_async()` instead, which properly awaits async shutdown methods.+ """+ for exporter in exporters:+ if hasattr(exporter, "shutdown"):+ try:+ result = exporter.shutdown()+ if inspect.isawaitable(result):+ try:+ loop = asyncio.get_running_loop()+ except RuntimeError:+ # No running event loop - use asyncio.run() to execute the coroutine+ asyncio.run(result)+ else:+ # There is a running event loop. We cannot block-wait from within+ # the event loop thread (would cause deadlock). Instead, schedule+ # the shutdown as a task and rely on the async shutdown path+ # (shutdown_metrics_async) for proper cleanup in async contexts.+ #+ # This is best-effort for synchronous re-initialization paths.+ # The proper async shutdown path is used in app.py's lifespan.+ loop.create_task(result)+ except Exception as e:+ logger.warning(+ "[_shutdown_exporters_best_effort] failed to shutdown exporter: %s", e+ )++def configure_metric_account_dimension(*,enabled: bool | None = None,@@ -76,29 +116,34 @@ def init_metrics_from_server_config(config: ServerConfig, *, app=None, service=None, registry: MetricRegistry | None = None) -> None:"""- Initialize the process-global metrics registry, router, and exporter from server config.+ Initialize the process-global metrics registry, router, and exporters from server config.When metrics are enabled:- Creates a MetricRegistry (or uses provided one)- Registers default collectors (State/Probe/DomainStats)- Builds the in-process event router for Event-style metrics- - Creates PrometheusExporter and stores it in app.state.metrics_exporter+ - Creates configured exporters and stores them in app.state.metrics_exportersWhen metrics are disabled:- Clears all global metrics state- Resets account-dimension runtime to its disabled defaults- - Detaches any exporter reference previously stored on `app.state`+ - Detaches any exporter references previously stored on `app.state`"""enabled = config.observability.metrics.enabled- global _registry, _exporter, _event_router+ global _registry, _exporters, _event_routerwith _lock:if not enabled:+ _shutdown_exporters_best_effort(_exporters)_registry = None- _exporter = None+ _exporters = []_event_router = Nonereset_metric_account_dimension()if app is not None:- app.state.metrics_exporter = None+ app.state.metrics_exporters = []return++ # If re-initializing while already enabled, clean up existing exporters first.+ if _exporters:+ _shutdown_exporters_best_effort(_exporters)account_dimension = config.observability.metrics.account_dimensionconfigure_metric_account_dimension(enabled=account_dimension.enabled,@@ -108,26 +153,75 @@ def init_metrics_from_server_config(_registry = registry or MetricRegistry()collector_manager = create_default_collector_manager(app=app, service=service)_event_router = _build_event_router(_registry)- _exporter = PrometheusExporter(registry=_registry, collector_manager=collector_manager)++ _exporters = []+ exporters_config = config.observability.metrics.exporters++ # Initialize Prometheus exporter if enabled+ if exporters_config.prometheus.enabled:+ prometheus_exporter = PrometheusExporter(+ registry=_registry, collector_manager=collector_manager+ )+ _exporters.append(prometheus_exporter)++ # Initialize OTel exporter if enabled+ if exporters_config.otel.enabled:+ otel_exporter = OTelMetricExporter(+ registry=_registry,+ collector_manager=collector_manager,+ protocol=exporters_config.otel.protocol,+ insecure=exporters_config.otel.tls.insecure,+ endpoint=exporters_config.otel.endpoint,+ service_name=exporters_config.otel.service_name,+ export_interval_ms=exporters_config.otel.export_interval_ms,+ enabled=True,+ )+ otel_exporter.start()+ _exporters.append(otel_exporter)+if app is not None:- app.state.metrics_exporter = _exporter+ app.state.metrics_exporters = _exportersdef shutdown_metrics(*, app=None) -> None:"""- Clear all process-global metrics objects and detach the exporter from the application.+ Clear all process-global metrics objects and detach the exporters from the application.Shutdown is intentionally idempotent so tests and service teardown paths can call itrepeatedly without needing to coordinate current metrics initialization state."""- global _registry, _exporter, _event_router+ global _registry, _exporters, _event_router+ with _lock:+ _shutdown_exporters_best_effort(_exporters)++ _registry = None+ _exporters = []+ _event_router = None+ reset_metric_account_dimension()+ if app is not None:+ app.state.metrics_exporters = []+++async def shutdown_metrics_async(*, app=None) -> None:+ """Async variant of `shutdown_metrics` that awaits async exporter cleanup."""+ global _registry, _exporters, _event_routerwith _lock:+ exporters = list(_exporters)_registry = None- _exporter = None+ _exporters = []_event_router = Nonereset_metric_account_dimension()if app is not None:- app.state.metrics_exporter = None+ app.state.metrics_exporters = []++ for exporter in exporters:+ if hasattr(exporter, "shutdown"):+ try:+ result = exporter.shutdown()+ if inspect.isawaitable(result):+ await result+ except Exception as e:+ logger.warning("[shutdown_metrics_async] failed to shutdown exporter: %s", e)def get_metrics_registry() -> MetricRegistry:diff --git a/openviking/metrics/http_middleware.py b/openviking/metrics/http_middleware.pydeleted file mode 100644index ee2f9b18a..000000000--- a/openviking/metrics/http_middleware.py+++ /dev/null@@ -1,214 +0,0 @@-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.-# SPDX-License-Identifier: AGPL-3.0--"""-HTTP metrics middleware.--This middleware records request-level metrics in a best-effort manner:-- Inflight gauge per route template (to avoid high cardinality per raw path).-- Request count and duration histogram by method/route/status.--Implementation notes:-- Uses HttpRequestLifecycleDataSource, which emits events; collectors write into MetricRegistry.-- All metrics calls are wrapped in try/except to ensure observability never breaks request handling.-- Route label uses the Starlette route template (e.g., "/sessions/{session_id}") when available.-"""--from __future__ import annotations--import logging-import threading-import time-from typing import Callable--from starlette.requests import Request-from starlette.responses import Response--from openviking.metrics.account_context import (- bind_metric_account_context,- reset_metric_account_context,-)-from openviking.metrics.datasources import HttpRequestLifecycleDataSource--logger = logging.getLogger(__name__)--_HTTP_IGNORE_ROUTES = frozenset(- {- "/metrics",- "/health",- "/ready",- }-)---def create_http_metrics_middleware() -> Callable[[Request, Callable], Response]:- """- Create a Starlette-compatible middleware that records HTTP request metrics.-- The middleware emits events through `HttpRequestLifecycleDataSource`:- - `http.inflight` updates on request start/end- - `http.request` on request completion with status and duration-- Routes listed in `_HTTP_IGNORE_ROUTES` are intentionally skipped so internal endpoints such- as metrics or health probes do not pollute business-facing HTTP timeseries.- """-- async def middleware(request: Request, call_next: Callable) -> Response:- """- Wrap one request execution and emit best-effort inflight and request metrics events.-- The middleware never lets metrics failures affect request handling and will skip the- entire HTTP metrics flow for routes that are explicitly ignored.- """- # Ignore by raw path first so internal endpoints are skipped even when no route template- # is bound (e.g. middleware executed outside Starlette routing).- raw_path = str(request.url.path)- if raw_path in _HTTP_IGNORE_ROUTES:- return await call_next(request)-- initial_route = _get_route_template(request)- if initial_route in _HTTP_IGNORE_ROUTES:- return await call_next(request)- provisional_account_id = _extract_request_account_id(request)- account_token = bind_metric_account_context(account_id=provisional_account_id)- try:- HttpRequestLifecycleDataSource.set_inflight(- route=initial_route,- value=_inflight_delta(initial_route, provisional_account_id, +1),- account_id=provisional_account_id,- )- except Exception:- _log_metrics_failure("http.inflight increment failed", route=initial_route)-- start = time.perf_counter()- status_code = 500- try:- response = await call_next(request)- status_code = int(getattr(response, "status_code", 500))- return response- finally:- elapsed = time.perf_counter() - start- final_route = _get_route_template(request)- final_account_id = _extract_request_account_id(request)- if (final_route, final_account_id) != (initial_route, provisional_account_id):- try:- HttpRequestLifecycleDataSource.set_inflight(- route=initial_route,- value=_inflight_delta(initial_route, provisional_account_id, -1),- account_id=provisional_account_id,- )- HttpRequestLifecycleDataSource.set_inflight(- route=final_route,- value=_inflight_delta(final_route, final_account_id, +1),- account_id=final_account_id,- )- except Exception:- _log_metrics_failure(- "http.inflight rebalance failed",- route=final_route,- account_id=final_account_id,- )- try:- HttpRequestLifecycleDataSource.record_request(- method=request.method,- route=final_route,- status=str(status_code),- duration_seconds=elapsed,- account_id=final_account_id,- )- except Exception:- _log_metrics_failure(- "http.request recording failed",- route=final_route,- account_id=final_account_id,- )- try:- HttpRequestLifecycleDataSource.set_inflight(- route=final_route,- value=_inflight_delta(final_route, final_account_id, -1),- account_id=final_account_id,- )- except Exception:- _log_metrics_failure(- "http.inflight decrement failed",- route=final_route,- account_id=final_account_id,- )- reset_metric_account_context(account_token)-- return middleware---_INFLIGHT: dict[tuple[str, str | None], int] = {}-_INFLIGHT_LOCK = threading.Lock()---def _inflight_delta(route: str, account_id: str | None, delta: int) -> int:- """- Update and return the current in-process inflight count for a route template.-- This local counter is only used to provide a reasonable gauge value for event emission.- It is not a correctness mechanism and is clamped to never go below zero.- """- key = (route, account_id)- with _INFLIGHT_LOCK:- v = _INFLIGHT.get(key, 0) + delta- if v < 0:- v = 0- if v == 0:- _INFLIGHT.pop(key, None)- else:- _INFLIGHT[key] = v- return v---def _get_route_template(request: Request) -> str:- """- Resolve a low-cardinality route identifier for labeling request metrics.-- Prefer the Starlette route template (e.g. `/sessions/{session_id}`) when available, and- fall back to a fixed sentinel when the request is not bound to a recognized route template.- """- route = request.scope.get("route")- path = getattr(route, "path", None)- if path:- return str(path)- # Never label by raw path here: it may contain IDs and explode cardinality.- return "/__unmatched__"---def _extract_request_account_id(request: Request) -> str | None:- """- Return the best request-scoped account id currently available for metrics labeling.-- Only trust the authenticated request state (`request.state.metric_account_id`).-- The `/metrics` endpoint is intentionally best-effort and must not let unauthenticated or- rejected traffic control tenant labels via raw headers. When the authenticated account id- is not available yet, return `None` so collector-side policy can map it to `__unknown__`.- """- state_account_id = getattr(request.state, "metric_account_id", None)- if state_account_id:- return str(state_account_id)- return None---def _log_metrics_failure(message: str, *, route: str, account_id: str | None = None) -> None:- """- Emit a debug log for best-effort metrics failures without affecting request handling.-- This middleware intentionally treats observability as a side channel. The helper keeps- failure reporting centralized so all swallowed exceptions still leave a trace when debug- logging is enabled.- """- if not logger.isEnabledFor(logging.DEBUG):- return- logger.debug(- "http metrics write failed: %s",- message,- extra={- "route": route,- "account_id": account_id,- },- exc_info=True,- )diff --git a/openviking/models/embedder/base.py b/openviking/models/embedder/base.pyindex 91352cd65..c43a7b1a4 100644--- a/openviking/models/embedder/base.py+++ b/openviking/models/embedder/base.py@@ -328,8 +328,10 @@ class EmbedderBase(ABC):completion_tokens=completion_tokens,)try:- from openviking.metrics.account_context import get_metric_account_contextfrom openviking.metrics.datasources import EmbeddingEventDataSource+ from openviking.observability.context import get_root_observability_context++ root_context = get_root_observability_context()EmbeddingEventDataSource.record_call(provider=str(provider),@@ -337,7 +339,7 @@ class EmbedderBase(ABC):duration_seconds=self._resolve_metrics_duration_seconds(duration_seconds),prompt_tokens=int(prompt_tokens),completion_tokens=int(completion_tokens),- account_id=get_metric_account_context().http_account_id,+ account_id=root_context.account_id if root_context is not None else None,)except Exception as e:# Metrics must never break embedding execution.diff --git a/openviking/models/rerank/base.py b/openviking/models/rerank/base.pyindex 61942746c..dcc25acdb 100644--- a/openviking/models/rerank/base.py+++ b/openviking/models/rerank/base.py@@ -92,8 +92,10 @@ class RerankBase:e,)try:- from openviking.metrics.account_context import get_metric_account_contextfrom openviking.metrics.datasources import RerankEventDataSource+ from openviking.observability.context import get_root_observability_context++ root_context = get_root_observability_context()RerankEventDataSource.record_call(provider=str(provider),@@ -101,7 +103,7 @@ class RerankBase:duration_seconds=max(float(duration_seconds), 0.0),prompt_tokens=int(prompt_tokens),completion_tokens=int(completion_tokens),- account_id=get_metric_account_context().http_account_id,+ account_id=root_context.account_id if root_context is not None else None,)except Exception as e:# Metrics must never break rerank execution.diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.pyindex dc9017dfa..3cfbee579 100644--- a/openviking/models/vlm/base.py+++ b/openviking/models/vlm/base.py@@ -219,8 +219,10 @@ class VLMBase(ABC):# Record the VLM call in Prometheus metrics (if enabled).try:- from openviking.metrics.account_context import get_metric_account_contextfrom openviking.metrics.datasources import VLMEventDataSource+ from openviking.observability.context import get_root_observability_context++ root_context = get_root_observability_context()VLMEventDataSource.record_call(provider=str(provider),@@ -228,7 +230,7 @@ class VLMBase(ABC):duration_seconds=float(duration_seconds),prompt_tokens=int(prompt_tokens),completion_tokens=int(completion_tokens),- account_id=get_metric_account_context().http_account_id,+ account_id=root_context.account_id if root_context is not None else None,)except Exception as e:# Metrics must never break model inference.diff --git a/openviking/observability/__init__.py b/openviking/observability/__init__.pynew file mode 100644index 000000000..70854cd14--- /dev/null+++ b/openviking/observability/__init__.py@@ -0,0 +1,19 @@+"""Unified observability context helpers."""++from .context import (+ bind_operation_observability_context,+ bind_root_observability_context,+ get_operation_observability_context,+ get_root_observability_context,+ reset_operation_observability_context,+ reset_root_observability_context,+)++__all__ = [+ "bind_root_observability_context",+ "get_root_observability_context",+ "reset_root_observability_context",+ "bind_operation_observability_context",+ "get_operation_observability_context",+ "reset_operation_observability_context",+]diff --git a/openviking/observability/context.py b/openviking/observability/context.pynew file mode 100644index 000000000..5ae46e87a--- /dev/null+++ b/openviking/observability/context.py@@ -0,0 +1,345 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Unified observability context management.++This module provides a unified context management system for observability,+integrating:+- Root context (HTTP request-level)+- Operation context (business operation-level)+- Execution context (non-request scenarios)++Key features:+- Single contextvar for all observability context+- Unified binding, retrieval, and reset interfaces+- Type-safe data structures+- Automatic OTel trace/span ID integration+"""++from __future__ import annotations++import contextvars+import logging+import uuid+from contextlib import contextmanager+from dataclasses import dataclass, field+from typing import Any, Dict, Optional++from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributes++# Try to import opentelemetry+try:+ from opentelemetry import trace as otel_trace+ from opentelemetry.trace import format_span_id, format_trace_id+except ImportError:+ otel_trace = None+ format_span_id = None+ format_trace_id = None++logger = logging.getLogger(__name__)+++@dataclass+class ObservabilityContext:+ """+ Unified observability context data structure.++ This class integrates all observability-related context information:+ - Root context (HTTP request-level)+ - Operation context (business operation-level)+ - Execution context (non-request scenarios)++ Attributes:+ root: Root span attributes for HTTP requests.+ operation: Operation span attributes for business operations.+ execution_trace_id: Trace ID for non-request execution contexts.+ execution_span_id: Span ID for non-request execution contexts.+ extra: Additional custom attributes.+ """++ root: Optional[RootSpanAttributes] = None+ operation: Optional[OperationSpanAttributes] = None+ execution_trace_id: Optional[str] = None+ execution_span_id: Optional[str] = None+ extra: Dict[str, Any] = field(default_factory=dict)++ def get_trace_id(self) -> str:+ """+ Get the effective trace ID.++ Priority:+ 1. OTel current span trace_id+ 2. Execution context trace_id+ 3. Empty string++ Returns:+ The effective trace ID, or empty string if not available.+ """+ # 1. Check OTel current span+ if otel_trace is not None and format_trace_id is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and hasattr(current_span, "context"):+ span_context = current_span.get_span_context()+ if span_context.is_valid:+ return format_trace_id(span_context.trace_id)+ except Exception:+ pass++ # 2. Check execution context+ if self.execution_trace_id:+ return self.execution_trace_id++ # 3. Return empty string+ return ""++ def get_span_id(self) -> str:+ """+ Get the effective span ID.++ Priority:+ 1. OTel current span span_id+ 2. Execution context span_id+ 3. Empty string++ Returns:+ The effective span ID, or empty string if not available.+ """+ # 1. Check OTel current span+ if otel_trace is not None and format_span_id is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and hasattr(current_span, "context"):+ span_context = current_span.get_span_context()+ if span_context.is_valid:+ return format_span_id(span_context.span_id)+ except Exception:+ pass++ # 2. Check execution context+ if self.execution_span_id:+ return self.execution_span_id++ # 3. Return empty string+ return ""++ def to_log_fields(self) -> Dict[str, Any]:+ """+ Convert the context to log fields.++ Returns:+ A dictionary of log fields including trace_id, span_id, and other context fields.+ """+ fields: Dict[str, Any] = {+ "trace_id": self.get_trace_id(),+ "span_id": self.get_span_id(),+ }++ # Add root context fields+ if self.root is not None:+ root_fields = self.root.to_log_fields()+ fields.update(root_fields)++ # Add operation context fields+ if self.operation is not None:+ operation_fields = self.operation.to_log_fields()+ fields.update(operation_fields)++ # Add extra fields+ fields.update(self.extra)++ return fields++ def bind_root(self, root_attrs: RootSpanAttributes) -> "ObservabilityContext":+ """+ Bind root context.++ Args:+ root_attrs: The root span attributes to bind.++ Returns:+ Self for method chaining.+ """+ self.root = root_attrs+ return self++ def bind_operation(self, operation_attrs: OperationSpanAttributes) -> "ObservabilityContext":+ """+ Bind operation context.++ Args:+ operation_attrs: The operation span attributes to bind.++ Returns:+ Self for method chaining.+ """+ self.operation = operation_attrs+ return self++ def bind_execution(+ self,+ trace_id: Optional[str] = None,+ span_id: Optional[str] = None,+ ) -> "ObservabilityContext":+ """+ Bind execution context for non-request scenarios.++ Args:+ trace_id: Optional trace ID. Generated automatically if not provided.+ span_id: Optional span ID. Derived from trace_id if not provided.++ Returns:+ Self for method chaining.+ """+ self.execution_trace_id = trace_id or uuid.uuid4().hex+ self.execution_span_id = span_id or self.execution_trace_id[:16]+ return self++ def set_extra(self, key: str, value: Any) -> "ObservabilityContext":+ """+ Set an extra attribute.++ Args:+ key: The attribute key.+ value: The attribute value.++ Returns:+ Self for method chaining.+ """+ self.extra[key] = value+ return self+++# Global context variable+_OBSERVABILITY_CONTEXT: contextvars.ContextVar[Optional[ObservabilityContext]] = (+ contextvars.ContextVar(+ "openviking_observability_context",+ default=None,+ )+)+++def get_observability_context() -> ObservabilityContext:+ """+ Get the current observability context.++ If no context is bound, a new empty context is created.++ Returns:+ The current observability context.+ """+ ctx = _OBSERVABILITY_CONTEXT.get()+ if ctx is None:+ ctx = ObservabilityContext()+ _OBSERVABILITY_CONTEXT.set(ctx)+ return ctx+++def set_observability_context(ctx: ObservabilityContext) -> contextvars.Token:+ """+ Set the observability context.++ Args:+ ctx: The observability context to set.++ Returns:+ A token that can be used to reset the context.+ """+ return _OBSERVABILITY_CONTEXT.set(ctx)+++def reset_observability_context(token: contextvars.Token) -> None:+ """+ Reset the observability context to a previous state.++ Args:+ token: The token returned by set_observability_context.+ """+ _OBSERVABILITY_CONTEXT.reset(token)+++def bind_root_context(root_attrs: RootSpanAttributes) -> contextvars.Token:+ """+ Bind root context for HTTP requests.++ Args:+ root_attrs: The root span attributes to bind.++ Returns:+ A token that can be used to reset the context.+ """+ ctx = get_observability_context()+ ctx.bind_root(root_attrs)+ return set_observability_context(ctx)+++def bind_operation_context(operation_attrs: OperationSpanAttributes) -> contextvars.Token:+ """+ Bind operation context for business operations.++ Args:+ operation_attrs: The operation span attributes to bind.++ Returns:+ A token that can be used to reset the context.+ """+ ctx = get_observability_context()+ ctx.bind_operation(operation_attrs)+ return set_observability_context(ctx)+++@contextmanager+def bind_execution_context(+ trace_id: Optional[str] = None,+ span_id: Optional[str] = None,+):+ """+ Context manager for binding execution context in non-request scenarios.++ This is useful for:+ - Process startup+ - CLI execution+ - Background tasks+ - Any code that doesn't run under an HTTP request++ Args:+ trace_id: Optional trace ID. Generated automatically if not provided.+ span_id: Optional span ID. Derived from trace_id if not provided.++ Yields:+ A tuple of (trace_id, span_id).+ """+ # Create new context with execution trace/span IDs+ ctx = ObservabilityContext()+ ctx.bind_execution(trace_id, span_id)++ token = set_observability_context(ctx)++ try:+ yield ctx.execution_trace_id, ctx.execution_span_id+ finally:+ reset_observability_context(token)+++# Convenience aliases for unified context management+bind_root_observability_context = bind_root_context+reset_root_observability_context = reset_observability_context+bind_operation_observability_context = bind_operation_context+++def get_root_observability_context() -> Optional[RootSpanAttributes]:+ """Get the root context from the unified observability context."""+ ctx = get_observability_context()+ return ctx.root+++def get_operation_observability_context() -> Optional[OperationSpanAttributes]:+ """Get the operation context from the unified observability context."""+ ctx = get_observability_context()+ return ctx.operation+++def reset_operation_observability_context(token: contextvars.Token) -> None:+ """Reset the observability context."""+ reset_observability_context(token)diff --git a/openviking/observability/http_observability_middleware.py b/openviking/observability/http_observability_middleware.pynew file mode 100644index 000000000..77c353428--- /dev/null+++ b/openviking/observability/http_observability_middleware.py@@ -0,0 +1,869 @@+"""Unified HTTP observability middleware.++This module intentionally owns both HTTP metrics and HTTP tracing logic.++Rationale:+- A single middleware entry point avoids ordering issues between separate Starlette/FastAPI+ middlewares (e.g. authentication mutating request-scoped fields that metrics need).+- Route template resolution happens after routing, so both tracing and metrics need a shared+ convention for when and how to finalize `http_route`.+"""++from __future__ import annotations++import logging+import threading+import time+import uuid+from contextlib import nullcontext+from dataclasses import dataclass+from typing import Any, Awaitable, Callable, ContextManager, Optional++from starlette.requests import Request+from starlette.responses import Response++from openviking.metrics.datasources import HttpRequestLifecycleDataSource+from openviking.observability.context import (+ bind_root_observability_context,+ reset_root_observability_context,+)+from openviking.telemetry.span_models import RootSpanAttributes, create_root_span_attributes++# Try to import opentelemetry - will be None if not installed+try:+ from opentelemetry import trace as otel_trace+ from opentelemetry.propagate import extract+ from opentelemetry.trace import Status, StatusCode+except ImportError:+ otel_trace = None+ extract = None+ Status = None+ StatusCode = None++logger = logging.getLogger(__name__)++_HTTP_IGNORE_ROUTES = frozenset(+ {+ "/metrics",+ "/health",+ "/ready",+ }+)+++class ShardedInflightCounter:+ """+ Sharded inflight counter to reduce lock contention under high concurrency.++ This class splits the key space into multiple shards, each with its own lock.+ This reduces lock contention compared to a single global lock.++ The number of shards should be a power of 2 for efficient hash-based sharding.+ """++ def __init__(self, num_shards: int = 16):+ """+ Initialize the sharded inflight counter.++ Args:+ num_shards: Number of shards to use. Should be a power of 2.+ Default is 16, which provides good balance between+ memory overhead and lock contention reduction.+ """+ self._num_shards = num_shards+ self._shards: list[dict] = [{} for _ in range(num_shards)]+ self._locks: list[threading.Lock] = [threading.Lock() for _ in range(num_shards)]++ def _get_shard_index(self, key: tuple) -> int:+ """+ Calculate the shard index for a given key.++ Uses the built-in hash function to distribute keys across shards.++ Args:+ key: The key to hash (route, account_id tuple).++ Returns:+ The shard index (0 to num_shards - 1).+ """+ return hash(key) % self._num_shards++ def increment(self, route: str, account_id: Optional[str] = None) -> int:+ """+ Increment the inflight count for a route and account_id.++ Args:+ route: The route template.+ account_id: The account identifier, or None.++ Returns:+ The new inflight count after increment.+ """+ key = (route, account_id)+ shard_idx = self._get_shard_index(key)++ with self._locks[shard_idx]:+ shard = self._shards[shard_idx]+ shard[key] = shard.get(key, 0) + 1+ return shard[key]++ def decrement(self, route: str, account_id: Optional[str] = None) -> int:+ """+ Decrement the inflight count for a route and account_id.++ The count is clamped to never go below zero.++ Args:+ route: The route template.+ account_id: The account identifier, or None.++ Returns:+ The new inflight count after decrement (clamped to >= 0).+ """+ key = (route, account_id)+ shard_idx = self._get_shard_index(key)++ with self._locks[shard_idx]:+ shard = self._shards[shard_idx]+ if key in shard:+ shard[key] -= 1+ if shard[key] <= 0:+ del shard[key]+ return 0+ return shard[key]+ return 0++ def get(self, route: str, account_id: Optional[str] = None) -> int:+ """+ Get the current inflight count for a route and account_id.++ This method uses lock-free reading for better performance under high concurrency.+ In CPython, a single dict.get() call is atomic due to the GIL, so no lock is needed.++ Args:+ route: The route template.+ account_id: The account identifier, or None.++ Returns:+ The current inflight count (0 if not found).+ """+ key = (route, account_id)+ shard_idx = self._get_shard_index(key)++ # Lock-free read: single dict.get() is atomic in CPython+ # If the read fails for any reason, fall back to locked read+ try:+ return self._shards[shard_idx].get(key, 0)+ except Exception:+ # Fall back to locked read if lock-free read fails+ with self._locks[shard_idx]:+ return self._shards[shard_idx].get(key, 0)++ def get_all(self) -> dict[tuple, int]:+ """+ Get all inflight counts across all shards.++ This method first attempts lock-free reading for better performance.+ If lock-free reading fails (e.g., dictionary modified during iteration),+ it falls back to locked reading.++ Note: Lock-free reading may return slightly stale or inconsistent data+ under high concurrency, but this is acceptable for monitoring purposes.++ Returns:+ A dictionary of all (route, account_id) -> count mappings.+ """+ result: dict[tuple, int] = {}++ # First attempt: lock-free reading+ try:+ for i in range(self._num_shards):+ # dict.copy() is atomic in CPython for small dicts+ # but may fail if the dict is being modified+ shard_copy = dict(self._shards[i])+ result.update(shard_copy)+ return result+ except Exception:+ # Lock-free read failed, fall back to locked reading+ pass++ # Fallback: locked reading+ for i in range(self._num_shards):+ with self._locks[i]:+ result.update(self._shards[i])+ return result+++# Global sharded inflight counter instance+# Using 16 shards provides good balance between memory and contention reduction+_INFLIGHT_COUNTER = ShardedInflightCounter(num_shards=16)+++def maybe_start_root_span(+ request: Request, root_attrs: RootSpanAttributes+) -> Optional[ContextManager[Any]]:+ """+ Return a root span context manager when OTel is available.++ Creates a server span for the incoming HTTP request, extracting trace context+ from request headers if present.++ Args:+ request: The incoming HTTP request.+ root_attrs: The root span attributes.++ Returns:+ A span context manager if OTel is available and span creation succeeds,+ None otherwise.+ """+ if otel_trace is None or extract is None:+ logger.debug("OTel trace or extract not available, skipping root span creation")+ return None++ try:+ tracer = otel_trace.get_tracer(__name__)+ if tracer is None:+ logger.warning("OTel tracer is None, cannot create root span")+ return None++ carrier = dict(request.headers)+ context = extract(carrier)+ span_name = f"{request.method} {root_attrs.http_route}"++ return tracer.start_as_current_span(+ name=span_name,+ context=context,+ kind=otel_trace.SpanKind.SERVER,+ )++ except ImportError as e:+ logger.warning("OTel import error when creating root span: %s", str(e))+ return None++ except (TypeError, ValueError, AttributeError) as e:+ logger.warning("OTel configuration error when creating root span: %s", str(e))+ return None++ except Exception as e:+ logger.error(+ "Unexpected error when creating root span: %s",+ str(e),+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+ return None+++def maybe_apply_root_span_attributes(root_attrs: RootSpanAttributes) -> None:+ """+ Best-effort sync of root attributes into the current OTel span.++ Sets the root span attributes (request_id, http_method, http_route, account_id, etc.)+ on the currently active OTel span.++ Args:+ root_attrs: The root span attributes to apply.+ """+ if otel_trace is None:+ logger.debug("OTel trace not available, skipping root span attributes application")+ return++ try:+ current_span = otel_trace.get_current_span()+ if current_span is None:+ logger.debug("No current span found, skipping root span attributes application")+ return++ if not current_span.is_recording():+ logger.debug("Current span is not recording, skipping root span attributes application")+ return++ current_span.set_attributes(root_attrs.to_otel_attributes())++ except (TypeError, ValueError, AttributeError) as e:+ logger.warning("Error applying root span attributes: %s", str(e))++ except Exception as e:+ logger.error(+ "Unexpected error applying root span attributes: %s",+ str(e),+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+++def maybe_apply_root_span_response(root_attrs: RootSpanAttributes) -> None:+ """+ Best-effort sync of HTTP response information into the current root span.++ Sets the HTTP status code and updates the span status based on the response:+ - 5xx: ERROR status+ - 4xx: UNSET status+ - 2xx/3xx: OK status (implicit)++ Args:+ root_attrs: The root span attributes containing the HTTP status code.+ """+ if otel_trace is None or Status is None or StatusCode is None:+ logger.debug(+ "OTel trace or Status/StatusCode not available, skipping response state application"+ )+ return++ try:+ current_span = otel_trace.get_current_span()+ if current_span is None or not current_span.is_recording():+ logger.debug("No active recording span found, skipping response state application")+ return++ current_span.set_attributes(root_attrs.to_otel_attributes())++ status_code = int(root_attrs.http_status_code or 500)++ if status_code >= 500:+ current_span.set_status(Status(StatusCode.ERROR, description=f"HTTP {status_code}"))+ elif status_code >= 400:+ current_span.set_status(Status(StatusCode.UNSET))+ # 2xx/3xx implicitly get OK status++ except (TypeError, ValueError, AttributeError) as e:+ logger.warning("Error applying root span response state: %s", str(e))++ except Exception as e:+ logger.error(+ "Unexpected error applying root span response state: %s",+ str(e),+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+++def maybe_apply_root_span_error(root_attrs: RootSpanAttributes, exc: Exception) -> None:+ """+ Best-effort exception recording for the current root span.++ Records the exception on the currently active OTel span and sets the span+ status to ERROR.++ Args:+ root_attrs: The root span attributes.+ exc: The exception to record.+ """+ if otel_trace is None or Status is None or StatusCode is None:+ logger.debug(+ "OTel trace or Status/StatusCode not available, skipping error state application"+ )+ return++ try:+ current_span = otel_trace.get_current_span()+ if current_span is None or not current_span.is_recording():+ logger.debug("No active recording span found, skipping error state application")+ return++ current_span.set_attributes(root_attrs.to_otel_attributes())+ current_span.record_exception(exc)+ current_span.set_status(Status(StatusCode.ERROR, description=str(exc)))++ except (TypeError, ValueError, AttributeError) as e:+ logger.warning("Error applying root span error state: %s", str(e))++ except Exception as e:+ logger.error(+ "Unexpected error applying root span error state: %s",+ str(e),+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+++def _get_route_template(request: Request) -> str:+ """+ Resolve a shared low-cardinality route template for request observability.++ Extracts the route template from the request scope after routing has occurred.+ If routing has not yet occurred, returns "/__unmatched__".++ Args:+ request: The incoming HTTP request.++ Returns:+ The route template string (e.g., "/api/v1/sessions/{session_id}") if routing+ has occurred, otherwise "/__unmatched__".+ """+ try:+ route = request.scope.get("route")+ path = getattr(route, "path", None)+ if path:+ return str(path)+ return "/__unmatched__"+ except (TypeError, AttributeError) as e:+ logger.debug("Error getting route template: %s", str(e))+ return "/__unmatched__"+ except Exception as e:+ logger.error(+ "Unexpected error getting route template: %s",+ str(e),+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+ return "/__unmatched__"+++def _maybe_update_route_and_span_name(+ *,+ request: Request,+ root_attrs: Any,+ span: Any,+) -> None:+ """+ Update `root_attrs.http_route` and the active span name after routing.++ Starlette/FastAPI perform route matching in the downstream app (i.e. inside `call_next`).+ This means `request.scope["route"]` is often unavailable before `call_next` executes.++ We intentionally:+ - keep `url_path` as the raw request path (always available),+ - fill `http_route` with a low-cardinality route template once routing completes.++ Args:+ request: The incoming HTTP request.+ root_attrs: The root span attributes to update.+ span: The active OTel span (if any).+ """+ try:+ final_route = _get_route_template(request)++ if final_route and final_route != "/__unmatched__":+ if getattr(root_attrs, "http_route", None) != final_route:+ root_attrs.http_route = final_route++ # Keep span name aligned with the finalized route template for better aggregation.+ if span is not None:+ update_name = getattr(span, "update_name", None)+ if callable(update_name):+ update_name(f"{request.method} {final_route}")++ except (TypeError, ValueError, AttributeError) as e:+ logger.warning("Error updating route and span name: %s", str(e))++ except Exception as e:+ logger.error(+ "Unexpected error updating route and span name: %s",+ str(e),+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+++def should_skip_http_metrics(request: Request) -> bool:+ """+ Return whether the request should skip HTTP metrics/tracing entirely.++ Checks if the request path matches any of the ignore routes (health checks,+ metrics endpoint, etc.).++ Args:+ request: The incoming HTTP request.++ Returns:+ True if the request should be skipped, False otherwise.+ """+ try:+ raw_path = str(request.url.path)+ if raw_path in _HTTP_IGNORE_ROUTES:+ return True++ route_template = _get_route_template(request)+ return route_template in _HTTP_IGNORE_ROUTES++ except Exception as e:+ logger.warning("Error checking if should skip HTTP metrics: %s", str(e))+ # Default to not skipping on error+ return False+++@dataclass(slots=True)+class HTTPMetricsStartState:+ """Capture the initial metrics labels used when a request starts."""++ route: str+ account_id: Optional[str]+++def _inflight_delta(route: str, account_id: Optional[str], delta: int) -> int:+ """+ Update and return the current in-process inflight count for a route template.++ This function uses a sharded counter to reduce lock contention under high concurrency.+ The local counter is only used to provide a reasonable gauge value for event emission.+ It is not a correctness mechanism and is clamped to never go below zero.++ Args:+ route: The route template.+ account_id: The account identifier, or None.+ delta: The change to apply (+1 for increment, -1 for decrement).++ Returns:+ The new inflight count after the delta is applied.+ """+ if delta > 0:+ return _INFLIGHT_COUNTER.increment(route, account_id)+ elif delta < 0:+ return _INFLIGHT_COUNTER.decrement(route, account_id)+ else:+ return _INFLIGHT_COUNTER.get(route, account_id)+++def _log_metrics_failure(+ message: str,+ *,+ route: str,+ account_id: Optional[str] = None,+ error: Optional[Exception] = None,+) -> None:+ """+ Emit a log for best-effort metrics failures without affecting request handling.++ This middleware intentionally treats observability as a side channel. The helper keeps+ failure reporting centralized so all swallowed exceptions still leave a trace.++ Logs at WARNING level for known error types, and ERROR level for unexpected errors.+ Stack traces are only included when debug logging is enabled.++ Args:+ message: The error message.+ route: The route template.+ account_id: The account identifier, or None.+ error: The exception that occurred, or None.+ """+ log_level = logging.WARNING if error else logging.DEBUG++ extra = {+ "route": route,+ "account_id": account_id,+ }++ if error:+ logger.log(+ log_level,+ "http metrics write failed: %s - %s",+ message,+ str(error),+ extra=extra,+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+ else:+ logger.log(+ log_level,+ "http metrics write failed: %s",+ message,+ extra=extra,+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+++def apply_http_metrics_start(+ *,+ request: Request,+ root_attrs: RootSpanAttributes,+) -> HTTPMetricsStartState:+ """+ Record the initial inflight sample and store the start-state on the request.++ Called at the beginning of request handling to increment the inflight counter+ and capture the initial route/account_id for later rebalancing.++ Args:+ request: The incoming HTTP request.+ root_attrs: The root span attributes.++ Returns:+ The HTTPMetricsStartState containing the initial route and account_id.+ """+ initial_route = root_attrs.http_route+ initial_account_id = root_attrs.account_id++ state = HTTPMetricsStartState(route=initial_route, account_id=initial_account_id)+ request.state.http_metrics_start_state = state++ try:+ HttpRequestLifecycleDataSource.set_inflight(+ route=initial_route,+ value=_inflight_delta(initial_route, initial_account_id, +1),+ account_id=initial_account_id,+ )++ except (TypeError, ValueError, AttributeError) as e:+ _log_metrics_failure(+ "http.inflight increment failed",+ route=initial_route,+ account_id=initial_account_id,+ error=e,+ )++ except Exception as e:+ logger.error(+ "Unexpected error in http.inflight increment: %s",+ str(e),+ extra={+ "route": initial_route,+ "account_id": initial_account_id,+ },+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )++ return state+++def apply_http_metrics_finalize(+ *,+ request: Request,+ root_attrs: RootSpanAttributes,+ elapsed: float,+) -> None:+ """+ Record final request metrics while preserving current inflight semantics.++ Called at the end of request handling to:+ 1. Rebalance inflight counter if route/account_id changed during request+ 2. Record the request duration and status code+ 3. Decrement the inflight counter++ Args:+ request: The incoming HTTP request.+ root_attrs: The root span attributes.+ elapsed: The request duration in seconds.+ """+ start_state = getattr(request.state, "http_metrics_start_state", None)+ initial_route = getattr(start_state, "route", root_attrs.http_route)+ initial_account_id = getattr(start_state, "account_id", root_attrs.account_id)+ final_route = _get_route_template(request)+ final_account_id = root_attrs.account_id++ # Rebalance inflight counter if route/account_id changed during request+ if (final_route, final_account_id) != (initial_route, initial_account_id):+ try:+ HttpRequestLifecycleDataSource.set_inflight(+ route=initial_route,+ value=_inflight_delta(initial_route, initial_account_id, -1),+ account_id=initial_account_id,+ )+ HttpRequestLifecycleDataSource.set_inflight(+ route=final_route,+ value=_inflight_delta(final_route, final_account_id, +1),+ account_id=final_account_id,+ )++ except (TypeError, ValueError, AttributeError) as e:+ _log_metrics_failure(+ "http.inflight rebalance failed",+ route=final_route,+ account_id=final_account_id,+ error=e,+ )++ except Exception as e:+ logger.error(+ "Unexpected error in http.inflight rebalance: %s",+ str(e),+ extra={+ "route": final_route,+ "account_id": final_account_id,+ },+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )++ # Record request duration and status code+ try:+ HttpRequestLifecycleDataSource.record_request(+ method=root_attrs.http_method,+ route=final_route,+ status=str(root_attrs.http_status_code or 500),+ duration_seconds=elapsed,+ account_id=final_account_id,+ )++ except (TypeError, ValueError, AttributeError) as e:+ _log_metrics_failure(+ "http.request recording failed",+ route=final_route,+ account_id=final_account_id,+ error=e,+ )++ except Exception as e:+ logger.error(+ "Unexpected error in http.request recording: %s",+ str(e),+ extra={+ "route": final_route,+ "account_id": final_account_id,+ },+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )++ # Decrement inflight counter+ try:+ HttpRequestLifecycleDataSource.set_inflight(+ route=final_route,+ value=_inflight_delta(final_route, final_account_id, -1),+ account_id=final_account_id,+ )++ except (TypeError, ValueError, AttributeError) as e:+ _log_metrics_failure(+ "http.inflight decrement failed",+ route=final_route,+ account_id=final_account_id,+ error=e,+ )++ except Exception as e:+ logger.error(+ "Unexpected error in http.inflight decrement: %s",+ str(e),+ extra={+ "route": final_route,+ "account_id": final_account_id,+ },+ exc_info=logger.isEnabledFor(logging.DEBUG),+ )+++async def _execute_request_with_span(+ request: Request,+ call_next: Callable[[Request], Awaitable[Response]],+ root_attrs: RootSpanAttributes,+ span_cm: Optional[Any],+) -> Response:+ """+ Execute the request with optional span context management.++ This function encapsulates the common logic for both cases:+ 1. When OTel is available and a span context manager exists+ 2. When OTel is not available or span creation failed++ Uses `contextlib.nullcontext` to unify the handling of both cases,+ eliminating code duplication.++ Args:+ request: The incoming HTTP request.+ call_next: The next middleware/handler in the chain.+ root_attrs: The root span attributes.+ span_cm: The span context manager (if available), or None.++ Returns:+ The response from the next handler.++ Raises:+ Any exception raised by the next handler.+ """+ # Use nullcontext when span_cm is None to unify handling+ context = span_cm if span_cm is not None else nullcontext()++ with context as span:+ try:+ response = await call_next(request)++ # Update route and span name if we have a real span+ if span is not None:+ _maybe_update_route_and_span_name(+ request=request,+ root_attrs=root_attrs,+ span=span,+ )+ # Apply root attributes to span+ maybe_apply_root_span_attributes(root_attrs)++ # Set status code from response+ root_attrs.http_status_code = int(getattr(response, "status_code", 500))++ # Apply response state (status code, etc.)+ maybe_apply_root_span_response(root_attrs)++ return response++ except Exception as exc:+ # Update route and span name if we have a real span+ if span is not None:+ _maybe_update_route_and_span_name(+ request=request,+ root_attrs=root_attrs,+ span=span,+ )+ # Apply root attributes to span+ maybe_apply_root_span_attributes(root_attrs)++ # Apply error state to span+ maybe_apply_root_span_error(root_attrs, exc)++ raise+++def create_http_observability_middleware() -> Callable[[Request, Callable], Response]:+ """+ Create the unified HTTP observability middleware.++ This middleware handles:+ - HTTP metrics collection (inflight, request duration, status codes)+ - OTel trace span creation and management+ - Context propagation (root observability context binding)++ Returns:+ The middleware function.+ """++ async def middleware(request: Request, call_next: Callable) -> Response:+ # Skip metrics/tracing for health check and metrics endpoints+ if should_skip_http_metrics(request):+ return await call_next(request)++ # Extract request information+ raw_path = str(request.url.path)+ route_template = _get_route_template(request)+ request_id = request.headers.get("x-request-id") or str(uuid.uuid4())++ # Create root span attributes+ root_attrs = create_root_span_attributes(+ http_method=request.method,+ http_route=route_template,+ request_id=request_id,+ url_path=raw_path,+ url_scheme=request.url.scheme,+ http_host=request.url.netloc,+ source_type=request.headers.get("x-source-type"),+ source_version=request.headers.get("x-source-version"),+ )++ # Attach to request state for later access+ request.state.root_span_attrs = root_attrs+ request.state.request_id = request_id++ # Bind root context and start metrics+ root_token = bind_root_observability_context(root_attrs)+ apply_http_metrics_start(request=request, root_attrs=root_attrs)++ start = time.perf_counter()++ try:+ # Try to create a root span if OTel is available+ span_cm = maybe_start_root_span(request, root_attrs)++ # Execute the request with unified span handling+ response = await _execute_request_with_span(+ request=request,+ call_next=call_next,+ root_attrs=root_attrs,+ span_cm=span_cm,+ )++ return response++ finally:+ # Calculate elapsed time and finalize metrics+ elapsed = time.perf_counter() - start+ apply_http_metrics_finalize(+ request=request,+ root_attrs=root_attrs,+ elapsed=elapsed,+ )++ # Reset root context+ reset_root_observability_context(root_token)++ return middlewarediff --git a/openviking/observability/log_trace_bridge.py b/openviking/observability/log_trace_bridge.pynew file mode 100644index 000000000..c63ea68d3--- /dev/null+++ b/openviking/observability/log_trace_bridge.py@@ -0,0 +1,215 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Log to Trace Bridge Module.++Provides convenience functions for developers to log messages and span events/exceptions+simultaneously, reducing redundant code and simplifying development.++Key features:+- log_and_add_event: Log message and add span event simultaneously+- log_exception: Log error and record span exception simultaneously+- Preserves original types for OTel attributes (str, int, float, bool, arrays)+- Safe fallback: only logs when OTel is not available+"""++from __future__ import annotations++import logging+from typing import Any, Sequence, Union++# Try to import opentelemetry - will be None if not installed+try:+ from opentelemetry import trace as otel_trace+ from opentelemetry.trace import Status, StatusCode+except ImportError:+ otel_trace = None+ Status = None+ StatusCode = None+++# Define OTel-compatible attribute value types+OTelAttributeValue = Union[+ str,+ int,+ float,+ bool,+ Sequence[str],+ Sequence[int],+ Sequence[float],+ Sequence[bool],+]+++def _to_otel_attributes(kwargs: dict[str, Any]) -> dict[str, OTelAttributeValue]:+ """+ Convert kwargs to OTel-compatible span attributes.++ Preserves original types for supported types (str, int, float, bool, and homogeneous arrays).+ Unsupported types are converted to strings.++ Args:+ kwargs: Dictionary of key-value pairs to convert.++ Returns:+ Dictionary of OTel-compatible attributes.+ """+ attributes: dict[str, OTelAttributeValue] = {}++ for key, value in kwargs.items():+ if value is None:+ # Skip None values+ continue+ elif isinstance(value, (str, int, float, bool)):+ # Primitive types - use as-is+ attributes[key] = value+ elif isinstance(value, (list, tuple)):+ # Array types - check if homogeneous+ if not value:+ # Empty array - skip+ continue++ # Check if all elements are of the same primitive type+ first_type = type(value[0])+ if first_type not in (str, int, float, bool):+ # Mixed or unsupported types - convert to string array+ attributes[key] = [str(v) for v in value]+ continue++ # Verify all elements are of the same type+ all_same_type = all(isinstance(v, first_type) for v in value)+ if all_same_type:+ # Homogeneous array - use as-is+ attributes[key] = list(value)+ else:+ # Mixed types - convert to string array+ attributes[key] = [str(v) for v in value]+ else:+ # Other types - convert to string+ attributes[key] = str(value)++ return attributes+++def log_and_add_event(logger: logging.Logger, level: int, message: str, **kwargs: Any) -> None:+ """+ Log a message and add it as a span event simultaneously.++ This function allows developers to log messages and record span events+ in a single call, reducing redundant code.++ Args:+ logger: The logger instance to use.+ level: The logging level (logging.INFO, logging.DEBUG, etc.).+ message: The log message.+ **kwargs: Additional key-value pairs to be included as log extra+ and span attributes. Supported types: str, int, float, bool,+ and homogeneous arrays of these types. Unsupported types are+ converted to strings.+ """+ # Log the message+ logger.log(level, message, extra=kwargs)++ # If OTel is available, also record as a span event+ if otel_trace is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and current_span.is_recording():+ # Convert kwargs to OTel-compatible attributes, preserving types+ attributes = _to_otel_attributes(kwargs)+ current_span.add_event(message, attributes=attributes)+ except Exception:+ # Best-effort only - don't break logging if span event fails+ pass+++def log_exception(logger: logging.Logger, message: str, exc: Exception, **kwargs: Any) -> None:+ """+ Log an error and record it as a span exception simultaneously.++ This function logs the exception with full stack trace and also records+ it in the current OTel span if available.++ Args:+ logger: The logger instance to use.+ message: The error message.+ exc: The exception object.+ **kwargs: Additional key-value pairs to be included as log extra+ and span attributes.+ """+ # Log the error with exception stack trace+ logger.error(message, exc_info=exc, extra=kwargs)++ # If OTel is available, also record the exception in the span+ if otel_trace is not None and Status is not None and StatusCode is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and current_span.is_recording():+ # Convert kwargs to OTel-compatible attributes+ attributes = _to_otel_attributes(kwargs)+ attributes["message"] = message++ # Record the exception in the span+ current_span.record_exception(exc, attributes=attributes)+ current_span.set_status(Status(StatusCode.ERROR, description=str(exc)))+ except Exception:+ # Best-effort only - don't break logging if span recording fails+ pass+++def log_debug(logger: logging.Logger, message: str, **kwargs: Any) -> None:+ """+ Log a DEBUG level message and add it as a span event.++ Convenience wrapper around log_and_add_event with DEBUG level.++ Args:+ logger: The logger instance to use.+ message: The log message.+ **kwargs: Additional key-value pairs.+ """+ log_and_add_event(logger, logging.DEBUG, message, **kwargs)+++def log_info(logger: logging.Logger, message: str, **kwargs: Any) -> None:+ """+ Log an INFO level message and add it as a span event.++ Convenience wrapper around log_and_add_event with INFO level.++ Args:+ logger: The logger instance to use.+ message: The log message.+ **kwargs: Additional key-value pairs.+ """+ log_and_add_event(logger, logging.INFO, message, **kwargs)+++def log_warning(logger: logging.Logger, message: str, **kwargs: Any) -> None:+ """+ Log a WARNING level message and add it as a span event.++ Convenience wrapper around log_and_add_event with WARNING level.++ Args:+ logger: The logger instance to use.+ message: The log message.+ **kwargs: Additional key-value pairs.+ """+ log_and_add_event(logger, logging.WARNING, message, **kwargs)+++def log_error(logger: logging.Logger, message: str, **kwargs: Any) -> None:+ """+ Log an ERROR level message and add it as a span event.++ Convenience wrapper around log_and_add_event with ERROR level.+ Note: This does not record an exception. Use log_exception for that.++ Args:+ logger: The logger instance to use.+ message: The log message.+ **kwargs: Additional key-value pairs.+ """+ log_and_add_event(logger, logging.ERROR, message, **kwargs)diff --git a/openviking/server/app.py b/openviking/server/app.pyindex 6e1cb00ee..8c04d136e 100644--- a/openviking/server/app.py+++ b/openviking/server/app.py@@ -46,6 +46,7 @@ from openviking.service.core import OpenVikingServicefrom openviking.service.task_tracker import get_task_trackerfrom openviking_cli.exceptions import OpenVikingErrorfrom openviking_cli.utils import get_logger+from openviking_cli.utils.logger import init_otel_log_handler_from_server_configlogger = get_logger(__name__)@@ -126,17 +127,18 @@ def create_app(task_tracker = get_task_tracker()task_tracker.start_cleanup_loop()- # Initialize tracer+ # Initialize tracing and OTLP log export from server.observability.from openviking.telemetry import tracer_module- tracer_module.init_tracer_from_config()+ tracer_module.init_tracer_from_server_config(config)+ init_otel_log_handler_from_server_config(config)yield# Cleanup- from openviking.metrics.global_api import shutdown_metrics+ from openviking.metrics.global_api import shutdown_metrics_async- shutdown_metrics(app=app)+ await shutdown_metrics_async(app=app)task_tracker.stop_cleanup_loop()if owns_service and service:try:@@ -165,6 +167,22 @@ def create_app(allow_headers=["*"],)+ # Add HTTP observability middleware first (metrics, tracing)+ # Note: In FastAPI/Starlette, middleware added later executes first (outer layer).+ # We want timing to be the outermost layer to measure the full request duration.+ from openviking.observability.http_observability_middleware import (+ create_http_observability_middleware,+ )++ http_observability_middleware = create_http_observability_middleware()++ @app.middleware("http")+ async def add_http_observability(request: Request, call_next: Callable):+ return await http_observability_middleware(request, call_next)++ # Add request timing middleware last (so it executes first as the outermost layer)+ # This ensures X-Process-Time includes the full request duration including+ # observability middleware overhead.# Add request header logging middleware (for debug)@app.middleware("http")async def log_request_headers(request: Request, call_next: Callable):@@ -181,19 +199,25 @@ def create_app(# Add request timing middleware@app.middleware("http")async def add_timing(request: Request, call_next: Callable):- start_time = time.time()- response = await call_next(request)- process_time = time.time() - start_time- response.headers["X-Process-Time"] = str(process_time)- return response+ """+ Middleware to measure request processing time.- from openviking.metrics.http_middleware import create_http_metrics_middleware+ This middleware is added last so it executes as the outermost layer,+ ensuring X-Process-Time includes the full request duration including+ all other middleware overhead.- http_metrics_middleware = create_http_metrics_middleware()+ Args:+ request: The incoming HTTP request.+ call_next: The next middleware/handler in the chain.- @app.middleware("http")- async def add_http_metrics(request: Request, call_next: Callable):- return await http_metrics_middleware(request, call_next)+ Returns:+ The response with X-Process-Time header added.+ """+ start_time = time.perf_counter()+ response = await call_next(request)+ process_time = time.perf_counter() - start_time+ response.headers["X-Process-Time"] = str(process_time)+ return response# Add exception handler for OpenVikingError@app.exception_handler(OpenVikingError)diff --git a/openviking/server/auth.py b/openviking/server/auth.pyindex 2b18aabb7..e6517de82 100644--- a/openviking/server/auth.py+++ b/openviking/server/auth.py@@ -7,8 +7,8 @@ from typing import Optionalfrom fastapi import Depends, Header, Request-from openviking.metrics.account_context import set_metric_account_contextfrom openviking.server.identity import AuthMode, RequestContext, ResolvedIdentity, Role+from openviking.telemetry.span_models import update_root_span_identityfrom openviking_cli.exceptions import (InvalidArgumentError,PermissionDeniedError,@@ -274,8 +274,14 @@ async def get_request_context(else identity.namespace_policy),)- request.state.metric_account_id = ctx.account_id- set_metric_account_context(account_id=ctx.account_id)+ # Update the unified root observability context after authentication succeeds.+ update_root_span_identity(+ request_state=request.state,+ account_id=identity.account_id,+ user_id=identity.user_id,+ agent_id=identity.agent_id,+ )+return ctxdiff --git a/openviking/server/config.py b/openviking/server/config.pyindex c48bb8daa..6f1153192 100644--- a/openviking/server/config.py+++ b/openviking/server/config.py@@ -35,6 +35,43 @@ class MetricsAccountDimensionConfig(BaseModel):model_config = {"extra": "forbid"}+class PrometheusExporterConfig(BaseModel):+ """Prometheus exporter configuration."""++ enabled: bool = True++ model_config = {"extra": "forbid"}+++class OTelExporterConfig(BaseModel):+ """OpenTelemetry exporter configuration."""++ class TLSConfig(BaseModel):+ """TLS configuration for OTLP exporters."""++ insecure: bool = False++ model_config = {"extra": "forbid"}++ enabled: bool = False+ protocol: str = "grpc" # "grpc" or "http"+ tls: TLSConfig = Field(default_factory=TLSConfig)+ endpoint: str = "localhost:4317" # gRPC default: 4317; HTTP default: 4318+ service_name: str = "openviking-server"+ export_interval_ms: int = 10000++ model_config = {"extra": "forbid"}+++class MetricsExportersConfig(BaseModel):+ """Metrics exporters configuration."""++ prometheus: PrometheusExporterConfig = Field(default_factory=PrometheusExporterConfig)+ otel: OTelExporterConfig = Field(default_factory=OTelExporterConfig)++ model_config = {"extra": "forbid"}++class MetricsConfig(BaseModel):"""Metrics subsystem configuration."""@@ -42,6 +79,7 @@ class MetricsConfig(BaseModel):account_dimension: MetricsAccountDimensionConfig = Field(default_factory=MetricsAccountDimensionConfig)+ exporters: MetricsExportersConfig = Field(default_factory=MetricsExportersConfig)model_config = {"extra": "forbid"}@@ -50,6 +88,8 @@ class ObservabilityConfig(BaseModel):"""Server-side observability configuration."""metrics: MetricsConfig = Field(default_factory=MetricsConfig)+ traces: OTelExporterConfig = Field(default_factory=OTelExporterConfig)+ logs: OTelExporterConfig = Field(default_factory=OTelExporterConfig)model_config = {"extra": "forbid"}diff --git a/openviking/server/routers/metrics.py b/openviking/server/routers/metrics.pyindex 65cb7d863..848908025 100644--- a/openviking/server/routers/metrics.py+++ b/openviking/server/routers/metrics.py@@ -5,17 +5,24 @@from fastapi import APIRouter, Requestfrom fastapi.responses import PlainTextResponse+from openviking.metrics.exporters.prometheus import PrometheusExporter+router = APIRouter(tags=["metrics"])@router.get("/metrics")async def metrics(request: Request):"""Return Prometheus metrics in text exposition format."""- exporter = getattr(request.app.state, "metrics_exporter", None)- if exporter is None:+ exporters = getattr(request.app.state, "metrics_exporters", [])+ # Find PrometheusExporter from the list+ prometheus_exporter = next(+ (e for e in exporters if isinstance(e, PrometheusExporter)),+ None,+ )+ if prometheus_exporter is None:return PlainTextResponse(status_code=404, content="Prometheus metrics are disabled.\n")return PlainTextResponse(- content=await exporter.export(),+ content=await prometheus_exporter.export(),media_type="text/plain; version=0.0.4; charset=utf-8",)diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.pyindex 32984fe1e..7e7b2ba64 100644--- a/openviking/storage/queuefs/semantic_processor.py+++ b/openviking/storage/queuefs/semantic_processor.py@@ -9,9 +9,9 @@ from dataclasses import dataclass, fieldfrom typing import Any, Dict, List, Optional, Set, Tuplefrom openviking.core.namespace import agent_space_fragment, user_space_fragment-from openviking.metrics.account_context import (- bind_metric_account_context,- reset_metric_account_context,+from openviking.observability.context import (+ bind_root_observability_context,+ reset_root_observability_context,)from openviking.parse.parsers.constants import (CODE_EXTENSIONS,@@ -34,6 +34,7 @@ from openviking.storage.queuefs.semantic_msg import SemanticMsgfrom openviking.storage.viking_fs import get_viking_fsfrom openviking.telemetry import bind_telemetry, bind_telemetry_stage, resolve_telemetryfrom openviking.telemetry.request_wait_tracker import get_request_wait_tracker+from openviking.telemetry.span_models import create_root_span_attributesfrom openviking.utils.circuit_breaker import (CircuitBreaker,CircuitBreakerOpen,@@ -274,7 +275,16 @@ class SemanticProcessor(DequeueHandlerBase):collector = resolve_telemetry(msg.telemetry_id)telemetry_ctx = bind_telemetry(collector) if collector is not None else nullcontext()with telemetry_ctx:- metric_account_token = bind_metric_account_context(account_id=msg.account_id)+ root_attrs = create_root_span_attributes(+ http_method="QUEUE",+ http_route=msg.context_type or "/queuefs/semantic",+ request_id=msg.telemetry_id or msg.id,+ url_path=msg.uri,+ )+ root_attrs.account_id = msg.account_id+ root_attrs.user_id = msg.user_id+ root_attrs.agent_id = msg.agent_id+ root_context_token = bind_root_observability_context(root_attrs)try:self._current_msg = msgself._current_ctx = self._ctx_from_semantic_msg(msg)@@ -337,7 +347,7 @@ class SemanticProcessor(DequeueHandlerBase):self._circuit_breaker.record_success()return Nonefinally:- reset_metric_account_context(metric_account_token)+ reset_root_observability_context(root_context_token)except Exception as e:error_class = classify_api_error(e)diff --git a/openviking/telemetry/execution.py b/openviking/telemetry/execution.pyindex df1679639..019946828 100644--- a/openviking/telemetry/execution.py+++ b/openviking/telemetry/execution.py@@ -7,12 +7,17 @@ from __future__ import annotationsfrom dataclasses import dataclassfrom typing import Any, Awaitable, Callable, Generic, Optional, TypeVar+from openviking.observability.context import (+ bind_operation_observability_context,+ reset_operation_observability_context,+)from openviking_cli.exceptions import InvalidArgumentErrorfrom openviking_cli.utils import get_loggerfrom .context import bind_telemetryfrom .operation import OperationTelemetry, TelemetrySnapshotfrom .request import TelemetryRequest, TelemetrySelection, normalize_telemetry_request+from .span_models import OperationSpanAttributesT = TypeVar("T")logger = get_logger(__name__)@@ -36,16 +41,16 @@ def parse_telemetry_selection(telemetry: TelemetryRequest) -> TelemetrySelectiondef build_telemetry_payload(- snapshot: TelemetrySnapshot | None,+ snapshot: Optional[TelemetrySnapshot],selection: TelemetrySelection,-) -> dict[str, Any] | None:+) -> Optional[dict[str, Any]]:"""Build a telemetry payload from a finished snapshot."""if snapshot is None or not selection.include_payload:return Nonereturn snapshot.to_dict(include_summary=selection.include_summary)-def _log_telemetry_summary(snapshot: TelemetrySnapshot | None) -> None:+def _log_telemetry_summary(snapshot: Optional[TelemetrySnapshot]) -> None:if snapshot is None:returnlogger.info(@@ -75,6 +80,78 @@ def attach_telemetry_payload(return result+# Try to import opentelemetry - will be None if not installed+try:+ from opentelemetry import trace as otel_trace+ from opentelemetry.trace import Status, StatusCode+except ImportError:+ otel_trace = None+ Status = None+ StatusCode = None+++def _fill_telemetry_summary_to_span(snapshot, span):+ """fill OperationTelemetry summary into span++ Args:+ snapshot: TelemetrySnapshot object+ span: current OTel span+ """+ if snapshot is None or span is None:+ return++ try:+ # Build OperationSpanAttributes from snapshot for consistent span attributes.+ op_attrs = OperationSpanAttributes.from_telemetry_snapshot(+ operation=snapshot.operation,+ telemetry_id=snapshot.telemetry_id,+ status=snapshot.status,+ summary=snapshot.summary,+ )++ # Apply to span attributes (best-effort).+ span.set_attributes(op_attrs.to_otel_attributes())++ # Attach summary as a span event for debugging (best-effort).+ if snapshot.summary:+ span.add_event("telemetry_summary", attributes={"summary": str(snapshot.summary)})+ except Exception:+ logger.debug("failed to fill telemetry summary into span", exc_info=True)+++def _fill_operation_context_from_snapshot(+ operation_attrs: OperationSpanAttributes,+ snapshot: Optional[TelemetrySnapshot],+) -> OperationSpanAttributes:+ """+ Update the bound operation context from a finished telemetry snapshot.++ Returns a new OperationSpanAttributes instance with the updated fields,+ rather than modifying the input object in place. This avoids unintended+ side effects when the same object is referenced elsewhere.++ Args:+ operation_attrs: The original operation attributes.+ snapshot: The telemetry snapshot (optional).++ Returns:+ A new OperationSpanAttributes instance with updated fields, or the original+ instance if snapshot is None.+ """+ if snapshot is None:+ return operation_attrs++ # Create a new instance from the snapshot+ updated = OperationSpanAttributes.from_telemetry_snapshot(+ operation=operation_attrs.operation,+ telemetry_id=snapshot.telemetry_id,+ status=snapshot.summary.get("status"),+ summary=snapshot.summary,+ )++ return updated++async def run_with_telemetry(*,operation: str,@@ -88,36 +165,93 @@ async def run_with_telemetry(operation=operation,enabled=True,)+ operation_attrs = OperationSpanAttributes(+ operation=operation,+ telemetry_id=collector.telemetry_id,+ )+ operation_token = bind_operation_observability_context(operation_attrs)++ # If OTel is available, create an operation span (best-effort).+ span_context_manager = None+ if otel_trace is not None:+ try:+ tracer = otel_trace.get_tracer(__name__)+ span_context_manager = tracer.start_as_current_span(+ name=operation,+ kind=otel_trace.SpanKind.INTERNAL,+ )+ except Exception:+ logger.debug("failed to start operation span", exc_info=True)+ span_context_manager = Nonetry:- with bind_telemetry(collector):- result = await fn()- except Exception as exc:- collector.set_error(operation, type(exc).__name__, str(exc))- snapshot = collector.finish(status=error_status)+ try:+ with bind_telemetry(collector):+ if span_context_manager is not None:+ with span_context_manager as span:+ try:+ span.set_attributes(operation_attrs.to_otel_attributes())+ except Exception:+ logger.debug(+ "failed to set operation span attributes (initial)",+ exc_info=True,+ )+ result = await fn()+ else:+ result = await fn()+ except Exception as exc:+ collector.set_error(operation, type(exc).__name__, str(exc))+ snapshot = collector.finish(status=error_status)+ _log_telemetry_summary(snapshot)++ # If a span exists, record exception and backfill summary (best-effort).+ if otel_trace is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and current_span.is_recording():+ current_span.record_exception(exc)+ current_span.set_status(Status(StatusCode.ERROR, description=str(exc)))+ _fill_telemetry_summary_to_span(snapshot, current_span)+ except Exception:+ logger.debug(+ "failed to record exception into operation span",+ exc_info=True,+ )++ raise++ snapshot = collector.finish(status="ok")_log_telemetry_summary(snapshot)- raise- snapshot = collector.finish(status="ok")- _log_telemetry_summary(snapshot)- try:- if snapshot is not None:- from openviking.metrics.datasources.telemetry_bridge import (- TelemetryBridgeEventDataSource,- )+ # If a span exists, backfill summary (best-effort).+ if otel_trace is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and current_span.is_recording():+ _fill_telemetry_summary_to_span(snapshot, current_span)+ except Exception:+ logger.debug("failed to backfill telemetry summary into span", exc_info=True)- TelemetryBridgeEventDataSource.record_summary(snapshot.summary)- except Exception:- pass- telemetry_payload = build_telemetry_payload(- snapshot,- selection,- )- return TelemetryExecutionResult(- result=result,- telemetry=telemetry_payload,- selection=selection,- )+ try:+ if snapshot is not None:+ from openviking.metrics.datasources.telemetry_bridge import (+ TelemetryBridgeEventDataSource,+ )++ TelemetryBridgeEventDataSource.record_summary(snapshot.summary)+ except Exception:+ logger.debug("failed to record telemetry summary into metrics bridge", exc_info=True)+ telemetry_payload = build_telemetry_payload(+ snapshot,+ selection,+ )+ return TelemetryExecutionResult(+ result=result,+ telemetry=telemetry_payload,+ selection=selection,+ )+ finally:+ reset_operation_observability_context(operation_token)__all__ = [diff --git a/openviking/telemetry/span_models.py b/openviking/telemetry/span_models.pynew file mode 100644index 000000000..49d5ff0f6--- /dev/null+++ b/openviking/telemetry/span_models.py@@ -0,0 +1,376 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Span model definitions.++This module defines the data structures for root spans and operation spans and+provides shared field management, validation-friendly constructors, and+serialization helpers.++Key features:+- Shared field definitions to avoid hard-coded attribute names+- Type-safe dataclass-based models+- Factory helpers for consistent span/context creation+- JSON and dict serialization helpers+- Lightweight integration with OpenTelemetry attribute APIs+"""++from __future__ import annotations++import json+from dataclasses import asdict, dataclass+from typing import Any, Dict, Optional+++@dataclass+class RootSpanAttributes:+ """Attributes for the root span associated with one HTTP request."""++ # HTTP semantic fields+ http_method: str+ """HTTP method, such as GET or POST."""++ http_route: str+ """Low-cardinality route template, such as `/sessions/{session_id}`."""++ # Request-level fields+ request_id: str+ """Unique request identifier."""++ http_status_code: Optional[int] = None+ """HTTP status code, populated after the response is available."""++ url_path: Optional[str] = None+ """Raw request path for debugging. This may be high-cardinality."""++ url_scheme: Optional[str] = None+ """URL scheme, such as `http` or `https`."""++ http_host: Optional[str] = None+ """HTTP host value."""++ source_type: Optional[str] = None+ """Client source type."""++ source_version: Optional[str] = None+ """Client source version."""++ # Identity fields populated after authentication+ account_id: Optional[str] = None+ """Tenant or account identifier."""++ user_id: Optional[str] = None+ """User identifier."""++ agent_id: Optional[str] = None+ """Agent identifier."""++ def to_otel_attributes(self) -> Dict[str, Any]:+ """Convert the model into an OpenTelemetry span attribute mapping."""+ attrs = {+ "http.method": self.http_method,+ "http.route": self.http_route,+ "request_id": self.request_id,+ }++ if self.http_status_code is not None:+ attrs["http.status_code"] = self.http_status_code+ if self.url_path is not None:+ attrs["url.path"] = self.url_path+ if self.url_scheme is not None:+ attrs["url.scheme"] = self.url_scheme+ if self.http_host is not None:+ attrs["http.host"] = self.http_host+ if self.source_type is not None:+ attrs["source_type"] = self.source_type+ if self.source_version is not None:+ attrs["source_version"] = self.source_version+ if self.account_id is not None:+ attrs["account_id"] = self.account_id+ if self.user_id is not None:+ attrs["user_id"] = self.user_id+ if self.agent_id is not None:+ attrs["agent_id"] = self.agent_id++ return attrs++ def to_log_fields(self) -> Dict[str, Any]:+ """Convert the root context into structured log fields."""+ fields: Dict[str, Any] = {+ "request_id": self.request_id,+ "http_method": self.http_method,+ "http_route": self.http_route,+ }+ if self.http_status_code is not None:+ fields["http_status_code"] = self.http_status_code+ if self.account_id is not None:+ fields["account_id"] = self.account_id+ if self.user_id is not None:+ fields["user_id"] = self.user_id+ if self.agent_id is not None:+ fields["agent_id"] = self.agent_id+ return fields++ def to_dict(self) -> Dict[str, Any]:+ """Return all fields as a plain dictionary."""+ return asdict(self)++ def to_json(self) -> str:+ """Serialize the model into a JSON string."""+ return json.dumps(self.to_dict(), ensure_ascii=False)++ @classmethod+ def from_dict(cls, data: Dict[str, Any]) -> RootSpanAttributes:+ """Build a `RootSpanAttributes` instance from a dictionary."""+ return cls(**data)++ @classmethod+ def from_json(cls, json_str: str) -> RootSpanAttributes:+ """Build a `RootSpanAttributes` instance from a JSON string."""+ data = json.loads(json_str)+ return cls.from_dict(data)+++@dataclass+class OperationSpanAttributes:+ """Attributes for one business-level operation span."""++ # Business-level fields+ operation: str+ """Operation name, such as `search.find`."""++ telemetry_id: str+ """Telemetry identifier for one operation execution."""++ status: Optional[str] = None+ """Business outcome, such as `ok` or `error`."""++ # Token-related fields+ tokens_total: Optional[int] = None+ """Total token count."""++ # Vector-related fields+ vector_searches: Optional[int] = None+ """Number of vector search operations."""++ vector_scanned: Optional[int] = None+ """Number of scanned vectors."""++ vector_returned: Optional[int] = None+ """Number of returned vectors."""++ # Memory-related fields+ memory_extracted: Optional[int] = None+ """Number of extracted memory items."""++ # Error-related fields+ errors_stage: Optional[str] = None+ """Stage where the error occurred."""++ errors_error_code: Optional[str] = None+ """Error code."""++ errors_message: Optional[str] = None+ """Error message."""++ def to_otel_attributes(self) -> Dict[str, Any]:+ """Convert the model into an OpenTelemetry span attribute mapping."""+ attrs = {+ "operation": self.operation,+ "telemetry_id": self.telemetry_id,+ }++ if self.status is not None:+ attrs["status"] = self.status+ if self.tokens_total is not None:+ attrs["tokens.total"] = self.tokens_total+ if self.vector_searches is not None:+ attrs["vector.searches"] = self.vector_searches+ if self.vector_scanned is not None:+ attrs["vector.scanned"] = self.vector_scanned+ if self.vector_returned is not None:+ attrs["vector.returned"] = self.vector_returned+ if self.memory_extracted is not None:+ attrs["memory.extracted"] = self.memory_extracted+ if self.errors_stage is not None:+ attrs["errors.stage"] = self.errors_stage+ if self.errors_error_code is not None:+ attrs["errors.error_code"] = self.errors_error_code+ if self.errors_message is not None:+ attrs["errors.message"] = self.errors_message++ return attrs++ def to_log_fields(self) -> Dict[str, Any]:+ """Convert the operation context into structured log fields."""+ fields: Dict[str, Any] = {+ "operation": self.operation,+ "telemetry_id": self.telemetry_id,+ }+ if self.status is not None:+ fields["status"] = self.status+ if self.tokens_total is not None:+ fields["tokens_total"] = self.tokens_total+ return fields++ def to_dict(self) -> Dict[str, Any]:+ """Return all fields as a plain dictionary."""+ return asdict(self)++ def to_json(self) -> str:+ """Serialize the model into a JSON string."""+ return json.dumps(self.to_dict(), ensure_ascii=False)++ @classmethod+ def from_dict(cls, data: Dict[str, Any]) -> OperationSpanAttributes:+ """Build an `OperationSpanAttributes` instance from a dictionary."""+ return cls(**data)++ @classmethod+ def from_json(cls, json_str: str) -> OperationSpanAttributes:+ """Build an `OperationSpanAttributes` instance from a JSON string."""+ data = json.loads(json_str)+ return cls.from_dict(data)++ @classmethod+ def from_telemetry_snapshot(+ cls,+ operation: str,+ telemetry_id: str,+ status: Optional[str] = None,+ summary: Optional[Dict[str, Any]] = None,+ ) -> OperationSpanAttributes:+ """Build an instance from an `OperationTelemetry` summary snapshot."""+ attrs = cls(+ operation=operation,+ telemetry_id=telemetry_id,+ status=status,+ )++ if summary:+ # Extract token-related fields from the summary payload.+ if "tokens" in summary:+ tokens = summary["tokens"]+ if "total" in tokens:+ attrs.tokens_total = tokens["total"]++ # Extract vector-related fields from the summary payload.+ if "vector" in summary:+ vector = summary["vector"]+ if "searches" in vector:+ attrs.vector_searches = vector["searches"]+ if "scanned" in vector:+ attrs.vector_scanned = vector["scanned"]+ if "returned" in vector:+ attrs.vector_returned = vector["returned"]++ # Extract memory-related fields from the summary payload.+ if "memory" in summary:+ memory = summary["memory"]+ if "extracted" in memory:+ attrs.memory_extracted = memory["extracted"]++ # Extract error-related fields from the summary payload.+ if "errors" in summary:+ errors = summary["errors"]+ if "stage" in errors:+ attrs.errors_stage = errors["stage"]+ if "error_code" in errors:+ attrs.errors_error_code = errors["error_code"]+ if "message" in errors:+ attrs.errors_message = errors["message"]++ return attrs+++def create_root_span_attributes(+ *,+ http_method: str,+ http_route: str,+ request_id: str,+ url_path: Optional[str] = None,+ url_scheme: Optional[str] = None,+ http_host: Optional[str] = None,+ source_type: Optional[str] = None,+ source_version: Optional[str] = None,+) -> RootSpanAttributes:+ """Factory helper for `RootSpanAttributes`."""+ return RootSpanAttributes(+ http_method=http_method,+ http_route=http_route,+ request_id=request_id,+ url_path=url_path,+ url_scheme=url_scheme,+ http_host=http_host,+ source_type=source_type,+ source_version=source_version,+ )+++def create_operation_span_attributes(+ *,+ operation: str,+ telemetry_id: str,+ status: Optional[str] = None,+) -> OperationSpanAttributes:+ """Factory helper for `OperationSpanAttributes`."""+ return OperationSpanAttributes(+ operation=operation,+ telemetry_id=telemetry_id,+ status=status,+ )+++def update_root_span_identity(+ *,+ request_state: Any,+ account_id: Optional[str] = None,+ user_id: Optional[str] = None,+ agent_id: Optional[str] = None,+) -> bool:+ """+ Update identity fields on `RootSpanAttributes` and mirror them into the current span.++ This helper is used after authentication completes, when account or user identity becomes+ available only after the root request span has already been created.+ """+ try:+ # Ensure a root observability model is attached to `request.state`.+ if not hasattr(request_state, "root_span_attrs"):+ return False++ root_attrs = request_state.root_span_attrs+ if not isinstance(root_attrs, RootSpanAttributes):+ return False++ # Update identity fields in place.+ if account_id is not None:+ root_attrs.account_id = account_id+ if user_id is not None:+ root_attrs.user_id = user_id+ if agent_id is not None:+ root_attrs.agent_id = agent_id++ # Best-effort sync into the active OTel span, if one exists.+ try:+ from opentelemetry import trace as otel_trace++ current_span = otel_trace.get_current_span()+ if current_span is not None and current_span.is_recording():+ current_span.set_attributes(root_attrs.to_otel_attributes())+ except Exception:+ pass++ return True+ except Exception:+ return False+++__all__ = [+ "RootSpanAttributes",+ "OperationSpanAttributes",+ "create_root_span_attributes",+ "create_operation_span_attributes",+ "update_root_span_identity",+]diff --git a/openviking/telemetry/tracer.py b/openviking/telemetry/tracer.pyindex 8afdcdc8a..298b03e99 100644--- a/openviking/telemetry/tracer.py+++ b/openviking/telemetry/tracer.py@@ -14,19 +14,33 @@ from loguru import loggertry:from opentelemetry import trace as otel_tracefrom opentelemetry.context import Context- from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporterfrom opentelemetry.propagate import extract, injectfrom opentelemetry.sdk.resources import Resourcefrom opentelemetry.sdk.trace import Status, StatusCode, TracerProviderfrom opentelemetry.sdk.trace.export import BatchSpanProcessorfrom opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator++ try:+ from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (+ OTLPSpanExporter as OTLPGrpcSpanExporter,+ )+ except ImportError:+ OTLPGrpcSpanExporter = None++ try:+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import (+ OTLPSpanExporter as OTLPHttpSpanExporter,+ )+ except ImportError:+ OTLPHttpSpanExporter = Noneexcept ImportError:otel_trace = NoneTracerProvider = NoneStatus = NoneStatusCode = NoneBatchSpanProcessor = None- OTLPSpanExporter = None+ OTLPGrpcSpanExporter = None+ OTLPHttpSpanExporter = NoneTraceContextTextMapPropagator = NoneContext = Noneextract = None@@ -80,32 +94,34 @@ def _setup_logging():_log_trace_internal_failure("[TRACER] failed to attach standard logging trace_id filter")-def init_tracer_from_config() -> Any:- """Initialize tracer from OpenViking config."""- try:- from openviking_cli.utils.config import get_openviking_config+def init_tracer_from_server_config(server_config: Any) -> Any:+ """Initialize tracer from server.observability.traces config.- config = get_openviking_config()- tracer_cfg = config.telemetry.tracer+ Args:+ server_config: The server configuration containing observability settings.- if not tracer_cfg.enabled:- logger.info("[TRACER] disabled in config")+ Returns:+ The initialized tracer, or None if initialization failed or disabled.+ """+ try:+ trace_cfg = server_config.observability.traces+ if not trace_cfg.enabled:+ logger.info("[TRACER] disabled in server.observability.traces")return None- if not tracer_cfg.endpoint:- logger.warning("[TRACER] endpoint not configured")+ if not trace_cfg.endpoint:+ logger.warning("[TRACER] server.observability.traces.endpoint not configured")return Nonereturn init_tracer(- endpoint=tracer_cfg.endpoint,- service_name=tracer_cfg.service_name,- topic=tracer_cfg.topic,- ak=tracer_cfg.ak,- sk=tracer_cfg.sk,- enabled=tracer_cfg.enabled,+ endpoint=trace_cfg.endpoint,+ service_name=trace_cfg.service_name,+ protocol=trace_cfg.protocol,+ insecure=trace_cfg.tls.insecure,+ enabled=trace_cfg.enabled,)except Exception as e:- logger.warning(f"[TRACER] init from config failed: {e}")+ logger.warning(f"[TRACER] init from server config failed: {e}")return None@@ -125,19 +141,17 @@ def _init_asyncio_instrumentation() -> None:def init_tracer(endpoint: str,service_name: str,- topic: str,- ak: str,- sk: str,+ protocol: str = "grpc",+ insecure: bool = False,enabled: bool = True,) -> Any:"""Initialize the OpenTelemetry tracer.Args:- endpoint: OTLP endpoint URL+ endpoint: OTLP endpointservice_name: Service name for tracing- topic: Trace topic- ak: Access key- sk: Secret key+ protocol: OTLP protocol ("grpc" or "http")+ insecure: For OTLP/gRPC only. When True, use plaintext instead of TLS.enabled: Whether to enable tracingReturns:@@ -157,22 +171,36 @@ def init_tracer(return Nonetry:- headers = {- "x-tls-otel-tracetopic": topic,- "x-tls-otel-ak": ak,- "x-tls-otel-sk": sk,- "x-tls-otel-region": "cn-beijing",- }-resource_attributes = {"service.name": service_name,}resource = Resource.create(resource_attributes)- trace_exporter = OTLPSpanExporter(- endpoint=endpoint,- headers=headers,- )+ protocol = protocol.lower()+ if protocol == "grpc":+ if OTLPGrpcSpanExporter is None:+ raise ImportError("gRPC OTLP trace exporter not available")+ try:+ trace_exporter = OTLPGrpcSpanExporter(+ endpoint=endpoint,+ insecure=insecure,+ )+ except TypeError:+ trace_exporter = OTLPGrpcSpanExporter(+ endpoint=endpoint,+ )+ elif protocol == "http":+ if OTLPHttpSpanExporter is None:+ raise ImportError("HTTP OTLP trace exporter not available")+ if not endpoint.startswith("http://") and not endpoint.startswith("https://"):+ raise ValueError(+ "OTLP/HTTP trace endpoint must include scheme, e.g. 'http://localhost:4318/v1/traces'"+ )+ trace_exporter = OTLPHttpSpanExporter(+ endpoint=endpoint,+ )+ else:+ raise ValueError(f"Unsupported trace protocol: {protocol}")trace_provider = TracerProvider(resource=resource)trace_provider.add_span_processor(@@ -194,7 +222,12 @@ def init_tracer(# Initialize asyncio instrumentation to create child spans for create_task_init_asyncio_instrumentation()- logger.info(f"[TRACER] initialized with service_name={service_name}, endpoint={endpoint}")+ logger.info(+ "[TRACER] initialized with service_name=%s, protocol=%s, endpoint=%s",+ service_name,+ protocol,+ endpoint,+ )return _otel_tracerexcept Exception as e:diff --git a/openviking_cli/server_bootstrap.py b/openviking_cli/server_bootstrap.pyindex 36bbe86af..523f6606b 100644--- a/openviking_cli/server_bootstrap.py+++ b/openviking_cli/server_bootstrap.py@@ -21,19 +21,11 @@ from openviking_cli.utils.config import OPENVIKING_CONFIG_ENVdef main():- # Intercept subcommands that don't need the server.- if len(sys.argv) > 1 and sys.argv[1] == "init":- from openviking_cli.setup_wizard import main as init_main-- sys.exit(init_main())-- if len(sys.argv) > 1 and sys.argv[1] == "doctor":- from openviking_cli.doctor import main as doctor_main-- sys.exit(doctor_main())-+ """Bootstrap the server while binding a stable execution-level log trace ID."""# Pre-parse --config from sys.argv before any openviking imports,# so the env var is visible when the config singleton first initialises.+ # This is done for all subcommands (init, doctor, server) to ensure+ # consistent behavior.for i, arg in enumerate(sys.argv):if arg == "--config" and i + 1 < len(sys.argv):os.environ[OPENVIKING_CONFIG_ENV] = sys.argv[i + 1]@@ -42,9 +34,27 @@ def main():os.environ[OPENVIKING_CONFIG_ENV] = arg.split("=", 1)[1]break+ # Import after config pre-parse to avoid early config singleton initialization via+ # module-level loggers.+ from openviking_cli.utils.logger import bind_log_execution_trace # noqa: PLC0415++ # Intercept subcommands that don't need the server.+ if len(sys.argv) > 1 and sys.argv[1] == "init":+ from openviking_cli.setup_wizard import main as init_main++ with bind_log_execution_trace():+ sys.exit(init_main())++ if len(sys.argv) > 1 and sys.argv[1] == "doctor":+ from openviking_cli.doctor import main as doctor_main++ with bind_log_execution_trace():+ sys.exit(doctor_main())+from openviking.server.bootstrap import main as _real_main- _real_main()+ with bind_log_execution_trace():+ _real_main()if __name__ == "__main__":diff --git a/openviking_cli/utils/logger.py b/openviking_cli/utils/logger.pyindex 629bb7d86..cd5e0235e 100644--- a/openviking_cli/utils/logger.py+++ b/openviking_cli/utils/logger.py@@ -6,9 +6,586 @@ Logging utilities for OpenViking.import loggingimport sys+from contextlib import contextmanagerfrom logging.handlers import TimedRotatingFileHandlerfrom pathlib import Pathfrom typing import Any, Optional, Tuple+from uuid import uuid4++from openviking.observability.context import (+ bind_execution_context,+ get_observability_context,+)++# Try to import opentelemetry - will be None if not installed+try:+ from opentelemetry import trace as otel_trace+ from opentelemetry._logs import set_logger_provider+ from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler+ from opentelemetry.sdk._logs.export import BatchLogRecordProcessor+ from opentelemetry.sdk.resources import Resource+ from opentelemetry.trace import Status, StatusCode, format_span_id, format_trace_id++ # Try to import the gRPC exporter.+ try:+ from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (+ OTLPLogExporter as OTLPGrpcLogExporter,+ )+ except ImportError:+ OTLPGrpcLogExporter = None++ # Try to import the HTTP exporter.+ try:+ from opentelemetry.exporter.otlp.proto.http._log_exporter import (+ OTLPLogExporter as OTLPHttpLogExporter,+ )+ except ImportError:+ OTLPHttpLogExporter = None+except ImportError:+ otel_trace = None+ set_logger_provider = None+ LoggerProvider = None+ LoggingHandler = None+ BatchLogRecordProcessor = None+ OTLPGrpcLogExporter = None+ OTLPHttpLogExporter = None+ Resource = None+ Status = None+ StatusCode = None+ format_span_id = None+ format_trace_id = None+++# Global OTel log handler state+_otel_log_handler_initialized = False+_otel_log_handler: Any = None+++def _get_log_context() -> dict[str, Any]:+ """+ Build the log context dynamically for the current context.++ This function builds the log context from scratch on each call, ensuring+ that any context changes (e.g., account_id updated after authentication)+ are immediately reflected in subsequent log records.++ Context sources (in priority order):+ 1. OTel current span (for trace_id, span_id)+ 2. Root observability context (for request_id, account_id, etc.)+ 3. Operation observability context (for operation, telemetry_id, etc.)+ 4. Execution-level trace context (fallback when no OTel span)++ Returns:+ A dictionary containing all log context fields.+ """+ context: dict[str, Any] = {+ "trace_id": "",+ "span_id": "",+ "request_id": "",+ "operation": "",+ "account_id": "",+ "user_id": "",+ "agent_id": "",+ }++ # 1. Get trace_id and span_id from OTel span if available+ if otel_trace is not None and format_trace_id is not None and format_span_id is not None:+ try:+ current_span = otel_trace.get_current_span()+ if current_span is not None and hasattr(current_span, "context"):+ span_context = current_span.get_span_context()+ if span_context.is_valid:+ context["trace_id"] = format_trace_id(span_context.trace_id)+ context["span_id"] = format_span_id(span_context.span_id)+ except Exception:+ # Best-effort only - don't fail if OTel context retrieval fails+ pass++ # 2. Get unified observability context (contains root and operation)+ obs_ctx = get_observability_context()++ # Get fields from root context+ if obs_ctx.root is not None:+ root_fields = obs_ctx.root.to_log_fields()+ for key, value in root_fields.items():+ if value is not None:+ context[key] = value++ # Get fields from operation context+ if obs_ctx.operation is not None:+ operation_fields = obs_ctx.operation.to_log_fields()+ for key, value in operation_fields.items():+ if value is not None:+ context[key] = value++ # 3. Fall back to execution-level trace context if no OTel span+ if not context["trace_id"]:+ context["trace_id"] = obs_ctx.get_trace_id()+ if not context["span_id"]:+ context["span_id"] = obs_ctx.get_span_id()++ return context+++def set_log_execution_trace_id(trace_id: Optional[str] = None) -> str:+ """+ Set the execution-level trace ID used by logs outside real OTel spans.++ This function uses the unified ObservabilityContext to store the trace ID.++ Args:+ trace_id: Optional 32-character lowercase hex trace ID. When omitted, a new+ trace ID is generated automatically.++ Returns:+ The effective execution trace ID stored in the current context.+ """+ ctx = get_observability_context()+ effective_trace_id = trace_id or uuid4().hex+ ctx.execution_trace_id = effective_trace_id+ return effective_trace_id+++def set_log_execution_span_id(span_id: Optional[str] = None) -> str:+ """+ Set the execution-level span ID used by logs outside real OTel spans.++ This function uses the unified ObservabilityContext to store the span ID.++ Args:+ span_id: Optional 16-character lowercase hex span ID. When omitted, a new+ span ID is generated automatically.++ Returns:+ The effective execution span ID stored in the current context.+ """+ ctx = get_observability_context()+ effective_span_id = span_id or uuid4().hex[:16]+ ctx.execution_span_id = effective_span_id+ return effective_span_id+++def get_effective_trace_id() -> str:+ """+ Return the current execution-level trace ID, or an empty string when unset.++ This function uses the unified ObservabilityContext to retrieve the trace ID.+ It also checks the OTel current span for a valid trace ID.++ Returns:+ The effective trace ID, or empty string if not available.+ """+ ctx = get_observability_context()+ return ctx.get_trace_id()+++def get_effective_span_id() -> str:+ """+ Return the current execution-level span ID, or an empty string when unset.++ This function uses the unified ObservabilityContext to retrieve the span ID.+ It also checks the OTel current span for a valid span ID.++ Returns:+ The effective span ID, or empty string if not available.+ """+ ctx = get_observability_context()+ return ctx.get_span_id()+++def clear_log_execution_trace_id() -> None:+ """+ Clear the execution-level trace ID from the current logging context.++ This function clears the execution trace ID and span ID from the+ unified ObservabilityContext.+ """+ ctx = get_observability_context()+ ctx.execution_trace_id = None+ ctx.execution_span_id = None+++@contextmanager+def bind_log_execution_trace(trace_id: Optional[str] = None, span_id: Optional[str] = None):+ """+ Bind a stable execution-level trace ID and span ID for logs emitted in this context.++ This context manager is intended for non-request entry points, such as process startup,+ CLI execution, or background tasks that do not run under a real OTel request span.++ This is a convenience wrapper around the unified `bind_execution_context` from+ the observability context module.++ Args:+ trace_id: Optional 32-character lowercase hex trace ID. When omitted, a new+ trace ID is generated automatically for this execution unit.+ span_id: Optional 16-character lowercase hex span ID. When omitted, the first 16+ characters of the trace_id are used as the span_id.++ Yields:+ A tuple of (trace_id, span_id).+ """+ with bind_execution_context(trace_id, span_id) as (effective_trace_id, effective_span_id):+ yield effective_trace_id, effective_span_id+++def init_otel_log_handler(+ protocol: str = "grpc", # "grpc" or "http"+ endpoint: str = "localhost:4317", # gRPC default: 4317; HTTP default: 4318+ service_name: str = "openviking-server",+ insecure: bool = False,+ enabled: bool = True,+) -> Any:+ """Initialize OTel LoggingHandler.++ Args:+ protocol: OTLP protocol ("grpc" or "http").+ endpoint: OTLP logs endpoint.+ service_name: Service name.+ insecure: For OTLP/gRPC only. When True, use plaintext instead of TLS.+ enabled: Whether to enable the handler.++ Returns:+ The initialized LoggingHandler instance, or None if initialization fails.+ """+ global _otel_log_handler_initialized, _otel_log_handler++ if not enabled:+ return None++ if _otel_log_handler_initialized:+ return _otel_log_handler++ # Validate protocol+ protocol = protocol.lower()+ if protocol not in ["grpc", "http"]:+ protocol = "grpc"++ if (+ set_logger_provider is None+ or LoggerProvider is None+ or LoggingHandler is None+ or BatchLogRecordProcessor is None+ or Resource is None+ ):+ return None++ try:+ # Create resource+ resource = Resource.create(+ {+ "service.name": service_name,+ }+ )++ # Create logger provider+ logger_provider = LoggerProvider(resource=resource)++ # Select exporter based on protocol+ otlp_exporter = None+ if protocol == "grpc":+ if OTLPGrpcLogExporter is None:+ raise ImportError("gRPC OTLP log exporter not available")+ try:+ otlp_exporter = OTLPGrpcLogExporter(endpoint=endpoint, insecure=insecure)+ except TypeError:+ otlp_exporter = OTLPGrpcLogExporter(endpoint=endpoint)+ elif protocol == "http":+ if OTLPHttpLogExporter is None:+ raise ImportError("HTTP OTLP log exporter not available")+ # HTTP endpoint must be a full URL (with scheme). Do not auto-prefix to avoid implicit behavior.+ if not endpoint.startswith("http://") and not endpoint.startswith("https://"):+ raise ValueError(+ "OTLP/HTTP endpoint must include scheme, e.g. 'http://localhost:4318/v1/logs'"+ )+ otlp_exporter = OTLPHttpLogExporter(endpoint=endpoint)++ if otlp_exporter is None:+ raise ValueError(f"Failed to create log exporter for protocol={protocol}")++ # Create batch log processor+ log_processor = BatchLogRecordProcessor(otlp_exporter)+ logger_provider.add_log_record_processor(log_processor)++ # Set global logger provider+ set_logger_provider(logger_provider)++ # Create logging handler+ _otel_log_handler = LoggingHandler(+ level=logging.INFO,+ logger_provider=logger_provider,+ )++ _otel_log_handler_initialized = True++ logger = get_logger(__name__)+ logger.info(+ "[OTelLogHandler] initialized with protocol=%s, endpoint=%s, service_name=%s",+ protocol,+ endpoint,+ service_name,+ )++ return _otel_log_handler+ except Exception as e:+ logger = get_logger(__name__)+ logger.warning("[OTelLogHandler] initialization failed: %s", e)+ return None+++def attach_otel_log_handler_to_existing_loggers() -> None:+ """Attach the global OTel log handler to already-created loggers."""+ if _otel_log_handler is None:+ return++ add_otel_log_handler_to_logger(logging.getLogger())++ for logger_obj in logging.Logger.manager.loggerDict.values():+ if isinstance(logger_obj, logging.Logger):+ add_otel_log_handler_to_logger(logger_obj)+++def init_otel_log_handler_from_server_config(server_config: Any) -> Any:+ """Initialize OTLP log export from server.observability.logs."""+ logs_cfg = server_config.observability.logs+ if not logs_cfg.enabled:+ return None++ handler = init_otel_log_handler(+ protocol=logs_cfg.protocol,+ endpoint=logs_cfg.endpoint,+ service_name=logs_cfg.service_name,+ insecure=logs_cfg.tls.insecure,+ enabled=logs_cfg.enabled,+ )+ attach_otel_log_handler_to_existing_loggers()+ return handler+++def add_otel_log_handler_to_logger(logger: logging.Logger) -> None:+ """Add OTel LoggingHandler to the specified logger.++ Args:+ logger: The logger instance to add the handler to.+ """+ if _otel_log_handler is not None:+ # Check if the handler has already been added+ if not any(isinstance(h, type(_otel_log_handler)) for h in logger.handlers):+ logger.addHandler(_otel_log_handler)+++class TraceContextFilter(logging.Filter):+ """+ Log filter that injects OTel trace context information.++ Automatically extracts trace_id, span_id, and other fields from the current span+ and injects them into log records, enabling automatic correlation between logs and traces.++ This filter builds the context dynamically on each call to ensure that any context+ changes (e.g., account_id updated after authentication) are immediately reflected+ in subsequent log records.+ """++ def filter(self, record: logging.LogRecord) -> bool:+ """+ Inject trace context information into the log record.++ Builds the context dynamically on each call to ensure freshness.++ Args:+ record: The log record object.++ Returns:+ bool: Always returns True, indicating the log should be recorded.+ """+ # Build context dynamically to ensure freshness+ context = _get_log_context()++ # Apply context fields to the log record+ record.trace_id = context.get("trace_id", "")+ record.span_id = context.get("span_id", "")+ record.request_id = context.get("request_id", "")+ record.operation = context.get("operation", "")+ record.account_id = context.get("account_id", "")+ record.user_id = context.get("user_id", "")+ record.agent_id = context.get("agent_id", "")++ return True+++class LogToSpanEventFilter(logging.Filter):+ """+ Log filter that automatically maps log records to OTel span events.++ This filter allows developers to only write log.xxx() calls, and the logs+ will automatically be recorded as span events when OTel is available.+ This eliminates the need to write duplicate code for both logging and tracing.++ Key features:+ - Logs are automatically added as span events when a span is active+ - Exceptions are automatically recorded with record_exception()+ - Extra fields from log records are included as span attributes+ - Best-effort only: never breaks logging if OTel is not available++ Usage:+ handler.addFilter(LogToSpanEventFilter(+ min_level=logging.INFO,+ excluded_loggers={"uvicorn.access", "uvicorn.error"}+ ))+ """++ def __init__(+ self,+ name: str = "",+ min_level: int = logging.INFO,+ excluded_loggers: Optional[set] = None,+ ):+ """+ Initialize the LogToSpanEventFilter.++ Args:+ name: Filter name.+ min_level: Minimum log level to be mapped to span events.+ Logs below this level will only be recorded, not added as span events.+ excluded_loggers: Set of logger names to exclude from span event mapping.+ Useful for excluding noisy loggers like access logs.+ """+ super().__init__(name)+ self._min_level = min_level+ self._excluded_loggers = excluded_loggers or set()++ def _extract_attributes(self, record: logging.LogRecord) -> dict:+ """+ Extract span attributes from a log record.++ Extracts standard fields and extra fields from the log record+ to be used as span event attributes.++ Args:+ record: The log record object.++ Returns:+ Dictionary of attributes to be included in the span event.+ """+ attributes = {+ "log.level": record.levelname,+ "log.logger": record.name,+ }++ # Add location information if available+ if record.pathname:+ attributes["code.filepath"] = record.pathname+ if record.lineno:+ attributes["code.lineno"] = record.lineno+ if record.funcName:+ attributes["code.function"] = record.funcName++ # Add extra fields from the log record+ # Standard fields to exclude from extra attributes+ standard_fields = {+ "name",+ "msg",+ "args",+ "levelname",+ "levelno",+ "pathname",+ "filename",+ "module",+ "exc_info",+ "exc_text",+ "stack_info",+ "lineno",+ "funcName",+ "created",+ "msecs",+ "relativeCreated",+ "thread",+ "threadName",+ "processName",+ "process",+ "message",+ "asctime",+ "trace_id",+ "span_id",+ "request_id",+ "operation",+ "account_id",+ "user_id",+ "agent_id",+ }++ for key, value in record.__dict__.items():+ if key not in standard_fields and not key.startswith("_"):+ # Convert to OTel-compatible types+ if isinstance(value, (str, int, float, bool)):+ attributes[key] = value+ elif value is not None:+ attributes[key] = str(value)++ return attributes++ def filter(self, record: logging.LogRecord) -> bool:+ """+ Map log record to span event (best-effort).++ This method is called for each log record. It:+ 1. Checks if the logger should be excluded+ 2. Checks if the log level is above the minimum threshold+ 3. If OTel is available and a span is active, adds the log as a span event+ 4. For exceptions, also records the exception with record_exception()++ Args:+ record: The log record object.++ Returns:+ bool: Always returns True, indicating the log should be recorded.+ """+ # Check if this logger should be excluded+ if record.name in self._excluded_loggers:+ return True++ # Check if log level is below the minimum threshold for span events+ if record.levelno < self._min_level:+ return True++ # Check if OTel is available+ if otel_trace is None:+ return True++ try:+ # Get the current span+ current_span = otel_trace.get_current_span()+ if current_span is None or not current_span.is_recording():+ return True++ # Extract attributes from the log record+ attributes = self._extract_attributes(record)++ # Handle exceptions specially+ if record.exc_info is not None:+ exc_type, exc_value, exc_tb = record.exc_info+ if exc_value is not None:+ # Record the exception in the span+ current_span.record_exception(exc_value, attributes=attributes)++ # Set span status to ERROR if Status/StatusCode are available+ if Status is not None and StatusCode is not None:+ try:+ current_span.set_status(+ Status(StatusCode.ERROR, description=str(exc_value))+ )+ except Exception:+ pass+ else:+ # Add as a regular span event+ event_name = record.getMessage() or "log"+ current_span.add_event(event_name, attributes=attributes)++ except Exception:+ # Best-effort only: never break logging+ pass++ return Truedef _load_log_config() -> Tuple[str, str, str, Optional[Any]]:@@ -76,7 +653,18 @@ def _create_log_handler(log_output: str, config: Optional[Any]) -> logging.Handldef get_logger(name: str = "openviking",format_string: Optional[str] = None,+ add_otel_handler: bool = False,) -> logging.Logger:+ """Get a configured logger instance.++ Args:+ name: Logger name, defaults to "openviking".+ format_string: Custom log format string, optional.+ add_otel_handler: Whether to add OTel LoggingHandler, defaults to False.++ Returns:+ logging.Logger: Configured logger instance with automatic trace context injection.+ """logger = logging.getLogger(name)if not logger.handlers:@@ -88,10 +676,18 @@ def get_logger(format_string = log_formatformatter = logging.Formatter(format_string)handler.setFormatter(formatter)++ # Add unified trace context filter+ handler.addFilter(TraceContextFilter())+logger.addHandler(handler)logger.propagate = Falselogger.setLevel(level)+ # If OTel log export is globally initialized, attach the handler automatically.+ if add_otel_handler or _otel_log_handler_initialized:+ add_otel_log_handler_to_logger(logger)+return logger@@ -100,10 +696,10 @@ default_logger = get_logger()def configure_uvicorn_logging() -> None:- """Configure Uvicorn loggers to use OpenViking's logging configuration.+ """Configure Uvicorn logging to use OpenViking's logging configuration.- This function configures the 'uvicorn', 'uvicorn.error', and 'uvicorn.access'- loggers to use the same handlers and format as our openviking loggers.+ Configures the 'uvicorn', 'uvicorn.error', and 'uvicorn.access' loggers+ to use the same handlers, format, and trace context injection as openviking logs."""log_level_str, log_format, log_output, config = _load_log_config()level = getattr(logging, log_level_str, logging.INFO)@@ -111,6 +707,9 @@ def configure_uvicorn_logging() -> None:formatter = logging.Formatter(log_format)handler.setFormatter(formatter)+ # Add unified trace context filter+ handler.addFilter(TraceContextFilter())+# Configure all Uvicorn loggersuvicorn_logger_names = ["uvicorn", "uvicorn.error", "uvicorn.access"]for logger_name in uvicorn_logger_names:diff --git a/pyproject.toml b/pyproject.tomlindex 089bcef9a..759e7573d 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -75,6 +75,7 @@ dependencies = ["opentelemetry-api>=1.14","opentelemetry-sdk>=1.14","opentelemetry-exporter-otlp-proto-grpc>=1.14",+ "opentelemetry-exporter-otlp-proto-http>=1.14","opentelemetry-instrumentation-asyncio>=0.61b0","loguru>=0.7.3","cryptography>=42.0.0",diff --git a/tests/api_test/tools/tests/test_logger.py b/tests/api_test/tools/tests/test_logger.pyindex bd100f12a..40049cd93 100644--- a/tests/api_test/tools/tests/test_logger.py+++ b/tests/api_test/tools/tests/test_logger.py@@ -1,24 +1,138 @@import logging+from openviking.observability.context import (+ bind_operation_observability_context,+ bind_root_observability_context,+ reset_operation_observability_context,+ reset_root_observability_context,+)+from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributesfrom openviking_cli.utils import get_logger+from openviking_cli.utils.logger import TraceContextFilter, bind_log_execution_trace-print("Testing logger...")-print("=" * 80)-logger = get_logger(__name__)-print(f"Logger: {logger}")-print(f"Logger level: {logger.level}")+def test_trace_context_filter_reuses_execution_trace_id_without_active_span():+ """TraceContextFilter should fall back to execution-trace context when no span exists.-print(f"\nRoot logger level: {logging.getLogger().level}")+ The span_id is derived from the first 16 characters of the trace_id.+ """+ record = logging.LogRecord(+ name="test",+ level=logging.INFO,+ pathname=__file__,+ lineno=1,+ msg="hello",+ args=(),+ exc_info=None,+ )-print("\nTesting log messages:")-logger.debug("Debug message")-logger.info("Info message")-logger.warning("Warning message")-logger.error("Error message")+ with bind_log_execution_trace("1234567890abcdef1234567890abcdef"):+ passed = TraceContextFilter().filter(record)-print("\n" + "=" * 80)-print("Checking openviking.server.app logger...")+ assert passed is True+ assert record.trace_id == "1234567890abcdef1234567890abcdef"+ # span_id is derived from first 16 chars of trace_id+ assert record.span_id == "1234567890abcdef"-app_logger = get_logger("openviking.server.app")-print(f"App logger level: {app_logger.level}")++def test_trace_context_filter_generates_span_id_from_trace_id():+ """TraceContextFilter should generate span_id from trace_id when not explicitly provided."""+ record = logging.LogRecord(+ name="test",+ level=logging.INFO,+ pathname=__file__,+ lineno=1,+ msg="hello",+ args=(),+ exc_info=None,+ )++ # Test with explicitly provided span_id+ with bind_log_execution_trace(+ trace_id="1234567890abcdef1234567890abcdef",+ span_id="fedcba0987654321",+ ):+ passed = TraceContextFilter().filter(record)++ assert passed is True+ assert record.trace_id == "1234567890abcdef1234567890abcdef"+ assert record.span_id == "fedcba0987654321"+++def test_trace_context_filter_injects_root_observability_fields_without_span():+ record = logging.LogRecord(+ name="test",+ level=logging.INFO,+ pathname=__file__,+ lineno=1,+ msg="hello",+ args=(),+ exc_info=None,+ )+ root = RootSpanAttributes(http_method="GET", http_route="/demo", request_id="req-1")+ root.account_id = "acct-1"+ root.user_id = "user-1"+ root.agent_id = "agent-1"+ token = bind_root_observability_context(root)+ try:+ passed = TraceContextFilter().filter(record)+ finally:+ reset_root_observability_context(token)++ assert passed is True+ assert record.request_id == "req-1"+ assert record.account_id == "acct-1"+ assert record.user_id == "user-1"+ assert record.agent_id == "agent-1"+++def test_trace_context_filter_injects_operation_observability_fields_without_span():+ record = logging.LogRecord(+ name="test",+ level=logging.INFO,+ pathname=__file__,+ lineno=1,+ msg="hello",+ args=(),+ exc_info=None,+ )+ operation = OperationSpanAttributes(operation="search.find", telemetry_id="tm-1")+ operation.status = "ok"+ token = bind_operation_observability_context(operation)+ try:+ passed = TraceContextFilter().filter(record)+ finally:+ reset_operation_observability_context(token)++ assert passed is True+ assert record.operation == "search.find"+ assert record.telemetry_id == "tm-1"+ assert record.status == "ok"+++def _manual_smoke_check() -> None:+ """Run a manual logger smoke check when this module is executed directly."""+ print("Testing logger...")+ print("=" * 80)++ logger = get_logger(__name__)+ print(f"Logger: {logger}")+ print(f"Logger level: {logger.level}")++ print(f"\nRoot logger level: {logging.getLogger().level}")++ print("\nTesting log messages:")+ logger.debug("Debug message")+ logger.info("Info message")+ logger.warning("Warning message")+ logger.error("Error message")++ print("\n" + "=" * 80)+ print("Checking openviking.server.app logger...")++ app_logger = get_logger("openviking.server.app")+ print(f"App logger level: {app_logger.level}")+++if __name__ == "__main__":+ _manual_smoke_check()diff --git a/tests/metrics/core/test_exporter.py b/tests/metrics/core/test_exporter.pyindex 1782f0ce8..459ad300a 100644--- a/tests/metrics/core/test_exporter.py+++ b/tests/metrics/core/test_exporter.py@@ -1,6 +1,12 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+import asyncio++import pytest++from openviking.metrics.exporters.otel import OTelMetricExporter, _compute_next_tick_and_sleep+def test_exporter_outputs_help_type_and_empty_metrics(registry, render_prometheus):registry.counter("openviking_empty_counter_total")@@ -12,3 +18,80 @@ def test_exporter_outputs_help_type_and_empty_metrics(registry, render_prometheuassert "# TYPE openviking_empty_histogram_seconds histogram" in textassert "openviking_empty_histogram_seconds_count 0" in textassert 'openviking_empty_histogram_seconds_bucket{le="+Inf"} 0' in text+++async def test_otel_exporter_start_and_shutdown_manage_background_loop(registry, monkeypatch):+ """OTel exporter should own its periodic export loop lifecycle."""+ exporter = OTelMetricExporter(registry=registry, enabled=False, export_interval_ms=10)+ exporter._enabled = True++ calls = 0+ tick = asyncio.Event()++ async def _fake_export():+ nonlocal calls+ calls += 1+ tick.set()+ return ""++ monkeypatch.setattr(exporter, "export", _fake_export)++ exporter.start()+ await asyncio.wait_for(tick.wait(), timeout=0.2)+ assert calls >= 1+ assert exporter._export_task is not None++ task_before = exporter._export_task+ exporter.start()+ assert exporter._export_task is task_before++ await exporter.shutdown()+ assert exporter._export_task is None+++async def test_otel_export_loop_aligns_to_next_fixed_tick_after_overrun():+ """Scheduling helper should advance next tick and compute sleep after one interval overrun."""+ next_tick, sleep_s = _compute_next_tick_and_sleep(now=1.15, interval_s=1.0, next_tick=1.0)+ assert next_tick == pytest.approx(2.0)+ assert sleep_s == pytest.approx(0.85)+++def test_otel_http_send_uses_export_timeout_not_refresh_deadline(registry):+ """HTTP transport timeout should be decoupled from collector refresh timeout."""+ exporter = OTelMetricExporter(+ registry=registry,+ enabled=False,+ protocol="http",+ endpoint="http://localhost:4318/v1/metrics",+ refresh_deadline_seconds=1.0,+ )++ class _FakeSession:+ def __init__(self):+ self.timeout = None++ def post(self, _url, **kwargs):+ self.timeout = kwargs.get("timeout")++ class _Resp:+ @staticmethod+ def raise_for_status():+ return None++ return _Resp()++ fake_session = _FakeSession()+ exporter._http_session = fake_session++ request = exporter._build_export_request()+ assert request is None++ from opentelemetry.proto.collector.metrics.v1.metrics_service_pb2 import (+ ExportMetricsServiceRequest,+ )++ exporter._send_http_request(ExportMetricsServiceRequest())++ assert fake_session.timeout == 5.0diff --git a/tests/metrics/integration/test_account_dimension.py b/tests/metrics/integration/test_account_dimension.pyindex 2cfc52640..1e9d4d0f7 100644--- a/tests/metrics/integration/test_account_dimension.py+++ b/tests/metrics/integration/test_account_dimension.py@@ -4,15 +4,13 @@from __future__ import annotationsimport asyncio-from types import SimpleNamespaceimport pytest+from fastapi import FastAPI+from fastapi.testclient import TestClient+from starlette.requests import Request-from openviking.metrics.account_context import (- bind_metric_account_context,- get_metric_account_context,- reset_metric_account_context,-)+import openviking.observability.http_observability_middleware as http_middlewarefrom openviking.metrics.account_dimension import (configure_metric_account_dimension,reset_metric_account_dimension,@@ -23,18 +21,43 @@ from openviking.metrics.collectors.http import HTTPCollectorfrom openviking.metrics.core.registry import MetricRegistryfrom openviking.metrics.datasources.base import EventMetricDataSourcefrom openviking.metrics.exporters.prometheus import PrometheusExporter-from openviking.metrics.http_middleware import create_http_metrics_middleware+from openviking.observability.context import (+ bind_root_observability_context,+ get_root_observability_context,+ reset_root_observability_context,+)+from openviking.observability.http_observability_middleware import (+ create_http_observability_middleware,+)+from openviking.telemetry.span_models import RootSpanAttributes+++def _bind_root_context_for_account(account_id: str | None):+ root = RootSpanAttributes(http_method="GET", http_route="/test", request_id="req-test")+ root.account_id = account_id+ return bind_root_observability_context(root)+++def _build_test_app(middleware_factory, *, route_path: str, handler):+ app = FastAPI()++ @app.middleware("http")+ async def middleware_entry(request, call_next):+ return await middleware_factory(request, call_next)++ app.get(route_path)(handler)+ return appdef test_metric_account_context_binds_and_resets_account_id():- """Binding a context must be observable via `get_metric_account_context()` and resettable."""- token = bind_metric_account_context(account_id="acct_123")+ """Binding a root context must be observable and resettable."""+ token = _bind_root_context_for_account("acct_123")try:- assert get_metric_account_context().http_account_id == "acct_123"+ assert get_root_observability_context().account_id == "acct_123"finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)- assert get_metric_account_context().http_account_id is None+ assert get_root_observability_context() is None@pytest.mark.asyncio@@ -42,12 +65,13 @@ async def test_metric_account_context_is_isolated_per_task():"""Two concurrent tasks must not leak metric account context into each other."""async def _worker(account_id: str, seen: list[str | None]):- token = bind_metric_account_context(account_id=account_id)+ token = _bind_root_context_for_account(account_id)try:await asyncio.sleep(0)- seen.append(get_metric_account_context().http_account_id)+ root_context = get_root_observability_context()+ seen.append(root_context.account_id if root_context is not None else None)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)seen: list[str | None] = []await asyncio.gather(_worker("acct_a", seen), _worker("acct_b", seen))@@ -86,34 +110,30 @@ def test_http_collector_uses_unknown_account_when_unbound():)-async def test_http_metrics_middleware_propagates_state_account_to_collector(monkeypatch):+def test_http_metrics_middleware_propagates_state_account_to_collector(monkeypatch):"""HTTP middleware must attach the authenticated account id to emitted http events."""captured: list[tuple[str, dict]] = []- middleware = create_http_metrics_middleware()+ middleware = create_http_observability_middleware()def _fake_emit(event_name: str, payload: dict) -> None:captured.append((event_name, dict(payload)))monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)- async def _call_next(request):- request.state.metric_account_id = "acct-mdw"- return SimpleNamespace(status_code=200)-- from starlette.requests import Request+ async def resources_handler(request: Request):+ request.state.root_span_attrs.account_id = "acct-mdw"+ return {"ok": True}- request = Request(- {- "type": "http",- "method": "GET",- "path": "/api/v1/resources",- "route": SimpleNamespace(path="/api/v1/resources"),- "headers": [],- "state": {},- }+ app = _build_test_app(+ middleware,+ route_path="/api/v1/resources",+ handler=resources_handler,)- await middleware(request, _call_next)++ with TestClient(app) as client:+ resp = client.get("/api/v1/resources")+ assert resp.status_code == 200assert any(event_name == "http.request" and payload.get("account_id") == "acct-mdw"@@ -129,7 +149,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_exact():metric_allowlist={EmbeddingCollector.CALLS_TOTAL},max_active_accounts=10,)- token = bind_metric_account_context(account_id="acct-embed-1")+ token = _bind_root_context_for_account("acct-embed-1")try:EmbeddingCollector().receive("embedding.call",@@ -143,7 +163,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_exact():registry,)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)reset_metric_account_dimension()text = PrometheusExporter(registry=registry).render()@@ -158,7 +178,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_by_prefix(metric_allowlist={"openviking_embedding_*"},max_active_accounts=10,)- token = bind_metric_account_context(account_id="acct-embed-2")+ token = _bind_root_context_for_account("acct-embed-2")try:EmbeddingCollector().receive("embedding.success",@@ -166,7 +186,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_by_prefix(registry,)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)reset_metric_account_dimension()text = PrometheusExporter(registry=registry).render()@@ -186,7 +206,7 @@ def test_account_dimension_returns_overflow_when_max_active_accounts_exceeded():)try:for account_id in ("acct-a", "acct-b", "acct-c"):- token = bind_metric_account_context(account_id=account_id)+ token = _bind_root_context_for_account(account_id)try:EmbeddingCollector().receive("embedding.call",@@ -200,7 +220,7 @@ def test_account_dimension_returns_overflow_when_max_active_accounts_exceeded():registry,)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)finally:reset_metric_account_dimension()@@ -217,7 +237,7 @@ def test_account_dimension_returns_unknown_for_not_allowlisted_metric_even_with_metric_allowlist={HTTPCollector.REQUESTS_TOTAL},max_active_accounts=10,)- token = bind_metric_account_context(account_id="acct-x")+ token = _bind_root_context_for_account("acct-x")try:EmbeddingCollector().receive("embedding.success",@@ -225,7 +245,7 @@ def test_account_dimension_returns_unknown_for_not_allowlisted_metric_even_with_registry,)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)reset_metric_account_dimension()text = PrometheusExporter(registry=registry).render()@@ -250,11 +270,11 @@ def test_account_dimension_does_not_inject_account_id_for_unsupported_metric_nammetric_allowlist={"openviking_unsupported_total"},max_active_accounts=10,)- token = bind_metric_account_context(account_id="acct-z")+ token = _bind_root_context_for_account("acct-z")try:DummyCollector().receive("demo", {}, registry)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)reset_metric_account_dimension()text = PrometheusExporter(registry=registry).render()diff --git a/tests/metrics/integration/test_http_metrics.py b/tests/metrics/integration/test_http_metrics.pyindex 1af2b5334..42c877dfd 100644--- a/tests/metrics/integration/test_http_metrics.py+++ b/tests/metrics/integration/test_http_metrics.py@@ -3,16 +3,11 @@from __future__ import annotations-from types import SimpleNamespace--import pytest+from fastapi import FastAPI+from fastapi.testclient import TestClientfrom starlette.requests import Request-import openviking.metrics.http_middleware as http_middleware-from openviking.metrics.account_context import (- bind_metric_account_context,- reset_metric_account_context,-)+import openviking.observability.http_observability_middleware as http_middlewarefrom openviking.metrics.collectors.http import HTTPCollectorfrom openviking.metrics.core.registry import MetricRegistryfrom openviking.metrics.datasources.base import EventMetricDataSource@@ -20,14 +15,41 @@ from openviking.metrics.datasources.http import HttpRequestLifecycleDataSourcefrom openviking.metrics.datasources.model_usage import VLMEventDataSourcefrom openviking.metrics.exporters.prometheus import PrometheusExporterfrom openviking.metrics.global_api import configure_metric_account_dimension, shutdown_metrics-from openviking.metrics.http_middleware import (+from openviking.models.vlm.base import VLMBase+from openviking.observability.context import (+ bind_root_observability_context,+ reset_root_observability_context,+)+from openviking.observability.http_observability_middleware import (_INFLIGHT,- _extract_request_account_id,_get_route_template,_inflight_delta,- create_http_metrics_middleware,+ create_http_observability_middleware,)-from openviking.models.vlm.base import VLMBase+from openviking.telemetry.span_models import RootSpanAttributes+++def _bind_root_context_for_account(account_id: str | None):+ root = RootSpanAttributes(http_method="GET", http_route="/items", request_id="req-test")+ root.account_id = account_id+ return bind_root_observability_context(root)+++def _build_test_app(middleware_factory, *, route_path: str, method: str, handler):+ app = FastAPI()++ @app.middleware("http")+ async def middleware_entry(request, call_next):+ return await middleware_factory(request, call_next)++ if method == "GET":+ app.get(route_path)(handler)+ elif method == "POST":+ app.post(route_path)(handler)+ else:+ raise ValueError(f"unsupported method: {method}")++ return appdef test_http_collector_emits_account_label():@@ -43,7 +65,7 @@ def test_http_collector_emits_account_label():)collector = HTTPCollector()- token = bind_metric_account_context(account_id="acct-1")+ token = _bind_root_context_for_account("acct-1")try:collector.receive("http.request",@@ -56,7 +78,7 @@ def test_http_collector_emits_account_label():registry,)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)shutdown_metrics(app=None)text = PrometheusExporter(registry=registry).render()@@ -109,34 +131,6 @@ def test_http_request_datasource_propagates_explicit_account_id(monkeypatch):]-def test_extract_request_account_id_prefers_authenticated_request_state():- request = Request(- {- "type": "http",- "method": "GET",- "path": "/api/v1/resources",- "headers": [(b"x-openviking-account", b"header-account")],- "state": {},- }- )- request.state.metric_account_id = "state-account"-- assert _extract_request_account_id(request) == "state-account"---def test_extract_request_account_id_does_not_trust_raw_header_when_unauthenticated():- request = Request(- {- "type": "http",- "method": "GET",- "path": "/api/v1/resources",- "headers": [(b"x-openviking-account", b"header-account")],- "state": {},- }- )- assert _extract_request_account_id(request) is None--def test_get_route_template_uses_low_cardinality_fallback_for_unmatched_route():request = Request({@@ -161,32 +155,34 @@ def test_inflight_delta_removes_zero_value_entries():assert ("/api/v1/resources", None) not in _INFLIGHT-async def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatch):+def test_http_metrics_module_exposes_only_unified_middleware_entrypoint():+ assert not hasattr(http_middleware, "create_http_metrics_middleware")+++def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatch):captured: list[tuple[str, dict]] = []- middleware = create_http_metrics_middleware()+ middleware = create_http_observability_middleware()def _fake_emit(event_name: str, payload: dict) -> None:captured.append((event_name, dict(payload)))monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)- async def _call_next(request: Request):- request.state.metric_account_id = "acct-real"- return SimpleNamespace(status_code=200)+ async def resources_handler(request: Request):+ request.state.root_span_attrs.account_id = "acct-real"+ return {"ok": True}- request = Request(- {- "type": "http",- "method": "POST",- "path": "/api/v1/resources",- "route": SimpleNamespace(path="/api/v1/resources"),- "headers": [],- "state": {},- }+ app = _build_test_app(+ middleware,+ route_path="/api/v1/resources",+ method="POST",+ handler=resources_handler,)- await middleware(request, _call_next)+ with TestClient(app) as client:+ resp = client.post("/api/v1/resources")+ assert resp.status_code == 200request_events = [payload for event_name, payload in captured if event_name == "http.request"]assert request_events@@ -205,98 +201,92 @@ async def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatc)-async def test_http_metrics_middleware_ignores_internal_metrics_route(monkeypatch):+def test_http_metrics_middleware_ignores_internal_metrics_route(monkeypatch):captured: list[tuple[str, dict]] = []- middleware = create_http_metrics_middleware()+ middleware = create_http_observability_middleware()def _fake_emit(event_name: str, payload: dict) -> None:captured.append((event_name, dict(payload)))monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)- async def _call_next(_request: Request):- return SimpleNamespace(status_code=200)+ async def metrics_handler():+ return {"ok": True}- request = Request(- {- "type": "http",- "method": "GET",- "path": "/metrics",- "headers": [],- "state": {},- }+ app = _build_test_app(+ middleware,+ route_path="/metrics",+ method="GET",+ handler=metrics_handler,)- await middleware(request, _call_next)+ with TestClient(app) as client:+ resp = client.get("/metrics")+ assert resp.status_code == 200assert captured == []-async def test_http_metrics_middleware_still_records_business_route(monkeypatch):+def test_http_metrics_middleware_still_records_business_route(monkeypatch):captured: list[tuple[str, dict]] = []- middleware = create_http_metrics_middleware()+ middleware = create_http_observability_middleware()def _fake_emit(event_name: str, payload: dict) -> None:captured.append((event_name, dict(payload)))monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)- async def _call_next(request: Request):- request.state.metric_account_id = "acct-real"- return SimpleNamespace(status_code=200)+ async def resources_handler(request: Request):+ request.state.root_span_attrs.account_id = "acct-real"+ return {"ok": True}- request = Request(- {- "type": "http",- "method": "POST",- "path": "/api/v1/resources",- "route": SimpleNamespace(path="/api/v1/resources"),- "headers": [],- "state": {},- }+ app = _build_test_app(+ middleware,+ route_path="/api/v1/resources",+ method="POST",+ handler=resources_handler,)- await middleware(request, _call_next)+ with TestClient(app) as client:+ resp = client.post("/api/v1/resources")+ assert resp.status_code == 200assert any(event_name == "http.request" for event_name, _payload in captured)-async def test_http_metrics_middleware_uses_route_bound_during_call_next(monkeypatch):+def test_http_metrics_middleware_uses_route_bound_during_call_next(monkeypatch):captured: list[tuple[str, dict]] = []- middleware = create_http_metrics_middleware()+ middleware = create_http_observability_middleware()def _fake_emit(event_name: str, payload: dict) -> None:captured.append((event_name, dict(payload)))monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)- async def _call_next(request: Request):- request.scope["route"] = SimpleNamespace(path="/api/v1/sessions/{session_id}")- return SimpleNamespace(status_code=200)+ async def get_session_handler(session_id: str):+ return {"session_id": session_id}- request = Request(- {- "type": "http",- "method": "GET",- "path": "/api/v1/sessions/session_123",- "headers": [],- "state": {},- }+ app = _build_test_app(+ middleware,+ route_path="/api/v1/sessions/{session_id}",+ method="GET",+ handler=get_session_handler,)- await middleware(request, _call_next)+ with TestClient(app) as client:+ resp = client.get("/api/v1/sessions/session_123")+ assert resp.status_code == 200request_events = [payload for event_name, payload in captured if event_name == "http.request"]assert request_eventsassert request_events[0]["route"] == "/api/v1/sessions/{session_id}"-async def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monkeypatch):- middleware = create_http_metrics_middleware()+def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monkeypatch):+ middleware = create_http_observability_middleware()debug_calls: list[tuple[str, tuple, dict]] = []def _boom(**_kwargs):@@ -308,22 +298,22 @@ async def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monkemonkeypatch.setattr(HttpRequestLifecycleDataSource, "set_inflight", staticmethod(_boom))monkeypatch.setattr(http_middleware.logger, "isEnabledFor", lambda _level: True)monkeypatch.setattr(http_middleware.logger, "debug", _debug)+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)- async def _call_next(request: Request):- request.state.metric_account_id = "acct-real"- return SimpleNamespace(status_code=200)+ async def resources_handler(request: Request):+ request.state.root_span_attrs.account_id = "acct-real"+ return {"ok": True}- request = Request(- {- "type": "http",- "method": "POST",- "path": "/api/v1/resources",- "headers": [],- "state": {},- }+ app = _build_test_app(+ middleware,+ route_path="/api/v1/resources",+ method="POST",+ handler=resources_handler,)- await middleware(request, _call_next)+ with TestClient(app) as client:+ resp = client.post("/api/v1/resources")+ assert resp.status_code == 200assert any(message == "http metrics write failed: %s" for message, _args, _kwargs in debug_calls@@ -353,7 +343,7 @@ def test_vlm_base_update_token_usage_propagates_current_account(monkeypatch):monkeypatch.setattr(VLMEventDataSource, "record_call", staticmethod(_fake_record_call))- token = bind_metric_account_context(account_id="acct-vlm-callsite")+ token = _bind_root_context_for_account("acct-vlm-callsite")try:_DummyVLM({"provider": "volcengine", "model": "m1"}).update_token_usage(model_name="m1",@@ -363,6 +353,6 @@ def test_vlm_base_update_token_usage_propagates_current_account(monkeypatch):duration_seconds=0.5,)finally:- reset_metric_account_context(token)+ reset_root_observability_context(token)assert captured["account_id"] == "acct-vlm-callsite"diff --git a/tests/server/test_prometheus_metrics.py b/tests/server/test_prometheus_metrics.pyindex 1d82d08a0..a1ddc8a9c 100644--- a/tests/server/test_prometheus_metrics.py+++ b/tests/server/test_prometheus_metrics.py@@ -12,14 +12,17 @@ from openviking.metrics.global_api import (get_metrics_registry,init_metrics_from_server_config,shutdown_metrics,+ shutdown_metrics_async,)from openviking.server.app import create_appfrom openviking.server.config import (MetricsAccountDimensionConfig,MetricsConfig,- PrometheusConfig,+ MetricsExportersConfig,+ ObservabilityConfig,+ OTelExporterConfig,+ PrometheusExporterConfig,ServerConfig,- TelemetryConfig,)@@ -72,7 +75,16 @@ class TestRetrievalStatsMetricsIntegration:registry = MetricRegistry()shutdown_metrics(app=None)init_metrics_from_server_config(- ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True))),+ ServerConfig(+ observability=ObservabilityConfig(+ metrics=MetricsConfig(+ enabled=True,+ exporters=MetricsExportersConfig(+ prometheus=PrometheusExporterConfig(enabled=True)+ ),+ )+ )+ ),app=None,registry=registry,)@@ -115,7 +127,16 @@ class TestMetricsEndpoint:assert resp.status_code == 404async def test_metrics_enabled_returns_200(self):- config = ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True)))+ config = ServerConfig(+ observability=ObservabilityConfig(+ metrics=MetricsConfig(+ enabled=True,+ exporters=MetricsExportersConfig(+ prometheus=PrometheusExporterConfig(enabled=True)+ ),+ )+ )+ )app = create_app(config=config, service=None)init_metrics_from_server_config(config, app=app)transport = httpx.ASGITransport(app=app)@@ -133,7 +154,9 @@ class TestMetricsEndpoint:shutdown_metrics(app=app)async def test_metrics_enabled_by_new_server_metrics_flag(self):- config = ServerConfig(metrics=MetricsConfig(enabled=True))+ config = ServerConfig(+ observability=ObservabilityConfig(metrics=MetricsConfig(enabled=True))+ )app = create_app(config=config, service=None)init_metrics_from_server_config(config, app=app)transport = httpx.ASGITransport(app=app)@@ -144,34 +167,128 @@ class TestMetricsEndpoint:finally:shutdown_metrics(app=app)- async def test_metrics_enabled_by_legacy_or_new_flag(self):- config = ServerConfig(- telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=False)),- metrics=MetricsConfig(enabled=True),- )- app = create_app(config=config, service=None)- init_metrics_from_server_config(config, app=app)- assert getattr(app.state, "metrics_exporter", None) is not None- shutdown_metrics(app=app)-async def test_metrics_account_dimension_config_is_loaded(self):config = ServerConfig(- metrics=MetricsConfig(- enabled=True,- account_dimension=MetricsAccountDimensionConfig(+ observability=ObservabilityConfig(+ metrics=MetricsConfig(enabled=True,- max_active_accounts=3,- metric_allowlist=["openviking_http_requests_total"],- ),+ account_dimension=MetricsAccountDimensionConfig(+ enabled=True,+ max_active_accounts=3,+ metric_allowlist=["openviking_http_requests_total"],+ ),+ )))app = create_app(config=config, service=None)init_metrics_from_server_config(config, app=app)try:- assert config.metrics.account_dimension.enabled is True- assert config.metrics.account_dimension.max_active_accounts == 3- assert config.metrics.account_dimension.metric_allowlist == [+ assert config.observability.metrics.account_dimension.enabled is True+ assert config.observability.metrics.account_dimension.max_active_accounts == 3+ assert config.observability.metrics.account_dimension.metric_allowlist == ["openviking_http_requests_total"]finally:shutdown_metrics(app=app)+++def test_reinitializing_metrics_shuts_down_existing_exporters(monkeypatch):+ class FakeOTelExporter:+ instances = []++ def __init__(self, **kwargs):+ self.kwargs = kwargs+ self.started = 0+ self.shutdown_calls = 0+ self.__class__.instances.append(self)++ def start(self):+ self.started += 1++ async def shutdown(self):+ self.shutdown_calls += 1++ from openviking.metrics import global_api as global_api_module++ config = ServerConfig(+ observability=ObservabilityConfig(+ metrics=MetricsConfig(+ enabled=True,+ exporters=MetricsExportersConfig(+ prometheus=PrometheusExporterConfig(enabled=False),+ otel=OTelExporterConfig(enabled=True),+ ),+ )+ )+ )++ shutdown_metrics(app=None)+ monkeypatch.setattr(global_api_module, "OTelMetricExporter", FakeOTelExporter)+ init_metrics_from_server_config(config, app=None)+ first = FakeOTelExporter.instances[0]++ init_metrics_from_server_config(config, app=None)++ assert first.started == 1+ assert first.shutdown_calls == 1+ assert len(FakeOTelExporter.instances) == 2++ shutdown_metrics(app=None)+++async def test_async_shutdown_properly_awaits_exporter_cleanup(monkeypatch):+ """Async shutdown should properly await async exporter cleanup methods.++ This test verifies that shutdown_metrics_async properly awaits async shutdown+ methods, which is the recommended path for async contexts (e.g., app.py lifespan).+ """++ class FakeAsyncExporter:+ instances = []+ shutdown_awaited = False++ def __init__(self, **kwargs):+ self.kwargs = kwargs+ self.started = 0+ self.shutdown_calls = 0+ self.__class__.instances.append(self)++ def start(self):+ self.started += 1++ async def shutdown(self):+ self.shutdown_calls += 1+ FakeAsyncExporter.shutdown_awaited = True++ from openviking.metrics import global_api as global_api_module++ config = ServerConfig(+ observability=ObservabilityConfig(+ metrics=MetricsConfig(+ enabled=True,+ exporters=MetricsExportersConfig(+ prometheus=PrometheusExporterConfig(enabled=False),+ otel=OTelExporterConfig(enabled=True),+ ),+ )+ )+ )++ shutdown_metrics(app=None)+ FakeAsyncExporter.instances.clear()+ FakeAsyncExporter.shutdown_awaited = False+ monkeypatch.setattr(global_api_module, "OTelMetricExporter", FakeAsyncExporter)+ init_metrics_from_server_config(config, app=None)++ assert len(FakeAsyncExporter.instances) == 1+ first = FakeAsyncExporter.instances[0]+ assert first.started == 1+ assert first.shutdown_calls == 0+ assert FakeAsyncExporter.shutdown_awaited is False++ # Use async shutdown path - this should properly await the async shutdown+ await shutdown_metrics_async(app=None)++ assert first.shutdown_calls == 1+ assert FakeAsyncExporter.shutdown_awaited is Truediff --git a/tests/telemetry/test_http_observability_middleware.py b/tests/telemetry/test_http_observability_middleware.pynew file mode 100644index 000000000..a7dea3b7c--- /dev/null+++ b/tests/telemetry/test_http_observability_middleware.py@@ -0,0 +1,75 @@+from fastapi import FastAPI+from fastapi.testclient import TestClient+++class _DummySpan:+ def __init__(self) -> None:+ self.updated_names: list[str] = []++ def update_name(self, name: str) -> None:+ self.updated_names.append(name)+++class _DummySpanCM:+ def __init__(self, span: _DummySpan) -> None:+ self._span = span++ def __enter__(self) -> _DummySpan:+ return self._span++ def __exit__(self, exc_type, exc, tb) -> None:+ return None+++def test_http_observability_middleware_updates_route_template_after_routing(monkeypatch) -> None:+ """+ Ensure `http_route` is finalized after routing has occurred (post-call_next).++ Starlette/FastAPI route matching happens downstream of middleware, so reading+ `request.scope["route"]` before `call_next` may yield no route.+ """+ from openviking.observability import http_observability_middleware as mw_mod++ dummy_span = _DummySpan()++ # Avoid pulling in metrics/otel side effects in this unit test.+ monkeypatch.setattr(mw_mod, "should_skip_http_metrics", lambda request: False)+ monkeypatch.setattr(mw_mod, "apply_http_metrics_start", lambda **kwargs: None)+ monkeypatch.setattr(mw_mod, "apply_http_metrics_finalize", lambda **kwargs: None)+ monkeypatch.setattr(mw_mod, "maybe_apply_root_span_attributes", lambda *a, **k: None)+ monkeypatch.setattr(mw_mod, "maybe_apply_root_span_response", lambda *a, **k: None)+ monkeypatch.setattr(mw_mod, "maybe_apply_root_span_error", lambda *a, **k: None)++ # Force a "span" so we can validate operation_name update behavior.+ monkeypatch.setattr(+ mw_mod, "maybe_start_root_span", lambda request, root_attrs: _DummySpanCM(dummy_span)+ )++ captured = {}+ real_create = mw_mod.create_root_span_attributes++ def _capture_create_root_span_attributes(**kwargs):+ root_attrs = real_create(**kwargs)+ captured["root_attrs"] = root_attrs+ return root_attrs++ monkeypatch.setattr(mw_mod, "create_root_span_attributes", _capture_create_root_span_attributes)++ app = FastAPI()+ http_mw = mw_mod.create_http_observability_middleware()++ @app.middleware("http")+ async def _mw(request, call_next):+ return await http_mw(request, call_next)++ @app.get("/hello")+ async def hello():+ return {"ok": True}++ with TestClient(app) as client:+ resp = client.get("/hello")+ assert resp.status_code == 200++ root_attrs = captured["root_attrs"]+ assert root_attrs.http_route == "/hello"+ assert "GET /hello" in dummy_span.updated_namesdiff --git a/tests/telemetry/test_observability_context.py b/tests/telemetry/test_observability_context.pynew file mode 100644index 000000000..a60ea57c4--- /dev/null+++ b/tests/telemetry/test_observability_context.py@@ -0,0 +1,537 @@+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0++"""+Integration tests for unified observability context management.++This module tests the unified context management system that integrates:+- Root context (HTTP request-level)+- Operation context (business operation-level)+- Execution context (non-request scenarios)++Key features tested:+- Context binding and retrieval+- Context isolation between async tasks+- Integration with logging context+- Trace/span ID priority resolution+"""++from __future__ import annotations++import asyncio+from typing import Optional++import pytest++from openviking.observability.context import (+ ObservabilityContext,+ bind_execution_context,+ bind_operation_context,+ bind_root_context,+ get_observability_context,+ get_operation_observability_context,+ get_root_observability_context,+ reset_observability_context,+ set_observability_context,+)+from openviking.telemetry.span_models import (+ OperationSpanAttributes,+ RootSpanAttributes,+)+++class TestObservabilityContextBasics:+ """+ Basic tests for ObservabilityContext data class.+ """++ def test_default_context_creation(self) -> None:+ """+ Test that a default ObservabilityContext is created with None values.+ """+ ctx = ObservabilityContext()++ assert ctx.root is None+ assert ctx.operation is None+ assert ctx.execution_trace_id is None+ assert ctx.execution_span_id is None+ assert ctx.extra == {}++ def test_get_trace_id_empty(self) -> None:+ """+ Test that get_trace_id returns empty string when no context is available.+ """+ ctx = ObservabilityContext()++ assert ctx.get_trace_id() == ""++ def test_get_span_id_empty(self) -> None:+ """+ Test that get_span_id returns empty string when no context is available.+ """+ ctx = ObservabilityContext()++ assert ctx.get_span_id() == ""++ def test_to_log_fields_empty(self) -> None:+ """+ Test that to_log_fields returns basic fields even when empty.+ """+ ctx = ObservabilityContext()++ fields = ctx.to_log_fields()++ assert "trace_id" in fields+ assert "span_id" in fields+ assert fields["trace_id"] == ""+ assert fields["span_id"] == ""++ def test_bind_execution_generates_ids(self) -> None:+ """+ Test that bind_execution generates trace_id and span_id when not provided.+ """+ ctx = ObservabilityContext()+ ctx.bind_execution()++ assert ctx.execution_trace_id is not None+ assert ctx.execution_span_id is not None+ assert len(ctx.execution_trace_id) == 32 # UUID hex is 32 chars+ assert len(ctx.execution_span_id) == 16 # First 16 chars of trace_id++ def test_bind_execution_uses_provided_ids(self) -> None:+ """+ Test that bind_execution uses provided trace_id and span_id.+ """+ ctx = ObservabilityContext()+ ctx.bind_execution(trace_id="custom_trace_id", span_id="custom_span_id")++ assert ctx.execution_trace_id == "custom_trace_id"+ assert ctx.execution_span_id == "custom_span_id"++ def test_set_extra(self) -> None:+ """+ Test that set_extra adds custom attributes.+ """+ ctx = ObservabilityContext()+ ctx.set_extra("custom_key", "custom_value")+ ctx.set_extra("number_key", 42)++ assert ctx.extra["custom_key"] == "custom_value"+ assert ctx.extra["number_key"] == 42++ def test_to_log_fields_includes_extra(self) -> None:+ """+ Test that to_log_fields includes extra attributes.+ """+ ctx = ObservabilityContext()+ ctx.set_extra("custom_field", "custom_value")++ fields = ctx.to_log_fields()++ assert fields["custom_field"] == "custom_value"+++class TestRootContextBinding:+ """+ Tests for root context binding and retrieval.+ """++ def test_bind_root_context(self) -> None:+ """+ Test that root context can be bound and retrieved.+ """+ root_attrs = RootSpanAttributes(+ http_method="GET",+ http_route="/test",+ request_id="req-123",+ )+ root_attrs.account_id = "acct-456"++ token = bind_root_context(root_attrs)++ try:+ retrieved = get_root_observability_context()++ assert retrieved is not None+ assert retrieved.http_method == "GET"+ assert retrieved.http_route == "/test"+ assert retrieved.request_id == "req-123"+ assert retrieved.account_id == "acct-456"+ finally:+ reset_observability_context(token)++ def test_root_context_to_log_fields(self) -> None:+ """+ Test that root context fields are included in to_log_fields.+ """+ root_attrs = RootSpanAttributes(+ http_method="POST",+ http_route="/api/v1/sessions",+ request_id="req-test",+ )+ root_attrs.account_id = "acct-test"++ token = bind_root_context(root_attrs)++ try:+ ctx = get_observability_context()+ fields = ctx.to_log_fields()++ assert fields["request_id"] == "req-test"+ assert fields["account_id"] == "acct-test"+ assert fields["http_method"] == "POST"+ assert fields["http_route"] == "/api/v1/sessions"+ finally:+ reset_observability_context(token)++ def test_reset_root_context(self) -> None:+ """+ Test that root context can be reset.+ """+ root_attrs = RootSpanAttributes(+ http_method="GET",+ http_route="/test",+ request_id="req-123",+ )++ token = bind_root_context(root_attrs)++ try:+ assert get_root_observability_context() is not None+ finally:+ reset_observability_context(token)++ # After reset, get_root_observability_context should return None+ # because the context is reset to its previous state+ retrieved = get_root_observability_context()+ assert retrieved is None+++class TestOperationContextBinding:+ """+ Tests for operation context binding and retrieval.+ """++ def test_bind_operation_context(self) -> None:+ """+ Test that operation context can be bound and retrieved.+ """+ operation_attrs = OperationSpanAttributes(+ operation="search.find",+ telemetry_id="tm-123",+ )++ token = bind_operation_context(operation_attrs)++ try:+ retrieved = get_operation_observability_context()++ assert retrieved is not None+ assert retrieved.operation == "search.find"+ assert retrieved.telemetry_id == "tm-123"+ finally:+ reset_observability_context(token)++ def test_operation_context_to_log_fields(self) -> None:+ """+ Test that operation context fields are included in to_log_fields.+ """+ operation_attrs = OperationSpanAttributes(+ operation="session.commit",+ telemetry_id="tm-456",+ )+ operation_attrs.status = "ok"++ token = bind_operation_context(operation_attrs)++ try:+ ctx = get_observability_context()+ fields = ctx.to_log_fields()++ assert fields["operation"] == "session.commit"+ assert fields["telemetry_id"] == "tm-456"+ assert fields["status"] == "ok"+ finally:+ reset_observability_context(token)+++class TestExecutionContextBinding:+ """+ Tests for execution context binding (context manager.+ """++ def test_bind_execution_context_manager(self) -> None:+ """+ Test that execution context can be bound using context manager.+ """+ with bind_execution_context() as (trace_id, span_id):+ assert trace_id is not None+ assert span_id is not None+ assert len(trace_id) == 32+ assert len(span_id) == 16++ ctx = get_observability_context()+ assert ctx.execution_trace_id == trace_id+ assert ctx.execution_span_id == span_id++ # After exiting context manager, execution context should be reset+ ctx_after = get_observability_context()+ assert ctx_after.execution_trace_id is None+ assert ctx_after.execution_span_id is None++ def test_bind_execution_with_custom_ids(self) -> None:+ """+ Test that execution context can be bound with custom IDs.+ """+ custom_trace_id = "custom_trace_123456789012345678901234"+ custom_span_id = "custom_span_123456"++ with bind_execution_context(+ trace_id=custom_trace_id,+ span_id=custom_span_id,+ ) as (trace_id, span_id):+ assert trace_id == custom_trace_id+ assert span_id == custom_span_id++ ctx = get_observability_context()+ assert ctx.execution_trace_id == custom_trace_id+ assert ctx.execution_span_id == custom_span_id++ def test_execution_context_trace_id_priority(self) -> None:+ """+ Test that execution context trace_id is used when no OTel span is available.+ """+ with bind_execution_context() as (trace_id, span_id):+ ctx = get_observability_context()++ # get_trace_id should return the execution trace_id+ # since no OTel span is available+ effective_trace_id = ctx.get_trace_id()+ effective_span_id = ctx.get_span_id()++ assert effective_trace_id == trace_id+ assert effective_span_id == span_id+++class TestContextIsolation:+ """+ Tests for context isolation between async tasks.+ """++ @pytest.mark.asyncio+ async def test_async_task_context_isolation(self) -> None:+ """+ Test that context is isolated between concurrent async tasks.+ """+ seen: list[Optional[str]] = []++ async def _worker(account_id: str):+ root_attrs = RootSpanAttributes(+ http_method="GET",+ http_route="/test",+ request_id=f"req-{account_id}",+ )+ root_attrs.account_id = account_id++ token = bind_root_context(root_attrs)+ try:+ await asyncio.sleep(0) # Yield to other tasks+ root_context = get_root_observability_context()+ seen.append(root_context.account_id if root_context else None)+ finally:+ reset_observability_context(token)++ await asyncio.gather(+ _worker("acct-a"),+ _worker("acct-b"),+ _worker("acct-c"),+ )++ # All three tasks should see their own context, not leaked from others+ assert sorted(seen) == ["acct-a", "acct-b", "acct-c"]++ @pytest.mark.asyncio+ async def test_nested_context_binding(self) -> None:+ """+ Test that nested context binding works correctly.+ """+ root_attrs_outer = RootSpanAttributes(+ http_method="GET",+ http_route="/outer",+ request_id="req-outer",+ )+ root_attrs_outer.account_id = "acct-outer"++ token_outer = bind_root_context(root_attrs_outer)++ try:+ # Verify outer context+ outer_ctx = get_root_observability_context()+ assert outer_ctx is not None+ assert outer_ctx.account_id == "acct-outer"++ # Bind inner context+ root_attrs_inner = RootSpanAttributes(+ http_method="POST",+ http_route="/inner",+ request_id="req-inner",+ )+ root_attrs_inner.account_id = "acct-inner"++ token_inner = bind_root_context(root_attrs_inner)++ try:+ # Verify inner context+ inner_ctx = get_root_observability_context()+ assert inner_ctx is not None+ assert inner_ctx.account_id == "acct-inner"+ finally:+ reset_observability_context(token_inner)++ # After resetting inner context, should be back to outer+ after_inner_reset = get_root_observability_context()+ assert after_inner_reset is not None+ assert after_inner_reset.account_id == "acct-outer"++ finally:+ reset_observability_context(token_outer)++ # After resetting outer context+ after_outer_reset = get_root_observability_context()+ assert after_outer_reset is None+++class TestCombinedContexts:+ """+ Tests for combining multiple context types.+ """++ def test_combined_root_and_operation_context(self) -> None:+ """+ Test that root and operation contexts can be combined.+ """+ root_attrs = RootSpanAttributes(+ http_method="POST",+ http_route="/api/v1/sessions",+ request_id="req-123",+ )+ root_attrs.account_id = "acct-456"++ operation_attrs = OperationSpanAttributes(+ operation="session.commit",+ telemetry_id="tm-789",+ )+ operation_attrs.status = "ok"++ # Bind root context first+ token_root = bind_root_context(root_attrs)++ try:+ # Then bind operation context+ token_operation = bind_operation_context(operation_attrs)++ try:+ ctx = get_observability_context()+ fields = ctx.to_log_fields()++ # Both root and operation fields should be present+ assert fields["request_id"] == "req-123"+ assert fields["account_id"] == "acct-456"+ assert fields["operation"] == "session.commit"+ assert fields["telemetry_id"] == "tm-789"+ assert fields["status"] == "ok"++ finally:+ reset_observability_context(token_operation)++ # After resetting operation context, root context should still be there+ after_op_reset = get_root_observability_context()+ assert after_op_reset is not None+ assert after_op_reset.account_id == "acct-456"++ finally:+ reset_observability_context(token_root)+++class TestContextManagerIntegration:+ """+ Tests for integration with context manager patterns.+ """++ def test_set_and_reset_observability_context(self) -> None:+ """+ Test that set_observability_context and reset_observability_context work correctly.+ """+ # Create a custom context+ custom_ctx = ObservabilityContext()+ custom_ctx.set_extra("custom_key", "custom_value")+ custom_ctx.bind_execution(trace_id="custom_trace", span_id="custom_span")++ # Set the context+ token = set_observability_context(custom_ctx)++ try:+ # Verify the context is set+ retrieved = get_observability_context()+ assert retrieved.execution_trace_id == "custom_trace"+ assert retrieved.execution_span_id == "custom_span"+ assert retrieved.extra["custom_key"] == "custom_value"++ finally:+ # Reset the context+ reset_observability_context(token)++ # After reset, should be back to default+ after_reset = get_observability_context()+ assert after_reset.execution_trace_id is None+ assert after_reset.execution_span_id is None+++class TestLogFieldsIntegration:+ """+ Tests for log fields integration.+ """++ def test_to_log_fields_combines_all_contexts(self) -> None:+ """+ Test that to_log_fields combines all context types correctly.+ """+ ctx = ObservabilityContext()++ # Add root context+ root_attrs = RootSpanAttributes(+ http_method="GET",+ http_route="/test",+ request_id="req-123",+ )+ root_attrs.account_id = "acct-456"+ ctx.bind_root(root_attrs)++ # Add operation context+ operation_attrs = OperationSpanAttributes(+ operation="search.find",+ telemetry_id="tm-789",+ )+ operation_attrs.status = "ok"+ ctx.bind_operation(operation_attrs)++ # Add execution context+ ctx.bind_execution(trace_id="exec_trace", span_id="exec_span")++ # Add extra fields+ ctx.set_extra("extra_field", "extra_value")++ # Get log fields+ fields = ctx.to_log_fields()++ # Verify all fields are present+ assert fields["trace_id"] == "exec_trace"+ assert fields["span_id"] == "exec_span"+ assert fields["request_id"] == "req-123"+ assert fields["account_id"] == "acct-456"+ assert fields["http_method"] == "GET"+ assert fields["http_route"] == "/test"+ assert fields["operation"] == "search.find"+ assert fields["telemetry_id"] == "tm-789"+ assert fields["status"] == "ok"+ assert fields["extra_field"] == "extra_value"diff --git a/tests/test_server_config_loader.py b/tests/test_server_config_loader.pyindex 155c95dd0..5e9c17df2 100644--- a/tests/test_server_config_loader.py+++ b/tests/test_server_config_loader.py@@ -21,11 +21,19 @@ def test_load_server_config_rejects_unknown_field(tmp_path):def test_load_server_config_rejects_unknown_nested_field(tmp_path):config_path = tmp_path / "ov.conf"- config_path.write_text(json.dumps({"server": {"telemetry": {"prometheus": {"enabld": True}}}}))+ config_path.write_text(+ json.dumps(+ {+ "server": {+ "observability": {"metrics": {"exporters": {"prometheus": {"enabld": True}}}}+ }+ }+ )+ )with pytest.raises(ValueError,- match=r"server\.telemetry\.prometheus\.enabld'.*server\.telemetry\.prometheus\.enabled",+ match=r"server\.observability\.metrics\.exporters\.prometheus\.enabld'.*server\.observability\.metrics\.exporters\.prometheus\.enabled",):load_server_config(str(config_path))@@ -50,7 +58,7 @@ def test_load_server_config_preserves_supported_fields(tmp_path):"auth_mode": "trusted","with_bot": True,"bot_api_url": "http://localhost:19999",- "telemetry": {"prometheus": {"enabled": True}},+ "observability": {"metrics": {"exporters": {"prometheus": {"enabled": True}}}},},"encryption": {"enabled": True},}@@ -65,23 +73,13 @@ def test_load_server_config_preserves_supported_fields(tmp_path):assert config.auth_mode == "trusted"assert config.with_bot is Trueassert config.bot_api_url == "http://localhost:19999"- assert config.telemetry.prometheus.enabled is True+ assert config.observability.metrics.exporters.prometheus.enabled is Trueassert config.encryption_enabled is Truedef test_load_bot_gateway_token_reads_token_from_bot_gateway_section(tmp_path):config_path = tmp_path / "ov.conf"- config_path.write_text(- json.dumps(- {- "bot": {- "gateway": {- "token": "gateway-token"- }- }- }- )- )+ config_path.write_text(json.dumps({"bot": {"gateway": {"token": "gateway-token"}}}))assert load_bot_gateway_token(str(config_path)) == "gateway-token"@@ -92,16 +90,18 @@ def test_load_server_config_preserves_metrics_account_dimension_fields(tmp_path)json.dumps({"server": {- "metrics": {- "enabled": True,- "account_dimension": {+ "observability": {+ "metrics": {"enabled": True,- "max_active_accounts": 5,- "metric_allowlist": [- "openviking_http_requests_total",- "openviking_task_pending",- ],- },+ "account_dimension": {+ "enabled": True,+ "max_active_accounts": 5,+ "metric_allowlist": [+ "openviking_http_requests_total",+ "openviking_task_pending",+ ],+ },+ }}}}@@ -110,10 +110,10 @@ def test_load_server_config_preserves_metrics_account_dimension_fields(tmp_path)config = load_server_config(str(config_path))- assert config.metrics.enabled is True- assert config.metrics.account_dimension.enabled is True- assert config.metrics.account_dimension.max_active_accounts == 5- assert config.metrics.account_dimension.metric_allowlist == [+ assert config.observability.metrics.enabled is True+ assert config.observability.metrics.account_dimension.enabled is True+ assert config.observability.metrics.account_dimension.max_active_accounts == 5+ assert config.observability.metrics.account_dimension.metric_allowlist == ["openviking_http_requests_total","openviking_task_pending",]diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.pyindex aa1ae9219..1a2e0274e 100644--- a/tests/test_telemetry_runtime.py+++ b/tests/test_telemetry_runtime.py@@ -9,9 +9,16 @@ from types import SimpleNamespaceimport pytest-from openviking.metrics.account_context import get_metric_account_contextfrom openviking.models.embedder.base import EmbedResult, embed_compatfrom openviking.models.vlm.base import VLMBase+from openviking.observability.context import (+ bind_operation_observability_context,+ bind_root_observability_context,+ get_operation_observability_context,+ get_root_observability_context,+ reset_operation_observability_context,+ reset_root_observability_context,+)from openviking.server.identity import RequestContext, Rolefrom openviking.service.resource_service import ResourceServicefrom openviking.storage.collection_schemas import TextEmbeddingHandler@@ -27,6 +34,7 @@ from openviking.telemetry import (from openviking.telemetry.backends.memory import MemoryOperationTelemetryfrom openviking.telemetry.context import bind_telemetry, bind_telemetry_stagefrom openviking.telemetry.snapshot import TelemetrySnapshot+from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributesfrom openviking_cli.session.user_id import UserIdentifier@@ -41,6 +49,26 @@ def test_telemetry_module_exports_snapshot_and_runtime():assert get_telemetry_runtime().meter() is not None+def test_root_observability_context_bind_and_reset():+ root = RootSpanAttributes(http_method="GET", http_route="/demo", request_id="req-1")+ token = bind_root_observability_context(root)+ try:+ assert get_root_observability_context() is root+ finally:+ reset_root_observability_context(token)+ assert get_root_observability_context() is None+++def test_operation_observability_context_bind_and_reset():+ operation = OperationSpanAttributes(operation="search.find", telemetry_id="tm-demo")+ token = bind_operation_observability_context(operation)+ try:+ assert get_operation_observability_context() is operation+ finally:+ reset_operation_observability_context(token)+ assert get_operation_observability_context() is None++def test_telemetry_snapshot_to_dict_supports_summary_only():snapshot = TelemetrySnapshot(telemetry_id="tm_demo",@@ -286,7 +314,9 @@ async def test_semantic_processor_binds_metric_account_context(monkeypatch):async def run(self, root_uri):ran["value"] = True- assert get_metric_account_context().http_account_id == "acct-semantic"+ root_context = get_root_observability_context()+ assert root_context is not None+ assert root_context.account_id == "acct-semantic"def get_stats(self):return DagStats()diff --git a/uv.lock b/uv.lockindex 77594d811..11e8ad6b3 100644--- a/uv.lock+++ b/uv.lock@@ -3509,6 +3509,7 @@ dependencies = [{ name = "openpyxl" },{ name = "opentelemetry-api" },{ name = "opentelemetry-exporter-otlp-proto-grpc" },+ { name = "opentelemetry-exporter-otlp-proto-http" },{ name = "opentelemetry-instrumentation-asyncio" },{ name = "opentelemetry-sdk" },{ name = "pdfminer-six" },@@ -3746,6 +3747,7 @@ requires-dist = [{ name = "opensandbox-server", marker = "extra == 'bot-sandbox'", specifier = ">=0.1.0" },{ name = "opentelemetry-api", specifier = ">=1.14" },{ name = "opentelemetry-exporter-otlp-proto-grpc", specifier = ">=1.14" },+ { name = "opentelemetry-exporter-otlp-proto-http", specifier = ">=1.14" },{ name = "opentelemetry-instrumentation-asyncio", specifier = ">=0.61b0" },{ name = "opentelemetry-sdk", specifier = ">=1.14" },{ name = "openviking", extras = ["bot", "bot-dingtalk", "bot-feishu", "bot-fuse", "bot-langfuse", "bot-opencode", "bot-qq", "bot-sandbox", "bot-slack", "bot-telegram"], marker = "extra == 'bot-full'" },