Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(observability): unify observability context && support otel && etc. (#1666)

baojun-zhang17d2c5603e13
diff --git a/docs/design/metric-design.md b/docs/design/metric-design.md
index 618a2b260..a5daaf8b5 100644
--- a/docs/design/metric-design.md
+++ b/docs/design/metric-design.md
@@ -810,6 +810,7 @@ Openviking 指标体系采用如下兼容性策略:
- 保持现有 retrieval / embedding / vlm / cache 指标名尽量不变;
- `/metrics` 路由地址保持不变;
- `server.observability.metrics.enabled` 配置保持不变;
+- `server.observability.metrics.exporters.*` 作为 exporter 扩展点:默认 Prometheus exporter 可继续工作,OTLP exporter 可按需启用;
- `/api/v1/observer/*` 与 `/api/v1/stats/*` 行为保持不变。
## 6. 风险与缓解
diff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.md
index e4bf34275..689b6cc2c 100644
--- a/docs/en/concepts/12-metrics.md
+++ b/docs/en/concepts/12-metrics.md
@@ -365,6 +365,48 @@ Recommended mental model:
- `server.observability.metrics.enabled`: master switch for the metrics subsystem
- `server.observability.metrics.account_dimension`: controls whether `account_id` labels are enabled and where they are allowed
+### Exporters
+
+By default, OpenViking exports metrics via Prometheus exposition format at `/metrics`.
+You can also enable additional exporters under `server.observability.metrics.exporters`.
+
+Key fields:
+
+- `server.observability.metrics.exporters.prometheus.enabled`: enable the Prometheus exporter (serves `/metrics`)
+- `server.observability.metrics.exporters.otel.enabled`: enable OTLP export from the same in-process registry
+- `server.observability.metrics.exporters.otel.protocol`: `"grpc"` or `"http"`
+- `server.observability.metrics.exporters.otel.tls.insecure`: OTLP/gRPC only; `true` means plaintext (no TLS)
+- `server.observability.metrics.exporters.otel.endpoint`: OTLP endpoint (for gRPC, use `host:4317`; for HTTP, use a full URL)
+
+Example:
+
+```json
+{
+ "server": {
+ "observability": {
+ "metrics": {
+ "enabled": true,
+ "exporters": {
+ "prometheus": {
+ "enabled": true
+ },
+ "otel": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server",
+ "export_interval_ms": 10000
+ }
+ }
+ }
+ }
+ }
+}
+```
+
### Recommended `account_id` Usage
- enabled by default, but only allowlisted metric families will receive tenant ids (empty allowlist still yields `__unknown__`)
diff --git a/docs/en/guides/05-observability.md b/docs/en/guides/05-observability.md
index e9f68587e..3b0c5bd7e 100644
--- a/docs/en/guides/05-observability.md
+++ b/docs/en/guides/05-observability.md
@@ -270,6 +270,61 @@ Add the following to `~/.openviking/ov.conf` (or the path passed via `--config`)
Restart OpenViking Server after editing the config.
+### Observability config hierarchy
+
+OpenViking groups signal-level observability configuration under `server.observability`:
+
+- `server.observability.metrics`: metrics subsystem and exporters
+- `server.observability.traces`: trace export configuration
+- `server.observability.logs`: log export configuration
+
+Example:
+
+```json
+{
+ "server": {
+ "observability": {
+ "metrics": {
+ "enabled": true,
+ "exporters": {
+ "prometheus": {
+ "enabled": true
+ },
+ "otel": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server",
+ "export_interval_ms": 10000
+ }
+ }
+ },
+ "traces": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server"
+ },
+ "logs": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server"
+ }
+ }
+ }
+}
+```
+
For full fields, supported ranges, and more examples, see:
- [Metrics](../concepts/12-metrics.md)
diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.md
index a4f1668f8..1e3c8360e 100644
--- a/docs/zh/concepts/12-metrics.md
+++ b/docs/zh/concepts/12-metrics.md
@@ -368,6 +368,48 @@ scrape_configs:
- `server.observability.metrics.enabled`:指标体系总开关
- `server.observability.metrics.account_dimension`:控制 `account_id` 标签是否启用以及启用范围
+### Exporters 配置
+
+默认情况下,OpenViking 会通过 Prometheus exposition 格式在 `/metrics` 输出指标。
+如果希望在保留 `/metrics` 的同时把同一份进程内指标导出到 OTLP 后端,可以在 `server.observability.metrics.exporters` 下启用 exporter。
+
+关键字段:
+
+- `server.observability.metrics.exporters.prometheus.enabled`:是否启用 Prometheus exporter(提供 `/metrics`)
+- `server.observability.metrics.exporters.otel.enabled`:是否启用 OTLP 导出(复用同一份 registry)
+- `server.observability.metrics.exporters.otel.protocol`:`"grpc"` 或 `"http"`
+- `server.observability.metrics.exporters.otel.tls.insecure`:仅对 OTLP/gRPC 生效;`true` 表示明文连接(无 TLS)
+- `server.observability.metrics.exporters.otel.endpoint`:OTLP 端点(gRPC 用 `host:4317`;HTTP 必须是完整 URL)
+
+示例:
+
+```json
+{
+ "server": {
+ "observability": {
+ "metrics": {
+ "enabled": true,
+ "exporters": {
+ "prometheus": {
+ "enabled": true
+ },
+ "otel": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server",
+ "export_interval_ms": 10000
+ }
+ }
+ }
+ }
+ }
+}
+```
+
### `account_id` 标签的使用建议
- 默认开启,但仅对白名单指标启用(`metric_allowlist` 为空时仍会输出为 `__unknown__`)
diff --git a/docs/zh/guides/05-observability.md b/docs/zh/guides/05-observability.md
index 3ec1b1796..f5c0bcafc 100644
--- a/docs/zh/guides/05-observability.md
+++ b/docs/zh/guides/05-observability.md
@@ -269,6 +269,61 @@ curl -X POST http://localhost:1933/api/v1/search/find \
```
改完配置后需要**重启 OpenViking Server** 才会生效。
+### observability 配置层级
+
+OpenViking 将信号级别的可观测性配置统一放在 `server.observability` 下:
+
+- `server.observability.metrics`:metrics 子系统与 exporter 配置
+- `server.observability.traces`:trace 导出配置
+- `server.observability.logs`:log 导出配置
+
+示例:
+
+```json
+{
+ "server": {
+ "observability": {
+ "metrics": {
+ "enabled": true,
+ "exporters": {
+ "prometheus": {
+ "enabled": true
+ },
+ "otel": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server",
+ "export_interval_ms": 10000
+ }
+ }
+ },
+ "traces": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server"
+ },
+ "logs": {
+ "enabled": true,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": true
+ },
+ "endpoint": "otel-collector:4317",
+ "service_name": "openviking-server"
+ }
+ }
+ }
+}
+```
+
完整字段、支持范围和更多示例见:
- [指标](../concepts/12-metrics.md)
diff --git a/examples/ov.conf.example b/examples/ov.conf.example
index d0979655a..08ba4cf44 100644
--- a/examples/ov.conf.example
+++ b/examples/ov.conf.example
@@ -5,16 +5,49 @@
"root_api_key": null,
"cors_origins": ["*"],
"observability": {
- "metrics": {
- "enabled": true,
- "account_dimension": {
- "enabled": true,
- "max_active_accounts": 100,
- "metric_allowlist": [
- "openviking_*"
- ]
+ "metrics": {
+ "enabled": true,
+ "account_dimension": {
+ "enabled": true,
+ "max_active_accounts": 100,
+ "metric_allowlist": [
+ "openviking_*"
+ ]
+ },
+ "exporters": {
+ "prometheus": {
+ "enabled": true
+ },
+ "otel": {
+ "enabled": false,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": false
+ },
+ "endpoint": "localhost:4317",
+ "service_name": "openviking-server",
+ "export_interval_ms": 10000
+ }
+ }
+ },
+ "traces": {
+ "enabled": false,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": false
+ },
+ "endpoint": "localhost:4317",
+ "service_name": "openviking-server"
+ },
+ "logs": {
+ "enabled": false,
+ "protocol": "grpc",
+ "tls": {
+ "insecure": false
+ },
+ "endpoint": "localhost:4317",
+ "service_name": "openviking-server"
}
- }
}
},
"storage": {
diff --git a/openviking/metrics/account_context.py b/openviking/metrics/account_context.py
deleted file mode 100644
index 81d0b2591..000000000
--- a/openviking/metrics/account_context.py
+++ /dev/null
@@ -1,80 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-
-"""
-Request-scoped account context for metrics label injection.
-
-This module keeps metrics-specific request context separate from business-layer request
-objects so collectors can resolve `account_id` without depending on FastAPI internals.
-"""
-
-from __future__ import annotations
-
-import contextvars
-from dataclasses import dataclass
-
-
-@dataclass(frozen=True, slots=True)
-class MetricAccountContext:
- """
- Store request-scoped account information used during metrics label resolution.
-
- The context currently keeps the HTTP-derived account identifier that collectors and
- account-dimension policy code can consult without reaching back into framework-specific
- request objects.
- """
-
- http_account_id: str | None = None
-
-
-_CURRENT_METRIC_ACCOUNT_CONTEXT: contextvars.ContextVar[MetricAccountContext | None] = (
- contextvars.ContextVar(
- "openviking_metric_account_context",
- default=None,
- )
-)
-
-
-def get_metric_account_context() -> MetricAccountContext:
- """
- Return the metrics account context bound to the current execution flow.
-
- When no context has been bound yet, the function returns an empty `MetricAccountContext`
- instance so callers can treat the result as a stable object instead of handling `None`.
- """
- context = _CURRENT_METRIC_ACCOUNT_CONTEXT.get()
- if context is None:
- return MetricAccountContext()
- return context
-
-
-def bind_metric_account_context(
- *, account_id: str | None
-) -> contextvars.Token[MetricAccountContext | None]:
- """
- Bind a new metrics account context and return the token needed to restore the prior one.
-
- This helper is intended for request or worker entry points that need scoped lifetime
- management through `contextvars.Token`, typically via `try/finally`.
- """
- return _CURRENT_METRIC_ACCOUNT_CONTEXT.set(MetricAccountContext(http_account_id=account_id))
-
-
-def set_metric_account_context(*, account_id: str | None) -> None:
- """
- Overwrite the current metrics account context for the active execution flow.
-
- Unlike `bind_metric_account_context`, this helper does not provide a restoration token and
- is therefore best suited to one-way updates where the caller owns the full scope lifetime.
- """
- _CURRENT_METRIC_ACCOUNT_CONTEXT.set(MetricAccountContext(http_account_id=account_id))
-
-
-def reset_metric_account_context(token: contextvars.Token[MetricAccountContext | None]) -> None:
- """
- Restore the previous metrics account context captured by `bind_metric_account_context`.
-
- Callers should use the exact token returned by the matching bind operation so nested context
- updates unwind in the correct order.
- """
- _CURRENT_METRIC_ACCOUNT_CONTEXT.reset(token)
diff --git a/openviking/metrics/account_dimension.py b/openviking/metrics/account_dimension.py
index 96bb05166..96bc7d79b 100644
--- a/openviking/metrics/account_dimension.py
+++ b/openviking/metrics/account_dimension.py
@@ -13,7 +13,7 @@ from __future__ import annotations
import threading
from dataclasses import dataclass
-from .account_context import get_metric_account_context
+from openviking.observability.context import get_root_observability_context
UNKNOWN_ACCOUNT_ID = "__unknown__"
OVERFLOW_ACCOUNT_ID = "__overflow__"
@@ -96,7 +96,8 @@ class MetricAccountContextResolver:
if explicit_account_id and str(explicit_account_id).strip():
return str(explicit_account_id).strip()
- http_account_id = get_metric_account_context().http_account_id
+ root_context = get_root_observability_context()
+ http_account_id = root_context.account_id if root_context is not None else None
if http_account_id and str(http_account_id).strip():
return str(http_account_id).strip()
diff --git a/openviking/metrics/exporters/__init__.py b/openviking/metrics/exporters/__init__.py
index 118902a47..f912459bd 100644
--- a/openviking/metrics/exporters/__init__.py
+++ b/openviking/metrics/exporters/__init__.py
@@ -3,6 +3,7 @@
"""Exporter entry points that render the shared metrics registry into external formats."""
+from .otel import OTelMetricExporter
from .prometheus import PrometheusExporter
-__all__ = ["PrometheusExporter"]
+__all__ = ["PrometheusExporter", "OTelMetricExporter"]
diff --git a/openviking/metrics/exporters/otel.py b/openviking/metrics/exporters/otel.py
new file mode 100644
index 000000000..de3af171a
--- /dev/null
+++ b/openviking/metrics/exporters/otel.py
@@ -0,0 +1,713 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""
+OpenTelemetry OTLP metrics exporter implementation.
+
+This exporter serializes the in-process MetricRegistry directly into OTLP protobuf
+messages and sends them to an OTLP endpoint over HTTP or gRPC.
+
+Key design goals:
+- Export the same MetricRegistry snapshot semantics that PrometheusExporter renders
+- Preserve histogram bucket/count/sum semantics without re-aggregation
+- Keep exporter lifecycle self-contained through start()/shutdown()
+- Refresh collectors before export so snapshot-based metrics stay aligned
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from typing import Any, Iterable, Optional, Tuple
+
+import requests
+
+from openviking.metrics.core.base import MetricExporter
+
+logger = logging.getLogger(__name__)
+
+try:
+ import grpc
+except ImportError:
+ grpc = None
+
+try:
+ from requests import RequestException
+except Exception:
+ RequestException = Exception
+
+try:
+ from opentelemetry.proto.collector.metrics.v1.metrics_service_pb2 import (
+ ExportMetricsServiceRequest,
+ )
+ from opentelemetry.proto.collector.metrics.v1.metrics_service_pb2_grpc import (
+ MetricsServiceStub,
+ )
+ from opentelemetry.proto.common.v1.common_pb2 import AnyValue, InstrumentationScope, KeyValue
+ from opentelemetry.proto.metrics.v1.metrics_pb2 import (
+ AGGREGATION_TEMPORALITY_CUMULATIVE,
+ Gauge,
+ Histogram,
+ HistogramDataPoint,
+ Metric,
+ NumberDataPoint,
+ Sum,
+ )
+ from opentelemetry.proto.resource.v1.resource_pb2 import Resource
+except ImportError:
+ ExportMetricsServiceRequest = None
+ MetricsServiceStub = None
+ AnyValue = None
+ InstrumentationScope = None
+ KeyValue = None
+ Gauge = None
+ Histogram = None
+ HistogramDataPoint = None
+ Metric = None
+ NumberDataPoint = None
+ Sum = None
+ Resource = None
+ AGGREGATION_TEMPORALITY_CUMULATIVE = None
+
+
+def _compute_next_tick_and_sleep(
+ *,
+ now: float,
+ interval_s: float,
+ next_tick: float,
+) -> tuple[float, float]:
+ """
+ Compute next tick time and sleep duration for a fixed-interval loop.
+
+ This is a pure function to keep scheduling behavior testable without adding
+ production-only hooks.
+
+ Rules:
+ - If the loop overruns (now >= next_tick), advance next_tick by interval until it is in the future.
+ - Sleep only until the next_tick.
+
+ Args:
+ now: Current monotonic time in seconds.
+ interval_s: Target loop interval in seconds.
+ next_tick: Previously scheduled tick time in seconds.
+
+ Returns:
+ (next_tick, sleep_seconds)
+ """
+ while next_tick <= now:
+ next_tick += interval_s
+ return next_tick, max(0.0, next_tick - now)
+
+
+class OTelMetricExporter(MetricExporter):
+ """
+ Export MetricRegistry snapshots to OTLP metrics.
+
+ This exporter intentionally avoids OTel SDK metric aggregation. Instead, it
+ periodically snapshots MetricRegistry, converts the snapshot into OTLP protobuf
+ payloads, and pushes them to the configured collector endpoint.
+ """
+
+ def __init__(
+ self,
+ *,
+ registry,
+ collector_manager=None,
+ refresh_deadline_seconds: float = 1.0,
+ protocol: str = "grpc",
+ insecure: bool = False,
+ endpoint: str = "localhost:4317",
+ service_name: str = "openviking-server",
+ export_interval_ms: int = 10000,
+ export_timeout_seconds: Optional[float] = None,
+ enabled: bool = True,
+ ) -> None:
+ """
+ Initialize the OTLP metrics exporter.
+
+ Args:
+ registry: The in-process metric registry backing all exporters.
+ collector_manager: Optional collector refresh orchestrator.
+ refresh_deadline_seconds: Best-effort refresh timeout before each export.
+ protocol: Transport protocol, either "grpc" or "http".
+ insecure: Whether gRPC transport should use plaintext.
+ endpoint: OTLP collector endpoint.
+ service_name: Service name written into OTLP resource attributes.
+ export_interval_ms: Periodic export interval in milliseconds.
+ export_timeout_seconds: Transport timeout for one OTLP export request.
+ enabled: Whether the exporter is enabled.
+ """
+ self._registry = registry
+ self._collector_manager = collector_manager
+ self._refresh_deadline_seconds = float(refresh_deadline_seconds)
+ self._protocol = protocol.lower()
+ self._insecure = bool(insecure)
+ self._endpoint = endpoint
+ self._service_name = service_name
+ self._export_interval_ms = max(1000, int(export_interval_ms))
+ self._export_timeout_seconds = float(
+ export_timeout_seconds
+ if export_timeout_seconds is not None
+ else max(5.0, self._refresh_deadline_seconds)
+ )
+ self._enabled = bool(enabled)
+ self._export_task: Optional[asyncio.Task[Any]] = None
+ self._http_session: Optional[requests.Session] = None
+ self._grpc_channel: Optional[Any] = None
+ self._grpc_stub: Optional[Any] = None
+ self._start_time_ns = time.time_ns()
+
+ if self._protocol not in {"grpc", "http"}:
+ logger.warning(
+ "[OTelMetricExporter] invalid protocol=%s, defaulting to grpc",
+ self._protocol,
+ )
+ self._protocol = "grpc"
+
+ if self._enabled:
+ self._init_transport()
+
+ def _init_transport(self) -> None:
+ """
+ Initialize the transport client for the configured OTLP protocol.
+
+ The exporter keeps the transport client alive across export cycles so the
+ periodic loop only pays the cost of serializing current registry snapshots.
+ """
+ if not self._protobuf_available():
+ logger.warning(
+ "[OTelMetricExporter] initialization failed: OTLP metrics protobuf dependencies are not available"
+ )
+ self._enabled = False
+ return
+
+ try:
+ if self._protocol == "http":
+ self._validate_http_endpoint(self._endpoint)
+ self._http_session = requests.Session()
+ else:
+ if grpc is None or MetricsServiceStub is None:
+ raise ImportError("gRPC OTLP metrics dependencies are not available")
+ if self._insecure:
+ self._grpc_channel = grpc.insecure_channel(self._endpoint)
+ else:
+ self._grpc_channel = grpc.secure_channel(
+ self._endpoint,
+ grpc.ssl_channel_credentials(),
+ )
+ self._grpc_stub = MetricsServiceStub(self._grpc_channel)
+
+ logger.info(
+ "[OTelMetricExporter] initialized: phase=init_transport protocol=%s endpoint=%s service_name=%s export_interval_ms=%s export_timeout_seconds=%s",
+ self._protocol,
+ self._endpoint,
+ self._service_name,
+ self._export_interval_ms,
+ self._export_timeout_seconds,
+ )
+ except Exception as exc:
+ logger.warning(
+ "[OTelMetricExporter] initialization failed: phase=init_transport protocol=%s endpoint=%s service_name=%s export_timeout_seconds=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ self._service_name,
+ self._export_timeout_seconds,
+ exc,
+ )
+ self._enabled = False
+
+ def _protobuf_available(self) -> bool:
+ """Return whether all required OTLP protobuf symbols are available."""
+ return all(
+ symbol is not None
+ for symbol in (
+ ExportMetricsServiceRequest,
+ AnyValue,
+ InstrumentationScope,
+ KeyValue,
+ Metric,
+ NumberDataPoint,
+ HistogramDataPoint,
+ Gauge,
+ Sum,
+ Histogram,
+ Resource,
+ AGGREGATION_TEMPORALITY_CUMULATIVE,
+ )
+ )
+
+ def _validate_http_endpoint(self, endpoint: str) -> None:
+ """
+ Validate that an OTLP/HTTP endpoint includes a URL scheme and path.
+
+ Args:
+ endpoint: OTLP/HTTP metrics endpoint.
+ """
+ if not endpoint.startswith("http://") and not endpoint.startswith("https://"):
+ raise ValueError(
+ "OTLP/HTTP endpoint must include scheme, e.g. 'http://localhost:4318/v1/metrics'"
+ )
+
+ async def export(self) -> str:
+ """
+ Refresh collectors and export the current MetricRegistry snapshot.
+
+ Returns:
+ An empty string to preserve the MetricExporter interface contract.
+ """
+ if not self._enabled:
+ return ""
+
+ await self._refresh_collectors()
+ request = self._build_export_request()
+ if request is None:
+ return ""
+
+ await self._send_request(request)
+ return ""
+
+ async def _refresh_collectors(self) -> None:
+ """
+ Best-effort refresh of snapshot-driven collectors before serialization.
+
+ Export must never break application flow, so refresh failures are swallowed.
+ """
+ if self._collector_manager is None:
+ return
+ started = time.monotonic()
+ try:
+ await asyncio.wait_for(
+ self._collector_manager.refresh_all(
+ self._registry,
+ deadline_seconds=self._refresh_deadline_seconds,
+ ),
+ timeout=self._refresh_deadline_seconds,
+ )
+ except asyncio.TimeoutError as exc:
+ elapsed_ms = int((time.monotonic() - started) * 1000)
+ logger.warning(
+ "[OTelMetricExporter] collector refresh failed: phase=refresh_collectors reason=timeout protocol=%s endpoint=%s deadline_seconds=%s elapsed_ms=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ self._refresh_deadline_seconds,
+ elapsed_ms,
+ exc,
+ )
+ except Exception as exc:
+ elapsed_ms = int((time.monotonic() - started) * 1000)
+ logger.warning(
+ "[OTelMetricExporter] collector refresh failed: phase=refresh_collectors reason=exception protocol=%s endpoint=%s deadline_seconds=%s elapsed_ms=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ self._refresh_deadline_seconds,
+ elapsed_ms,
+ exc,
+ )
+
+ def _build_export_request(self) -> Optional[ExportMetricsServiceRequest]:
+ """
+ Build an OTLP ExportMetricsServiceRequest from the current registry snapshot.
+
+ Returns:
+ A populated OTLP export request, or None when there is no metric family to send.
+ """
+ if not self._protobuf_available():
+ return None
+
+ now_ns = time.time_ns()
+ request = ExportMetricsServiceRequest()
+ resource_metrics = request.resource_metrics.add()
+ resource_metrics.resource.CopyFrom(self._build_resource())
+ scope_metrics = resource_metrics.scope_metrics.add()
+ scope_metrics.scope.CopyFrom(
+ InstrumentationScope(name="openviking.metrics.exporter.otel", version="1")
+ )
+
+ self._append_counter_metrics(scope_metrics.metrics, now_ns)
+ self._append_gauge_metrics(scope_metrics.metrics, now_ns)
+ self._append_histogram_metrics(scope_metrics.metrics, now_ns)
+ self._append_dropped_series_metrics(scope_metrics.metrics, now_ns)
+
+ if not scope_metrics.metrics:
+ return None
+ return request
+
+ def _build_resource(self) -> Resource:
+ """
+ Build the OTLP resource envelope for this exporter.
+
+ Returns:
+ A Resource message containing the service name.
+ """
+ resource = Resource()
+ resource.attributes.append(
+ KeyValue(key="service.name", value=AnyValue(string_value=self._service_name))
+ )
+ return resource
+
+ def _append_counter_metrics(self, metrics, now_ns: int) -> None:
+ """
+ Append all counter families from MetricRegistry as cumulative OTLP sums.
+
+ Empty unlabeled counters emit a single zero data point to match PrometheusExporter.
+ """
+ for name, counter_series in self._registry.iter_counters():
+ metric = metrics.add()
+ metric.name = name
+ metric.description = "OpenViking metric."
+ metric.sum.CopyFrom(self._build_sum(counter_series, now_ns, name=name))
+
+ def _append_gauge_metrics(self, metrics, now_ns: int) -> None:
+ """
+ Append all gauge families from MetricRegistry as OTLP gauges.
+
+ Empty unlabeled gauges emit a single zero data point to match PrometheusExporter.
+ """
+ for name, gauge_series in self._registry.iter_gauges():
+ metric = metrics.add()
+ metric.name = name
+ metric.description = "OpenViking metric."
+ metric.gauge.CopyFrom(self._build_gauge(gauge_series, now_ns, name=name))
+
+ def _append_histogram_metrics(self, metrics, now_ns: int) -> None:
+ """
+ Append all histogram families from MetricRegistry as cumulative OTLP histograms.
+
+ Bucket counts and explicit bounds are copied directly from the registry snapshot.
+ """
+ for name, label_names, bucket_bounds, series_iter in self._registry.iter_histograms():
+ series_list = list(series_iter)
+ metric = metrics.add()
+ metric.name = name
+ metric.description = "OpenViking metric."
+ metric.histogram.CopyFrom(
+ self._build_histogram(
+ label_names=label_names,
+ bucket_bounds=bucket_bounds,
+ series_list=series_list,
+ now_ns=now_ns,
+ )
+ )
+
+ def _append_dropped_series_metrics(self, metrics, now_ns: int) -> None:
+ """
+ Append dropped-series diagnostics as a cumulative OTLP sum family.
+
+ This mirrors the `openviking_metrics_dropped_series_total` metric emitted by
+ PrometheusExporter.
+ """
+ dropped_points: list[tuple[tuple[tuple[str, str], ...], float]] = []
+ for metric_name, dropped in self._registry.iter_dropped_series():
+ dropped_points.append(((("metric", metric_name),), float(dropped)))
+
+ if not dropped_points:
+ return
+
+ metric = metrics.add()
+ metric.name = "openviking_metrics_dropped_series_total"
+ metric.description = "OpenViking metric."
+ metric.sum.CopyFrom(self._build_sum(dropped_points, now_ns, name=metric.name))
+
+ def _build_sum(self, series, now_ns: int, *, name: str) -> Sum:
+ """
+ Build an OTLP Sum message from a counter-like registry family.
+
+ Args:
+ series: List of `(labels, value)` tuples from MetricRegistry.
+ now_ns: Export timestamp in nanoseconds.
+ name: Metric family name for zero-series handling.
+ """
+ sum_metric = Sum(
+ aggregation_temporality=AGGREGATION_TEMPORALITY_CUMULATIVE,
+ is_monotonic=True,
+ )
+
+ if not series and not self._registry.counter_label_names(name):
+ point = NumberDataPoint(
+ start_time_unix_nano=self._start_time_ns,
+ time_unix_nano=now_ns,
+ as_int=0,
+ )
+ sum_metric.data_points.append(point)
+ return sum_metric
+
+ for labels, value in series:
+ point = NumberDataPoint(
+ start_time_unix_nano=self._start_time_ns,
+ time_unix_nano=now_ns,
+ )
+ point.attributes.extend(self._build_attributes(labels))
+ if float(value).is_integer():
+ point.as_int = int(value)
+ else:
+ point.as_double = float(value)
+ sum_metric.data_points.append(point)
+ return sum_metric
+
+ def _build_gauge(self, series, now_ns: int, *, name: str) -> Gauge:
+ """
+ Build an OTLP Gauge message from a registry gauge family.
+
+ Args:
+ series: List of `(labels, value)` tuples from MetricRegistry.
+ now_ns: Export timestamp in nanoseconds.
+ name: Metric family name for zero-series handling.
+ """
+ gauge_metric = Gauge()
+
+ if not series and not self._registry.gauge_label_names(name):
+ point = NumberDataPoint(time_unix_nano=now_ns, as_int=0)
+ gauge_metric.data_points.append(point)
+ return gauge_metric
+
+ for labels, value in series:
+ point = NumberDataPoint(time_unix_nano=now_ns)
+ point.attributes.extend(self._build_attributes(labels))
+ if float(value).is_integer():
+ point.as_int = int(value)
+ else:
+ point.as_double = float(value)
+ gauge_metric.data_points.append(point)
+ return gauge_metric
+
+ def _build_histogram(
+ self,
+ *,
+ label_names: tuple[str, ...],
+ bucket_bounds: tuple[float, ...],
+ series_list,
+ now_ns: int,
+ ) -> Histogram:
+ """
+ Build an OTLP Histogram message from a registry histogram family.
+
+ Args:
+ name: Histogram family name.
+ label_names: Registered label keys for the family.
+ bucket_bounds: Explicit bucket upper bounds from the registry.
+ series_list: Histogram series snapshot from the registry.
+ now_ns: Export timestamp in nanoseconds.
+ """
+ histogram_metric = Histogram(
+ aggregation_temporality=AGGREGATION_TEMPORALITY_CUMULATIVE,
+ )
+
+ if not series_list and not label_names:
+ point = HistogramDataPoint(
+ start_time_unix_nano=self._start_time_ns,
+ time_unix_nano=now_ns,
+ count=0,
+ sum=0.0,
+ )
+ point.bucket_counts.extend((0,) * (len(bucket_bounds) + 1))
+ point.explicit_bounds.extend(bucket_bounds)
+ histogram_metric.data_points.append(point)
+ return histogram_metric
+
+ for labels, bucket_counts, count, value_sum in series_list:
+ point = HistogramDataPoint(
+ start_time_unix_nano=self._start_time_ns,
+ time_unix_nano=now_ns,
+ count=int(count),
+ sum=float(value_sum),
+ )
+ point.attributes.extend(self._build_attributes(labels))
+ point.bucket_counts.extend(int(v) for v in bucket_counts)
+ point.explicit_bounds.extend(float(v) for v in bucket_bounds)
+ histogram_metric.data_points.append(point)
+ return histogram_metric
+
+ def _build_attributes(self, labels: Iterable[Tuple[str, str]]) -> list[KeyValue]:
+ """
+ Convert MetricRegistry labels into OTLP KeyValue attributes.
+
+ Args:
+ labels: Iterable of sorted `(key, value)` label pairs.
+ """
+ return [KeyValue(key=key, value=AnyValue(string_value=value)) for key, value in labels]
+
+ async def _send_request(self, request: ExportMetricsServiceRequest) -> None:
+ """
+ Send one OTLP metrics export request using the configured transport.
+
+ Args:
+ request: The populated OTLP metrics export payload.
+ """
+ metric_count = self._count_otlp_metrics(request)
+ request_bytes = request.ByteSize()
+ try:
+ if self._protocol == "http":
+ await asyncio.to_thread(self._send_http_request, request)
+ else:
+ await asyncio.to_thread(self._send_grpc_request, request)
+ except (RequestException, OSError, TimeoutError) as exc:
+ logger.warning(
+ "[OTelMetricExporter] export failed: phase=send_request protocol=%s endpoint=%s timeout_seconds=%s metric_count=%s request_bytes=%s error_type=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ self._export_timeout_seconds,
+ metric_count,
+ request_bytes,
+ type(exc).__name__,
+ exc,
+ )
+ except Exception as exc:
+ # Keep exporter failure best-effort. Log unexpected failures at warning level.
+ logger.warning(
+ "[OTelMetricExporter] export failed: phase=send_request protocol=%s endpoint=%s timeout_seconds=%s metric_count=%s request_bytes=%s error_type=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ self._export_timeout_seconds,
+ metric_count,
+ request_bytes,
+ type(exc).__name__,
+ exc,
+ )
+
+ def _count_otlp_metrics(self, request: ExportMetricsServiceRequest) -> int:
+ """
+ Count OTLP metric families in one export request for logging purposes.
+
+ Args:
+ request: The populated OTLP metrics export payload.
+
+ Returns:
+ Number of OTLP Metric messages in the request.
+ """
+ count = 0
+ for resource_metrics in request.resource_metrics:
+ for scope_metrics in resource_metrics.scope_metrics:
+ count += len(scope_metrics.metrics)
+ return count
+
+ def _send_http_request(self, request: ExportMetricsServiceRequest) -> None:
+ """
+ Send one OTLP/HTTP metrics request.
+
+ Args:
+ request: The populated OTLP metrics export payload.
+ """
+ if self._http_session is None:
+ return
+ payload = request.SerializeToString()
+ self._http_session.post(
+ self._endpoint,
+ data=payload,
+ headers={"Content-Type": "application/x-protobuf"},
+ timeout=self._export_timeout_seconds,
+ ).raise_for_status()
+
+ def _send_grpc_request(self, request: ExportMetricsServiceRequest) -> None:
+ """
+ Send one OTLP/gRPC metrics request.
+
+ Args:
+ request: The populated OTLP metrics export payload.
+ """
+ if self._grpc_stub is None:
+ return
+ self._grpc_stub.Export(request, timeout=self._export_timeout_seconds)
+
+ def start(self) -> None:
+ """
+ Start the exporter-owned periodic background loop.
+
+ The exporter keeps its own lifecycle so application startup can create the
+ exporter once and let it handle periodic exports internally.
+ """
+ if not self._enabled:
+ return
+ if self._export_task is not None and not self._export_task.done():
+ return
+ try:
+ self._export_task = asyncio.create_task(
+ self._run_export_loop(),
+ name="openviking-otel-metrics-export-loop",
+ )
+ except RuntimeError:
+ logger.warning(
+ "[OTelMetricExporter] periodic export loop not started: phase=start protocol=%s endpoint=%s reason=no running event loop",
+ self._protocol,
+ self._endpoint,
+ )
+ self._export_task = None
+
+ async def _run_export_loop(self) -> None:
+ """
+ Run the periodic export loop until shutdown is requested.
+ """
+ interval_s = float(self._export_interval_ms) / 1000.0
+ next_tick = time.monotonic() + interval_s
+ while self._enabled:
+ try:
+ await self.export()
+ except asyncio.CancelledError:
+ raise
+ except Exception as exc:
+ logger.warning(
+ "[OTelMetricExporter] periodic export loop iteration failed: phase=run_export_loop protocol=%s endpoint=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ exc,
+ )
+ now = time.monotonic()
+ next_tick, sleep_s = _compute_next_tick_and_sleep(
+ now=now,
+ interval_s=interval_s,
+ next_tick=next_tick,
+ )
+ await asyncio.sleep(sleep_s)
+
+ async def shutdown(self) -> None:
+ """
+ Stop the exporter loop and release transport resources.
+ """
+ self._enabled = False
+
+ if self._export_task is not None:
+ self._export_task.cancel()
+ try:
+ await self._export_task
+ except asyncio.CancelledError:
+ logger.info(
+ "[OTelMetricExporter] periodic export loop cancelled: phase=shutdown protocol=%s endpoint=%s",
+ self._protocol,
+ self._endpoint,
+ )
+ except Exception as exc:
+ logger.warning(
+ "[OTelMetricExporter] periodic export loop shutdown failed: phase=shutdown protocol=%s endpoint=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ exc,
+ )
+ self._export_task = None
+
+ if self._http_session is not None:
+ try:
+ self._http_session.close()
+ except Exception as exc:
+ logger.warning(
+ "[OTelMetricExporter] http session close failed: phase=shutdown protocol=%s endpoint=%s error_type=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ type(exc).__name__,
+ exc,
+ )
+ self._http_session = None
+
+ if self._grpc_channel is not None:
+ try:
+ # grpc.Channel.close() is synchronous and returns None.
+ self._grpc_channel.close()
+ except Exception as exc:
+ logger.warning(
+ "[OTelMetricExporter] grpc channel close failed: phase=shutdown protocol=%s endpoint=%s error_type=%s error=%s",
+ self._protocol,
+ self._endpoint,
+ type(exc).__name__,
+ exc,
+ )
+ self._grpc_channel = None
+ self._grpc_stub = None
diff --git a/openviking/metrics/global_api.py b/openviking/metrics/global_api.py
index d65ff03ff..1e35b98e3 100644
--- a/openviking/metrics/global_api.py
+++ b/openviking/metrics/global_api.py
@@ -17,6 +17,9 @@ only layer allowed to write into MetricRegistry, keeping the architecture consis
from __future__ import annotations
+import asyncio
+import inspect
+import logging
import threading
from typing import Optional
@@ -41,14 +44,51 @@ from .collectors.telemetry_bridge import TelemetryBridgeCollector
from .collectors.vlm import VLMCollector
from .core.registry import MetricRegistry
from .core.runtime import EventCollectorRouter
+from .exporters.otel import OTelMetricExporter
from .exporters.prometheus import PrometheusExporter
+logger = logging.getLogger(__name__)
+
_lock = threading.Lock()
_registry: Optional[MetricRegistry] = None
-_exporter: Optional[PrometheusExporter] = None
+_exporters: list = []
_event_router: EventCollectorRouter | None = None
+def _shutdown_exporters_best_effort(exporters: list) -> None:
+ """Shutdown exporters without raising.
+
+ This is used both for explicit shutdown and for re-initialization, to avoid leaking
+ background tasks (e.g. OTel periodic export loops) across multiple init cycles.
+
+ Note: This is a synchronous function. For proper async shutdown in async contexts,
+ use `shutdown_metrics_async()` instead, which properly awaits async shutdown methods.
+ """
+ for exporter in exporters:
+ if hasattr(exporter, "shutdown"):
+ try:
+ result = exporter.shutdown()
+ if inspect.isawaitable(result):
+ try:
+ loop = asyncio.get_running_loop()
+ except RuntimeError:
+ # No running event loop - use asyncio.run() to execute the coroutine
+ asyncio.run(result)
+ else:
+ # There is a running event loop. We cannot block-wait from within
+ # the event loop thread (would cause deadlock). Instead, schedule
+ # the shutdown as a task and rely on the async shutdown path
+ # (shutdown_metrics_async) for proper cleanup in async contexts.
+ #
+ # This is best-effort for synchronous re-initialization paths.
+ # The proper async shutdown path is used in app.py's lifespan.
+ loop.create_task(result)
+ except Exception as e:
+ logger.warning(
+ "[_shutdown_exporters_best_effort] failed to shutdown exporter: %s", e
+ )
+
+
def configure_metric_account_dimension(
*,
enabled: bool | None = None,
@@ -76,29 +116,34 @@ def init_metrics_from_server_config(
config: ServerConfig, *, app=None, service=None, registry: MetricRegistry | None = None
) -> None:
"""
- Initialize the process-global metrics registry, router, and exporter from server config.
+ Initialize the process-global metrics registry, router, and exporters from server config.
When metrics are enabled:
- Creates a MetricRegistry (or uses provided one)
- Registers default collectors (State/Probe/DomainStats)
- Builds the in-process event router for Event-style metrics
- - Creates PrometheusExporter and stores it in app.state.metrics_exporter
+ - Creates configured exporters and stores them in app.state.metrics_exporters
When metrics are disabled:
- Clears all global metrics state
- Resets account-dimension runtime to its disabled defaults
- - Detaches any exporter reference previously stored on `app.state`
+ - Detaches any exporter references previously stored on `app.state`
"""
enabled = config.observability.metrics.enabled
- global _registry, _exporter, _event_router
+ global _registry, _exporters, _event_router
with _lock:
if not enabled:
+ _shutdown_exporters_best_effort(_exporters)
_registry = None
- _exporter = None
+ _exporters = []
_event_router = None
reset_metric_account_dimension()
if app is not None:
- app.state.metrics_exporter = None
+ app.state.metrics_exporters = []
return
+
+ # If re-initializing while already enabled, clean up existing exporters first.
+ if _exporters:
+ _shutdown_exporters_best_effort(_exporters)
account_dimension = config.observability.metrics.account_dimension
configure_metric_account_dimension(
enabled=account_dimension.enabled,
@@ -108,26 +153,75 @@ def init_metrics_from_server_config(
_registry = registry or MetricRegistry()
collector_manager = create_default_collector_manager(app=app, service=service)
_event_router = _build_event_router(_registry)
- _exporter = PrometheusExporter(registry=_registry, collector_manager=collector_manager)
+
+ _exporters = []
+ exporters_config = config.observability.metrics.exporters
+
+ # Initialize Prometheus exporter if enabled
+ if exporters_config.prometheus.enabled:
+ prometheus_exporter = PrometheusExporter(
+ registry=_registry, collector_manager=collector_manager
+ )
+ _exporters.append(prometheus_exporter)
+
+ # Initialize OTel exporter if enabled
+ if exporters_config.otel.enabled:
+ otel_exporter = OTelMetricExporter(
+ registry=_registry,
+ collector_manager=collector_manager,
+ protocol=exporters_config.otel.protocol,
+ insecure=exporters_config.otel.tls.insecure,
+ endpoint=exporters_config.otel.endpoint,
+ service_name=exporters_config.otel.service_name,
+ export_interval_ms=exporters_config.otel.export_interval_ms,
+ enabled=True,
+ )
+ otel_exporter.start()
+ _exporters.append(otel_exporter)
+
if app is not None:
- app.state.metrics_exporter = _exporter
+ app.state.metrics_exporters = _exporters
def shutdown_metrics(*, app=None) -> None:
"""
- Clear all process-global metrics objects and detach the exporter from the application.
+ Clear all process-global metrics objects and detach the exporters from the application.
Shutdown is intentionally idempotent so tests and service teardown paths can call it
repeatedly without needing to coordinate current metrics initialization state.
"""
- global _registry, _exporter, _event_router
+ global _registry, _exporters, _event_router
+ with _lock:
+ _shutdown_exporters_best_effort(_exporters)
+
+ _registry = None
+ _exporters = []
+ _event_router = None
+ reset_metric_account_dimension()
+ if app is not None:
+ app.state.metrics_exporters = []
+
+
+async def shutdown_metrics_async(*, app=None) -> None:
+ """Async variant of `shutdown_metrics` that awaits async exporter cleanup."""
+ global _registry, _exporters, _event_router
with _lock:
+ exporters = list(_exporters)
_registry = None
- _exporter = None
+ _exporters = []
_event_router = None
reset_metric_account_dimension()
if app is not None:
- app.state.metrics_exporter = None
+ app.state.metrics_exporters = []
+
+ for exporter in exporters:
+ if hasattr(exporter, "shutdown"):
+ try:
+ result = exporter.shutdown()
+ if inspect.isawaitable(result):
+ await result
+ except Exception as e:
+ logger.warning("[shutdown_metrics_async] failed to shutdown exporter: %s", e)
def get_metrics_registry() -> MetricRegistry:
diff --git a/openviking/metrics/http_middleware.py b/openviking/metrics/http_middleware.py
deleted file mode 100644
index ee2f9b18a..000000000
--- a/openviking/metrics/http_middleware.py
+++ /dev/null
@@ -1,214 +0,0 @@
-# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
-# SPDX-License-Identifier: AGPL-3.0
-
-"""
-HTTP metrics middleware.
-
-This middleware records request-level metrics in a best-effort manner:
-- Inflight gauge per route template (to avoid high cardinality per raw path).
-- Request count and duration histogram by method/route/status.
-
-Implementation notes:
-- Uses HttpRequestLifecycleDataSource, which emits events; collectors write into MetricRegistry.
-- All metrics calls are wrapped in try/except to ensure observability never breaks request handling.
-- Route label uses the Starlette route template (e.g., "/sessions/{session_id}") when available.
-"""
-
-from __future__ import annotations
-
-import logging
-import threading
-import time
-from typing import Callable
-
-from starlette.requests import Request
-from starlette.responses import Response
-
-from openviking.metrics.account_context import (
- bind_metric_account_context,
- reset_metric_account_context,
-)
-from openviking.metrics.datasources import HttpRequestLifecycleDataSource
-
-logger = logging.getLogger(__name__)
-
-_HTTP_IGNORE_ROUTES = frozenset(
- {
- "/metrics",
- "/health",
- "/ready",
- }
-)
-
-
-def create_http_metrics_middleware() -> Callable[[Request, Callable], Response]:
- """
- Create a Starlette-compatible middleware that records HTTP request metrics.
-
- The middleware emits events through `HttpRequestLifecycleDataSource`:
- - `http.inflight` updates on request start/end
- - `http.request` on request completion with status and duration
-
- Routes listed in `_HTTP_IGNORE_ROUTES` are intentionally skipped so internal endpoints such
- as metrics or health probes do not pollute business-facing HTTP timeseries.
- """
-
- async def middleware(request: Request, call_next: Callable) -> Response:
- """
- Wrap one request execution and emit best-effort inflight and request metrics events.
-
- The middleware never lets metrics failures affect request handling and will skip the
- entire HTTP metrics flow for routes that are explicitly ignored.
- """
- # Ignore by raw path first so internal endpoints are skipped even when no route template
- # is bound (e.g. middleware executed outside Starlette routing).
- raw_path = str(request.url.path)
- if raw_path in _HTTP_IGNORE_ROUTES:
- return await call_next(request)
-
- initial_route = _get_route_template(request)
- if initial_route in _HTTP_IGNORE_ROUTES:
- return await call_next(request)
- provisional_account_id = _extract_request_account_id(request)
- account_token = bind_metric_account_context(account_id=provisional_account_id)
- try:
- HttpRequestLifecycleDataSource.set_inflight(
- route=initial_route,
- value=_inflight_delta(initial_route, provisional_account_id, +1),
- account_id=provisional_account_id,
- )
- except Exception:
- _log_metrics_failure("http.inflight increment failed", route=initial_route)
-
- start = time.perf_counter()
- status_code = 500
- try:
- response = await call_next(request)
- status_code = int(getattr(response, "status_code", 500))
- return response
- finally:
- elapsed = time.perf_counter() - start
- final_route = _get_route_template(request)
- final_account_id = _extract_request_account_id(request)
- if (final_route, final_account_id) != (initial_route, provisional_account_id):
- try:
- HttpRequestLifecycleDataSource.set_inflight(
- route=initial_route,
- value=_inflight_delta(initial_route, provisional_account_id, -1),
- account_id=provisional_account_id,
- )
- HttpRequestLifecycleDataSource.set_inflight(
- route=final_route,
- value=_inflight_delta(final_route, final_account_id, +1),
- account_id=final_account_id,
- )
- except Exception:
- _log_metrics_failure(
- "http.inflight rebalance failed",
- route=final_route,
- account_id=final_account_id,
- )
- try:
- HttpRequestLifecycleDataSource.record_request(
- method=request.method,
- route=final_route,
- status=str(status_code),
- duration_seconds=elapsed,
- account_id=final_account_id,
- )
- except Exception:
- _log_metrics_failure(
- "http.request recording failed",
- route=final_route,
- account_id=final_account_id,
- )
- try:
- HttpRequestLifecycleDataSource.set_inflight(
- route=final_route,
- value=_inflight_delta(final_route, final_account_id, -1),
- account_id=final_account_id,
- )
- except Exception:
- _log_metrics_failure(
- "http.inflight decrement failed",
- route=final_route,
- account_id=final_account_id,
- )
- reset_metric_account_context(account_token)
-
- return middleware
-
-
-_INFLIGHT: dict[tuple[str, str | None], int] = {}
-_INFLIGHT_LOCK = threading.Lock()
-
-
-def _inflight_delta(route: str, account_id: str | None, delta: int) -> int:
- """
- Update and return the current in-process inflight count for a route template.
-
- This local counter is only used to provide a reasonable gauge value for event emission.
- It is not a correctness mechanism and is clamped to never go below zero.
- """
- key = (route, account_id)
- with _INFLIGHT_LOCK:
- v = _INFLIGHT.get(key, 0) + delta
- if v < 0:
- v = 0
- if v == 0:
- _INFLIGHT.pop(key, None)
- else:
- _INFLIGHT[key] = v
- return v
-
-
-def _get_route_template(request: Request) -> str:
- """
- Resolve a low-cardinality route identifier for labeling request metrics.
-
- Prefer the Starlette route template (e.g. `/sessions/{session_id}`) when available, and
- fall back to a fixed sentinel when the request is not bound to a recognized route template.
- """
- route = request.scope.get("route")
- path = getattr(route, "path", None)
- if path:
- return str(path)
- # Never label by raw path here: it may contain IDs and explode cardinality.
- return "/__unmatched__"
-
-
-def _extract_request_account_id(request: Request) -> str | None:
- """
- Return the best request-scoped account id currently available for metrics labeling.
-
- Only trust the authenticated request state (`request.state.metric_account_id`).
-
- The `/metrics` endpoint is intentionally best-effort and must not let unauthenticated or
- rejected traffic control tenant labels via raw headers. When the authenticated account id
- is not available yet, return `None` so collector-side policy can map it to `__unknown__`.
- """
- state_account_id = getattr(request.state, "metric_account_id", None)
- if state_account_id:
- return str(state_account_id)
- return None
-
-
-def _log_metrics_failure(message: str, *, route: str, account_id: str | None = None) -> None:
- """
- Emit a debug log for best-effort metrics failures without affecting request handling.
-
- This middleware intentionally treats observability as a side channel. The helper keeps
- failure reporting centralized so all swallowed exceptions still leave a trace when debug
- logging is enabled.
- """
- if not logger.isEnabledFor(logging.DEBUG):
- return
- logger.debug(
- "http metrics write failed: %s",
- message,
- extra={
- "route": route,
- "account_id": account_id,
- },
- exc_info=True,
- )
diff --git a/openviking/models/embedder/base.py b/openviking/models/embedder/base.py
index 91352cd65..c43a7b1a4 100644
--- a/openviking/models/embedder/base.py
+++ b/openviking/models/embedder/base.py
@@ -328,8 +328,10 @@ class EmbedderBase(ABC):
completion_tokens=completion_tokens,
)
try:
- from openviking.metrics.account_context import get_metric_account_context
from openviking.metrics.datasources import EmbeddingEventDataSource
+ from openviking.observability.context import get_root_observability_context
+
+ root_context = get_root_observability_context()
EmbeddingEventDataSource.record_call(
provider=str(provider),
@@ -337,7 +339,7 @@ class EmbedderBase(ABC):
duration_seconds=self._resolve_metrics_duration_seconds(duration_seconds),
prompt_tokens=int(prompt_tokens),
completion_tokens=int(completion_tokens),
- account_id=get_metric_account_context().http_account_id,
+ account_id=root_context.account_id if root_context is not None else None,
)
except Exception as e:
# Metrics must never break embedding execution.
diff --git a/openviking/models/rerank/base.py b/openviking/models/rerank/base.py
index 61942746c..dcc25acdb 100644
--- a/openviking/models/rerank/base.py
+++ b/openviking/models/rerank/base.py
@@ -92,8 +92,10 @@ class RerankBase:
e,
)
try:
- from openviking.metrics.account_context import get_metric_account_context
from openviking.metrics.datasources import RerankEventDataSource
+ from openviking.observability.context import get_root_observability_context
+
+ root_context = get_root_observability_context()
RerankEventDataSource.record_call(
provider=str(provider),
@@ -101,7 +103,7 @@ class RerankBase:
duration_seconds=max(float(duration_seconds), 0.0),
prompt_tokens=int(prompt_tokens),
completion_tokens=int(completion_tokens),
- account_id=get_metric_account_context().http_account_id,
+ account_id=root_context.account_id if root_context is not None else None,
)
except Exception as e:
# Metrics must never break rerank execution.
diff --git a/openviking/models/vlm/base.py b/openviking/models/vlm/base.py
index dc9017dfa..3cfbee579 100644
--- a/openviking/models/vlm/base.py
+++ b/openviking/models/vlm/base.py
@@ -219,8 +219,10 @@ class VLMBase(ABC):
# Record the VLM call in Prometheus metrics (if enabled).
try:
- from openviking.metrics.account_context import get_metric_account_context
from openviking.metrics.datasources import VLMEventDataSource
+ from openviking.observability.context import get_root_observability_context
+
+ root_context = get_root_observability_context()
VLMEventDataSource.record_call(
provider=str(provider),
@@ -228,7 +230,7 @@ class VLMBase(ABC):
duration_seconds=float(duration_seconds),
prompt_tokens=int(prompt_tokens),
completion_tokens=int(completion_tokens),
- account_id=get_metric_account_context().http_account_id,
+ account_id=root_context.account_id if root_context is not None else None,
)
except Exception as e:
# Metrics must never break model inference.
diff --git a/openviking/observability/__init__.py b/openviking/observability/__init__.py
new file mode 100644
index 000000000..70854cd14
--- /dev/null
+++ b/openviking/observability/__init__.py
@@ -0,0 +1,19 @@
+"""Unified observability context helpers."""
+
+from .context import (
+ bind_operation_observability_context,
+ bind_root_observability_context,
+ get_operation_observability_context,
+ get_root_observability_context,
+ reset_operation_observability_context,
+ reset_root_observability_context,
+)
+
+__all__ = [
+ "bind_root_observability_context",
+ "get_root_observability_context",
+ "reset_root_observability_context",
+ "bind_operation_observability_context",
+ "get_operation_observability_context",
+ "reset_operation_observability_context",
+]
diff --git a/openviking/observability/context.py b/openviking/observability/context.py
new file mode 100644
index 000000000..5ae46e87a
--- /dev/null
+++ b/openviking/observability/context.py
@@ -0,0 +1,345 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""
+Unified observability context management.
+
+This module provides a unified context management system for observability,
+integrating:
+- Root context (HTTP request-level)
+- Operation context (business operation-level)
+- Execution context (non-request scenarios)
+
+Key features:
+- Single contextvar for all observability context
+- Unified binding, retrieval, and reset interfaces
+- Type-safe data structures
+- Automatic OTel trace/span ID integration
+"""
+
+from __future__ import annotations
+
+import contextvars
+import logging
+import uuid
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from typing import Any, Dict, Optional
+
+from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributes
+
+# Try to import opentelemetry
+try:
+ from opentelemetry import trace as otel_trace
+ from opentelemetry.trace import format_span_id, format_trace_id
+except ImportError:
+ otel_trace = None
+ format_span_id = None
+ format_trace_id = None
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class ObservabilityContext:
+ """
+ Unified observability context data structure.
+
+ This class integrates all observability-related context information:
+ - Root context (HTTP request-level)
+ - Operation context (business operation-level)
+ - Execution context (non-request scenarios)
+
+ Attributes:
+ root: Root span attributes for HTTP requests.
+ operation: Operation span attributes for business operations.
+ execution_trace_id: Trace ID for non-request execution contexts.
+ execution_span_id: Span ID for non-request execution contexts.
+ extra: Additional custom attributes.
+ """
+
+ root: Optional[RootSpanAttributes] = None
+ operation: Optional[OperationSpanAttributes] = None
+ execution_trace_id: Optional[str] = None
+ execution_span_id: Optional[str] = None
+ extra: Dict[str, Any] = field(default_factory=dict)
+
+ def get_trace_id(self) -> str:
+ """
+ Get the effective trace ID.
+
+ Priority:
+ 1. OTel current span trace_id
+ 2. Execution context trace_id
+ 3. Empty string
+
+ Returns:
+ The effective trace ID, or empty string if not available.
+ """
+ # 1. Check OTel current span
+ if otel_trace is not None and format_trace_id is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and hasattr(current_span, "context"):
+ span_context = current_span.get_span_context()
+ if span_context.is_valid:
+ return format_trace_id(span_context.trace_id)
+ except Exception:
+ pass
+
+ # 2. Check execution context
+ if self.execution_trace_id:
+ return self.execution_trace_id
+
+ # 3. Return empty string
+ return ""
+
+ def get_span_id(self) -> str:
+ """
+ Get the effective span ID.
+
+ Priority:
+ 1. OTel current span span_id
+ 2. Execution context span_id
+ 3. Empty string
+
+ Returns:
+ The effective span ID, or empty string if not available.
+ """
+ # 1. Check OTel current span
+ if otel_trace is not None and format_span_id is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and hasattr(current_span, "context"):
+ span_context = current_span.get_span_context()
+ if span_context.is_valid:
+ return format_span_id(span_context.span_id)
+ except Exception:
+ pass
+
+ # 2. Check execution context
+ if self.execution_span_id:
+ return self.execution_span_id
+
+ # 3. Return empty string
+ return ""
+
+ def to_log_fields(self) -> Dict[str, Any]:
+ """
+ Convert the context to log fields.
+
+ Returns:
+ A dictionary of log fields including trace_id, span_id, and other context fields.
+ """
+ fields: Dict[str, Any] = {
+ "trace_id": self.get_trace_id(),
+ "span_id": self.get_span_id(),
+ }
+
+ # Add root context fields
+ if self.root is not None:
+ root_fields = self.root.to_log_fields()
+ fields.update(root_fields)
+
+ # Add operation context fields
+ if self.operation is not None:
+ operation_fields = self.operation.to_log_fields()
+ fields.update(operation_fields)
+
+ # Add extra fields
+ fields.update(self.extra)
+
+ return fields
+
+ def bind_root(self, root_attrs: RootSpanAttributes) -> "ObservabilityContext":
+ """
+ Bind root context.
+
+ Args:
+ root_attrs: The root span attributes to bind.
+
+ Returns:
+ Self for method chaining.
+ """
+ self.root = root_attrs
+ return self
+
+ def bind_operation(self, operation_attrs: OperationSpanAttributes) -> "ObservabilityContext":
+ """
+ Bind operation context.
+
+ Args:
+ operation_attrs: The operation span attributes to bind.
+
+ Returns:
+ Self for method chaining.
+ """
+ self.operation = operation_attrs
+ return self
+
+ def bind_execution(
+ self,
+ trace_id: Optional[str] = None,
+ span_id: Optional[str] = None,
+ ) -> "ObservabilityContext":
+ """
+ Bind execution context for non-request scenarios.
+
+ Args:
+ trace_id: Optional trace ID. Generated automatically if not provided.
+ span_id: Optional span ID. Derived from trace_id if not provided.
+
+ Returns:
+ Self for method chaining.
+ """
+ self.execution_trace_id = trace_id or uuid.uuid4().hex
+ self.execution_span_id = span_id or self.execution_trace_id[:16]
+ return self
+
+ def set_extra(self, key: str, value: Any) -> "ObservabilityContext":
+ """
+ Set an extra attribute.
+
+ Args:
+ key: The attribute key.
+ value: The attribute value.
+
+ Returns:
+ Self for method chaining.
+ """
+ self.extra[key] = value
+ return self
+
+
+# Global context variable
+_OBSERVABILITY_CONTEXT: contextvars.ContextVar[Optional[ObservabilityContext]] = (
+ contextvars.ContextVar(
+ "openviking_observability_context",
+ default=None,
+ )
+)
+
+
+def get_observability_context() -> ObservabilityContext:
+ """
+ Get the current observability context.
+
+ If no context is bound, a new empty context is created.
+
+ Returns:
+ The current observability context.
+ """
+ ctx = _OBSERVABILITY_CONTEXT.get()
+ if ctx is None:
+ ctx = ObservabilityContext()
+ _OBSERVABILITY_CONTEXT.set(ctx)
+ return ctx
+
+
+def set_observability_context(ctx: ObservabilityContext) -> contextvars.Token:
+ """
+ Set the observability context.
+
+ Args:
+ ctx: The observability context to set.
+
+ Returns:
+ A token that can be used to reset the context.
+ """
+ return _OBSERVABILITY_CONTEXT.set(ctx)
+
+
+def reset_observability_context(token: contextvars.Token) -> None:
+ """
+ Reset the observability context to a previous state.
+
+ Args:
+ token: The token returned by set_observability_context.
+ """
+ _OBSERVABILITY_CONTEXT.reset(token)
+
+
+def bind_root_context(root_attrs: RootSpanAttributes) -> contextvars.Token:
+ """
+ Bind root context for HTTP requests.
+
+ Args:
+ root_attrs: The root span attributes to bind.
+
+ Returns:
+ A token that can be used to reset the context.
+ """
+ ctx = get_observability_context()
+ ctx.bind_root(root_attrs)
+ return set_observability_context(ctx)
+
+
+def bind_operation_context(operation_attrs: OperationSpanAttributes) -> contextvars.Token:
+ """
+ Bind operation context for business operations.
+
+ Args:
+ operation_attrs: The operation span attributes to bind.
+
+ Returns:
+ A token that can be used to reset the context.
+ """
+ ctx = get_observability_context()
+ ctx.bind_operation(operation_attrs)
+ return set_observability_context(ctx)
+
+
+@contextmanager
+def bind_execution_context(
+ trace_id: Optional[str] = None,
+ span_id: Optional[str] = None,
+):
+ """
+ Context manager for binding execution context in non-request scenarios.
+
+ This is useful for:
+ - Process startup
+ - CLI execution
+ - Background tasks
+ - Any code that doesn't run under an HTTP request
+
+ Args:
+ trace_id: Optional trace ID. Generated automatically if not provided.
+ span_id: Optional span ID. Derived from trace_id if not provided.
+
+ Yields:
+ A tuple of (trace_id, span_id).
+ """
+ # Create new context with execution trace/span IDs
+ ctx = ObservabilityContext()
+ ctx.bind_execution(trace_id, span_id)
+
+ token = set_observability_context(ctx)
+
+ try:
+ yield ctx.execution_trace_id, ctx.execution_span_id
+ finally:
+ reset_observability_context(token)
+
+
+# Convenience aliases for unified context management
+bind_root_observability_context = bind_root_context
+reset_root_observability_context = reset_observability_context
+bind_operation_observability_context = bind_operation_context
+
+
+def get_root_observability_context() -> Optional[RootSpanAttributes]:
+ """Get the root context from the unified observability context."""
+ ctx = get_observability_context()
+ return ctx.root
+
+
+def get_operation_observability_context() -> Optional[OperationSpanAttributes]:
+ """Get the operation context from the unified observability context."""
+ ctx = get_observability_context()
+ return ctx.operation
+
+
+def reset_operation_observability_context(token: contextvars.Token) -> None:
+ """Reset the observability context."""
+ reset_observability_context(token)
diff --git a/openviking/observability/http_observability_middleware.py b/openviking/observability/http_observability_middleware.py
new file mode 100644
index 000000000..77c353428
--- /dev/null
+++ b/openviking/observability/http_observability_middleware.py
@@ -0,0 +1,869 @@
+"""Unified HTTP observability middleware.
+
+This module intentionally owns both HTTP metrics and HTTP tracing logic.
+
+Rationale:
+- A single middleware entry point avoids ordering issues between separate Starlette/FastAPI
+ middlewares (e.g. authentication mutating request-scoped fields that metrics need).
+- Route template resolution happens after routing, so both tracing and metrics need a shared
+ convention for when and how to finalize `http_route`.
+"""
+
+from __future__ import annotations
+
+import logging
+import threading
+import time
+import uuid
+from contextlib import nullcontext
+from dataclasses import dataclass
+from typing import Any, Awaitable, Callable, ContextManager, Optional
+
+from starlette.requests import Request
+from starlette.responses import Response
+
+from openviking.metrics.datasources import HttpRequestLifecycleDataSource
+from openviking.observability.context import (
+ bind_root_observability_context,
+ reset_root_observability_context,
+)
+from openviking.telemetry.span_models import RootSpanAttributes, create_root_span_attributes
+
+# Try to import opentelemetry - will be None if not installed
+try:
+ from opentelemetry import trace as otel_trace
+ from opentelemetry.propagate import extract
+ from opentelemetry.trace import Status, StatusCode
+except ImportError:
+ otel_trace = None
+ extract = None
+ Status = None
+ StatusCode = None
+
+logger = logging.getLogger(__name__)
+
+_HTTP_IGNORE_ROUTES = frozenset(
+ {
+ "/metrics",
+ "/health",
+ "/ready",
+ }
+)
+
+
+class ShardedInflightCounter:
+ """
+ Sharded inflight counter to reduce lock contention under high concurrency.
+
+ This class splits the key space into multiple shards, each with its own lock.
+ This reduces lock contention compared to a single global lock.
+
+ The number of shards should be a power of 2 for efficient hash-based sharding.
+ """
+
+ def __init__(self, num_shards: int = 16):
+ """
+ Initialize the sharded inflight counter.
+
+ Args:
+ num_shards: Number of shards to use. Should be a power of 2.
+ Default is 16, which provides good balance between
+ memory overhead and lock contention reduction.
+ """
+ self._num_shards = num_shards
+ self._shards: list[dict] = [{} for _ in range(num_shards)]
+ self._locks: list[threading.Lock] = [threading.Lock() for _ in range(num_shards)]
+
+ def _get_shard_index(self, key: tuple) -> int:
+ """
+ Calculate the shard index for a given key.
+
+ Uses the built-in hash function to distribute keys across shards.
+
+ Args:
+ key: The key to hash (route, account_id tuple).
+
+ Returns:
+ The shard index (0 to num_shards - 1).
+ """
+ return hash(key) % self._num_shards
+
+ def increment(self, route: str, account_id: Optional[str] = None) -> int:
+ """
+ Increment the inflight count for a route and account_id.
+
+ Args:
+ route: The route template.
+ account_id: The account identifier, or None.
+
+ Returns:
+ The new inflight count after increment.
+ """
+ key = (route, account_id)
+ shard_idx = self._get_shard_index(key)
+
+ with self._locks[shard_idx]:
+ shard = self._shards[shard_idx]
+ shard[key] = shard.get(key, 0) + 1
+ return shard[key]
+
+ def decrement(self, route: str, account_id: Optional[str] = None) -> int:
+ """
+ Decrement the inflight count for a route and account_id.
+
+ The count is clamped to never go below zero.
+
+ Args:
+ route: The route template.
+ account_id: The account identifier, or None.
+
+ Returns:
+ The new inflight count after decrement (clamped to >= 0).
+ """
+ key = (route, account_id)
+ shard_idx = self._get_shard_index(key)
+
+ with self._locks[shard_idx]:
+ shard = self._shards[shard_idx]
+ if key in shard:
+ shard[key] -= 1
+ if shard[key] <= 0:
+ del shard[key]
+ return 0
+ return shard[key]
+ return 0
+
+ def get(self, route: str, account_id: Optional[str] = None) -> int:
+ """
+ Get the current inflight count for a route and account_id.
+
+ This method uses lock-free reading for better performance under high concurrency.
+ In CPython, a single dict.get() call is atomic due to the GIL, so no lock is needed.
+
+ Args:
+ route: The route template.
+ account_id: The account identifier, or None.
+
+ Returns:
+ The current inflight count (0 if not found).
+ """
+ key = (route, account_id)
+ shard_idx = self._get_shard_index(key)
+
+ # Lock-free read: single dict.get() is atomic in CPython
+ # If the read fails for any reason, fall back to locked read
+ try:
+ return self._shards[shard_idx].get(key, 0)
+ except Exception:
+ # Fall back to locked read if lock-free read fails
+ with self._locks[shard_idx]:
+ return self._shards[shard_idx].get(key, 0)
+
+ def get_all(self) -> dict[tuple, int]:
+ """
+ Get all inflight counts across all shards.
+
+ This method first attempts lock-free reading for better performance.
+ If lock-free reading fails (e.g., dictionary modified during iteration),
+ it falls back to locked reading.
+
+ Note: Lock-free reading may return slightly stale or inconsistent data
+ under high concurrency, but this is acceptable for monitoring purposes.
+
+ Returns:
+ A dictionary of all (route, account_id) -> count mappings.
+ """
+ result: dict[tuple, int] = {}
+
+ # First attempt: lock-free reading
+ try:
+ for i in range(self._num_shards):
+ # dict.copy() is atomic in CPython for small dicts
+ # but may fail if the dict is being modified
+ shard_copy = dict(self._shards[i])
+ result.update(shard_copy)
+ return result
+ except Exception:
+ # Lock-free read failed, fall back to locked reading
+ pass
+
+ # Fallback: locked reading
+ for i in range(self._num_shards):
+ with self._locks[i]:
+ result.update(self._shards[i])
+ return result
+
+
+# Global sharded inflight counter instance
+# Using 16 shards provides good balance between memory and contention reduction
+_INFLIGHT_COUNTER = ShardedInflightCounter(num_shards=16)
+
+
+def maybe_start_root_span(
+ request: Request, root_attrs: RootSpanAttributes
+) -> Optional[ContextManager[Any]]:
+ """
+ Return a root span context manager when OTel is available.
+
+ Creates a server span for the incoming HTTP request, extracting trace context
+ from request headers if present.
+
+ Args:
+ request: The incoming HTTP request.
+ root_attrs: The root span attributes.
+
+ Returns:
+ A span context manager if OTel is available and span creation succeeds,
+ None otherwise.
+ """
+ if otel_trace is None or extract is None:
+ logger.debug("OTel trace or extract not available, skipping root span creation")
+ return None
+
+ try:
+ tracer = otel_trace.get_tracer(__name__)
+ if tracer is None:
+ logger.warning("OTel tracer is None, cannot create root span")
+ return None
+
+ carrier = dict(request.headers)
+ context = extract(carrier)
+ span_name = f"{request.method} {root_attrs.http_route}"
+
+ return tracer.start_as_current_span(
+ name=span_name,
+ context=context,
+ kind=otel_trace.SpanKind.SERVER,
+ )
+
+ except ImportError as e:
+ logger.warning("OTel import error when creating root span: %s", str(e))
+ return None
+
+ except (TypeError, ValueError, AttributeError) as e:
+ logger.warning("OTel configuration error when creating root span: %s", str(e))
+ return None
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error when creating root span: %s",
+ str(e),
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+ return None
+
+
+def maybe_apply_root_span_attributes(root_attrs: RootSpanAttributes) -> None:
+ """
+ Best-effort sync of root attributes into the current OTel span.
+
+ Sets the root span attributes (request_id, http_method, http_route, account_id, etc.)
+ on the currently active OTel span.
+
+ Args:
+ root_attrs: The root span attributes to apply.
+ """
+ if otel_trace is None:
+ logger.debug("OTel trace not available, skipping root span attributes application")
+ return
+
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is None:
+ logger.debug("No current span found, skipping root span attributes application")
+ return
+
+ if not current_span.is_recording():
+ logger.debug("Current span is not recording, skipping root span attributes application")
+ return
+
+ current_span.set_attributes(root_attrs.to_otel_attributes())
+
+ except (TypeError, ValueError, AttributeError) as e:
+ logger.warning("Error applying root span attributes: %s", str(e))
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error applying root span attributes: %s",
+ str(e),
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+
+def maybe_apply_root_span_response(root_attrs: RootSpanAttributes) -> None:
+ """
+ Best-effort sync of HTTP response information into the current root span.
+
+ Sets the HTTP status code and updates the span status based on the response:
+ - 5xx: ERROR status
+ - 4xx: UNSET status
+ - 2xx/3xx: OK status (implicit)
+
+ Args:
+ root_attrs: The root span attributes containing the HTTP status code.
+ """
+ if otel_trace is None or Status is None or StatusCode is None:
+ logger.debug(
+ "OTel trace or Status/StatusCode not available, skipping response state application"
+ )
+ return
+
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is None or not current_span.is_recording():
+ logger.debug("No active recording span found, skipping response state application")
+ return
+
+ current_span.set_attributes(root_attrs.to_otel_attributes())
+
+ status_code = int(root_attrs.http_status_code or 500)
+
+ if status_code >= 500:
+ current_span.set_status(Status(StatusCode.ERROR, description=f"HTTP {status_code}"))
+ elif status_code >= 400:
+ current_span.set_status(Status(StatusCode.UNSET))
+ # 2xx/3xx implicitly get OK status
+
+ except (TypeError, ValueError, AttributeError) as e:
+ logger.warning("Error applying root span response state: %s", str(e))
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error applying root span response state: %s",
+ str(e),
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+
+def maybe_apply_root_span_error(root_attrs: RootSpanAttributes, exc: Exception) -> None:
+ """
+ Best-effort exception recording for the current root span.
+
+ Records the exception on the currently active OTel span and sets the span
+ status to ERROR.
+
+ Args:
+ root_attrs: The root span attributes.
+ exc: The exception to record.
+ """
+ if otel_trace is None or Status is None or StatusCode is None:
+ logger.debug(
+ "OTel trace or Status/StatusCode not available, skipping error state application"
+ )
+ return
+
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is None or not current_span.is_recording():
+ logger.debug("No active recording span found, skipping error state application")
+ return
+
+ current_span.set_attributes(root_attrs.to_otel_attributes())
+ current_span.record_exception(exc)
+ current_span.set_status(Status(StatusCode.ERROR, description=str(exc)))
+
+ except (TypeError, ValueError, AttributeError) as e:
+ logger.warning("Error applying root span error state: %s", str(e))
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error applying root span error state: %s",
+ str(e),
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+
+def _get_route_template(request: Request) -> str:
+ """
+ Resolve a shared low-cardinality route template for request observability.
+
+ Extracts the route template from the request scope after routing has occurred.
+ If routing has not yet occurred, returns "/__unmatched__".
+
+ Args:
+ request: The incoming HTTP request.
+
+ Returns:
+ The route template string (e.g., "/api/v1/sessions/{session_id}") if routing
+ has occurred, otherwise "/__unmatched__".
+ """
+ try:
+ route = request.scope.get("route")
+ path = getattr(route, "path", None)
+ if path:
+ return str(path)
+ return "/__unmatched__"
+ except (TypeError, AttributeError) as e:
+ logger.debug("Error getting route template: %s", str(e))
+ return "/__unmatched__"
+ except Exception as e:
+ logger.error(
+ "Unexpected error getting route template: %s",
+ str(e),
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+ return "/__unmatched__"
+
+
+def _maybe_update_route_and_span_name(
+ *,
+ request: Request,
+ root_attrs: Any,
+ span: Any,
+) -> None:
+ """
+ Update `root_attrs.http_route` and the active span name after routing.
+
+ Starlette/FastAPI perform route matching in the downstream app (i.e. inside `call_next`).
+ This means `request.scope["route"]` is often unavailable before `call_next` executes.
+
+ We intentionally:
+ - keep `url_path` as the raw request path (always available),
+ - fill `http_route` with a low-cardinality route template once routing completes.
+
+ Args:
+ request: The incoming HTTP request.
+ root_attrs: The root span attributes to update.
+ span: The active OTel span (if any).
+ """
+ try:
+ final_route = _get_route_template(request)
+
+ if final_route and final_route != "/__unmatched__":
+ if getattr(root_attrs, "http_route", None) != final_route:
+ root_attrs.http_route = final_route
+
+ # Keep span name aligned with the finalized route template for better aggregation.
+ if span is not None:
+ update_name = getattr(span, "update_name", None)
+ if callable(update_name):
+ update_name(f"{request.method} {final_route}")
+
+ except (TypeError, ValueError, AttributeError) as e:
+ logger.warning("Error updating route and span name: %s", str(e))
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error updating route and span name: %s",
+ str(e),
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+
+def should_skip_http_metrics(request: Request) -> bool:
+ """
+ Return whether the request should skip HTTP metrics/tracing entirely.
+
+ Checks if the request path matches any of the ignore routes (health checks,
+ metrics endpoint, etc.).
+
+ Args:
+ request: The incoming HTTP request.
+
+ Returns:
+ True if the request should be skipped, False otherwise.
+ """
+ try:
+ raw_path = str(request.url.path)
+ if raw_path in _HTTP_IGNORE_ROUTES:
+ return True
+
+ route_template = _get_route_template(request)
+ return route_template in _HTTP_IGNORE_ROUTES
+
+ except Exception as e:
+ logger.warning("Error checking if should skip HTTP metrics: %s", str(e))
+ # Default to not skipping on error
+ return False
+
+
+@dataclass(slots=True)
+class HTTPMetricsStartState:
+ """Capture the initial metrics labels used when a request starts."""
+
+ route: str
+ account_id: Optional[str]
+
+
+def _inflight_delta(route: str, account_id: Optional[str], delta: int) -> int:
+ """
+ Update and return the current in-process inflight count for a route template.
+
+ This function uses a sharded counter to reduce lock contention under high concurrency.
+ The local counter is only used to provide a reasonable gauge value for event emission.
+ It is not a correctness mechanism and is clamped to never go below zero.
+
+ Args:
+ route: The route template.
+ account_id: The account identifier, or None.
+ delta: The change to apply (+1 for increment, -1 for decrement).
+
+ Returns:
+ The new inflight count after the delta is applied.
+ """
+ if delta > 0:
+ return _INFLIGHT_COUNTER.increment(route, account_id)
+ elif delta < 0:
+ return _INFLIGHT_COUNTER.decrement(route, account_id)
+ else:
+ return _INFLIGHT_COUNTER.get(route, account_id)
+
+
+def _log_metrics_failure(
+ message: str,
+ *,
+ route: str,
+ account_id: Optional[str] = None,
+ error: Optional[Exception] = None,
+) -> None:
+ """
+ Emit a log for best-effort metrics failures without affecting request handling.
+
+ This middleware intentionally treats observability as a side channel. The helper keeps
+ failure reporting centralized so all swallowed exceptions still leave a trace.
+
+ Logs at WARNING level for known error types, and ERROR level for unexpected errors.
+ Stack traces are only included when debug logging is enabled.
+
+ Args:
+ message: The error message.
+ route: The route template.
+ account_id: The account identifier, or None.
+ error: The exception that occurred, or None.
+ """
+ log_level = logging.WARNING if error else logging.DEBUG
+
+ extra = {
+ "route": route,
+ "account_id": account_id,
+ }
+
+ if error:
+ logger.log(
+ log_level,
+ "http metrics write failed: %s - %s",
+ message,
+ str(error),
+ extra=extra,
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+ else:
+ logger.log(
+ log_level,
+ "http metrics write failed: %s",
+ message,
+ extra=extra,
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+
+def apply_http_metrics_start(
+ *,
+ request: Request,
+ root_attrs: RootSpanAttributes,
+) -> HTTPMetricsStartState:
+ """
+ Record the initial inflight sample and store the start-state on the request.
+
+ Called at the beginning of request handling to increment the inflight counter
+ and capture the initial route/account_id for later rebalancing.
+
+ Args:
+ request: The incoming HTTP request.
+ root_attrs: The root span attributes.
+
+ Returns:
+ The HTTPMetricsStartState containing the initial route and account_id.
+ """
+ initial_route = root_attrs.http_route
+ initial_account_id = root_attrs.account_id
+
+ state = HTTPMetricsStartState(route=initial_route, account_id=initial_account_id)
+ request.state.http_metrics_start_state = state
+
+ try:
+ HttpRequestLifecycleDataSource.set_inflight(
+ route=initial_route,
+ value=_inflight_delta(initial_route, initial_account_id, +1),
+ account_id=initial_account_id,
+ )
+
+ except (TypeError, ValueError, AttributeError) as e:
+ _log_metrics_failure(
+ "http.inflight increment failed",
+ route=initial_route,
+ account_id=initial_account_id,
+ error=e,
+ )
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error in http.inflight increment: %s",
+ str(e),
+ extra={
+ "route": initial_route,
+ "account_id": initial_account_id,
+ },
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+ return state
+
+
+def apply_http_metrics_finalize(
+ *,
+ request: Request,
+ root_attrs: RootSpanAttributes,
+ elapsed: float,
+) -> None:
+ """
+ Record final request metrics while preserving current inflight semantics.
+
+ Called at the end of request handling to:
+ 1. Rebalance inflight counter if route/account_id changed during request
+ 2. Record the request duration and status code
+ 3. Decrement the inflight counter
+
+ Args:
+ request: The incoming HTTP request.
+ root_attrs: The root span attributes.
+ elapsed: The request duration in seconds.
+ """
+ start_state = getattr(request.state, "http_metrics_start_state", None)
+ initial_route = getattr(start_state, "route", root_attrs.http_route)
+ initial_account_id = getattr(start_state, "account_id", root_attrs.account_id)
+ final_route = _get_route_template(request)
+ final_account_id = root_attrs.account_id
+
+ # Rebalance inflight counter if route/account_id changed during request
+ if (final_route, final_account_id) != (initial_route, initial_account_id):
+ try:
+ HttpRequestLifecycleDataSource.set_inflight(
+ route=initial_route,
+ value=_inflight_delta(initial_route, initial_account_id, -1),
+ account_id=initial_account_id,
+ )
+ HttpRequestLifecycleDataSource.set_inflight(
+ route=final_route,
+ value=_inflight_delta(final_route, final_account_id, +1),
+ account_id=final_account_id,
+ )
+
+ except (TypeError, ValueError, AttributeError) as e:
+ _log_metrics_failure(
+ "http.inflight rebalance failed",
+ route=final_route,
+ account_id=final_account_id,
+ error=e,
+ )
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error in http.inflight rebalance: %s",
+ str(e),
+ extra={
+ "route": final_route,
+ "account_id": final_account_id,
+ },
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+ # Record request duration and status code
+ try:
+ HttpRequestLifecycleDataSource.record_request(
+ method=root_attrs.http_method,
+ route=final_route,
+ status=str(root_attrs.http_status_code or 500),
+ duration_seconds=elapsed,
+ account_id=final_account_id,
+ )
+
+ except (TypeError, ValueError, AttributeError) as e:
+ _log_metrics_failure(
+ "http.request recording failed",
+ route=final_route,
+ account_id=final_account_id,
+ error=e,
+ )
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error in http.request recording: %s",
+ str(e),
+ extra={
+ "route": final_route,
+ "account_id": final_account_id,
+ },
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+ # Decrement inflight counter
+ try:
+ HttpRequestLifecycleDataSource.set_inflight(
+ route=final_route,
+ value=_inflight_delta(final_route, final_account_id, -1),
+ account_id=final_account_id,
+ )
+
+ except (TypeError, ValueError, AttributeError) as e:
+ _log_metrics_failure(
+ "http.inflight decrement failed",
+ route=final_route,
+ account_id=final_account_id,
+ error=e,
+ )
+
+ except Exception as e:
+ logger.error(
+ "Unexpected error in http.inflight decrement: %s",
+ str(e),
+ extra={
+ "route": final_route,
+ "account_id": final_account_id,
+ },
+ exc_info=logger.isEnabledFor(logging.DEBUG),
+ )
+
+
+async def _execute_request_with_span(
+ request: Request,
+ call_next: Callable[[Request], Awaitable[Response]],
+ root_attrs: RootSpanAttributes,
+ span_cm: Optional[Any],
+) -> Response:
+ """
+ Execute the request with optional span context management.
+
+ This function encapsulates the common logic for both cases:
+ 1. When OTel is available and a span context manager exists
+ 2. When OTel is not available or span creation failed
+
+ Uses `contextlib.nullcontext` to unify the handling of both cases,
+ eliminating code duplication.
+
+ Args:
+ request: The incoming HTTP request.
+ call_next: The next middleware/handler in the chain.
+ root_attrs: The root span attributes.
+ span_cm: The span context manager (if available), or None.
+
+ Returns:
+ The response from the next handler.
+
+ Raises:
+ Any exception raised by the next handler.
+ """
+ # Use nullcontext when span_cm is None to unify handling
+ context = span_cm if span_cm is not None else nullcontext()
+
+ with context as span:
+ try:
+ response = await call_next(request)
+
+ # Update route and span name if we have a real span
+ if span is not None:
+ _maybe_update_route_and_span_name(
+ request=request,
+ root_attrs=root_attrs,
+ span=span,
+ )
+ # Apply root attributes to span
+ maybe_apply_root_span_attributes(root_attrs)
+
+ # Set status code from response
+ root_attrs.http_status_code = int(getattr(response, "status_code", 500))
+
+ # Apply response state (status code, etc.)
+ maybe_apply_root_span_response(root_attrs)
+
+ return response
+
+ except Exception as exc:
+ # Update route and span name if we have a real span
+ if span is not None:
+ _maybe_update_route_and_span_name(
+ request=request,
+ root_attrs=root_attrs,
+ span=span,
+ )
+ # Apply root attributes to span
+ maybe_apply_root_span_attributes(root_attrs)
+
+ # Apply error state to span
+ maybe_apply_root_span_error(root_attrs, exc)
+
+ raise
+
+
+def create_http_observability_middleware() -> Callable[[Request, Callable], Response]:
+ """
+ Create the unified HTTP observability middleware.
+
+ This middleware handles:
+ - HTTP metrics collection (inflight, request duration, status codes)
+ - OTel trace span creation and management
+ - Context propagation (root observability context binding)
+
+ Returns:
+ The middleware function.
+ """
+
+ async def middleware(request: Request, call_next: Callable) -> Response:
+ # Skip metrics/tracing for health check and metrics endpoints
+ if should_skip_http_metrics(request):
+ return await call_next(request)
+
+ # Extract request information
+ raw_path = str(request.url.path)
+ route_template = _get_route_template(request)
+ request_id = request.headers.get("x-request-id") or str(uuid.uuid4())
+
+ # Create root span attributes
+ root_attrs = create_root_span_attributes(
+ http_method=request.method,
+ http_route=route_template,
+ request_id=request_id,
+ url_path=raw_path,
+ url_scheme=request.url.scheme,
+ http_host=request.url.netloc,
+ source_type=request.headers.get("x-source-type"),
+ source_version=request.headers.get("x-source-version"),
+ )
+
+ # Attach to request state for later access
+ request.state.root_span_attrs = root_attrs
+ request.state.request_id = request_id
+
+ # Bind root context and start metrics
+ root_token = bind_root_observability_context(root_attrs)
+ apply_http_metrics_start(request=request, root_attrs=root_attrs)
+
+ start = time.perf_counter()
+
+ try:
+ # Try to create a root span if OTel is available
+ span_cm = maybe_start_root_span(request, root_attrs)
+
+ # Execute the request with unified span handling
+ response = await _execute_request_with_span(
+ request=request,
+ call_next=call_next,
+ root_attrs=root_attrs,
+ span_cm=span_cm,
+ )
+
+ return response
+
+ finally:
+ # Calculate elapsed time and finalize metrics
+ elapsed = time.perf_counter() - start
+ apply_http_metrics_finalize(
+ request=request,
+ root_attrs=root_attrs,
+ elapsed=elapsed,
+ )
+
+ # Reset root context
+ reset_root_observability_context(root_token)
+
+ return middleware
diff --git a/openviking/observability/log_trace_bridge.py b/openviking/observability/log_trace_bridge.py
new file mode 100644
index 000000000..c63ea68d3
--- /dev/null
+++ b/openviking/observability/log_trace_bridge.py
@@ -0,0 +1,215 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""
+Log to Trace Bridge Module.
+
+Provides convenience functions for developers to log messages and span events/exceptions
+simultaneously, reducing redundant code and simplifying development.
+
+Key features:
+- log_and_add_event: Log message and add span event simultaneously
+- log_exception: Log error and record span exception simultaneously
+- Preserves original types for OTel attributes (str, int, float, bool, arrays)
+- Safe fallback: only logs when OTel is not available
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Sequence, Union
+
+# Try to import opentelemetry - will be None if not installed
+try:
+ from opentelemetry import trace as otel_trace
+ from opentelemetry.trace import Status, StatusCode
+except ImportError:
+ otel_trace = None
+ Status = None
+ StatusCode = None
+
+
+# Define OTel-compatible attribute value types
+OTelAttributeValue = Union[
+ str,
+ int,
+ float,
+ bool,
+ Sequence[str],
+ Sequence[int],
+ Sequence[float],
+ Sequence[bool],
+]
+
+
+def _to_otel_attributes(kwargs: dict[str, Any]) -> dict[str, OTelAttributeValue]:
+ """
+ Convert kwargs to OTel-compatible span attributes.
+
+ Preserves original types for supported types (str, int, float, bool, and homogeneous arrays).
+ Unsupported types are converted to strings.
+
+ Args:
+ kwargs: Dictionary of key-value pairs to convert.
+
+ Returns:
+ Dictionary of OTel-compatible attributes.
+ """
+ attributes: dict[str, OTelAttributeValue] = {}
+
+ for key, value in kwargs.items():
+ if value is None:
+ # Skip None values
+ continue
+ elif isinstance(value, (str, int, float, bool)):
+ # Primitive types - use as-is
+ attributes[key] = value
+ elif isinstance(value, (list, tuple)):
+ # Array types - check if homogeneous
+ if not value:
+ # Empty array - skip
+ continue
+
+ # Check if all elements are of the same primitive type
+ first_type = type(value[0])
+ if first_type not in (str, int, float, bool):
+ # Mixed or unsupported types - convert to string array
+ attributes[key] = [str(v) for v in value]
+ continue
+
+ # Verify all elements are of the same type
+ all_same_type = all(isinstance(v, first_type) for v in value)
+ if all_same_type:
+ # Homogeneous array - use as-is
+ attributes[key] = list(value)
+ else:
+ # Mixed types - convert to string array
+ attributes[key] = [str(v) for v in value]
+ else:
+ # Other types - convert to string
+ attributes[key] = str(value)
+
+ return attributes
+
+
+def log_and_add_event(logger: logging.Logger, level: int, message: str, **kwargs: Any) -> None:
+ """
+ Log a message and add it as a span event simultaneously.
+
+ This function allows developers to log messages and record span events
+ in a single call, reducing redundant code.
+
+ Args:
+ logger: The logger instance to use.
+ level: The logging level (logging.INFO, logging.DEBUG, etc.).
+ message: The log message.
+ **kwargs: Additional key-value pairs to be included as log extra
+ and span attributes. Supported types: str, int, float, bool,
+ and homogeneous arrays of these types. Unsupported types are
+ converted to strings.
+ """
+ # Log the message
+ logger.log(level, message, extra=kwargs)
+
+ # If OTel is available, also record as a span event
+ if otel_trace is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and current_span.is_recording():
+ # Convert kwargs to OTel-compatible attributes, preserving types
+ attributes = _to_otel_attributes(kwargs)
+ current_span.add_event(message, attributes=attributes)
+ except Exception:
+ # Best-effort only - don't break logging if span event fails
+ pass
+
+
+def log_exception(logger: logging.Logger, message: str, exc: Exception, **kwargs: Any) -> None:
+ """
+ Log an error and record it as a span exception simultaneously.
+
+ This function logs the exception with full stack trace and also records
+ it in the current OTel span if available.
+
+ Args:
+ logger: The logger instance to use.
+ message: The error message.
+ exc: The exception object.
+ **kwargs: Additional key-value pairs to be included as log extra
+ and span attributes.
+ """
+ # Log the error with exception stack trace
+ logger.error(message, exc_info=exc, extra=kwargs)
+
+ # If OTel is available, also record the exception in the span
+ if otel_trace is not None and Status is not None and StatusCode is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and current_span.is_recording():
+ # Convert kwargs to OTel-compatible attributes
+ attributes = _to_otel_attributes(kwargs)
+ attributes["message"] = message
+
+ # Record the exception in the span
+ current_span.record_exception(exc, attributes=attributes)
+ current_span.set_status(Status(StatusCode.ERROR, description=str(exc)))
+ except Exception:
+ # Best-effort only - don't break logging if span recording fails
+ pass
+
+
+def log_debug(logger: logging.Logger, message: str, **kwargs: Any) -> None:
+ """
+ Log a DEBUG level message and add it as a span event.
+
+ Convenience wrapper around log_and_add_event with DEBUG level.
+
+ Args:
+ logger: The logger instance to use.
+ message: The log message.
+ **kwargs: Additional key-value pairs.
+ """
+ log_and_add_event(logger, logging.DEBUG, message, **kwargs)
+
+
+def log_info(logger: logging.Logger, message: str, **kwargs: Any) -> None:
+ """
+ Log an INFO level message and add it as a span event.
+
+ Convenience wrapper around log_and_add_event with INFO level.
+
+ Args:
+ logger: The logger instance to use.
+ message: The log message.
+ **kwargs: Additional key-value pairs.
+ """
+ log_and_add_event(logger, logging.INFO, message, **kwargs)
+
+
+def log_warning(logger: logging.Logger, message: str, **kwargs: Any) -> None:
+ """
+ Log a WARNING level message and add it as a span event.
+
+ Convenience wrapper around log_and_add_event with WARNING level.
+
+ Args:
+ logger: The logger instance to use.
+ message: The log message.
+ **kwargs: Additional key-value pairs.
+ """
+ log_and_add_event(logger, logging.WARNING, message, **kwargs)
+
+
+def log_error(logger: logging.Logger, message: str, **kwargs: Any) -> None:
+ """
+ Log an ERROR level message and add it as a span event.
+
+ Convenience wrapper around log_and_add_event with ERROR level.
+ Note: This does not record an exception. Use log_exception for that.
+
+ Args:
+ logger: The logger instance to use.
+ message: The log message.
+ **kwargs: Additional key-value pairs.
+ """
+ log_and_add_event(logger, logging.ERROR, message, **kwargs)
diff --git a/openviking/server/app.py b/openviking/server/app.py
index 6e1cb00ee..8c04d136e 100644
--- a/openviking/server/app.py
+++ b/openviking/server/app.py
@@ -46,6 +46,7 @@ from openviking.service.core import OpenVikingService
from openviking.service.task_tracker import get_task_tracker
from openviking_cli.exceptions import OpenVikingError
from openviking_cli.utils import get_logger
+from openviking_cli.utils.logger import init_otel_log_handler_from_server_config
logger = get_logger(__name__)
@@ -126,17 +127,18 @@ def create_app(
task_tracker = get_task_tracker()
task_tracker.start_cleanup_loop()
- # Initialize tracer
+ # Initialize tracing and OTLP log export from server.observability.
from openviking.telemetry import tracer_module
- tracer_module.init_tracer_from_config()
+ tracer_module.init_tracer_from_server_config(config)
+ init_otel_log_handler_from_server_config(config)
yield
# Cleanup
- from openviking.metrics.global_api import shutdown_metrics
+ from openviking.metrics.global_api import shutdown_metrics_async
- shutdown_metrics(app=app)
+ await shutdown_metrics_async(app=app)
task_tracker.stop_cleanup_loop()
if owns_service and service:
try:
@@ -165,6 +167,22 @@ def create_app(
allow_headers=["*"],
)
+ # Add HTTP observability middleware first (metrics, tracing)
+ # Note: In FastAPI/Starlette, middleware added later executes first (outer layer).
+ # We want timing to be the outermost layer to measure the full request duration.
+ from openviking.observability.http_observability_middleware import (
+ create_http_observability_middleware,
+ )
+
+ http_observability_middleware = create_http_observability_middleware()
+
+ @app.middleware("http")
+ async def add_http_observability(request: Request, call_next: Callable):
+ return await http_observability_middleware(request, call_next)
+
+ # Add request timing middleware last (so it executes first as the outermost layer)
+ # This ensures X-Process-Time includes the full request duration including
+ # observability middleware overhead.
# Add request header logging middleware (for debug)
@app.middleware("http")
async def log_request_headers(request: Request, call_next: Callable):
@@ -181,19 +199,25 @@ def create_app(
# Add request timing middleware
@app.middleware("http")
async def add_timing(request: Request, call_next: Callable):
- start_time = time.time()
- response = await call_next(request)
- process_time = time.time() - start_time
- response.headers["X-Process-Time"] = str(process_time)
- return response
+ """
+ Middleware to measure request processing time.
- from openviking.metrics.http_middleware import create_http_metrics_middleware
+ This middleware is added last so it executes as the outermost layer,
+ ensuring X-Process-Time includes the full request duration including
+ all other middleware overhead.
- http_metrics_middleware = create_http_metrics_middleware()
+ Args:
+ request: The incoming HTTP request.
+ call_next: The next middleware/handler in the chain.
- @app.middleware("http")
- async def add_http_metrics(request: Request, call_next: Callable):
- return await http_metrics_middleware(request, call_next)
+ Returns:
+ The response with X-Process-Time header added.
+ """
+ start_time = time.perf_counter()
+ response = await call_next(request)
+ process_time = time.perf_counter() - start_time
+ response.headers["X-Process-Time"] = str(process_time)
+ return response
# Add exception handler for OpenVikingError
@app.exception_handler(OpenVikingError)
diff --git a/openviking/server/auth.py b/openviking/server/auth.py
index 2b18aabb7..e6517de82 100644
--- a/openviking/server/auth.py
+++ b/openviking/server/auth.py
@@ -7,8 +7,8 @@ from typing import Optional
from fastapi import Depends, Header, Request
-from openviking.metrics.account_context import set_metric_account_context
from openviking.server.identity import AuthMode, RequestContext, ResolvedIdentity, Role
+from openviking.telemetry.span_models import update_root_span_identity
from openviking_cli.exceptions import (
InvalidArgumentError,
PermissionDeniedError,
@@ -274,8 +274,14 @@ async def get_request_context(
else identity.namespace_policy
),
)
- request.state.metric_account_id = ctx.account_id
- set_metric_account_context(account_id=ctx.account_id)
+ # Update the unified root observability context after authentication succeeds.
+ update_root_span_identity(
+ request_state=request.state,
+ account_id=identity.account_id,
+ user_id=identity.user_id,
+ agent_id=identity.agent_id,
+ )
+
return ctx
diff --git a/openviking/server/config.py b/openviking/server/config.py
index c48bb8daa..6f1153192 100644
--- a/openviking/server/config.py
+++ b/openviking/server/config.py
@@ -35,6 +35,43 @@ class MetricsAccountDimensionConfig(BaseModel):
model_config = {"extra": "forbid"}
+class PrometheusExporterConfig(BaseModel):
+ """Prometheus exporter configuration."""
+
+ enabled: bool = True
+
+ model_config = {"extra": "forbid"}
+
+
+class OTelExporterConfig(BaseModel):
+ """OpenTelemetry exporter configuration."""
+
+ class TLSConfig(BaseModel):
+ """TLS configuration for OTLP exporters."""
+
+ insecure: bool = False
+
+ model_config = {"extra": "forbid"}
+
+ enabled: bool = False
+ protocol: str = "grpc" # "grpc" or "http"
+ tls: TLSConfig = Field(default_factory=TLSConfig)
+ endpoint: str = "localhost:4317" # gRPC default: 4317; HTTP default: 4318
+ service_name: str = "openviking-server"
+ export_interval_ms: int = 10000
+
+ model_config = {"extra": "forbid"}
+
+
+class MetricsExportersConfig(BaseModel):
+ """Metrics exporters configuration."""
+
+ prometheus: PrometheusExporterConfig = Field(default_factory=PrometheusExporterConfig)
+ otel: OTelExporterConfig = Field(default_factory=OTelExporterConfig)
+
+ model_config = {"extra": "forbid"}
+
+
class MetricsConfig(BaseModel):
"""Metrics subsystem configuration."""
@@ -42,6 +79,7 @@ class MetricsConfig(BaseModel):
account_dimension: MetricsAccountDimensionConfig = Field(
default_factory=MetricsAccountDimensionConfig
)
+ exporters: MetricsExportersConfig = Field(default_factory=MetricsExportersConfig)
model_config = {"extra": "forbid"}
@@ -50,6 +88,8 @@ class ObservabilityConfig(BaseModel):
"""Server-side observability configuration."""
metrics: MetricsConfig = Field(default_factory=MetricsConfig)
+ traces: OTelExporterConfig = Field(default_factory=OTelExporterConfig)
+ logs: OTelExporterConfig = Field(default_factory=OTelExporterConfig)
model_config = {"extra": "forbid"}
diff --git a/openviking/server/routers/metrics.py b/openviking/server/routers/metrics.py
index 65cb7d863..848908025 100644
--- a/openviking/server/routers/metrics.py
+++ b/openviking/server/routers/metrics.py
@@ -5,17 +5,24 @@
from fastapi import APIRouter, Request
from fastapi.responses import PlainTextResponse
+from openviking.metrics.exporters.prometheus import PrometheusExporter
+
router = APIRouter(tags=["metrics"])
@router.get("/metrics")
async def metrics(request: Request):
"""Return Prometheus metrics in text exposition format."""
- exporter = getattr(request.app.state, "metrics_exporter", None)
- if exporter is None:
+ exporters = getattr(request.app.state, "metrics_exporters", [])
+ # Find PrometheusExporter from the list
+ prometheus_exporter = next(
+ (e for e in exporters if isinstance(e, PrometheusExporter)),
+ None,
+ )
+ if prometheus_exporter is None:
return PlainTextResponse(status_code=404, content="Prometheus metrics are disabled.\n")
return PlainTextResponse(
- content=await exporter.export(),
+ content=await prometheus_exporter.export(),
media_type="text/plain; version=0.0.4; charset=utf-8",
)
diff --git a/openviking/storage/queuefs/semantic_processor.py b/openviking/storage/queuefs/semantic_processor.py
index 32984fe1e..7e7b2ba64 100644
--- a/openviking/storage/queuefs/semantic_processor.py
+++ b/openviking/storage/queuefs/semantic_processor.py
@@ -9,9 +9,9 @@ from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Set, Tuple
from openviking.core.namespace import agent_space_fragment, user_space_fragment
-from openviking.metrics.account_context import (
- bind_metric_account_context,
- reset_metric_account_context,
+from openviking.observability.context import (
+ bind_root_observability_context,
+ reset_root_observability_context,
)
from openviking.parse.parsers.constants import (
CODE_EXTENSIONS,
@@ -34,6 +34,7 @@ from openviking.storage.queuefs.semantic_msg import SemanticMsg
from openviking.storage.viking_fs import get_viking_fs
from openviking.telemetry import bind_telemetry, bind_telemetry_stage, resolve_telemetry
from openviking.telemetry.request_wait_tracker import get_request_wait_tracker
+from openviking.telemetry.span_models import create_root_span_attributes
from openviking.utils.circuit_breaker import (
CircuitBreaker,
CircuitBreakerOpen,
@@ -274,7 +275,16 @@ class SemanticProcessor(DequeueHandlerBase):
collector = resolve_telemetry(msg.telemetry_id)
telemetry_ctx = bind_telemetry(collector) if collector is not None else nullcontext()
with telemetry_ctx:
- metric_account_token = bind_metric_account_context(account_id=msg.account_id)
+ root_attrs = create_root_span_attributes(
+ http_method="QUEUE",
+ http_route=msg.context_type or "/queuefs/semantic",
+ request_id=msg.telemetry_id or msg.id,
+ url_path=msg.uri,
+ )
+ root_attrs.account_id = msg.account_id
+ root_attrs.user_id = msg.user_id
+ root_attrs.agent_id = msg.agent_id
+ root_context_token = bind_root_observability_context(root_attrs)
try:
self._current_msg = msg
self._current_ctx = self._ctx_from_semantic_msg(msg)
@@ -337,7 +347,7 @@ class SemanticProcessor(DequeueHandlerBase):
self._circuit_breaker.record_success()
return None
finally:
- reset_metric_account_context(metric_account_token)
+ reset_root_observability_context(root_context_token)
except Exception as e:
error_class = classify_api_error(e)
diff --git a/openviking/telemetry/execution.py b/openviking/telemetry/execution.py
index df1679639..019946828 100644
--- a/openviking/telemetry/execution.py
+++ b/openviking/telemetry/execution.py
@@ -7,12 +7,17 @@ from __future__ import annotations
from dataclasses import dataclass
from typing import Any, Awaitable, Callable, Generic, Optional, TypeVar
+from openviking.observability.context import (
+ bind_operation_observability_context,
+ reset_operation_observability_context,
+)
from openviking_cli.exceptions import InvalidArgumentError
from openviking_cli.utils import get_logger
from .context import bind_telemetry
from .operation import OperationTelemetry, TelemetrySnapshot
from .request import TelemetryRequest, TelemetrySelection, normalize_telemetry_request
+from .span_models import OperationSpanAttributes
T = TypeVar("T")
logger = get_logger(__name__)
@@ -36,16 +41,16 @@ def parse_telemetry_selection(telemetry: TelemetryRequest) -> TelemetrySelection
def build_telemetry_payload(
- snapshot: TelemetrySnapshot | None,
+ snapshot: Optional[TelemetrySnapshot],
selection: TelemetrySelection,
-) -> dict[str, Any] | None:
+) -> Optional[dict[str, Any]]:
"""Build a telemetry payload from a finished snapshot."""
if snapshot is None or not selection.include_payload:
return None
return snapshot.to_dict(include_summary=selection.include_summary)
-def _log_telemetry_summary(snapshot: TelemetrySnapshot | None) -> None:
+def _log_telemetry_summary(snapshot: Optional[TelemetrySnapshot]) -> None:
if snapshot is None:
return
logger.info(
@@ -75,6 +80,78 @@ def attach_telemetry_payload(
return result
+# Try to import opentelemetry - will be None if not installed
+try:
+ from opentelemetry import trace as otel_trace
+ from opentelemetry.trace import Status, StatusCode
+except ImportError:
+ otel_trace = None
+ Status = None
+ StatusCode = None
+
+
+def _fill_telemetry_summary_to_span(snapshot, span):
+ """fill OperationTelemetry summary into span
+
+ Args:
+ snapshot: TelemetrySnapshot object
+ span: current OTel span
+ """
+ if snapshot is None or span is None:
+ return
+
+ try:
+ # Build OperationSpanAttributes from snapshot for consistent span attributes.
+ op_attrs = OperationSpanAttributes.from_telemetry_snapshot(
+ operation=snapshot.operation,
+ telemetry_id=snapshot.telemetry_id,
+ status=snapshot.status,
+ summary=snapshot.summary,
+ )
+
+ # Apply to span attributes (best-effort).
+ span.set_attributes(op_attrs.to_otel_attributes())
+
+ # Attach summary as a span event for debugging (best-effort).
+ if snapshot.summary:
+ span.add_event("telemetry_summary", attributes={"summary": str(snapshot.summary)})
+ except Exception:
+ logger.debug("failed to fill telemetry summary into span", exc_info=True)
+
+
+def _fill_operation_context_from_snapshot(
+ operation_attrs: OperationSpanAttributes,
+ snapshot: Optional[TelemetrySnapshot],
+) -> OperationSpanAttributes:
+ """
+ Update the bound operation context from a finished telemetry snapshot.
+
+ Returns a new OperationSpanAttributes instance with the updated fields,
+ rather than modifying the input object in place. This avoids unintended
+ side effects when the same object is referenced elsewhere.
+
+ Args:
+ operation_attrs: The original operation attributes.
+ snapshot: The telemetry snapshot (optional).
+
+ Returns:
+ A new OperationSpanAttributes instance with updated fields, or the original
+ instance if snapshot is None.
+ """
+ if snapshot is None:
+ return operation_attrs
+
+ # Create a new instance from the snapshot
+ updated = OperationSpanAttributes.from_telemetry_snapshot(
+ operation=operation_attrs.operation,
+ telemetry_id=snapshot.telemetry_id,
+ status=snapshot.summary.get("status"),
+ summary=snapshot.summary,
+ )
+
+ return updated
+
+
async def run_with_telemetry(
*,
operation: str,
@@ -88,36 +165,93 @@ async def run_with_telemetry(
operation=operation,
enabled=True,
)
+ operation_attrs = OperationSpanAttributes(
+ operation=operation,
+ telemetry_id=collector.telemetry_id,
+ )
+ operation_token = bind_operation_observability_context(operation_attrs)
+
+ # If OTel is available, create an operation span (best-effort).
+ span_context_manager = None
+ if otel_trace is not None:
+ try:
+ tracer = otel_trace.get_tracer(__name__)
+ span_context_manager = tracer.start_as_current_span(
+ name=operation,
+ kind=otel_trace.SpanKind.INTERNAL,
+ )
+ except Exception:
+ logger.debug("failed to start operation span", exc_info=True)
+ span_context_manager = None
try:
- with bind_telemetry(collector):
- result = await fn()
- except Exception as exc:
- collector.set_error(operation, type(exc).__name__, str(exc))
- snapshot = collector.finish(status=error_status)
+ try:
+ with bind_telemetry(collector):
+ if span_context_manager is not None:
+ with span_context_manager as span:
+ try:
+ span.set_attributes(operation_attrs.to_otel_attributes())
+ except Exception:
+ logger.debug(
+ "failed to set operation span attributes (initial)",
+ exc_info=True,
+ )
+ result = await fn()
+ else:
+ result = await fn()
+ except Exception as exc:
+ collector.set_error(operation, type(exc).__name__, str(exc))
+ snapshot = collector.finish(status=error_status)
+ _log_telemetry_summary(snapshot)
+
+ # If a span exists, record exception and backfill summary (best-effort).
+ if otel_trace is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and current_span.is_recording():
+ current_span.record_exception(exc)
+ current_span.set_status(Status(StatusCode.ERROR, description=str(exc)))
+ _fill_telemetry_summary_to_span(snapshot, current_span)
+ except Exception:
+ logger.debug(
+ "failed to record exception into operation span",
+ exc_info=True,
+ )
+
+ raise
+
+ snapshot = collector.finish(status="ok")
_log_telemetry_summary(snapshot)
- raise
- snapshot = collector.finish(status="ok")
- _log_telemetry_summary(snapshot)
- try:
- if snapshot is not None:
- from openviking.metrics.datasources.telemetry_bridge import (
- TelemetryBridgeEventDataSource,
- )
+ # If a span exists, backfill summary (best-effort).
+ if otel_trace is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and current_span.is_recording():
+ _fill_telemetry_summary_to_span(snapshot, current_span)
+ except Exception:
+ logger.debug("failed to backfill telemetry summary into span", exc_info=True)
- TelemetryBridgeEventDataSource.record_summary(snapshot.summary)
- except Exception:
- pass
- telemetry_payload = build_telemetry_payload(
- snapshot,
- selection,
- )
- return TelemetryExecutionResult(
- result=result,
- telemetry=telemetry_payload,
- selection=selection,
- )
+ try:
+ if snapshot is not None:
+ from openviking.metrics.datasources.telemetry_bridge import (
+ TelemetryBridgeEventDataSource,
+ )
+
+ TelemetryBridgeEventDataSource.record_summary(snapshot.summary)
+ except Exception:
+ logger.debug("failed to record telemetry summary into metrics bridge", exc_info=True)
+ telemetry_payload = build_telemetry_payload(
+ snapshot,
+ selection,
+ )
+ return TelemetryExecutionResult(
+ result=result,
+ telemetry=telemetry_payload,
+ selection=selection,
+ )
+ finally:
+ reset_operation_observability_context(operation_token)
__all__ = [
diff --git a/openviking/telemetry/span_models.py b/openviking/telemetry/span_models.py
new file mode 100644
index 000000000..49d5ff0f6
--- /dev/null
+++ b/openviking/telemetry/span_models.py
@@ -0,0 +1,376 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""
+Span model definitions.
+
+This module defines the data structures for root spans and operation spans and
+provides shared field management, validation-friendly constructors, and
+serialization helpers.
+
+Key features:
+- Shared field definitions to avoid hard-coded attribute names
+- Type-safe dataclass-based models
+- Factory helpers for consistent span/context creation
+- JSON and dict serialization helpers
+- Lightweight integration with OpenTelemetry attribute APIs
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import asdict, dataclass
+from typing import Any, Dict, Optional
+
+
+@dataclass
+class RootSpanAttributes:
+ """Attributes for the root span associated with one HTTP request."""
+
+ # HTTP semantic fields
+ http_method: str
+ """HTTP method, such as GET or POST."""
+
+ http_route: str
+ """Low-cardinality route template, such as `/sessions/{session_id}`."""
+
+ # Request-level fields
+ request_id: str
+ """Unique request identifier."""
+
+ http_status_code: Optional[int] = None
+ """HTTP status code, populated after the response is available."""
+
+ url_path: Optional[str] = None
+ """Raw request path for debugging. This may be high-cardinality."""
+
+ url_scheme: Optional[str] = None
+ """URL scheme, such as `http` or `https`."""
+
+ http_host: Optional[str] = None
+ """HTTP host value."""
+
+ source_type: Optional[str] = None
+ """Client source type."""
+
+ source_version: Optional[str] = None
+ """Client source version."""
+
+ # Identity fields populated after authentication
+ account_id: Optional[str] = None
+ """Tenant or account identifier."""
+
+ user_id: Optional[str] = None
+ """User identifier."""
+
+ agent_id: Optional[str] = None
+ """Agent identifier."""
+
+ def to_otel_attributes(self) -> Dict[str, Any]:
+ """Convert the model into an OpenTelemetry span attribute mapping."""
+ attrs = {
+ "http.method": self.http_method,
+ "http.route": self.http_route,
+ "request_id": self.request_id,
+ }
+
+ if self.http_status_code is not None:
+ attrs["http.status_code"] = self.http_status_code
+ if self.url_path is not None:
+ attrs["url.path"] = self.url_path
+ if self.url_scheme is not None:
+ attrs["url.scheme"] = self.url_scheme
+ if self.http_host is not None:
+ attrs["http.host"] = self.http_host
+ if self.source_type is not None:
+ attrs["source_type"] = self.source_type
+ if self.source_version is not None:
+ attrs["source_version"] = self.source_version
+ if self.account_id is not None:
+ attrs["account_id"] = self.account_id
+ if self.user_id is not None:
+ attrs["user_id"] = self.user_id
+ if self.agent_id is not None:
+ attrs["agent_id"] = self.agent_id
+
+ return attrs
+
+ def to_log_fields(self) -> Dict[str, Any]:
+ """Convert the root context into structured log fields."""
+ fields: Dict[str, Any] = {
+ "request_id": self.request_id,
+ "http_method": self.http_method,
+ "http_route": self.http_route,
+ }
+ if self.http_status_code is not None:
+ fields["http_status_code"] = self.http_status_code
+ if self.account_id is not None:
+ fields["account_id"] = self.account_id
+ if self.user_id is not None:
+ fields["user_id"] = self.user_id
+ if self.agent_id is not None:
+ fields["agent_id"] = self.agent_id
+ return fields
+
+ def to_dict(self) -> Dict[str, Any]:
+ """Return all fields as a plain dictionary."""
+ return asdict(self)
+
+ def to_json(self) -> str:
+ """Serialize the model into a JSON string."""
+ return json.dumps(self.to_dict(), ensure_ascii=False)
+
+ @classmethod
+ def from_dict(cls, data: Dict[str, Any]) -> RootSpanAttributes:
+ """Build a `RootSpanAttributes` instance from a dictionary."""
+ return cls(**data)
+
+ @classmethod
+ def from_json(cls, json_str: str) -> RootSpanAttributes:
+ """Build a `RootSpanAttributes` instance from a JSON string."""
+ data = json.loads(json_str)
+ return cls.from_dict(data)
+
+
+@dataclass
+class OperationSpanAttributes:
+ """Attributes for one business-level operation span."""
+
+ # Business-level fields
+ operation: str
+ """Operation name, such as `search.find`."""
+
+ telemetry_id: str
+ """Telemetry identifier for one operation execution."""
+
+ status: Optional[str] = None
+ """Business outcome, such as `ok` or `error`."""
+
+ # Token-related fields
+ tokens_total: Optional[int] = None
+ """Total token count."""
+
+ # Vector-related fields
+ vector_searches: Optional[int] = None
+ """Number of vector search operations."""
+
+ vector_scanned: Optional[int] = None
+ """Number of scanned vectors."""
+
+ vector_returned: Optional[int] = None
+ """Number of returned vectors."""
+
+ # Memory-related fields
+ memory_extracted: Optional[int] = None
+ """Number of extracted memory items."""
+
+ # Error-related fields
+ errors_stage: Optional[str] = None
+ """Stage where the error occurred."""
+
+ errors_error_code: Optional[str] = None
+ """Error code."""
+
+ errors_message: Optional[str] = None
+ """Error message."""
+
+ def to_otel_attributes(self) -> Dict[str, Any]:
+ """Convert the model into an OpenTelemetry span attribute mapping."""
+ attrs = {
+ "operation": self.operation,
+ "telemetry_id": self.telemetry_id,
+ }
+
+ if self.status is not None:
+ attrs["status"] = self.status
+ if self.tokens_total is not None:
+ attrs["tokens.total"] = self.tokens_total
+ if self.vector_searches is not None:
+ attrs["vector.searches"] = self.vector_searches
+ if self.vector_scanned is not None:
+ attrs["vector.scanned"] = self.vector_scanned
+ if self.vector_returned is not None:
+ attrs["vector.returned"] = self.vector_returned
+ if self.memory_extracted is not None:
+ attrs["memory.extracted"] = self.memory_extracted
+ if self.errors_stage is not None:
+ attrs["errors.stage"] = self.errors_stage
+ if self.errors_error_code is not None:
+ attrs["errors.error_code"] = self.errors_error_code
+ if self.errors_message is not None:
+ attrs["errors.message"] = self.errors_message
+
+ return attrs
+
+ def to_log_fields(self) -> Dict[str, Any]:
+ """Convert the operation context into structured log fields."""
+ fields: Dict[str, Any] = {
+ "operation": self.operation,
+ "telemetry_id": self.telemetry_id,
+ }
+ if self.status is not None:
+ fields["status"] = self.status
+ if self.tokens_total is not None:
+ fields["tokens_total"] = self.tokens_total
+ return fields
+
+ def to_dict(self) -> Dict[str, Any]:
+ """Return all fields as a plain dictionary."""
+ return asdict(self)
+
+ def to_json(self) -> str:
+ """Serialize the model into a JSON string."""
+ return json.dumps(self.to_dict(), ensure_ascii=False)
+
+ @classmethod
+ def from_dict(cls, data: Dict[str, Any]) -> OperationSpanAttributes:
+ """Build an `OperationSpanAttributes` instance from a dictionary."""
+ return cls(**data)
+
+ @classmethod
+ def from_json(cls, json_str: str) -> OperationSpanAttributes:
+ """Build an `OperationSpanAttributes` instance from a JSON string."""
+ data = json.loads(json_str)
+ return cls.from_dict(data)
+
+ @classmethod
+ def from_telemetry_snapshot(
+ cls,
+ operation: str,
+ telemetry_id: str,
+ status: Optional[str] = None,
+ summary: Optional[Dict[str, Any]] = None,
+ ) -> OperationSpanAttributes:
+ """Build an instance from an `OperationTelemetry` summary snapshot."""
+ attrs = cls(
+ operation=operation,
+ telemetry_id=telemetry_id,
+ status=status,
+ )
+
+ if summary:
+ # Extract token-related fields from the summary payload.
+ if "tokens" in summary:
+ tokens = summary["tokens"]
+ if "total" in tokens:
+ attrs.tokens_total = tokens["total"]
+
+ # Extract vector-related fields from the summary payload.
+ if "vector" in summary:
+ vector = summary["vector"]
+ if "searches" in vector:
+ attrs.vector_searches = vector["searches"]
+ if "scanned" in vector:
+ attrs.vector_scanned = vector["scanned"]
+ if "returned" in vector:
+ attrs.vector_returned = vector["returned"]
+
+ # Extract memory-related fields from the summary payload.
+ if "memory" in summary:
+ memory = summary["memory"]
+ if "extracted" in memory:
+ attrs.memory_extracted = memory["extracted"]
+
+ # Extract error-related fields from the summary payload.
+ if "errors" in summary:
+ errors = summary["errors"]
+ if "stage" in errors:
+ attrs.errors_stage = errors["stage"]
+ if "error_code" in errors:
+ attrs.errors_error_code = errors["error_code"]
+ if "message" in errors:
+ attrs.errors_message = errors["message"]
+
+ return attrs
+
+
+def create_root_span_attributes(
+ *,
+ http_method: str,
+ http_route: str,
+ request_id: str,
+ url_path: Optional[str] = None,
+ url_scheme: Optional[str] = None,
+ http_host: Optional[str] = None,
+ source_type: Optional[str] = None,
+ source_version: Optional[str] = None,
+) -> RootSpanAttributes:
+ """Factory helper for `RootSpanAttributes`."""
+ return RootSpanAttributes(
+ http_method=http_method,
+ http_route=http_route,
+ request_id=request_id,
+ url_path=url_path,
+ url_scheme=url_scheme,
+ http_host=http_host,
+ source_type=source_type,
+ source_version=source_version,
+ )
+
+
+def create_operation_span_attributes(
+ *,
+ operation: str,
+ telemetry_id: str,
+ status: Optional[str] = None,
+) -> OperationSpanAttributes:
+ """Factory helper for `OperationSpanAttributes`."""
+ return OperationSpanAttributes(
+ operation=operation,
+ telemetry_id=telemetry_id,
+ status=status,
+ )
+
+
+def update_root_span_identity(
+ *,
+ request_state: Any,
+ account_id: Optional[str] = None,
+ user_id: Optional[str] = None,
+ agent_id: Optional[str] = None,
+) -> bool:
+ """
+ Update identity fields on `RootSpanAttributes` and mirror them into the current span.
+
+ This helper is used after authentication completes, when account or user identity becomes
+ available only after the root request span has already been created.
+ """
+ try:
+ # Ensure a root observability model is attached to `request.state`.
+ if not hasattr(request_state, "root_span_attrs"):
+ return False
+
+ root_attrs = request_state.root_span_attrs
+ if not isinstance(root_attrs, RootSpanAttributes):
+ return False
+
+ # Update identity fields in place.
+ if account_id is not None:
+ root_attrs.account_id = account_id
+ if user_id is not None:
+ root_attrs.user_id = user_id
+ if agent_id is not None:
+ root_attrs.agent_id = agent_id
+
+ # Best-effort sync into the active OTel span, if one exists.
+ try:
+ from opentelemetry import trace as otel_trace
+
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and current_span.is_recording():
+ current_span.set_attributes(root_attrs.to_otel_attributes())
+ except Exception:
+ pass
+
+ return True
+ except Exception:
+ return False
+
+
+__all__ = [
+ "RootSpanAttributes",
+ "OperationSpanAttributes",
+ "create_root_span_attributes",
+ "create_operation_span_attributes",
+ "update_root_span_identity",
+]
diff --git a/openviking/telemetry/tracer.py b/openviking/telemetry/tracer.py
index 8afdcdc8a..298b03e99 100644
--- a/openviking/telemetry/tracer.py
+++ b/openviking/telemetry/tracer.py
@@ -14,19 +14,33 @@ from loguru import logger
try:
from opentelemetry import trace as otel_trace
from opentelemetry.context import Context
- from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import extract, inject
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Status, StatusCode, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
+
+ try:
+ from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
+ OTLPSpanExporter as OTLPGrpcSpanExporter,
+ )
+ except ImportError:
+ OTLPGrpcSpanExporter = None
+
+ try:
+ from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
+ OTLPSpanExporter as OTLPHttpSpanExporter,
+ )
+ except ImportError:
+ OTLPHttpSpanExporter = None
except ImportError:
otel_trace = None
TracerProvider = None
Status = None
StatusCode = None
BatchSpanProcessor = None
- OTLPSpanExporter = None
+ OTLPGrpcSpanExporter = None
+ OTLPHttpSpanExporter = None
TraceContextTextMapPropagator = None
Context = None
extract = None
@@ -80,32 +94,34 @@ def _setup_logging():
_log_trace_internal_failure("[TRACER] failed to attach standard logging trace_id filter")
-def init_tracer_from_config() -> Any:
- """Initialize tracer from OpenViking config."""
- try:
- from openviking_cli.utils.config import get_openviking_config
+def init_tracer_from_server_config(server_config: Any) -> Any:
+ """Initialize tracer from server.observability.traces config.
- config = get_openviking_config()
- tracer_cfg = config.telemetry.tracer
+ Args:
+ server_config: The server configuration containing observability settings.
- if not tracer_cfg.enabled:
- logger.info("[TRACER] disabled in config")
+ Returns:
+ The initialized tracer, or None if initialization failed or disabled.
+ """
+ try:
+ trace_cfg = server_config.observability.traces
+ if not trace_cfg.enabled:
+ logger.info("[TRACER] disabled in server.observability.traces")
return None
- if not tracer_cfg.endpoint:
- logger.warning("[TRACER] endpoint not configured")
+ if not trace_cfg.endpoint:
+ logger.warning("[TRACER] server.observability.traces.endpoint not configured")
return None
return init_tracer(
- endpoint=tracer_cfg.endpoint,
- service_name=tracer_cfg.service_name,
- topic=tracer_cfg.topic,
- ak=tracer_cfg.ak,
- sk=tracer_cfg.sk,
- enabled=tracer_cfg.enabled,
+ endpoint=trace_cfg.endpoint,
+ service_name=trace_cfg.service_name,
+ protocol=trace_cfg.protocol,
+ insecure=trace_cfg.tls.insecure,
+ enabled=trace_cfg.enabled,
)
except Exception as e:
- logger.warning(f"[TRACER] init from config failed: {e}")
+ logger.warning(f"[TRACER] init from server config failed: {e}")
return None
@@ -125,19 +141,17 @@ def _init_asyncio_instrumentation() -> None:
def init_tracer(
endpoint: str,
service_name: str,
- topic: str,
- ak: str,
- sk: str,
+ protocol: str = "grpc",
+ insecure: bool = False,
enabled: bool = True,
) -> Any:
"""Initialize the OpenTelemetry tracer.
Args:
- endpoint: OTLP endpoint URL
+ endpoint: OTLP endpoint
service_name: Service name for tracing
- topic: Trace topic
- ak: Access key
- sk: Secret key
+ protocol: OTLP protocol ("grpc" or "http")
+ insecure: For OTLP/gRPC only. When True, use plaintext instead of TLS.
enabled: Whether to enable tracing
Returns:
@@ -157,22 +171,36 @@ def init_tracer(
return None
try:
- headers = {
- "x-tls-otel-tracetopic": topic,
- "x-tls-otel-ak": ak,
- "x-tls-otel-sk": sk,
- "x-tls-otel-region": "cn-beijing",
- }
-
resource_attributes = {
"service.name": service_name,
}
resource = Resource.create(resource_attributes)
- trace_exporter = OTLPSpanExporter(
- endpoint=endpoint,
- headers=headers,
- )
+ protocol = protocol.lower()
+ if protocol == "grpc":
+ if OTLPGrpcSpanExporter is None:
+ raise ImportError("gRPC OTLP trace exporter not available")
+ try:
+ trace_exporter = OTLPGrpcSpanExporter(
+ endpoint=endpoint,
+ insecure=insecure,
+ )
+ except TypeError:
+ trace_exporter = OTLPGrpcSpanExporter(
+ endpoint=endpoint,
+ )
+ elif protocol == "http":
+ if OTLPHttpSpanExporter is None:
+ raise ImportError("HTTP OTLP trace exporter not available")
+ if not endpoint.startswith("http://") and not endpoint.startswith("https://"):
+ raise ValueError(
+ "OTLP/HTTP trace endpoint must include scheme, e.g. 'http://localhost:4318/v1/traces'"
+ )
+ trace_exporter = OTLPHttpSpanExporter(
+ endpoint=endpoint,
+ )
+ else:
+ raise ValueError(f"Unsupported trace protocol: {protocol}")
trace_provider = TracerProvider(resource=resource)
trace_provider.add_span_processor(
@@ -194,7 +222,12 @@ def init_tracer(
# Initialize asyncio instrumentation to create child spans for create_task
_init_asyncio_instrumentation()
- logger.info(f"[TRACER] initialized with service_name={service_name}, endpoint={endpoint}")
+ logger.info(
+ "[TRACER] initialized with service_name=%s, protocol=%s, endpoint=%s",
+ service_name,
+ protocol,
+ endpoint,
+ )
return _otel_tracer
except Exception as e:
diff --git a/openviking_cli/server_bootstrap.py b/openviking_cli/server_bootstrap.py
index 36bbe86af..523f6606b 100644
--- a/openviking_cli/server_bootstrap.py
+++ b/openviking_cli/server_bootstrap.py
@@ -21,19 +21,11 @@ from openviking_cli.utils.config import OPENVIKING_CONFIG_ENV
def main():
- # Intercept subcommands that don't need the server.
- if len(sys.argv) > 1 and sys.argv[1] == "init":
- from openviking_cli.setup_wizard import main as init_main
-
- sys.exit(init_main())
-
- if len(sys.argv) > 1 and sys.argv[1] == "doctor":
- from openviking_cli.doctor import main as doctor_main
-
- sys.exit(doctor_main())
-
+ """Bootstrap the server while binding a stable execution-level log trace ID."""
# Pre-parse --config from sys.argv before any openviking imports,
# so the env var is visible when the config singleton first initialises.
+ # This is done for all subcommands (init, doctor, server) to ensure
+ # consistent behavior.
for i, arg in enumerate(sys.argv):
if arg == "--config" and i + 1 < len(sys.argv):
os.environ[OPENVIKING_CONFIG_ENV] = sys.argv[i + 1]
@@ -42,9 +34,27 @@ def main():
os.environ[OPENVIKING_CONFIG_ENV] = arg.split("=", 1)[1]
break
+ # Import after config pre-parse to avoid early config singleton initialization via
+ # module-level loggers.
+ from openviking_cli.utils.logger import bind_log_execution_trace # noqa: PLC0415
+
+ # Intercept subcommands that don't need the server.
+ if len(sys.argv) > 1 and sys.argv[1] == "init":
+ from openviking_cli.setup_wizard import main as init_main
+
+ with bind_log_execution_trace():
+ sys.exit(init_main())
+
+ if len(sys.argv) > 1 and sys.argv[1] == "doctor":
+ from openviking_cli.doctor import main as doctor_main
+
+ with bind_log_execution_trace():
+ sys.exit(doctor_main())
+
from openviking.server.bootstrap import main as _real_main
- _real_main()
+ with bind_log_execution_trace():
+ _real_main()
if __name__ == "__main__":
diff --git a/openviking_cli/utils/logger.py b/openviking_cli/utils/logger.py
index 629bb7d86..cd5e0235e 100644
--- a/openviking_cli/utils/logger.py
+++ b/openviking_cli/utils/logger.py
@@ -6,9 +6,586 @@ Logging utilities for OpenViking.
import logging
import sys
+from contextlib import contextmanager
from logging.handlers import TimedRotatingFileHandler
from pathlib import Path
from typing import Any, Optional, Tuple
+from uuid import uuid4
+
+from openviking.observability.context import (
+ bind_execution_context,
+ get_observability_context,
+)
+
+# Try to import opentelemetry - will be None if not installed
+try:
+ from opentelemetry import trace as otel_trace
+ from opentelemetry._logs import set_logger_provider
+ from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
+ from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
+ from opentelemetry.sdk.resources import Resource
+ from opentelemetry.trace import Status, StatusCode, format_span_id, format_trace_id
+
+ # Try to import the gRPC exporter.
+ try:
+ from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
+ OTLPLogExporter as OTLPGrpcLogExporter,
+ )
+ except ImportError:
+ OTLPGrpcLogExporter = None
+
+ # Try to import the HTTP exporter.
+ try:
+ from opentelemetry.exporter.otlp.proto.http._log_exporter import (
+ OTLPLogExporter as OTLPHttpLogExporter,
+ )
+ except ImportError:
+ OTLPHttpLogExporter = None
+except ImportError:
+ otel_trace = None
+ set_logger_provider = None
+ LoggerProvider = None
+ LoggingHandler = None
+ BatchLogRecordProcessor = None
+ OTLPGrpcLogExporter = None
+ OTLPHttpLogExporter = None
+ Resource = None
+ Status = None
+ StatusCode = None
+ format_span_id = None
+ format_trace_id = None
+
+
+# Global OTel log handler state
+_otel_log_handler_initialized = False
+_otel_log_handler: Any = None
+
+
+def _get_log_context() -> dict[str, Any]:
+ """
+ Build the log context dynamically for the current context.
+
+ This function builds the log context from scratch on each call, ensuring
+ that any context changes (e.g., account_id updated after authentication)
+ are immediately reflected in subsequent log records.
+
+ Context sources (in priority order):
+ 1. OTel current span (for trace_id, span_id)
+ 2. Root observability context (for request_id, account_id, etc.)
+ 3. Operation observability context (for operation, telemetry_id, etc.)
+ 4. Execution-level trace context (fallback when no OTel span)
+
+ Returns:
+ A dictionary containing all log context fields.
+ """
+ context: dict[str, Any] = {
+ "trace_id": "",
+ "span_id": "",
+ "request_id": "",
+ "operation": "",
+ "account_id": "",
+ "user_id": "",
+ "agent_id": "",
+ }
+
+ # 1. Get trace_id and span_id from OTel span if available
+ if otel_trace is not None and format_trace_id is not None and format_span_id is not None:
+ try:
+ current_span = otel_trace.get_current_span()
+ if current_span is not None and hasattr(current_span, "context"):
+ span_context = current_span.get_span_context()
+ if span_context.is_valid:
+ context["trace_id"] = format_trace_id(span_context.trace_id)
+ context["span_id"] = format_span_id(span_context.span_id)
+ except Exception:
+ # Best-effort only - don't fail if OTel context retrieval fails
+ pass
+
+ # 2. Get unified observability context (contains root and operation)
+ obs_ctx = get_observability_context()
+
+ # Get fields from root context
+ if obs_ctx.root is not None:
+ root_fields = obs_ctx.root.to_log_fields()
+ for key, value in root_fields.items():
+ if value is not None:
+ context[key] = value
+
+ # Get fields from operation context
+ if obs_ctx.operation is not None:
+ operation_fields = obs_ctx.operation.to_log_fields()
+ for key, value in operation_fields.items():
+ if value is not None:
+ context[key] = value
+
+ # 3. Fall back to execution-level trace context if no OTel span
+ if not context["trace_id"]:
+ context["trace_id"] = obs_ctx.get_trace_id()
+ if not context["span_id"]:
+ context["span_id"] = obs_ctx.get_span_id()
+
+ return context
+
+
+def set_log_execution_trace_id(trace_id: Optional[str] = None) -> str:
+ """
+ Set the execution-level trace ID used by logs outside real OTel spans.
+
+ This function uses the unified ObservabilityContext to store the trace ID.
+
+ Args:
+ trace_id: Optional 32-character lowercase hex trace ID. When omitted, a new
+ trace ID is generated automatically.
+
+ Returns:
+ The effective execution trace ID stored in the current context.
+ """
+ ctx = get_observability_context()
+ effective_trace_id = trace_id or uuid4().hex
+ ctx.execution_trace_id = effective_trace_id
+ return effective_trace_id
+
+
+def set_log_execution_span_id(span_id: Optional[str] = None) -> str:
+ """
+ Set the execution-level span ID used by logs outside real OTel spans.
+
+ This function uses the unified ObservabilityContext to store the span ID.
+
+ Args:
+ span_id: Optional 16-character lowercase hex span ID. When omitted, a new
+ span ID is generated automatically.
+
+ Returns:
+ The effective execution span ID stored in the current context.
+ """
+ ctx = get_observability_context()
+ effective_span_id = span_id or uuid4().hex[:16]
+ ctx.execution_span_id = effective_span_id
+ return effective_span_id
+
+
+def get_effective_trace_id() -> str:
+ """
+ Return the current execution-level trace ID, or an empty string when unset.
+
+ This function uses the unified ObservabilityContext to retrieve the trace ID.
+ It also checks the OTel current span for a valid trace ID.
+
+ Returns:
+ The effective trace ID, or empty string if not available.
+ """
+ ctx = get_observability_context()
+ return ctx.get_trace_id()
+
+
+def get_effective_span_id() -> str:
+ """
+ Return the current execution-level span ID, or an empty string when unset.
+
+ This function uses the unified ObservabilityContext to retrieve the span ID.
+ It also checks the OTel current span for a valid span ID.
+
+ Returns:
+ The effective span ID, or empty string if not available.
+ """
+ ctx = get_observability_context()
+ return ctx.get_span_id()
+
+
+def clear_log_execution_trace_id() -> None:
+ """
+ Clear the execution-level trace ID from the current logging context.
+
+ This function clears the execution trace ID and span ID from the
+ unified ObservabilityContext.
+ """
+ ctx = get_observability_context()
+ ctx.execution_trace_id = None
+ ctx.execution_span_id = None
+
+
+@contextmanager
+def bind_log_execution_trace(trace_id: Optional[str] = None, span_id: Optional[str] = None):
+ """
+ Bind a stable execution-level trace ID and span ID for logs emitted in this context.
+
+ This context manager is intended for non-request entry points, such as process startup,
+ CLI execution, or background tasks that do not run under a real OTel request span.
+
+ This is a convenience wrapper around the unified `bind_execution_context` from
+ the observability context module.
+
+ Args:
+ trace_id: Optional 32-character lowercase hex trace ID. When omitted, a new
+ trace ID is generated automatically for this execution unit.
+ span_id: Optional 16-character lowercase hex span ID. When omitted, the first 16
+ characters of the trace_id are used as the span_id.
+
+ Yields:
+ A tuple of (trace_id, span_id).
+ """
+ with bind_execution_context(trace_id, span_id) as (effective_trace_id, effective_span_id):
+ yield effective_trace_id, effective_span_id
+
+
+def init_otel_log_handler(
+ protocol: str = "grpc", # "grpc" or "http"
+ endpoint: str = "localhost:4317", # gRPC default: 4317; HTTP default: 4318
+ service_name: str = "openviking-server",
+ insecure: bool = False,
+ enabled: bool = True,
+) -> Any:
+ """Initialize OTel LoggingHandler.
+
+ Args:
+ protocol: OTLP protocol ("grpc" or "http").
+ endpoint: OTLP logs endpoint.
+ service_name: Service name.
+ insecure: For OTLP/gRPC only. When True, use plaintext instead of TLS.
+ enabled: Whether to enable the handler.
+
+ Returns:
+ The initialized LoggingHandler instance, or None if initialization fails.
+ """
+ global _otel_log_handler_initialized, _otel_log_handler
+
+ if not enabled:
+ return None
+
+ if _otel_log_handler_initialized:
+ return _otel_log_handler
+
+ # Validate protocol
+ protocol = protocol.lower()
+ if protocol not in ["grpc", "http"]:
+ protocol = "grpc"
+
+ if (
+ set_logger_provider is None
+ or LoggerProvider is None
+ or LoggingHandler is None
+ or BatchLogRecordProcessor is None
+ or Resource is None
+ ):
+ return None
+
+ try:
+ # Create resource
+ resource = Resource.create(
+ {
+ "service.name": service_name,
+ }
+ )
+
+ # Create logger provider
+ logger_provider = LoggerProvider(resource=resource)
+
+ # Select exporter based on protocol
+ otlp_exporter = None
+ if protocol == "grpc":
+ if OTLPGrpcLogExporter is None:
+ raise ImportError("gRPC OTLP log exporter not available")
+ try:
+ otlp_exporter = OTLPGrpcLogExporter(endpoint=endpoint, insecure=insecure)
+ except TypeError:
+ otlp_exporter = OTLPGrpcLogExporter(endpoint=endpoint)
+ elif protocol == "http":
+ if OTLPHttpLogExporter is None:
+ raise ImportError("HTTP OTLP log exporter not available")
+ # HTTP endpoint must be a full URL (with scheme). Do not auto-prefix to avoid implicit behavior.
+ if not endpoint.startswith("http://") and not endpoint.startswith("https://"):
+ raise ValueError(
+ "OTLP/HTTP endpoint must include scheme, e.g. 'http://localhost:4318/v1/logs'"
+ )
+ otlp_exporter = OTLPHttpLogExporter(endpoint=endpoint)
+
+ if otlp_exporter is None:
+ raise ValueError(f"Failed to create log exporter for protocol={protocol}")
+
+ # Create batch log processor
+ log_processor = BatchLogRecordProcessor(otlp_exporter)
+ logger_provider.add_log_record_processor(log_processor)
+
+ # Set global logger provider
+ set_logger_provider(logger_provider)
+
+ # Create logging handler
+ _otel_log_handler = LoggingHandler(
+ level=logging.INFO,
+ logger_provider=logger_provider,
+ )
+
+ _otel_log_handler_initialized = True
+
+ logger = get_logger(__name__)
+ logger.info(
+ "[OTelLogHandler] initialized with protocol=%s, endpoint=%s, service_name=%s",
+ protocol,
+ endpoint,
+ service_name,
+ )
+
+ return _otel_log_handler
+ except Exception as e:
+ logger = get_logger(__name__)
+ logger.warning("[OTelLogHandler] initialization failed: %s", e)
+ return None
+
+
+def attach_otel_log_handler_to_existing_loggers() -> None:
+ """Attach the global OTel log handler to already-created loggers."""
+ if _otel_log_handler is None:
+ return
+
+ add_otel_log_handler_to_logger(logging.getLogger())
+
+ for logger_obj in logging.Logger.manager.loggerDict.values():
+ if isinstance(logger_obj, logging.Logger):
+ add_otel_log_handler_to_logger(logger_obj)
+
+
+def init_otel_log_handler_from_server_config(server_config: Any) -> Any:
+ """Initialize OTLP log export from server.observability.logs."""
+ logs_cfg = server_config.observability.logs
+ if not logs_cfg.enabled:
+ return None
+
+ handler = init_otel_log_handler(
+ protocol=logs_cfg.protocol,
+ endpoint=logs_cfg.endpoint,
+ service_name=logs_cfg.service_name,
+ insecure=logs_cfg.tls.insecure,
+ enabled=logs_cfg.enabled,
+ )
+ attach_otel_log_handler_to_existing_loggers()
+ return handler
+
+
+def add_otel_log_handler_to_logger(logger: logging.Logger) -> None:
+ """Add OTel LoggingHandler to the specified logger.
+
+ Args:
+ logger: The logger instance to add the handler to.
+ """
+ if _otel_log_handler is not None:
+ # Check if the handler has already been added
+ if not any(isinstance(h, type(_otel_log_handler)) for h in logger.handlers):
+ logger.addHandler(_otel_log_handler)
+
+
+class TraceContextFilter(logging.Filter):
+ """
+ Log filter that injects OTel trace context information.
+
+ Automatically extracts trace_id, span_id, and other fields from the current span
+ and injects them into log records, enabling automatic correlation between logs and traces.
+
+ This filter builds the context dynamically on each call to ensure that any context
+ changes (e.g., account_id updated after authentication) are immediately reflected
+ in subsequent log records.
+ """
+
+ def filter(self, record: logging.LogRecord) -> bool:
+ """
+ Inject trace context information into the log record.
+
+ Builds the context dynamically on each call to ensure freshness.
+
+ Args:
+ record: The log record object.
+
+ Returns:
+ bool: Always returns True, indicating the log should be recorded.
+ """
+ # Build context dynamically to ensure freshness
+ context = _get_log_context()
+
+ # Apply context fields to the log record
+ record.trace_id = context.get("trace_id", "")
+ record.span_id = context.get("span_id", "")
+ record.request_id = context.get("request_id", "")
+ record.operation = context.get("operation", "")
+ record.account_id = context.get("account_id", "")
+ record.user_id = context.get("user_id", "")
+ record.agent_id = context.get("agent_id", "")
+
+ return True
+
+
+class LogToSpanEventFilter(logging.Filter):
+ """
+ Log filter that automatically maps log records to OTel span events.
+
+ This filter allows developers to only write log.xxx() calls, and the logs
+ will automatically be recorded as span events when OTel is available.
+ This eliminates the need to write duplicate code for both logging and tracing.
+
+ Key features:
+ - Logs are automatically added as span events when a span is active
+ - Exceptions are automatically recorded with record_exception()
+ - Extra fields from log records are included as span attributes
+ - Best-effort only: never breaks logging if OTel is not available
+
+ Usage:
+ handler.addFilter(LogToSpanEventFilter(
+ min_level=logging.INFO,
+ excluded_loggers={"uvicorn.access", "uvicorn.error"}
+ ))
+ """
+
+ def __init__(
+ self,
+ name: str = "",
+ min_level: int = logging.INFO,
+ excluded_loggers: Optional[set] = None,
+ ):
+ """
+ Initialize the LogToSpanEventFilter.
+
+ Args:
+ name: Filter name.
+ min_level: Minimum log level to be mapped to span events.
+ Logs below this level will only be recorded, not added as span events.
+ excluded_loggers: Set of logger names to exclude from span event mapping.
+ Useful for excluding noisy loggers like access logs.
+ """
+ super().__init__(name)
+ self._min_level = min_level
+ self._excluded_loggers = excluded_loggers or set()
+
+ def _extract_attributes(self, record: logging.LogRecord) -> dict:
+ """
+ Extract span attributes from a log record.
+
+ Extracts standard fields and extra fields from the log record
+ to be used as span event attributes.
+
+ Args:
+ record: The log record object.
+
+ Returns:
+ Dictionary of attributes to be included in the span event.
+ """
+ attributes = {
+ "log.level": record.levelname,
+ "log.logger": record.name,
+ }
+
+ # Add location information if available
+ if record.pathname:
+ attributes["code.filepath"] = record.pathname
+ if record.lineno:
+ attributes["code.lineno"] = record.lineno
+ if record.funcName:
+ attributes["code.function"] = record.funcName
+
+ # Add extra fields from the log record
+ # Standard fields to exclude from extra attributes
+ standard_fields = {
+ "name",
+ "msg",
+ "args",
+ "levelname",
+ "levelno",
+ "pathname",
+ "filename",
+ "module",
+ "exc_info",
+ "exc_text",
+ "stack_info",
+ "lineno",
+ "funcName",
+ "created",
+ "msecs",
+ "relativeCreated",
+ "thread",
+ "threadName",
+ "processName",
+ "process",
+ "message",
+ "asctime",
+ "trace_id",
+ "span_id",
+ "request_id",
+ "operation",
+ "account_id",
+ "user_id",
+ "agent_id",
+ }
+
+ for key, value in record.__dict__.items():
+ if key not in standard_fields and not key.startswith("_"):
+ # Convert to OTel-compatible types
+ if isinstance(value, (str, int, float, bool)):
+ attributes[key] = value
+ elif value is not None:
+ attributes[key] = str(value)
+
+ return attributes
+
+ def filter(self, record: logging.LogRecord) -> bool:
+ """
+ Map log record to span event (best-effort).
+
+ This method is called for each log record. It:
+ 1. Checks if the logger should be excluded
+ 2. Checks if the log level is above the minimum threshold
+ 3. If OTel is available and a span is active, adds the log as a span event
+ 4. For exceptions, also records the exception with record_exception()
+
+ Args:
+ record: The log record object.
+
+ Returns:
+ bool: Always returns True, indicating the log should be recorded.
+ """
+ # Check if this logger should be excluded
+ if record.name in self._excluded_loggers:
+ return True
+
+ # Check if log level is below the minimum threshold for span events
+ if record.levelno < self._min_level:
+ return True
+
+ # Check if OTel is available
+ if otel_trace is None:
+ return True
+
+ try:
+ # Get the current span
+ current_span = otel_trace.get_current_span()
+ if current_span is None or not current_span.is_recording():
+ return True
+
+ # Extract attributes from the log record
+ attributes = self._extract_attributes(record)
+
+ # Handle exceptions specially
+ if record.exc_info is not None:
+ exc_type, exc_value, exc_tb = record.exc_info
+ if exc_value is not None:
+ # Record the exception in the span
+ current_span.record_exception(exc_value, attributes=attributes)
+
+ # Set span status to ERROR if Status/StatusCode are available
+ if Status is not None and StatusCode is not None:
+ try:
+ current_span.set_status(
+ Status(StatusCode.ERROR, description=str(exc_value))
+ )
+ except Exception:
+ pass
+ else:
+ # Add as a regular span event
+ event_name = record.getMessage() or "log"
+ current_span.add_event(event_name, attributes=attributes)
+
+ except Exception:
+ # Best-effort only: never break logging
+ pass
+
+ return True
def _load_log_config() -> Tuple[str, str, str, Optional[Any]]:
@@ -76,7 +653,18 @@ def _create_log_handler(log_output: str, config: Optional[Any]) -> logging.Handl
def get_logger(
name: str = "openviking",
format_string: Optional[str] = None,
+ add_otel_handler: bool = False,
) -> logging.Logger:
+ """Get a configured logger instance.
+
+ Args:
+ name: Logger name, defaults to "openviking".
+ format_string: Custom log format string, optional.
+ add_otel_handler: Whether to add OTel LoggingHandler, defaults to False.
+
+ Returns:
+ logging.Logger: Configured logger instance with automatic trace context injection.
+ """
logger = logging.getLogger(name)
if not logger.handlers:
@@ -88,10 +676,18 @@ def get_logger(
format_string = log_format
formatter = logging.Formatter(format_string)
handler.setFormatter(formatter)
+
+ # Add unified trace context filter
+ handler.addFilter(TraceContextFilter())
+
logger.addHandler(handler)
logger.propagate = False
logger.setLevel(level)
+ # If OTel log export is globally initialized, attach the handler automatically.
+ if add_otel_handler or _otel_log_handler_initialized:
+ add_otel_log_handler_to_logger(logger)
+
return logger
@@ -100,10 +696,10 @@ default_logger = get_logger()
def configure_uvicorn_logging() -> None:
- """Configure Uvicorn loggers to use OpenViking's logging configuration.
+ """Configure Uvicorn logging to use OpenViking's logging configuration.
- This function configures the 'uvicorn', 'uvicorn.error', and 'uvicorn.access'
- loggers to use the same handlers and format as our openviking loggers.
+ Configures the 'uvicorn', 'uvicorn.error', and 'uvicorn.access' loggers
+ to use the same handlers, format, and trace context injection as openviking logs.
"""
log_level_str, log_format, log_output, config = _load_log_config()
level = getattr(logging, log_level_str, logging.INFO)
@@ -111,6 +707,9 @@ def configure_uvicorn_logging() -> None:
formatter = logging.Formatter(log_format)
handler.setFormatter(formatter)
+ # Add unified trace context filter
+ handler.addFilter(TraceContextFilter())
+
# Configure all Uvicorn loggers
uvicorn_logger_names = ["uvicorn", "uvicorn.error", "uvicorn.access"]
for logger_name in uvicorn_logger_names:
diff --git a/pyproject.toml b/pyproject.toml
index 089bcef9a..759e7573d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -75,6 +75,7 @@ dependencies = [
"opentelemetry-api>=1.14",
"opentelemetry-sdk>=1.14",
"opentelemetry-exporter-otlp-proto-grpc>=1.14",
+ "opentelemetry-exporter-otlp-proto-http>=1.14",
"opentelemetry-instrumentation-asyncio>=0.61b0",
"loguru>=0.7.3",
"cryptography>=42.0.0",
diff --git a/tests/api_test/tools/tests/test_logger.py b/tests/api_test/tools/tests/test_logger.py
index bd100f12a..40049cd93 100644
--- a/tests/api_test/tools/tests/test_logger.py
+++ b/tests/api_test/tools/tests/test_logger.py
@@ -1,24 +1,138 @@
import logging
+from openviking.observability.context import (
+ bind_operation_observability_context,
+ bind_root_observability_context,
+ reset_operation_observability_context,
+ reset_root_observability_context,
+)
+from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributes
from openviking_cli.utils import get_logger
+from openviking_cli.utils.logger import TraceContextFilter, bind_log_execution_trace
-print("Testing logger...")
-print("=" * 80)
-logger = get_logger(__name__)
-print(f"Logger: {logger}")
-print(f"Logger level: {logger.level}")
+def test_trace_context_filter_reuses_execution_trace_id_without_active_span():
+ """TraceContextFilter should fall back to execution-trace context when no span exists.
-print(f"\nRoot logger level: {logging.getLogger().level}")
+ The span_id is derived from the first 16 characters of the trace_id.
+ """
+ record = logging.LogRecord(
+ name="test",
+ level=logging.INFO,
+ pathname=__file__,
+ lineno=1,
+ msg="hello",
+ args=(),
+ exc_info=None,
+ )
-print("\nTesting log messages:")
-logger.debug("Debug message")
-logger.info("Info message")
-logger.warning("Warning message")
-logger.error("Error message")
+ with bind_log_execution_trace("1234567890abcdef1234567890abcdef"):
+ passed = TraceContextFilter().filter(record)
-print("\n" + "=" * 80)
-print("Checking openviking.server.app logger...")
+ assert passed is True
+ assert record.trace_id == "1234567890abcdef1234567890abcdef"
+ # span_id is derived from first 16 chars of trace_id
+ assert record.span_id == "1234567890abcdef"
-app_logger = get_logger("openviking.server.app")
-print(f"App logger level: {app_logger.level}")
+
+def test_trace_context_filter_generates_span_id_from_trace_id():
+ """TraceContextFilter should generate span_id from trace_id when not explicitly provided."""
+ record = logging.LogRecord(
+ name="test",
+ level=logging.INFO,
+ pathname=__file__,
+ lineno=1,
+ msg="hello",
+ args=(),
+ exc_info=None,
+ )
+
+ # Test with explicitly provided span_id
+ with bind_log_execution_trace(
+ trace_id="1234567890abcdef1234567890abcdef",
+ span_id="fedcba0987654321",
+ ):
+ passed = TraceContextFilter().filter(record)
+
+ assert passed is True
+ assert record.trace_id == "1234567890abcdef1234567890abcdef"
+ assert record.span_id == "fedcba0987654321"
+
+
+def test_trace_context_filter_injects_root_observability_fields_without_span():
+ record = logging.LogRecord(
+ name="test",
+ level=logging.INFO,
+ pathname=__file__,
+ lineno=1,
+ msg="hello",
+ args=(),
+ exc_info=None,
+ )
+ root = RootSpanAttributes(http_method="GET", http_route="/demo", request_id="req-1")
+ root.account_id = "acct-1"
+ root.user_id = "user-1"
+ root.agent_id = "agent-1"
+ token = bind_root_observability_context(root)
+ try:
+ passed = TraceContextFilter().filter(record)
+ finally:
+ reset_root_observability_context(token)
+
+ assert passed is True
+ assert record.request_id == "req-1"
+ assert record.account_id == "acct-1"
+ assert record.user_id == "user-1"
+ assert record.agent_id == "agent-1"
+
+
+def test_trace_context_filter_injects_operation_observability_fields_without_span():
+ record = logging.LogRecord(
+ name="test",
+ level=logging.INFO,
+ pathname=__file__,
+ lineno=1,
+ msg="hello",
+ args=(),
+ exc_info=None,
+ )
+ operation = OperationSpanAttributes(operation="search.find", telemetry_id="tm-1")
+ operation.status = "ok"
+ token = bind_operation_observability_context(operation)
+ try:
+ passed = TraceContextFilter().filter(record)
+ finally:
+ reset_operation_observability_context(token)
+
+ assert passed is True
+ assert record.operation == "search.find"
+ assert record.telemetry_id == "tm-1"
+ assert record.status == "ok"
+
+
+def _manual_smoke_check() -> None:
+ """Run a manual logger smoke check when this module is executed directly."""
+ print("Testing logger...")
+ print("=" * 80)
+
+ logger = get_logger(__name__)
+ print(f"Logger: {logger}")
+ print(f"Logger level: {logger.level}")
+
+ print(f"\nRoot logger level: {logging.getLogger().level}")
+
+ print("\nTesting log messages:")
+ logger.debug("Debug message")
+ logger.info("Info message")
+ logger.warning("Warning message")
+ logger.error("Error message")
+
+ print("\n" + "=" * 80)
+ print("Checking openviking.server.app logger...")
+
+ app_logger = get_logger("openviking.server.app")
+ print(f"App logger level: {app_logger.level}")
+
+
+if __name__ == "__main__":
+ _manual_smoke_check()
diff --git a/tests/metrics/core/test_exporter.py b/tests/metrics/core/test_exporter.py
index 1782f0ce8..459ad300a 100644
--- a/tests/metrics/core/test_exporter.py
+++ b/tests/metrics/core/test_exporter.py
@@ -1,6 +1,12 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+import asyncio
+
+import pytest
+
+from openviking.metrics.exporters.otel import OTelMetricExporter, _compute_next_tick_and_sleep
+
def test_exporter_outputs_help_type_and_empty_metrics(registry, render_prometheus):
registry.counter("openviking_empty_counter_total")
@@ -12,3 +18,80 @@ def test_exporter_outputs_help_type_and_empty_metrics(registry, render_prometheu
assert "# TYPE openviking_empty_histogram_seconds histogram" in text
assert "openviking_empty_histogram_seconds_count 0" in text
assert 'openviking_empty_histogram_seconds_bucket{le="+Inf"} 0' in text
+
+
+async def test_otel_exporter_start_and_shutdown_manage_background_loop(registry, monkeypatch):
+ """OTel exporter should own its periodic export loop lifecycle."""
+ exporter = OTelMetricExporter(registry=registry, enabled=False, export_interval_ms=10)
+ exporter._enabled = True
+
+ calls = 0
+ tick = asyncio.Event()
+
+ async def _fake_export():
+ nonlocal calls
+ calls += 1
+ tick.set()
+ return ""
+
+ monkeypatch.setattr(exporter, "export", _fake_export)
+
+ exporter.start()
+ await asyncio.wait_for(tick.wait(), timeout=0.2)
+ assert calls >= 1
+ assert exporter._export_task is not None
+
+ task_before = exporter._export_task
+ exporter.start()
+ assert exporter._export_task is task_before
+
+ await exporter.shutdown()
+ assert exporter._export_task is None
+
+
+async def test_otel_export_loop_aligns_to_next_fixed_tick_after_overrun():
+ """Scheduling helper should advance next tick and compute sleep after one interval overrun."""
+ next_tick, sleep_s = _compute_next_tick_and_sleep(now=1.15, interval_s=1.0, next_tick=1.0)
+ assert next_tick == pytest.approx(2.0)
+ assert sleep_s == pytest.approx(0.85)
+
+
+def test_otel_http_send_uses_export_timeout_not_refresh_deadline(registry):
+ """HTTP transport timeout should be decoupled from collector refresh timeout."""
+ exporter = OTelMetricExporter(
+ registry=registry,
+ enabled=False,
+ protocol="http",
+ endpoint="http://localhost:4318/v1/metrics",
+ refresh_deadline_seconds=1.0,
+ )
+
+ class _FakeSession:
+ def __init__(self):
+ self.timeout = None
+
+ def post(self, _url, **kwargs):
+ self.timeout = kwargs.get("timeout")
+
+ class _Resp:
+ @staticmethod
+ def raise_for_status():
+ return None
+
+ return _Resp()
+
+ fake_session = _FakeSession()
+ exporter._http_session = fake_session
+
+ request = exporter._build_export_request()
+ assert request is None
+
+ from opentelemetry.proto.collector.metrics.v1.metrics_service_pb2 import (
+ ExportMetricsServiceRequest,
+ )
+
+ exporter._send_http_request(ExportMetricsServiceRequest())
+
+ assert fake_session.timeout == 5.0
diff --git a/tests/metrics/integration/test_account_dimension.py b/tests/metrics/integration/test_account_dimension.py
index 2cfc52640..1e9d4d0f7 100644
--- a/tests/metrics/integration/test_account_dimension.py
+++ b/tests/metrics/integration/test_account_dimension.py
@@ -4,15 +4,13 @@
from __future__ import annotations
import asyncio
-from types import SimpleNamespace
import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+from starlette.requests import Request
-from openviking.metrics.account_context import (
- bind_metric_account_context,
- get_metric_account_context,
- reset_metric_account_context,
-)
+import openviking.observability.http_observability_middleware as http_middleware
from openviking.metrics.account_dimension import (
configure_metric_account_dimension,
reset_metric_account_dimension,
@@ -23,18 +21,43 @@ from openviking.metrics.collectors.http import HTTPCollector
from openviking.metrics.core.registry import MetricRegistry
from openviking.metrics.datasources.base import EventMetricDataSource
from openviking.metrics.exporters.prometheus import PrometheusExporter
-from openviking.metrics.http_middleware import create_http_metrics_middleware
+from openviking.observability.context import (
+ bind_root_observability_context,
+ get_root_observability_context,
+ reset_root_observability_context,
+)
+from openviking.observability.http_observability_middleware import (
+ create_http_observability_middleware,
+)
+from openviking.telemetry.span_models import RootSpanAttributes
+
+
+def _bind_root_context_for_account(account_id: str | None):
+ root = RootSpanAttributes(http_method="GET", http_route="/test", request_id="req-test")
+ root.account_id = account_id
+ return bind_root_observability_context(root)
+
+
+def _build_test_app(middleware_factory, *, route_path: str, handler):
+ app = FastAPI()
+
+ @app.middleware("http")
+ async def middleware_entry(request, call_next):
+ return await middleware_factory(request, call_next)
+
+ app.get(route_path)(handler)
+ return app
def test_metric_account_context_binds_and_resets_account_id():
- """Binding a context must be observable via `get_metric_account_context()` and resettable."""
- token = bind_metric_account_context(account_id="acct_123")
+ """Binding a root context must be observable and resettable."""
+ token = _bind_root_context_for_account("acct_123")
try:
- assert get_metric_account_context().http_account_id == "acct_123"
+ assert get_root_observability_context().account_id == "acct_123"
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
- assert get_metric_account_context().http_account_id is None
+ assert get_root_observability_context() is None
@pytest.mark.asyncio
@@ -42,12 +65,13 @@ async def test_metric_account_context_is_isolated_per_task():
"""Two concurrent tasks must not leak metric account context into each other."""
async def _worker(account_id: str, seen: list[str | None]):
- token = bind_metric_account_context(account_id=account_id)
+ token = _bind_root_context_for_account(account_id)
try:
await asyncio.sleep(0)
- seen.append(get_metric_account_context().http_account_id)
+ root_context = get_root_observability_context()
+ seen.append(root_context.account_id if root_context is not None else None)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
seen: list[str | None] = []
await asyncio.gather(_worker("acct_a", seen), _worker("acct_b", seen))
@@ -86,34 +110,30 @@ def test_http_collector_uses_unknown_account_when_unbound():
)
-async def test_http_metrics_middleware_propagates_state_account_to_collector(monkeypatch):
+def test_http_metrics_middleware_propagates_state_account_to_collector(monkeypatch):
"""HTTP middleware must attach the authenticated account id to emitted http events."""
captured: list[tuple[str, dict]] = []
- middleware = create_http_metrics_middleware()
+ middleware = create_http_observability_middleware()
def _fake_emit(event_name: str, payload: dict) -> None:
captured.append((event_name, dict(payload)))
monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)
+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)
- async def _call_next(request):
- request.state.metric_account_id = "acct-mdw"
- return SimpleNamespace(status_code=200)
-
- from starlette.requests import Request
+ async def resources_handler(request: Request):
+ request.state.root_span_attrs.account_id = "acct-mdw"
+ return {"ok": True}
- request = Request(
- {
- "type": "http",
- "method": "GET",
- "path": "/api/v1/resources",
- "route": SimpleNamespace(path="/api/v1/resources"),
- "headers": [],
- "state": {},
- }
+ app = _build_test_app(
+ middleware,
+ route_path="/api/v1/resources",
+ handler=resources_handler,
)
- await middleware(request, _call_next)
+
+ with TestClient(app) as client:
+ resp = client.get("/api/v1/resources")
+ assert resp.status_code == 200
assert any(
event_name == "http.request" and payload.get("account_id") == "acct-mdw"
@@ -129,7 +149,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_exact():
metric_allowlist={EmbeddingCollector.CALLS_TOTAL},
max_active_accounts=10,
)
- token = bind_metric_account_context(account_id="acct-embed-1")
+ token = _bind_root_context_for_account("acct-embed-1")
try:
EmbeddingCollector().receive(
"embedding.call",
@@ -143,7 +163,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_exact():
registry,
)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
reset_metric_account_dimension()
text = PrometheusExporter(registry=registry).render()
@@ -158,7 +178,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_by_prefix(
metric_allowlist={"openviking_embedding_*"},
max_active_accounts=10,
)
- token = bind_metric_account_context(account_id="acct-embed-2")
+ token = _bind_root_context_for_account("acct-embed-2")
try:
EmbeddingCollector().receive(
"embedding.success",
@@ -166,7 +186,7 @@ def test_account_dimension_injects_account_id_when_metric_allowlisted_by_prefix(
registry,
)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
reset_metric_account_dimension()
text = PrometheusExporter(registry=registry).render()
@@ -186,7 +206,7 @@ def test_account_dimension_returns_overflow_when_max_active_accounts_exceeded():
)
try:
for account_id in ("acct-a", "acct-b", "acct-c"):
- token = bind_metric_account_context(account_id=account_id)
+ token = _bind_root_context_for_account(account_id)
try:
EmbeddingCollector().receive(
"embedding.call",
@@ -200,7 +220,7 @@ def test_account_dimension_returns_overflow_when_max_active_accounts_exceeded():
registry,
)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
finally:
reset_metric_account_dimension()
@@ -217,7 +237,7 @@ def test_account_dimension_returns_unknown_for_not_allowlisted_metric_even_with_
metric_allowlist={HTTPCollector.REQUESTS_TOTAL},
max_active_accounts=10,
)
- token = bind_metric_account_context(account_id="acct-x")
+ token = _bind_root_context_for_account("acct-x")
try:
EmbeddingCollector().receive(
"embedding.success",
@@ -225,7 +245,7 @@ def test_account_dimension_returns_unknown_for_not_allowlisted_metric_even_with_
registry,
)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
reset_metric_account_dimension()
text = PrometheusExporter(registry=registry).render()
@@ -250,11 +270,11 @@ def test_account_dimension_does_not_inject_account_id_for_unsupported_metric_nam
metric_allowlist={"openviking_unsupported_total"},
max_active_accounts=10,
)
- token = bind_metric_account_context(account_id="acct-z")
+ token = _bind_root_context_for_account("acct-z")
try:
DummyCollector().receive("demo", {}, registry)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
reset_metric_account_dimension()
text = PrometheusExporter(registry=registry).render()
diff --git a/tests/metrics/integration/test_http_metrics.py b/tests/metrics/integration/test_http_metrics.py
index 1af2b5334..42c877dfd 100644
--- a/tests/metrics/integration/test_http_metrics.py
+++ b/tests/metrics/integration/test_http_metrics.py
@@ -3,16 +3,11 @@
from __future__ import annotations
-from types import SimpleNamespace
-
-import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
from starlette.requests import Request
-import openviking.metrics.http_middleware as http_middleware
-from openviking.metrics.account_context import (
- bind_metric_account_context,
- reset_metric_account_context,
-)
+import openviking.observability.http_observability_middleware as http_middleware
from openviking.metrics.collectors.http import HTTPCollector
from openviking.metrics.core.registry import MetricRegistry
from openviking.metrics.datasources.base import EventMetricDataSource
@@ -20,14 +15,41 @@ from openviking.metrics.datasources.http import HttpRequestLifecycleDataSource
from openviking.metrics.datasources.model_usage import VLMEventDataSource
from openviking.metrics.exporters.prometheus import PrometheusExporter
from openviking.metrics.global_api import configure_metric_account_dimension, shutdown_metrics
-from openviking.metrics.http_middleware import (
+from openviking.models.vlm.base import VLMBase
+from openviking.observability.context import (
+ bind_root_observability_context,
+ reset_root_observability_context,
+)
+from openviking.observability.http_observability_middleware import (
_INFLIGHT,
- _extract_request_account_id,
_get_route_template,
_inflight_delta,
- create_http_metrics_middleware,
+ create_http_observability_middleware,
)
-from openviking.models.vlm.base import VLMBase
+from openviking.telemetry.span_models import RootSpanAttributes
+
+
+def _bind_root_context_for_account(account_id: str | None):
+ root = RootSpanAttributes(http_method="GET", http_route="/items", request_id="req-test")
+ root.account_id = account_id
+ return bind_root_observability_context(root)
+
+
+def _build_test_app(middleware_factory, *, route_path: str, method: str, handler):
+ app = FastAPI()
+
+ @app.middleware("http")
+ async def middleware_entry(request, call_next):
+ return await middleware_factory(request, call_next)
+
+ if method == "GET":
+ app.get(route_path)(handler)
+ elif method == "POST":
+ app.post(route_path)(handler)
+ else:
+ raise ValueError(f"unsupported method: {method}")
+
+ return app
def test_http_collector_emits_account_label():
@@ -43,7 +65,7 @@ def test_http_collector_emits_account_label():
)
collector = HTTPCollector()
- token = bind_metric_account_context(account_id="acct-1")
+ token = _bind_root_context_for_account("acct-1")
try:
collector.receive(
"http.request",
@@ -56,7 +78,7 @@ def test_http_collector_emits_account_label():
registry,
)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
shutdown_metrics(app=None)
text = PrometheusExporter(registry=registry).render()
@@ -109,34 +131,6 @@ def test_http_request_datasource_propagates_explicit_account_id(monkeypatch):
]
-def test_extract_request_account_id_prefers_authenticated_request_state():
- request = Request(
- {
- "type": "http",
- "method": "GET",
- "path": "/api/v1/resources",
- "headers": [(b"x-openviking-account", b"header-account")],
- "state": {},
- }
- )
- request.state.metric_account_id = "state-account"
-
- assert _extract_request_account_id(request) == "state-account"
-
-
-def test_extract_request_account_id_does_not_trust_raw_header_when_unauthenticated():
- request = Request(
- {
- "type": "http",
- "method": "GET",
- "path": "/api/v1/resources",
- "headers": [(b"x-openviking-account", b"header-account")],
- "state": {},
- }
- )
- assert _extract_request_account_id(request) is None
-
-
def test_get_route_template_uses_low_cardinality_fallback_for_unmatched_route():
request = Request(
{
@@ -161,32 +155,34 @@ def test_inflight_delta_removes_zero_value_entries():
assert ("/api/v1/resources", None) not in _INFLIGHT
-async def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatch):
+def test_http_metrics_module_exposes_only_unified_middleware_entrypoint():
+ assert not hasattr(http_middleware, "create_http_metrics_middleware")
+
+
+def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatch):
captured: list[tuple[str, dict]] = []
- middleware = create_http_metrics_middleware()
+ middleware = create_http_observability_middleware()
def _fake_emit(event_name: str, payload: dict) -> None:
captured.append((event_name, dict(payload)))
monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)
+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)
- async def _call_next(request: Request):
- request.state.metric_account_id = "acct-real"
- return SimpleNamespace(status_code=200)
+ async def resources_handler(request: Request):
+ request.state.root_span_attrs.account_id = "acct-real"
+ return {"ok": True}
- request = Request(
- {
- "type": "http",
- "method": "POST",
- "path": "/api/v1/resources",
- "route": SimpleNamespace(path="/api/v1/resources"),
- "headers": [],
- "state": {},
- }
+ app = _build_test_app(
+ middleware,
+ route_path="/api/v1/resources",
+ method="POST",
+ handler=resources_handler,
)
- await middleware(request, _call_next)
+ with TestClient(app) as client:
+ resp = client.post("/api/v1/resources")
+ assert resp.status_code == 200
request_events = [payload for event_name, payload in captured if event_name == "http.request"]
assert request_events
@@ -205,98 +201,92 @@ async def test_http_metrics_middleware_emits_authenticated_account_id(monkeypatc
)
-async def test_http_metrics_middleware_ignores_internal_metrics_route(monkeypatch):
+def test_http_metrics_middleware_ignores_internal_metrics_route(monkeypatch):
captured: list[tuple[str, dict]] = []
- middleware = create_http_metrics_middleware()
+ middleware = create_http_observability_middleware()
def _fake_emit(event_name: str, payload: dict) -> None:
captured.append((event_name, dict(payload)))
monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)
+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)
- async def _call_next(_request: Request):
- return SimpleNamespace(status_code=200)
+ async def metrics_handler():
+ return {"ok": True}
- request = Request(
- {
- "type": "http",
- "method": "GET",
- "path": "/metrics",
- "headers": [],
- "state": {},
- }
+ app = _build_test_app(
+ middleware,
+ route_path="/metrics",
+ method="GET",
+ handler=metrics_handler,
)
- await middleware(request, _call_next)
+ with TestClient(app) as client:
+ resp = client.get("/metrics")
+ assert resp.status_code == 200
assert captured == []
-async def test_http_metrics_middleware_still_records_business_route(monkeypatch):
+def test_http_metrics_middleware_still_records_business_route(monkeypatch):
captured: list[tuple[str, dict]] = []
- middleware = create_http_metrics_middleware()
+ middleware = create_http_observability_middleware()
def _fake_emit(event_name: str, payload: dict) -> None:
captured.append((event_name, dict(payload)))
monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)
+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)
- async def _call_next(request: Request):
- request.state.metric_account_id = "acct-real"
- return SimpleNamespace(status_code=200)
+ async def resources_handler(request: Request):
+ request.state.root_span_attrs.account_id = "acct-real"
+ return {"ok": True}
- request = Request(
- {
- "type": "http",
- "method": "POST",
- "path": "/api/v1/resources",
- "route": SimpleNamespace(path="/api/v1/resources"),
- "headers": [],
- "state": {},
- }
+ app = _build_test_app(
+ middleware,
+ route_path="/api/v1/resources",
+ method="POST",
+ handler=resources_handler,
)
- await middleware(request, _call_next)
+ with TestClient(app) as client:
+ resp = client.post("/api/v1/resources")
+ assert resp.status_code == 200
assert any(event_name == "http.request" for event_name, _payload in captured)
-async def test_http_metrics_middleware_uses_route_bound_during_call_next(monkeypatch):
+def test_http_metrics_middleware_uses_route_bound_during_call_next(monkeypatch):
captured: list[tuple[str, dict]] = []
- middleware = create_http_metrics_middleware()
+ middleware = create_http_observability_middleware()
def _fake_emit(event_name: str, payload: dict) -> None:
captured.append((event_name, dict(payload)))
monkeypatch.setattr(EventMetricDataSource, "_emit", staticmethod(_fake_emit), raising=False)
+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)
- async def _call_next(request: Request):
- request.scope["route"] = SimpleNamespace(path="/api/v1/sessions/{session_id}")
- return SimpleNamespace(status_code=200)
+ async def get_session_handler(session_id: str):
+ return {"session_id": session_id}
- request = Request(
- {
- "type": "http",
- "method": "GET",
- "path": "/api/v1/sessions/session_123",
- "headers": [],
- "state": {},
- }
+ app = _build_test_app(
+ middleware,
+ route_path="/api/v1/sessions/{session_id}",
+ method="GET",
+ handler=get_session_handler,
)
- await middleware(request, _call_next)
+ with TestClient(app) as client:
+ resp = client.get("/api/v1/sessions/session_123")
+ assert resp.status_code == 200
request_events = [payload for event_name, payload in captured if event_name == "http.request"]
assert request_events
assert request_events[0]["route"] == "/api/v1/sessions/{session_id}"
-async def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monkeypatch):
- middleware = create_http_metrics_middleware()
+def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monkeypatch):
+ middleware = create_http_observability_middleware()
debug_calls: list[tuple[str, tuple, dict]] = []
def _boom(**_kwargs):
@@ -308,22 +298,22 @@ async def test_http_metrics_middleware_logs_debug_when_metrics_write_fails(monke
monkeypatch.setattr(HttpRequestLifecycleDataSource, "set_inflight", staticmethod(_boom))
monkeypatch.setattr(http_middleware.logger, "isEnabledFor", lambda _level: True)
monkeypatch.setattr(http_middleware.logger, "debug", _debug)
+ monkeypatch.setattr(http_middleware, "maybe_start_root_span", lambda _req, _attrs: None)
- async def _call_next(request: Request):
- request.state.metric_account_id = "acct-real"
- return SimpleNamespace(status_code=200)
+ async def resources_handler(request: Request):
+ request.state.root_span_attrs.account_id = "acct-real"
+ return {"ok": True}
- request = Request(
- {
- "type": "http",
- "method": "POST",
- "path": "/api/v1/resources",
- "headers": [],
- "state": {},
- }
+ app = _build_test_app(
+ middleware,
+ route_path="/api/v1/resources",
+ method="POST",
+ handler=resources_handler,
)
- await middleware(request, _call_next)
+ with TestClient(app) as client:
+ resp = client.post("/api/v1/resources")
+ assert resp.status_code == 200
assert any(
message == "http metrics write failed: %s" for message, _args, _kwargs in debug_calls
@@ -353,7 +343,7 @@ def test_vlm_base_update_token_usage_propagates_current_account(monkeypatch):
monkeypatch.setattr(VLMEventDataSource, "record_call", staticmethod(_fake_record_call))
- token = bind_metric_account_context(account_id="acct-vlm-callsite")
+ token = _bind_root_context_for_account("acct-vlm-callsite")
try:
_DummyVLM({"provider": "volcengine", "model": "m1"}).update_token_usage(
model_name="m1",
@@ -363,6 +353,6 @@ def test_vlm_base_update_token_usage_propagates_current_account(monkeypatch):
duration_seconds=0.5,
)
finally:
- reset_metric_account_context(token)
+ reset_root_observability_context(token)
assert captured["account_id"] == "acct-vlm-callsite"
diff --git a/tests/server/test_prometheus_metrics.py b/tests/server/test_prometheus_metrics.py
index 1d82d08a0..a1ddc8a9c 100644
--- a/tests/server/test_prometheus_metrics.py
+++ b/tests/server/test_prometheus_metrics.py
@@ -12,14 +12,17 @@ from openviking.metrics.global_api import (
get_metrics_registry,
init_metrics_from_server_config,
shutdown_metrics,
+ shutdown_metrics_async,
)
from openviking.server.app import create_app
from openviking.server.config import (
MetricsAccountDimensionConfig,
MetricsConfig,
- PrometheusConfig,
+ MetricsExportersConfig,
+ ObservabilityConfig,
+ OTelExporterConfig,
+ PrometheusExporterConfig,
ServerConfig,
- TelemetryConfig,
)
@@ -72,7 +75,16 @@ class TestRetrievalStatsMetricsIntegration:
registry = MetricRegistry()
shutdown_metrics(app=None)
init_metrics_from_server_config(
- ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True))),
+ ServerConfig(
+ observability=ObservabilityConfig(
+ metrics=MetricsConfig(
+ enabled=True,
+ exporters=MetricsExportersConfig(
+ prometheus=PrometheusExporterConfig(enabled=True)
+ ),
+ )
+ )
+ ),
app=None,
registry=registry,
)
@@ -115,7 +127,16 @@ class TestMetricsEndpoint:
assert resp.status_code == 404
async def test_metrics_enabled_returns_200(self):
- config = ServerConfig(telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=True)))
+ config = ServerConfig(
+ observability=ObservabilityConfig(
+ metrics=MetricsConfig(
+ enabled=True,
+ exporters=MetricsExportersConfig(
+ prometheus=PrometheusExporterConfig(enabled=True)
+ ),
+ )
+ )
+ )
app = create_app(config=config, service=None)
init_metrics_from_server_config(config, app=app)
transport = httpx.ASGITransport(app=app)
@@ -133,7 +154,9 @@ class TestMetricsEndpoint:
shutdown_metrics(app=app)
async def test_metrics_enabled_by_new_server_metrics_flag(self):
- config = ServerConfig(metrics=MetricsConfig(enabled=True))
+ config = ServerConfig(
+ observability=ObservabilityConfig(metrics=MetricsConfig(enabled=True))
+ )
app = create_app(config=config, service=None)
init_metrics_from_server_config(config, app=app)
transport = httpx.ASGITransport(app=app)
@@ -144,34 +167,128 @@ class TestMetricsEndpoint:
finally:
shutdown_metrics(app=app)
- async def test_metrics_enabled_by_legacy_or_new_flag(self):
- config = ServerConfig(
- telemetry=TelemetryConfig(prometheus=PrometheusConfig(enabled=False)),
- metrics=MetricsConfig(enabled=True),
- )
- app = create_app(config=config, service=None)
- init_metrics_from_server_config(config, app=app)
- assert getattr(app.state, "metrics_exporter", None) is not None
- shutdown_metrics(app=app)
-
async def test_metrics_account_dimension_config_is_loaded(self):
config = ServerConfig(
- metrics=MetricsConfig(
- enabled=True,
- account_dimension=MetricsAccountDimensionConfig(
+ observability=ObservabilityConfig(
+ metrics=MetricsConfig(
enabled=True,
- max_active_accounts=3,
- metric_allowlist=["openviking_http_requests_total"],
- ),
+ account_dimension=MetricsAccountDimensionConfig(
+ enabled=True,
+ max_active_accounts=3,
+ metric_allowlist=["openviking_http_requests_total"],
+ ),
+ )
)
)
app = create_app(config=config, service=None)
init_metrics_from_server_config(config, app=app)
try:
- assert config.metrics.account_dimension.enabled is True
- assert config.metrics.account_dimension.max_active_accounts == 3
- assert config.metrics.account_dimension.metric_allowlist == [
+ assert config.observability.metrics.account_dimension.enabled is True
+ assert config.observability.metrics.account_dimension.max_active_accounts == 3
+ assert config.observability.metrics.account_dimension.metric_allowlist == [
"openviking_http_requests_total"
]
finally:
shutdown_metrics(app=app)
+
+
+def test_reinitializing_metrics_shuts_down_existing_exporters(monkeypatch):
+ class FakeOTelExporter:
+ instances = []
+
+ def __init__(self, **kwargs):
+ self.kwargs = kwargs
+ self.started = 0
+ self.shutdown_calls = 0
+ self.__class__.instances.append(self)
+
+ def start(self):
+ self.started += 1
+
+ async def shutdown(self):
+ self.shutdown_calls += 1
+
+ from openviking.metrics import global_api as global_api_module
+
+ config = ServerConfig(
+ observability=ObservabilityConfig(
+ metrics=MetricsConfig(
+ enabled=True,
+ exporters=MetricsExportersConfig(
+ prometheus=PrometheusExporterConfig(enabled=False),
+ otel=OTelExporterConfig(enabled=True),
+ ),
+ )
+ )
+ )
+
+ shutdown_metrics(app=None)
+ monkeypatch.setattr(global_api_module, "OTelMetricExporter", FakeOTelExporter)
+ init_metrics_from_server_config(config, app=None)
+ first = FakeOTelExporter.instances[0]
+
+ init_metrics_from_server_config(config, app=None)
+
+ assert first.started == 1
+ assert first.shutdown_calls == 1
+ assert len(FakeOTelExporter.instances) == 2
+
+ shutdown_metrics(app=None)
+
+
+async def test_async_shutdown_properly_awaits_exporter_cleanup(monkeypatch):
+ """Async shutdown should properly await async exporter cleanup methods.
+
+ This test verifies that shutdown_metrics_async properly awaits async shutdown
+ methods, which is the recommended path for async contexts (e.g., app.py lifespan).
+ """
+
+ class FakeAsyncExporter:
+ instances = []
+ shutdown_awaited = False
+
+ def __init__(self, **kwargs):
+ self.kwargs = kwargs
+ self.started = 0
+ self.shutdown_calls = 0
+ self.__class__.instances.append(self)
+
+ def start(self):
+ self.started += 1
+
+ async def shutdown(self):
+ self.shutdown_calls += 1
+ FakeAsyncExporter.shutdown_awaited = True
+
+ from openviking.metrics import global_api as global_api_module
+
+ config = ServerConfig(
+ observability=ObservabilityConfig(
+ metrics=MetricsConfig(
+ enabled=True,
+ exporters=MetricsExportersConfig(
+ prometheus=PrometheusExporterConfig(enabled=False),
+ otel=OTelExporterConfig(enabled=True),
+ ),
+ )
+ )
+ )
+
+ shutdown_metrics(app=None)
+ FakeAsyncExporter.instances.clear()
+ FakeAsyncExporter.shutdown_awaited = False
+ monkeypatch.setattr(global_api_module, "OTelMetricExporter", FakeAsyncExporter)
+ init_metrics_from_server_config(config, app=None)
+
+ assert len(FakeAsyncExporter.instances) == 1
+ first = FakeAsyncExporter.instances[0]
+ assert first.started == 1
+ assert first.shutdown_calls == 0
+ assert FakeAsyncExporter.shutdown_awaited is False
+
+ # Use async shutdown path - this should properly await the async shutdown
+ await shutdown_metrics_async(app=None)
+
+ assert first.shutdown_calls == 1
+ assert FakeAsyncExporter.shutdown_awaited is True
diff --git a/tests/telemetry/test_http_observability_middleware.py b/tests/telemetry/test_http_observability_middleware.py
new file mode 100644
index 000000000..a7dea3b7c
--- /dev/null
+++ b/tests/telemetry/test_http_observability_middleware.py
@@ -0,0 +1,75 @@
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+
+class _DummySpan:
+ def __init__(self) -> None:
+ self.updated_names: list[str] = []
+
+ def update_name(self, name: str) -> None:
+ self.updated_names.append(name)
+
+
+class _DummySpanCM:
+ def __init__(self, span: _DummySpan) -> None:
+ self._span = span
+
+ def __enter__(self) -> _DummySpan:
+ return self._span
+
+ def __exit__(self, exc_type, exc, tb) -> None:
+ return None
+
+
+def test_http_observability_middleware_updates_route_template_after_routing(monkeypatch) -> None:
+ """
+ Ensure `http_route` is finalized after routing has occurred (post-call_next).
+
+ Starlette/FastAPI route matching happens downstream of middleware, so reading
+ `request.scope["route"]` before `call_next` may yield no route.
+ """
+ from openviking.observability import http_observability_middleware as mw_mod
+
+ dummy_span = _DummySpan()
+
+ # Avoid pulling in metrics/otel side effects in this unit test.
+ monkeypatch.setattr(mw_mod, "should_skip_http_metrics", lambda request: False)
+ monkeypatch.setattr(mw_mod, "apply_http_metrics_start", lambda **kwargs: None)
+ monkeypatch.setattr(mw_mod, "apply_http_metrics_finalize", lambda **kwargs: None)
+ monkeypatch.setattr(mw_mod, "maybe_apply_root_span_attributes", lambda *a, **k: None)
+ monkeypatch.setattr(mw_mod, "maybe_apply_root_span_response", lambda *a, **k: None)
+ monkeypatch.setattr(mw_mod, "maybe_apply_root_span_error", lambda *a, **k: None)
+
+ # Force a "span" so we can validate operation_name update behavior.
+ monkeypatch.setattr(
+ mw_mod, "maybe_start_root_span", lambda request, root_attrs: _DummySpanCM(dummy_span)
+ )
+
+ captured = {}
+ real_create = mw_mod.create_root_span_attributes
+
+ def _capture_create_root_span_attributes(**kwargs):
+ root_attrs = real_create(**kwargs)
+ captured["root_attrs"] = root_attrs
+ return root_attrs
+
+ monkeypatch.setattr(mw_mod, "create_root_span_attributes", _capture_create_root_span_attributes)
+
+ app = FastAPI()
+ http_mw = mw_mod.create_http_observability_middleware()
+
+ @app.middleware("http")
+ async def _mw(request, call_next):
+ return await http_mw(request, call_next)
+
+ @app.get("/hello")
+ async def hello():
+ return {"ok": True}
+
+ with TestClient(app) as client:
+ resp = client.get("/hello")
+ assert resp.status_code == 200
+
+ root_attrs = captured["root_attrs"]
+ assert root_attrs.http_route == "/hello"
+ assert "GET /hello" in dummy_span.updated_names
diff --git a/tests/telemetry/test_observability_context.py b/tests/telemetry/test_observability_context.py
new file mode 100644
index 000000000..a60ea57c4
--- /dev/null
+++ b/tests/telemetry/test_observability_context.py
@@ -0,0 +1,537 @@
+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
+# SPDX-License-Identifier: AGPL-3.0
+
+"""
+Integration tests for unified observability context management.
+
+This module tests the unified context management system that integrates:
+- Root context (HTTP request-level)
+- Operation context (business operation-level)
+- Execution context (non-request scenarios)
+
+Key features tested:
+- Context binding and retrieval
+- Context isolation between async tasks
+- Integration with logging context
+- Trace/span ID priority resolution
+"""
+
+from __future__ import annotations
+
+import asyncio
+from typing import Optional
+
+import pytest
+
+from openviking.observability.context import (
+ ObservabilityContext,
+ bind_execution_context,
+ bind_operation_context,
+ bind_root_context,
+ get_observability_context,
+ get_operation_observability_context,
+ get_root_observability_context,
+ reset_observability_context,
+ set_observability_context,
+)
+from openviking.telemetry.span_models import (
+ OperationSpanAttributes,
+ RootSpanAttributes,
+)
+
+
+class TestObservabilityContextBasics:
+ """
+ Basic tests for ObservabilityContext data class.
+ """
+
+ def test_default_context_creation(self) -> None:
+ """
+ Test that a default ObservabilityContext is created with None values.
+ """
+ ctx = ObservabilityContext()
+
+ assert ctx.root is None
+ assert ctx.operation is None
+ assert ctx.execution_trace_id is None
+ assert ctx.execution_span_id is None
+ assert ctx.extra == {}
+
+ def test_get_trace_id_empty(self) -> None:
+ """
+ Test that get_trace_id returns empty string when no context is available.
+ """
+ ctx = ObservabilityContext()
+
+ assert ctx.get_trace_id() == ""
+
+ def test_get_span_id_empty(self) -> None:
+ """
+ Test that get_span_id returns empty string when no context is available.
+ """
+ ctx = ObservabilityContext()
+
+ assert ctx.get_span_id() == ""
+
+ def test_to_log_fields_empty(self) -> None:
+ """
+ Test that to_log_fields returns basic fields even when empty.
+ """
+ ctx = ObservabilityContext()
+
+ fields = ctx.to_log_fields()
+
+ assert "trace_id" in fields
+ assert "span_id" in fields
+ assert fields["trace_id"] == ""
+ assert fields["span_id"] == ""
+
+ def test_bind_execution_generates_ids(self) -> None:
+ """
+ Test that bind_execution generates trace_id and span_id when not provided.
+ """
+ ctx = ObservabilityContext()
+ ctx.bind_execution()
+
+ assert ctx.execution_trace_id is not None
+ assert ctx.execution_span_id is not None
+ assert len(ctx.execution_trace_id) == 32 # UUID hex is 32 chars
+ assert len(ctx.execution_span_id) == 16 # First 16 chars of trace_id
+
+ def test_bind_execution_uses_provided_ids(self) -> None:
+ """
+ Test that bind_execution uses provided trace_id and span_id.
+ """
+ ctx = ObservabilityContext()
+ ctx.bind_execution(trace_id="custom_trace_id", span_id="custom_span_id")
+
+ assert ctx.execution_trace_id == "custom_trace_id"
+ assert ctx.execution_span_id == "custom_span_id"
+
+ def test_set_extra(self) -> None:
+ """
+ Test that set_extra adds custom attributes.
+ """
+ ctx = ObservabilityContext()
+ ctx.set_extra("custom_key", "custom_value")
+ ctx.set_extra("number_key", 42)
+
+ assert ctx.extra["custom_key"] == "custom_value"
+ assert ctx.extra["number_key"] == 42
+
+ def test_to_log_fields_includes_extra(self) -> None:
+ """
+ Test that to_log_fields includes extra attributes.
+ """
+ ctx = ObservabilityContext()
+ ctx.set_extra("custom_field", "custom_value")
+
+ fields = ctx.to_log_fields()
+
+ assert fields["custom_field"] == "custom_value"
+
+
+class TestRootContextBinding:
+ """
+ Tests for root context binding and retrieval.
+ """
+
+ def test_bind_root_context(self) -> None:
+ """
+ Test that root context can be bound and retrieved.
+ """
+ root_attrs = RootSpanAttributes(
+ http_method="GET",
+ http_route="/test",
+ request_id="req-123",
+ )
+ root_attrs.account_id = "acct-456"
+
+ token = bind_root_context(root_attrs)
+
+ try:
+ retrieved = get_root_observability_context()
+
+ assert retrieved is not None
+ assert retrieved.http_method == "GET"
+ assert retrieved.http_route == "/test"
+ assert retrieved.request_id == "req-123"
+ assert retrieved.account_id == "acct-456"
+ finally:
+ reset_observability_context(token)
+
+ def test_root_context_to_log_fields(self) -> None:
+ """
+ Test that root context fields are included in to_log_fields.
+ """
+ root_attrs = RootSpanAttributes(
+ http_method="POST",
+ http_route="/api/v1/sessions",
+ request_id="req-test",
+ )
+ root_attrs.account_id = "acct-test"
+
+ token = bind_root_context(root_attrs)
+
+ try:
+ ctx = get_observability_context()
+ fields = ctx.to_log_fields()
+
+ assert fields["request_id"] == "req-test"
+ assert fields["account_id"] == "acct-test"
+ assert fields["http_method"] == "POST"
+ assert fields["http_route"] == "/api/v1/sessions"
+ finally:
+ reset_observability_context(token)
+
+ def test_reset_root_context(self) -> None:
+ """
+ Test that root context can be reset.
+ """
+ root_attrs = RootSpanAttributes(
+ http_method="GET",
+ http_route="/test",
+ request_id="req-123",
+ )
+
+ token = bind_root_context(root_attrs)
+
+ try:
+ assert get_root_observability_context() is not None
+ finally:
+ reset_observability_context(token)
+
+ # After reset, get_root_observability_context should return None
+ # because the context is reset to its previous state
+ retrieved = get_root_observability_context()
+ assert retrieved is None
+
+
+class TestOperationContextBinding:
+ """
+ Tests for operation context binding and retrieval.
+ """
+
+ def test_bind_operation_context(self) -> None:
+ """
+ Test that operation context can be bound and retrieved.
+ """
+ operation_attrs = OperationSpanAttributes(
+ operation="search.find",
+ telemetry_id="tm-123",
+ )
+
+ token = bind_operation_context(operation_attrs)
+
+ try:
+ retrieved = get_operation_observability_context()
+
+ assert retrieved is not None
+ assert retrieved.operation == "search.find"
+ assert retrieved.telemetry_id == "tm-123"
+ finally:
+ reset_observability_context(token)
+
+ def test_operation_context_to_log_fields(self) -> None:
+ """
+ Test that operation context fields are included in to_log_fields.
+ """
+ operation_attrs = OperationSpanAttributes(
+ operation="session.commit",
+ telemetry_id="tm-456",
+ )
+ operation_attrs.status = "ok"
+
+ token = bind_operation_context(operation_attrs)
+
+ try:
+ ctx = get_observability_context()
+ fields = ctx.to_log_fields()
+
+ assert fields["operation"] == "session.commit"
+ assert fields["telemetry_id"] == "tm-456"
+ assert fields["status"] == "ok"
+ finally:
+ reset_observability_context(token)
+
+
+class TestExecutionContextBinding:
+ """
+ Tests for execution context binding (context manager.
+ """
+
+ def test_bind_execution_context_manager(self) -> None:
+ """
+ Test that execution context can be bound using context manager.
+ """
+ with bind_execution_context() as (trace_id, span_id):
+ assert trace_id is not None
+ assert span_id is not None
+ assert len(trace_id) == 32
+ assert len(span_id) == 16
+
+ ctx = get_observability_context()
+ assert ctx.execution_trace_id == trace_id
+ assert ctx.execution_span_id == span_id
+
+ # After exiting context manager, execution context should be reset
+ ctx_after = get_observability_context()
+ assert ctx_after.execution_trace_id is None
+ assert ctx_after.execution_span_id is None
+
+ def test_bind_execution_with_custom_ids(self) -> None:
+ """
+ Test that execution context can be bound with custom IDs.
+ """
+ custom_trace_id = "custom_trace_123456789012345678901234"
+ custom_span_id = "custom_span_123456"
+
+ with bind_execution_context(
+ trace_id=custom_trace_id,
+ span_id=custom_span_id,
+ ) as (trace_id, span_id):
+ assert trace_id == custom_trace_id
+ assert span_id == custom_span_id
+
+ ctx = get_observability_context()
+ assert ctx.execution_trace_id == custom_trace_id
+ assert ctx.execution_span_id == custom_span_id
+
+ def test_execution_context_trace_id_priority(self) -> None:
+ """
+ Test that execution context trace_id is used when no OTel span is available.
+ """
+ with bind_execution_context() as (trace_id, span_id):
+ ctx = get_observability_context()
+
+ # get_trace_id should return the execution trace_id
+ # since no OTel span is available
+ effective_trace_id = ctx.get_trace_id()
+ effective_span_id = ctx.get_span_id()
+
+ assert effective_trace_id == trace_id
+ assert effective_span_id == span_id
+
+
+class TestContextIsolation:
+ """
+ Tests for context isolation between async tasks.
+ """
+
+ @pytest.mark.asyncio
+ async def test_async_task_context_isolation(self) -> None:
+ """
+ Test that context is isolated between concurrent async tasks.
+ """
+ seen: list[Optional[str]] = []
+
+ async def _worker(account_id: str):
+ root_attrs = RootSpanAttributes(
+ http_method="GET",
+ http_route="/test",
+ request_id=f"req-{account_id}",
+ )
+ root_attrs.account_id = account_id
+
+ token = bind_root_context(root_attrs)
+ try:
+ await asyncio.sleep(0) # Yield to other tasks
+ root_context = get_root_observability_context()
+ seen.append(root_context.account_id if root_context else None)
+ finally:
+ reset_observability_context(token)
+
+ await asyncio.gather(
+ _worker("acct-a"),
+ _worker("acct-b"),
+ _worker("acct-c"),
+ )
+
+ # All three tasks should see their own context, not leaked from others
+ assert sorted(seen) == ["acct-a", "acct-b", "acct-c"]
+
+ @pytest.mark.asyncio
+ async def test_nested_context_binding(self) -> None:
+ """
+ Test that nested context binding works correctly.
+ """
+ root_attrs_outer = RootSpanAttributes(
+ http_method="GET",
+ http_route="/outer",
+ request_id="req-outer",
+ )
+ root_attrs_outer.account_id = "acct-outer"
+
+ token_outer = bind_root_context(root_attrs_outer)
+
+ try:
+ # Verify outer context
+ outer_ctx = get_root_observability_context()
+ assert outer_ctx is not None
+ assert outer_ctx.account_id == "acct-outer"
+
+ # Bind inner context
+ root_attrs_inner = RootSpanAttributes(
+ http_method="POST",
+ http_route="/inner",
+ request_id="req-inner",
+ )
+ root_attrs_inner.account_id = "acct-inner"
+
+ token_inner = bind_root_context(root_attrs_inner)
+
+ try:
+ # Verify inner context
+ inner_ctx = get_root_observability_context()
+ assert inner_ctx is not None
+ assert inner_ctx.account_id == "acct-inner"
+ finally:
+ reset_observability_context(token_inner)
+
+ # After resetting inner context, should be back to outer
+ after_inner_reset = get_root_observability_context()
+ assert after_inner_reset is not None
+ assert after_inner_reset.account_id == "acct-outer"
+
+ finally:
+ reset_observability_context(token_outer)
+
+ # After resetting outer context
+ after_outer_reset = get_root_observability_context()
+ assert after_outer_reset is None
+
+
+class TestCombinedContexts:
+ """
+ Tests for combining multiple context types.
+ """
+
+ def test_combined_root_and_operation_context(self) -> None:
+ """
+ Test that root and operation contexts can be combined.
+ """
+ root_attrs = RootSpanAttributes(
+ http_method="POST",
+ http_route="/api/v1/sessions",
+ request_id="req-123",
+ )
+ root_attrs.account_id = "acct-456"
+
+ operation_attrs = OperationSpanAttributes(
+ operation="session.commit",
+ telemetry_id="tm-789",
+ )
+ operation_attrs.status = "ok"
+
+ # Bind root context first
+ token_root = bind_root_context(root_attrs)
+
+ try:
+ # Then bind operation context
+ token_operation = bind_operation_context(operation_attrs)
+
+ try:
+ ctx = get_observability_context()
+ fields = ctx.to_log_fields()
+
+ # Both root and operation fields should be present
+ assert fields["request_id"] == "req-123"
+ assert fields["account_id"] == "acct-456"
+ assert fields["operation"] == "session.commit"
+ assert fields["telemetry_id"] == "tm-789"
+ assert fields["status"] == "ok"
+
+ finally:
+ reset_observability_context(token_operation)
+
+ # After resetting operation context, root context should still be there
+ after_op_reset = get_root_observability_context()
+ assert after_op_reset is not None
+ assert after_op_reset.account_id == "acct-456"
+
+ finally:
+ reset_observability_context(token_root)
+
+
+class TestContextManagerIntegration:
+ """
+ Tests for integration with context manager patterns.
+ """
+
+ def test_set_and_reset_observability_context(self) -> None:
+ """
+ Test that set_observability_context and reset_observability_context work correctly.
+ """
+ # Create a custom context
+ custom_ctx = ObservabilityContext()
+ custom_ctx.set_extra("custom_key", "custom_value")
+ custom_ctx.bind_execution(trace_id="custom_trace", span_id="custom_span")
+
+ # Set the context
+ token = set_observability_context(custom_ctx)
+
+ try:
+ # Verify the context is set
+ retrieved = get_observability_context()
+ assert retrieved.execution_trace_id == "custom_trace"
+ assert retrieved.execution_span_id == "custom_span"
+ assert retrieved.extra["custom_key"] == "custom_value"
+
+ finally:
+ # Reset the context
+ reset_observability_context(token)
+
+ # After reset, should be back to default
+ after_reset = get_observability_context()
+ assert after_reset.execution_trace_id is None
+ assert after_reset.execution_span_id is None
+
+
+class TestLogFieldsIntegration:
+ """
+ Tests for log fields integration.
+ """
+
+ def test_to_log_fields_combines_all_contexts(self) -> None:
+ """
+ Test that to_log_fields combines all context types correctly.
+ """
+ ctx = ObservabilityContext()
+
+ # Add root context
+ root_attrs = RootSpanAttributes(
+ http_method="GET",
+ http_route="/test",
+ request_id="req-123",
+ )
+ root_attrs.account_id = "acct-456"
+ ctx.bind_root(root_attrs)
+
+ # Add operation context
+ operation_attrs = OperationSpanAttributes(
+ operation="search.find",
+ telemetry_id="tm-789",
+ )
+ operation_attrs.status = "ok"
+ ctx.bind_operation(operation_attrs)
+
+ # Add execution context
+ ctx.bind_execution(trace_id="exec_trace", span_id="exec_span")
+
+ # Add extra fields
+ ctx.set_extra("extra_field", "extra_value")
+
+ # Get log fields
+ fields = ctx.to_log_fields()
+
+ # Verify all fields are present
+ assert fields["trace_id"] == "exec_trace"
+ assert fields["span_id"] == "exec_span"
+ assert fields["request_id"] == "req-123"
+ assert fields["account_id"] == "acct-456"
+ assert fields["http_method"] == "GET"
+ assert fields["http_route"] == "/test"
+ assert fields["operation"] == "search.find"
+ assert fields["telemetry_id"] == "tm-789"
+ assert fields["status"] == "ok"
+ assert fields["extra_field"] == "extra_value"
diff --git a/tests/test_server_config_loader.py b/tests/test_server_config_loader.py
index 155c95dd0..5e9c17df2 100644
--- a/tests/test_server_config_loader.py
+++ b/tests/test_server_config_loader.py
@@ -21,11 +21,19 @@ def test_load_server_config_rejects_unknown_field(tmp_path):
def test_load_server_config_rejects_unknown_nested_field(tmp_path):
config_path = tmp_path / "ov.conf"
- config_path.write_text(json.dumps({"server": {"telemetry": {"prometheus": {"enabld": True}}}}))
+ config_path.write_text(
+ json.dumps(
+ {
+ "server": {
+ "observability": {"metrics": {"exporters": {"prometheus": {"enabld": True}}}}
+ }
+ }
+ )
+ )
with pytest.raises(
ValueError,
- match=r"server\.telemetry\.prometheus\.enabld'.*server\.telemetry\.prometheus\.enabled",
+ match=r"server\.observability\.metrics\.exporters\.prometheus\.enabld'.*server\.observability\.metrics\.exporters\.prometheus\.enabled",
):
load_server_config(str(config_path))
@@ -50,7 +58,7 @@ def test_load_server_config_preserves_supported_fields(tmp_path):
"auth_mode": "trusted",
"with_bot": True,
"bot_api_url": "http://localhost:19999",
- "telemetry": {"prometheus": {"enabled": True}},
+ "observability": {"metrics": {"exporters": {"prometheus": {"enabled": True}}}},
},
"encryption": {"enabled": True},
}
@@ -65,23 +73,13 @@ def test_load_server_config_preserves_supported_fields(tmp_path):
assert config.auth_mode == "trusted"
assert config.with_bot is True
assert config.bot_api_url == "http://localhost:19999"
- assert config.telemetry.prometheus.enabled is True
+ assert config.observability.metrics.exporters.prometheus.enabled is True
assert config.encryption_enabled is True
def test_load_bot_gateway_token_reads_token_from_bot_gateway_section(tmp_path):
config_path = tmp_path / "ov.conf"
- config_path.write_text(
- json.dumps(
- {
- "bot": {
- "gateway": {
- "token": "gateway-token"
- }
- }
- }
- )
- )
+ config_path.write_text(json.dumps({"bot": {"gateway": {"token": "gateway-token"}}}))
assert load_bot_gateway_token(str(config_path)) == "gateway-token"
@@ -92,16 +90,18 @@ def test_load_server_config_preserves_metrics_account_dimension_fields(tmp_path)
json.dumps(
{
"server": {
- "metrics": {
- "enabled": True,
- "account_dimension": {
+ "observability": {
+ "metrics": {
"enabled": True,
- "max_active_accounts": 5,
- "metric_allowlist": [
- "openviking_http_requests_total",
- "openviking_task_pending",
- ],
- },
+ "account_dimension": {
+ "enabled": True,
+ "max_active_accounts": 5,
+ "metric_allowlist": [
+ "openviking_http_requests_total",
+ "openviking_task_pending",
+ ],
+ },
+ }
}
}
}
@@ -110,10 +110,10 @@ def test_load_server_config_preserves_metrics_account_dimension_fields(tmp_path)
config = load_server_config(str(config_path))
- assert config.metrics.enabled is True
- assert config.metrics.account_dimension.enabled is True
- assert config.metrics.account_dimension.max_active_accounts == 5
- assert config.metrics.account_dimension.metric_allowlist == [
+ assert config.observability.metrics.enabled is True
+ assert config.observability.metrics.account_dimension.enabled is True
+ assert config.observability.metrics.account_dimension.max_active_accounts == 5
+ assert config.observability.metrics.account_dimension.metric_allowlist == [
"openviking_http_requests_total",
"openviking_task_pending",
]
diff --git a/tests/test_telemetry_runtime.py b/tests/test_telemetry_runtime.py
index aa1ae9219..1a2e0274e 100644
--- a/tests/test_telemetry_runtime.py
+++ b/tests/test_telemetry_runtime.py
@@ -9,9 +9,16 @@ from types import SimpleNamespace
import pytest
-from openviking.metrics.account_context import get_metric_account_context
from openviking.models.embedder.base import EmbedResult, embed_compat
from openviking.models.vlm.base import VLMBase
+from openviking.observability.context import (
+ bind_operation_observability_context,
+ bind_root_observability_context,
+ get_operation_observability_context,
+ get_root_observability_context,
+ reset_operation_observability_context,
+ reset_root_observability_context,
+)
from openviking.server.identity import RequestContext, Role
from openviking.service.resource_service import ResourceService
from openviking.storage.collection_schemas import TextEmbeddingHandler
@@ -27,6 +34,7 @@ from openviking.telemetry import (
from openviking.telemetry.backends.memory import MemoryOperationTelemetry
from openviking.telemetry.context import bind_telemetry, bind_telemetry_stage
from openviking.telemetry.snapshot import TelemetrySnapshot
+from openviking.telemetry.span_models import OperationSpanAttributes, RootSpanAttributes
from openviking_cli.session.user_id import UserIdentifier
@@ -41,6 +49,26 @@ def test_telemetry_module_exports_snapshot_and_runtime():
assert get_telemetry_runtime().meter() is not None
+def test_root_observability_context_bind_and_reset():
+ root = RootSpanAttributes(http_method="GET", http_route="/demo", request_id="req-1")
+ token = bind_root_observability_context(root)
+ try:
+ assert get_root_observability_context() is root
+ finally:
+ reset_root_observability_context(token)
+ assert get_root_observability_context() is None
+
+
+def test_operation_observability_context_bind_and_reset():
+ operation = OperationSpanAttributes(operation="search.find", telemetry_id="tm-demo")
+ token = bind_operation_observability_context(operation)
+ try:
+ assert get_operation_observability_context() is operation
+ finally:
+ reset_operation_observability_context(token)
+ assert get_operation_observability_context() is None
+
+
def test_telemetry_snapshot_to_dict_supports_summary_only():
snapshot = TelemetrySnapshot(
telemetry_id="tm_demo",
@@ -286,7 +314,9 @@ async def test_semantic_processor_binds_metric_account_context(monkeypatch):
async def run(self, root_uri):
ran["value"] = True
- assert get_metric_account_context().http_account_id == "acct-semantic"
+ root_context = get_root_observability_context()
+ assert root_context is not None
+ assert root_context.account_id == "acct-semantic"
def get_stats(self):
return DagStats()
diff --git a/uv.lock b/uv.lock
index 77594d811..11e8ad6b3 100644
--- a/uv.lock
+++ b/uv.lock
@@ -3509,6 +3509,7 @@ dependencies = [
{ name = "openpyxl" },
{ name = "opentelemetry-api" },
{ name = "opentelemetry-exporter-otlp-proto-grpc" },
+ { name = "opentelemetry-exporter-otlp-proto-http" },
{ name = "opentelemetry-instrumentation-asyncio" },
{ name = "opentelemetry-sdk" },
{ name = "pdfminer-six" },
@@ -3746,6 +3747,7 @@ requires-dist = [
{ name = "opensandbox-server", marker = "extra == 'bot-sandbox'", specifier = ">=0.1.0" },
{ name = "opentelemetry-api", specifier = ">=1.14" },
{ name = "opentelemetry-exporter-otlp-proto-grpc", specifier = ">=1.14" },
+ { name = "opentelemetry-exporter-otlp-proto-http", specifier = ">=1.14" },
{ name = "opentelemetry-instrumentation-asyncio", specifier = ">=0.61b0" },
{ name = "opentelemetry-sdk", specifier = ">=1.14" },
{ name = "openviking", extras = ["bot", "bot-dingtalk", "bot-feishu", "bot-fuse", "bot-langfuse", "bot-opencode", "bot-qq", "bot-sandbox", "bot-slack", "bot-telegram"], marker = "extra == 'bot-full'" },