Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(metric): add executor && task metric export (#5105)

baojun-zhang3b6f37e9d288
diff --git a/docs/design/metric-design.md b/docs/design/metric-design.md
index 130ac7e57..46471c7e7 100644
--- a/docs/design/metric-design.md
+++ b/docs/design/metric-design.md
@@ -312,7 +312,7 @@ graph LR
| `StorageProbeDataSource` | `ProbeMetricDataSource` | AGFS / VikingFS、系统表访问检查 | 存储探针结果 | storage readability、storage writability、system table readiness |
| `RetrievalBackendProbeDataSource` | `ProbeMetricDataSource` | VikingDB、检索后端最小能力检查 | 检索后端探针结果 | backend readiness、collection availability |
| `ModelProviderProbeDataSource` | `ProbeMetricDataSource` | VLM / Embedding / Rerank provider 可用性检查 | 模型依赖探针结果 | provider readiness、credential availability |
-| `AsyncSystemProbeDataSource` | `ProbeMetricDataSource` | Queue、TaskTracker、后台消费线程检查 | 异步系统探针结果 | queue readiness、worker liveness |
+| `AsyncSystemProbeDataSource` | `ProbeMetricDataSource` | Queue、默认 asyncio executor monitor | 异步系统探针与 executor 指标 | queue readiness、executor threads / tasks |
| `EncryptionProbeDataSource` | `ProbeMetricDataSource` | Root Key、KMS / Vault Provider、加密组件检查 | 加密探针结果 | root key readiness、kms availability、encryption component health |
设计边界如下:
@@ -427,7 +427,7 @@ Collector 与 DataSource 的主映射关系如下:
| `StorageProbeCollector` | `ProbeMetricCollector` | StorageProbeDataSource | storage readiness、system table availability |
| `RetrievalBackendProbeCollector` | `ProbeMetricCollector` | RetrievalBackendProbeDataSource | backend readiness、collection availability |
| `ModelProviderProbeCollector` | `ProbeMetricCollector` | ModelProviderProbeDataSource | provider readiness、credential availability |
-| `AsyncSystemProbeCollector` | `ProbeMetricCollector` | AsyncSystemProbeDataSource | queue readiness、worker liveness |
+| `AsyncSystemProbeCollector` | `ProbeMetricCollector` | AsyncSystemProbeDataSource | queue readiness、默认 executor 线程与任务指标 |
| `EncryptionProbeCollector` | `ProbeMetricCollector` | EncryptionProbeDataSource | root key readiness、kms availability、encryption component health |
设计理由:在引入四类 Collector 之后,整个体系的语义边界更清晰:
@@ -644,6 +644,7 @@ sequenceDiagram
| RAGFS 监控 | `RagfsMetricDataSource` | `RagfsMetricCollector` | Counter、Histogram、Gauge | 文件操作、Cache、multi-backend、Lock |
| 加密监控 | `EncryptionEventDataSource`、`EncryptionProbeDataSource` | `EncryptionCollector`、`EncryptionProbeCollector` | Counter、Histogram、Gauge | encrypt count、decrypt duration、root key readiness |
| 系统探针监控 | 各类 `*ProbeDataSource` | 各类 `*ProbeCollector` | Gauge、Health | service readiness、storage readiness、kms availability |
+| Python executor 监控 | `AsyncSystemProbeDataSource` | `AsyncSystemProbeCollector` | Gauge、Counter | 默认 executor 线程数、任务数、提交/完成/失败计数 |
| 操作级 Telemetry 指标化 | telemetry adapter / bridge | `TelemetryBridgeCollector` | Counter、Histogram、Gauge | operation requests、vector scanned、memory extracted |
### 3.2 指标对象模型
@@ -808,7 +809,81 @@ sequenceDiagram
| `summary.semantic_nodes.{total|done|pending|running}` | `openviking_semantic_nodes_total{status=...}` |
| `summary.memory.extracted` | `openviking_memory_extracted_total{memory_type=...}` |
-### 3.8 多租户支持范围
+### 3.8 Python 默认 executor 指标
+
+Python executor 指标用于观察当前服务进程内 asyncio 默认 executor 的线程和任务状态。
+它只覆盖 Python 默认 executor,不覆盖 Rust / RAGFS 内部 Tokio runtime。
+
+当前代码路径:
+
+| 文件 | 符号 | 职责 |
+| --- | --- | --- |
+| `openviking/server/app.py` | `lifespan()` | 在 `_configure_default_executor(config)` 后安装 monitor,在 shutdown metrics 后卸载 |
+| `openviking/metrics/core/runtime.py` | `DefaultExecutorMonitor` | 保存 event loop、原始 `run_in_executor`、计数器和锁 |
+| `openviking/metrics/core/runtime.py` | `install_executor_monitor()` / `uninstall_executor_monitor()` | 安装和恢复当前 event loop 的默认 executor 入口 |
+| `openviking/metrics/datasources/probes.py` | `AsyncSystemProbeDataSource.read_async_system_state()` | 合并 queue readiness 和 executor metrics |
+| `openviking/metrics/collectors/async_system_probe.py` | `AsyncSystemProbeCollector` | 写 `openviking_async_system_readiness` 和 `openviking_executor_*` |
+
+采集链路:
+
+```text
+/metrics
+-> CollectorManager.collect_all()
+-> asyncio.to_thread(collector.collect, registry)
+-> AsyncSystemProbeCollector.read_metric_input()
+-> AsyncSystemProbeDataSource.read_async_system_state()
+-> get_executor_monitor().read_metrics()
+-> AsyncSystemProbeCollector.collect_hook()
+-> MetricRegistry
+```
+
+统计范围:
+
+```text
+统计:loop.run_in_executor(None, func, *args)
+统计:asyncio.to_thread(func, *args, **kwargs)
+不统计:loop.run_in_executor(custom_executor, func, *args)
+不统计:concurrent.futures.ThreadPoolExecutor(...) 直接创建的 executor
+不统计:Rust / RAGFS 内部 runtime 或线程
+```
+
+指标定义:
+
+| 指标名 | 类型 | 标签 | 说明 |
+| --- | --- | --- | --- |
+| `openviking_executor_max_workers` | Gauge | `pool,process_role,worker` | 默认 executor 最大 worker 数 |
+| `openviking_executor_threads` | Gauge | `pool,process_role,worker` | 默认 executor 已创建线程数 |
+| `openviking_executor_active_tasks` | Gauge | `pool,process_role,worker` | 当前正在执行的默认 executor 任务数 |
+| `openviking_executor_pending_tasks` | Gauge | `pool,process_role,worker` | 当前等待执行的默认 executor 任务数 |
+| `openviking_executor_submitted_total` | Counter | `pool,process_role,worker` | 累计提交到默认 executor 的任务数 |
+| `openviking_executor_completed_total` | Counter | `pool,process_role,worker` | 累计执行结束的默认 executor 任务数,失败也计入 |
+| `openviking_executor_failed_total` | Counter | `pool,process_role,worker` | 默认 executor callable 抛异常的累计次数 |
+
+固定标签:
+
+```text
+pool="asyncio_default"
+process_role="legacy_server"
+worker=multiprocessing.current_process().name
+```
+
+`failed_total` 的语义:
+
+- 该指标统计 worker callable 抛出的 `Exception`。
+- 如果异常被上层业务捕获并作为正常分支处理,也会计入。
+- 它不等同于业务请求失败数。
+- `/metrics` scrape 自身会通过 `asyncio.to_thread()` 刷新 collector,因此也会计入 executor 指标。
+
+运行观测:
+
+- monitor 在服务初始化前安装,会统计启动阶段的默认 executor 调用。
+- AGFS 初始化中的文件不存在、目录已存在等探测式异常会计入 `failed_total`。
+- 当前运行时如果加载旧版 `ragfs_python` binding,`RagfsMetricCollector` 调用
+ `service._agfs_client.metrics()` 会失败。
+- repo 当前源码 `crates/ragfs-python/src/lib.rs` 已定义 `metrics()`;若运行时对象没有该方法,
+ 说明 `.venv` 里的 native binding 产物与源码能力不一致。
+
+### 3.9 多租户支持范围
当前代码中,只有进入 `ACCOUNT_DIMENSION_SUPPORTED_METRICS` 支持集的指标族允许注入 `account_id`。支持集如下:
diff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.md
index f40bdafa4..07191d558 100644
--- a/docs/en/concepts/12-metrics.md
+++ b/docs/en/concepts/12-metrics.md
@@ -228,6 +228,13 @@ Notes:
| `openviking_queue_errors_total` | Counter | `queue` | total error count per queue |
| `openviking_queue_pending` | Gauge | `queue` | pending queue items |
| `openviking_queue_in_progress` | Gauge | `queue` | in-progress queue items |
+| `openviking_executor_max_workers` | Gauge | `pool, process_role, worker` | maximum workers in the asyncio default executor |
+| `openviking_executor_threads` | Gauge | `pool, process_role, worker` | threads created by the asyncio default executor |
+| `openviking_executor_active_tasks` | Gauge | `pool, process_role, worker` | default executor tasks currently running |
+| `openviking_executor_pending_tasks` | Gauge | `pool, process_role, worker` | default executor tasks waiting to run |
+| `openviking_executor_submitted_total` | Counter | `pool, process_role, worker` | total tasks submitted to the default executor |
+| `openviking_executor_completed_total` | Counter | `pool, process_role, worker` | default executor tasks that finished, including failed tasks |
+| `openviking_executor_failed_total` | Counter | `pool, process_role, worker` | default executor callables that raised exceptions |
| `openviking_lock_active` | Gauge | none | current published lock leases |
| `openviking_lock_waiting` | Gauge | none | requests currently waiting for locks |
| `openviking_lock_stale` | Gauge | none | cumulative count of stale lock tokens removed |
@@ -240,6 +247,14 @@ These help answer:
- Is there queue backlog?
- Is there lock contention or stale locking?
+- Is the default executor near its worker limit or building a queue?
+
+Executor metrics only cover `loop.run_in_executor(None, ...)` and
+`asyncio.to_thread(...)`. Calls using an explicit custom executor and Rust /
+RAGFS internal runtimes or threads are not included. `failed_total` counts
+callables that raise exceptions. If the caller catches that exception and treats
+it as a normal branch, it still counts here, so this is not the business request
+failure count.
### RAGFS
diff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.md
index 725ac7f96..50c123003 100644
--- a/docs/zh/concepts/12-metrics.md
+++ b/docs/zh/concepts/12-metrics.md
@@ -231,6 +231,13 @@ scrape_configs:
| `openviking_queue_errors_total` | Counter | `queue` | 队列累计错误量 |
| `openviking_queue_pending` | Gauge | `queue` | 队列待处理数 |
| `openviking_queue_in_progress` | Gauge | `queue` | 队列执行中数量 |
+| `openviking_executor_max_workers` | Gauge | `pool, process_role, worker` | asyncio 默认 executor 最大 worker 数 |
+| `openviking_executor_threads` | Gauge | `pool, process_role, worker` | asyncio 默认 executor 已创建线程数 |
+| `openviking_executor_active_tasks` | Gauge | `pool, process_role, worker` | 默认 executor 当前执行中的任务数 |
+| `openviking_executor_pending_tasks` | Gauge | `pool, process_role, worker` | 默认 executor 当前等待执行的任务数 |
+| `openviking_executor_submitted_total` | Counter | `pool, process_role, worker` | 默认 executor 累计提交任务数 |
+| `openviking_executor_completed_total` | Counter | `pool, process_role, worker` | 默认 executor 累计执行结束任务数,失败也计入 |
+| `openviking_executor_failed_total` | Counter | `pool, process_role, worker` | 默认 executor callable 抛异常次数 |
| `openviking_lock_active` | Gauge | 无 | 当前已发布的锁租约数 |
| `openviking_lock_waiting` | Gauge | 无 | 当前等待锁的请求数 |
| `openviking_lock_stale` | Gauge | 无 | 累计已清理的过期锁 token 数 |
@@ -243,6 +250,12 @@ scrape_configs:
- 是否有队列堆积?
- 是否有锁竞争或 stale lock?
+- 默认 executor 是否接近线程上限或出现排队?
+
+executor 指标只统计 `loop.run_in_executor(None, ...)` 和 `asyncio.to_thread(...)`。
+显式传入自定义 executor 的调用、Rust / RAGFS 内部 runtime 或线程不计入。
+`failed_total` 统计 callable 抛异常的次数。如果异常被上层捕获并作为正常分支处理,
+也会计入该指标,因此它不等同于业务请求失败数。
### RAGFS
diff --git a/openviking/metrics/collectors/async_system_probe.py b/openviking/metrics/collectors/async_system_probe.py
index 89fcae2c8..02c6f77f0 100644
--- a/openviking/metrics/collectors/async_system_probe.py
+++ b/openviking/metrics/collectors/async_system_probe.py
@@ -15,17 +15,27 @@ from .base import CollectorConfig, ProbeMetricCollector
@dataclass
class AsyncSystemProbeCollector(ProbeMetricCollector):
"""
- Export readiness for individual async-system probe checks.
+ Export async-system readiness and Python default executor metrics.
- The datasource returns a bounded mapping of probe names to booleans. This collector converts
- that mapping into one readiness gauge series per probe so dashboards can distinguish which
- async subsystem is currently unhealthy.
+ Readiness remains one gauge series per probe. Executor metrics use `pool`,
+ `process_role`, and `worker` labels.
"""
DOMAIN: ClassVar[str] = "async_system"
# rule: <METRICS_NAMESPACE>_<DOMAIN>_readiness
# e.g.: openviking_async_system_readiness
READINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "readiness")
+ EXECUTOR_GAUGES: ClassVar[tuple[tuple[str, str], ...]] = (
+ (MetricCollector.metric_name("executor", "max_workers"), "max_workers"),
+ (MetricCollector.metric_name("executor", "threads"), "threads"),
+ (MetricCollector.metric_name("executor", "active_tasks"), "active_tasks"),
+ (MetricCollector.metric_name("executor", "pending_tasks"), "pending_tasks"),
+ )
+ EXECUTOR_COUNTERS: ClassVar[tuple[tuple[str, str], ...]] = (
+ (MetricCollector.metric_name("executor", "submitted", unit="total"), "submitted_total"),
+ (MetricCollector.metric_name("executor", "completed", unit="total"), "completed_total"),
+ (MetricCollector.metric_name("executor", "failed", unit="total"), "failed_total"),
+ )
data_source: AsyncSystemProbeDataSource
config: CollectorConfig = CollectorConfig(timeout_seconds=0.5)
@@ -38,36 +48,63 @@ class AsyncSystemProbeCollector(ProbeMetricCollector):
self._known_probes = ["queue"]
def read_metric_input(self):
- """Read the latest async-system probe mapping from the datasource."""
- return self.data_source.read_probe_state()
+ """Read the latest async-system probe and executor metrics from the datasource."""
+ return self.data_source.read_async_system_state()
def collect_hook(self, registry, metric_input) -> None:
"""
- Publish one readiness sample per async-system probe name.
+ Publish readiness samples and default-executor runtime samples.
- The helper normalizes probe names to strings and writes a fixed 0/1 readiness gauge using
- `probe` as the only series label.
+ The readiness helper normalizes probe names to strings and writes a fixed 0/1 readiness
+ gauge. Executor samples keep a low-cardinality `pool/process_role/worker` label set.
"""
- normalized = {str(k): bool(v) for k, v in metric_input.items()}
+ probes = metric_input["probes"]
+ probes_valid = bool(metric_input["probes_valid"])
+ executor = metric_input["executor"]
+ normalized = {str(k): bool(v) for k, v in probes.items()}
self._known_probes = list(normalized.keys())
- for probe, is_ready in normalized.items():
+ self._write_readiness(registry, normalized, valid=probes_valid)
+ self._write_executor_metrics(registry, executor)
+
+ def _write_readiness(self, registry, probes: dict[str, bool], *, valid: bool) -> None:
+ """Write readiness gauges for async-system probes."""
+ for probe, is_ready in probes.items():
self.replace_gauge(
registry,
self.READINESS,
- 1.0 if bool(is_ready) else 0.0,
+ 1.0 if valid and bool(is_ready) else 0.0,
match_labels={"probe": str(probe)},
- labels={"probe": str(probe), "valid": "1"},
+ labels={"probe": str(probe), "valid": "1" if valid else "0"},
label_names=("probe", "valid"),
)
+ def _write_executor_metrics(self, registry, metrics: dict[str, int | str]) -> None:
+ """Write Python default executor metrics using stable labels."""
+ labels = {
+ "pool": str(metrics["pool"]),
+ "process_role": str(metrics["process_role"]),
+ "worker": str(metrics["worker"]),
+ }
+ label_names = ("pool", "process_role", "worker")
+ for metric_name, key in self.EXECUTOR_GAUGES:
+ registry.set_gauge(
+ metric_name,
+ float(metrics[key]),
+ labels=labels,
+ label_names=label_names,
+ )
+ for metric_name, key in self.EXECUTOR_COUNTERS:
+ registry.set_counter(
+ metric_name,
+ float(metrics[key]),
+ labels=labels,
+ label_names=label_names,
+ )
+
def collect_stale_hook(self, registry, error: Exception) -> None:
"""Emit invalid readiness series for the last known probe set on failure."""
- for probe in self._known_probes:
- self.replace_gauge(
- registry,
- self.READINESS,
- 0.0,
- match_labels={"probe": str(probe)},
- labels={"probe": str(probe), "valid": "0"},
- label_names=("probe", "valid"),
- )
+ self._write_readiness(
+ registry,
+ {str(probe): False for probe in self._known_probes},
+ valid=False,
+ )
diff --git a/openviking/metrics/core/runtime.py b/openviking/metrics/core/runtime.py
index 31aa07316..08afcc97c 100644
--- a/openviking/metrics/core/runtime.py
+++ b/openviking/metrics/core/runtime.py
@@ -4,7 +4,8 @@
"""
Runtime utilities for the metrics subsystem.
-Currently this module defines the metrics subscriber's event router:
+Currently this module defines shared in-process metrics runtime helpers:
+- Default executor instrumentation used by async-system metrics.
- DataSources publish events to the shared observability event bus.
- Metrics subscribes with this router and routes each event to the matching Collector handler.
@@ -14,9 +15,185 @@ restricted to Collector implementations.
from __future__ import annotations
+import asyncio
+import multiprocessing
+import threading
from collections.abc import Callable
from typing import Any
+DEFAULT_EXECUTOR_POOL = "asyncio_default"
+DEFAULT_PROCESS_ROLE = "legacy_server"
+
+
+class DefaultExecutorMonitor:
+ """Monitor tasks submitted to one event loop's default executor."""
+
+ def __init__(
+ self,
+ *,
+ loop: asyncio.AbstractEventLoop,
+ process_role: str = DEFAULT_PROCESS_ROLE,
+ ) -> None:
+ """Create a monitor for a single event loop without installing it."""
+ self._loop = loop
+ self._process_role = str(process_role)
+ self._lock = threading.Lock()
+ self._original_run_in_executor: Callable[..., Any] | None = None
+ self._active_tasks = 0
+ self._pending_tasks = 0
+ self._submitted_total = 0
+ self._completed_total = 0
+ self._failed_total = 0
+
+ def install(self) -> "DefaultExecutorMonitor":
+ """Patch the event loop's default executor entrypoint and return this monitor."""
+ if self._original_run_in_executor is not None:
+ return self
+ original_run_in_executor = self._loop.run_in_executor
+ self._original_run_in_executor = original_run_in_executor
+
+ def _run_in_executor(executor, func, *args):
+ """Wrap default executor submissions and leave custom executors untouched."""
+ if executor is not None:
+ return original_run_in_executor(executor, func, *args)
+ return self._submit_default_executor(func, *args)
+
+ self._loop.run_in_executor = _run_in_executor # type: ignore[method-assign]
+ return self
+
+ def uninstall(self) -> None:
+ """Restore the original event loop entrypoint when this monitor installed it."""
+ global _current_executor_monitor
+ if self._original_run_in_executor is not None:
+ self._loop.run_in_executor = self._original_run_in_executor # type: ignore[method-assign]
+ self._original_run_in_executor = None
+ with _executor_monitor_lock:
+ if _current_executor_monitor is self:
+ _current_executor_monitor = None
+
+ def read_metrics(self) -> dict[str, int | str]:
+ """Return the latest default executor counters and thread state."""
+ executor = getattr(self._loop, "_default_executor", None)
+ max_workers = int(getattr(executor, "_max_workers", 0) or 0)
+ threads = len(getattr(executor, "_threads", ()) or ())
+ with self._lock:
+ return {
+ "process_role": self._process_role,
+ "worker": multiprocessing.current_process().name,
+ "pool": DEFAULT_EXECUTOR_POOL,
+ "max_workers": max_workers,
+ "threads": threads,
+ "active_tasks": self._active_tasks,
+ "pending_tasks": self._pending_tasks,
+ "submitted_total": self._submitted_total,
+ "completed_total": self._completed_total,
+ "failed_total": self._failed_total,
+ }
+
+ def _submit_default_executor(self, func, *args):
+ """Submit one wrapped call to the original default executor and update counters."""
+ run_in_executor = self._original_run_in_executor
+ if run_in_executor is None:
+ raise RuntimeError("executor monitor is not installed")
+ started = False
+ finished = False
+ with self._lock:
+ self._submitted_total += 1
+ self._pending_tasks += 1
+
+ def _wrapped():
+ """Run the original callable while tracking pending, active, and final states."""
+ nonlocal started, finished
+ with self._lock:
+ started = True
+ self._pending_tasks = max(0, self._pending_tasks - 1)
+ self._active_tasks += 1
+ try:
+ return func(*args)
+ except Exception:
+ with self._lock:
+ self._failed_total += 1
+ raise
+ finally:
+ with self._lock:
+ finished = True
+ self._active_tasks = max(0, self._active_tasks - 1)
+ self._completed_total += 1
+
+ try:
+ future = run_in_executor(None, _wrapped)
+ except Exception:
+ with self._lock:
+ self._pending_tasks = max(0, self._pending_tasks - 1)
+ self._failed_total += 1
+ raise
+
+ def _on_done(done_future) -> None:
+ """Drop pending state when cancellation prevents a queued call from starting."""
+ nonlocal finished
+ if not done_future.cancelled():
+ return
+ with self._lock:
+ if started or finished:
+ return
+ finished = True
+ self._pending_tasks = max(0, self._pending_tasks - 1)
+
+ future.add_done_callback(_on_done)
+ return future
+
+
+_executor_monitor_lock = threading.RLock()
+_current_executor_monitor: DefaultExecutorMonitor | None = None
+
+
+def install_executor_monitor(
+ *,
+ loop: asyncio.AbstractEventLoop | None = None,
+ process_role: str = DEFAULT_PROCESS_ROLE,
+) -> DefaultExecutorMonitor:
+ """Install the process-wide default executor monitor on the selected event loop."""
+ global _current_executor_monitor
+ target_loop = loop or asyncio.get_running_loop()
+ with _executor_monitor_lock:
+ if _current_executor_monitor is not None:
+ _current_executor_monitor.uninstall()
+ monitor = DefaultExecutorMonitor(
+ loop=target_loop,
+ process_role=process_role,
+ ).install()
+ _current_executor_monitor = monitor
+ return monitor
+
+
+def uninstall_executor_monitor() -> None:
+ """Uninstall the current process-wide default executor monitor if one exists."""
+ with _executor_monitor_lock:
+ if _current_executor_monitor is not None:
+ _current_executor_monitor.uninstall()
+
+
+def get_executor_monitor() -> DefaultExecutorMonitor | None:
+ """Return the current process-wide default executor monitor, if installed."""
+ with _executor_monitor_lock:
+ return _current_executor_monitor
+
+
+def empty_executor_metrics() -> dict[str, int | str]:
+ """Return a zero-valued default executor metric payload."""
+ return {
+ "process_role": DEFAULT_PROCESS_ROLE,
+ "worker": multiprocessing.current_process().name,
+ "pool": DEFAULT_EXECUTOR_POOL,
+ "max_workers": 0,
+ "threads": 0,
+ "active_tasks": 0,
+ "pending_tasks": 0,
+ "submitted_total": 0,
+ "completed_total": 0,
+ "failed_total": 0,
+ }
+
class EventCollectorRouter:
"""
diff --git a/openviking/metrics/datasources/probes.py b/openviking/metrics/datasources/probes.py
index 444428b09..972089ac9 100644
--- a/openviking/metrics/datasources/probes.py
+++ b/openviking/metrics/datasources/probes.py
@@ -6,6 +6,7 @@ from __future__ import annotations
from typing import Any, Callable
from openviking.metrics.core.base import ReadEnvelope
+from openviking.metrics.core.runtime import empty_executor_metrics, get_executor_monitor
from openviking.storage.queuefs import get_queue_manager
from openviking.storage.viking_fs import get_viking_fs
from openviking_cli.utils import run_async
@@ -136,3 +137,19 @@ class AsyncSystemProbeDataSource(ProbeMetricDataSource):
def read_probe_state(self) -> ReadEnvelope[dict[str, bool]]:
"""Check whether the shared async queue subsystem can be resolved successfully."""
return self.safe_bool_probe("queue", lambda: bool(get_queue_manager()))
+
+ def read_async_system_state(self) -> ReadEnvelope[dict[str, Any]]:
+ """Read readiness and executor state in one async-system scrape input."""
+ probe_state = self.read_probe_state()
+ monitor = get_executor_monitor()
+ executor_metrics = (
+ monitor.read_metrics() if monitor is not None else empty_executor_metrics()
+ )
+ return ReadEnvelope(
+ ok=True,
+ value={
+ "probes": probe_state.value,
+ "probes_valid": probe_state.ok,
+ "executor": executor_metrics,
+ },
+ )
diff --git a/openviking/server/app.py b/openviking/server/app.py
index 835adfdfd..671235bdc 100644
--- a/openviking/server/app.py
+++ b/openviking/server/app.py
@@ -319,6 +319,10 @@ def create_app(
"""Application lifespan handler."""
nonlocal service
_configure_default_executor(config)
+ if config.observability.metrics.enabled:
+ from openviking.metrics.core.runtime import install_executor_monitor
+
+ install_executor_monitor()
owns_service = service is None
if owns_service:
service = OpenVikingService()
@@ -386,6 +390,10 @@ def create_app(
await shutdown_usage_audit(app=app)
await shutdown_metrics_async(app=app)
+ if config.observability.metrics.enabled:
+ from openviking.metrics.core.runtime import uninstall_executor_monitor
+
+ uninstall_executor_monitor()
task_tracker.stop_cleanup_loop()
auth_plugin_state = getattr(app.state, "auth_plugin", None)
if auth_plugin_state is not None:
diff --git a/tests/metrics/collectors/test_collector_contracts.py b/tests/metrics/collectors/test_collector_contracts.py
index dc9e425ef..68f82705e 100644
--- a/tests/metrics/collectors/test_collector_contracts.py
+++ b/tests/metrics/collectors/test_collector_contracts.py
@@ -22,6 +22,7 @@ from openviking.metrics.collectors.service_probe import ServiceProbeCollector
from openviking.metrics.collectors.storage_probe import StorageProbeCollector
from openviking.metrics.core.base import ReadEnvelope
from openviking.metrics.core.registry import MetricRegistry
+from openviking.metrics.datasources import probes as probe_datasources
class _FailingProbeCollector(ProbeMetricCollector):
@@ -279,14 +280,20 @@ def test_model_provider_probe_collector_uses_last_provider_on_failure(registry,
assert 'openviking_model_provider_readiness{provider="volcengine",valid="0"} 0.0' in text2
-def test_async_system_probe_collector_marks_invalid_on_failure(registry, render_prometheus):
- ds = _ProbeDataSource(value={"queue": True}, ok=True)
+def test_async_system_probe_collector_marks_invalid_on_failure(
+ monkeypatch, registry, render_prometheus
+):
+ monkeypatch.setattr(probe_datasources, "get_queue_manager", lambda: object())
+ ds = probe_datasources.AsyncSystemProbeDataSource()
c = AsyncSystemProbeCollector(data_source=ds)
c.collect(registry)
text = render_prometheus(registry)
assert 'openviking_async_system_readiness{probe="queue",valid="1"} 1.0' in text
- ds.raises = True
+ def _raise_queue_manager():
+ raise RuntimeError("probe read failed")
+
+ monkeypatch.setattr(probe_datasources, "get_queue_manager", _raise_queue_manager)
c.collect(registry)
text2 = render_prometheus(registry)
assert 'openviking_async_system_readiness{probe="queue",valid="0"} 0.0' in text2
diff --git a/tests/metrics/collectors/test_state_collectors.py b/tests/metrics/collectors/test_state_collectors.py
index 67ef5b2c1..57b5795cd 100644
--- a/tests/metrics/collectors/test_state_collectors.py
+++ b/tests/metrics/collectors/test_state_collectors.py
@@ -3,17 +3,22 @@
from __future__ import annotations
+import asyncio
+import multiprocessing
from concurrent.futures import ThreadPoolExecutor
from threading import Event
from types import SimpleNamespace
import pytest
+from openviking.metrics.collectors.async_system_probe import AsyncSystemProbeCollector
from openviking.metrics.collectors.observer_health import ObserverHealthCollector
from openviking.metrics.collectors.queue import QueueCollector
from openviking.metrics.collectors.task_tracker import TaskTrackerCollector
from openviking.metrics.collectors.vikingdb import VikingDBCollector
+from openviking.metrics.core.base import ReadEnvelope
from openviking.metrics.core.registry import MetricRegistry
+from openviking.metrics.core.runtime import install_executor_monitor, uninstall_executor_monitor
from openviking.metrics.datasources.observer_state import (
ObserverStateDataSource,
VikingDBStateDataSource,
@@ -23,6 +28,107 @@ from openviking.metrics.datasources.task import TaskStateDataSource
from openviking.metrics.exporters.prometheus import PrometheusExporter
+def test_executor_collector_exports_default_executor_metrics(registry, render_prometheus):
+ class DummyDataSource:
+ def read_async_system_state(self):
+ return ReadEnvelope(
+ ok=True,
+ value={
+ "probes": {"queue": True},
+ "probes_valid": True,
+ "executor": {
+ "process_role": "legacy_server",
+ "worker": "MainProcess",
+ "pool": "asyncio_default",
+ "max_workers": 4,
+ "threads": 2,
+ "active_tasks": 1,
+ "pending_tasks": 3,
+ "submitted_total": 5,
+ "completed_total": 4,
+ "failed_total": 1,
+ },
+ },
+ )
+
+ AsyncSystemProbeCollector(data_source=DummyDataSource()).collect(registry)
+ text = render_prometheus(registry)
+ labels = 'pool="asyncio_default",process_role="legacy_server",worker="MainProcess"'
+ expected = {
+ "openviking_executor_max_workers": "4.0",
+ "openviking_executor_threads": "2.0",
+ "openviking_executor_active_tasks": "1.0",
+ "openviking_executor_pending_tasks": "3.0",
+ "openviking_executor_submitted_total": "5",
+ "openviking_executor_completed_total": "4",
+ "openviking_executor_failed_total": "1",
+ }
+ for name, value in expected.items():
+ assert f"{name}{{{labels}}} {value}" in text
+
+
+async def test_executor_monitor_tracks_default_executor_only():
+ loop = asyncio.get_running_loop()
+ previous_default = getattr(loop, "_default_executor", None)
+ default_executor = ThreadPoolExecutor(max_workers=1)
+ custom_executor = ThreadPoolExecutor(max_workers=1)
+ loop.set_default_executor(default_executor)
+ monitor = install_executor_monitor(
+ loop=loop,
+ process_role="legacy_server",
+ )
+ started, release = Event(), Event()
+
+ def blocking_default():
+ started.set()
+ release.wait(5)
+ return "ok"
+
+ def raising_default():
+ raise RuntimeError("boom")
+
+ try:
+ first = loop.run_in_executor(None, blocking_default)
+ while not started.is_set():
+ await asyncio.sleep(0.01)
+ second = loop.run_in_executor(None, lambda: "queued")
+
+ metrics = monitor.read_metrics()
+ assert metrics["active_tasks"] == 1
+ assert metrics["pending_tasks"] == 1
+ assert metrics["submitted_total"] == 2
+
+ release.set()
+ assert await first == "ok"
+ assert await second == "queued"
+ with pytest.raises(RuntimeError, match="boom"):
+ await loop.run_in_executor(None, raising_default)
+ assert await loop.run_in_executor(custom_executor, lambda: "custom") == "custom"
+
+ final = monitor.read_metrics()
+ assert final == {
+ "process_role": "legacy_server",
+ "worker": multiprocessing.current_process().name,
+ "pool": "asyncio_default",
+ "max_workers": 1,
+ "threads": 1,
+ "active_tasks": 0,
+ "pending_tasks": 0,
+ "submitted_total": 3,
+ "completed_total": 3,
+ "failed_total": 1,
+ }
+ finally:
+ uninstall_executor_monitor()
+ if previous_default is not None:
+ loop.set_default_executor(previous_default)
+ else:
+ loop._default_executor = None # type: ignore[attr-defined]
+ custom_executor.shutdown(wait=True)
+ default_executor.shutdown(wait=True)
+
+
def test_queue_collector_maps_status(monkeypatch):
class DummyQueueStatus:
def __init__(