Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(metric): add executor && task metric export (#5105)
baojun-zhang3b6f37e9d288
diff --git a/docs/design/metric-design.md b/docs/design/metric-design.mdindex 130ac7e57..46471c7e7 100644--- a/docs/design/metric-design.md+++ b/docs/design/metric-design.md@@ -312,7 +312,7 @@ graph LR| `StorageProbeDataSource` | `ProbeMetricDataSource` | AGFS / VikingFS、系统表访问检查 | 存储探针结果 | storage readability、storage writability、system table readiness || `RetrievalBackendProbeDataSource` | `ProbeMetricDataSource` | VikingDB、检索后端最小能力检查 | 检索后端探针结果 | backend readiness、collection availability || `ModelProviderProbeDataSource` | `ProbeMetricDataSource` | VLM / Embedding / Rerank provider 可用性检查 | 模型依赖探针结果 | provider readiness、credential availability |-| `AsyncSystemProbeDataSource` | `ProbeMetricDataSource` | Queue、TaskTracker、后台消费线程检查 | 异步系统探针结果 | queue readiness、worker liveness |+| `AsyncSystemProbeDataSource` | `ProbeMetricDataSource` | Queue、默认 asyncio executor monitor | 异步系统探针与 executor 指标 | queue readiness、executor threads / tasks || `EncryptionProbeDataSource` | `ProbeMetricDataSource` | Root Key、KMS / Vault Provider、加密组件检查 | 加密探针结果 | root key readiness、kms availability、encryption component health |设计边界如下:@@ -427,7 +427,7 @@ Collector 与 DataSource 的主映射关系如下:| `StorageProbeCollector` | `ProbeMetricCollector` | StorageProbeDataSource | storage readiness、system table availability || `RetrievalBackendProbeCollector` | `ProbeMetricCollector` | RetrievalBackendProbeDataSource | backend readiness、collection availability || `ModelProviderProbeCollector` | `ProbeMetricCollector` | ModelProviderProbeDataSource | provider readiness、credential availability |-| `AsyncSystemProbeCollector` | `ProbeMetricCollector` | AsyncSystemProbeDataSource | queue readiness、worker liveness |+| `AsyncSystemProbeCollector` | `ProbeMetricCollector` | AsyncSystemProbeDataSource | queue readiness、默认 executor 线程与任务指标 || `EncryptionProbeCollector` | `ProbeMetricCollector` | EncryptionProbeDataSource | root key readiness、kms availability、encryption component health |设计理由:在引入四类 Collector 之后,整个体系的语义边界更清晰:@@ -644,6 +644,7 @@ sequenceDiagram| RAGFS 监控 | `RagfsMetricDataSource` | `RagfsMetricCollector` | Counter、Histogram、Gauge | 文件操作、Cache、multi-backend、Lock || 加密监控 | `EncryptionEventDataSource`、`EncryptionProbeDataSource` | `EncryptionCollector`、`EncryptionProbeCollector` | Counter、Histogram、Gauge | encrypt count、decrypt duration、root key readiness || 系统探针监控 | 各类 `*ProbeDataSource` | 各类 `*ProbeCollector` | Gauge、Health | service readiness、storage readiness、kms availability |+| Python executor 监控 | `AsyncSystemProbeDataSource` | `AsyncSystemProbeCollector` | Gauge、Counter | 默认 executor 线程数、任务数、提交/完成/失败计数 || 操作级 Telemetry 指标化 | telemetry adapter / bridge | `TelemetryBridgeCollector` | Counter、Histogram、Gauge | operation requests、vector scanned、memory extracted |### 3.2 指标对象模型@@ -808,7 +809,81 @@ sequenceDiagram| `summary.semantic_nodes.{total|done|pending|running}` | `openviking_semantic_nodes_total{status=...}` || `summary.memory.extracted` | `openviking_memory_extracted_total{memory_type=...}` |-### 3.8 多租户支持范围+### 3.8 Python 默认 executor 指标++Python executor 指标用于观察当前服务进程内 asyncio 默认 executor 的线程和任务状态。+它只覆盖 Python 默认 executor,不覆盖 Rust / RAGFS 内部 Tokio runtime。++当前代码路径:++| 文件 | 符号 | 职责 |+| --- | --- | --- |+| `openviking/server/app.py` | `lifespan()` | 在 `_configure_default_executor(config)` 后安装 monitor,在 shutdown metrics 后卸载 |+| `openviking/metrics/core/runtime.py` | `DefaultExecutorMonitor` | 保存 event loop、原始 `run_in_executor`、计数器和锁 |+| `openviking/metrics/core/runtime.py` | `install_executor_monitor()` / `uninstall_executor_monitor()` | 安装和恢复当前 event loop 的默认 executor 入口 |+| `openviking/metrics/datasources/probes.py` | `AsyncSystemProbeDataSource.read_async_system_state()` | 合并 queue readiness 和 executor metrics |+| `openviking/metrics/collectors/async_system_probe.py` | `AsyncSystemProbeCollector` | 写 `openviking_async_system_readiness` 和 `openviking_executor_*` |++采集链路:++```text+/metrics+-> CollectorManager.collect_all()+-> asyncio.to_thread(collector.collect, registry)+-> AsyncSystemProbeCollector.read_metric_input()+-> AsyncSystemProbeDataSource.read_async_system_state()+-> get_executor_monitor().read_metrics()+-> AsyncSystemProbeCollector.collect_hook()+-> MetricRegistry+```++统计范围:++```text+统计:loop.run_in_executor(None, func, *args)+统计:asyncio.to_thread(func, *args, **kwargs)+不统计:loop.run_in_executor(custom_executor, func, *args)+不统计:concurrent.futures.ThreadPoolExecutor(...) 直接创建的 executor+不统计:Rust / RAGFS 内部 runtime 或线程+```++指标定义:++| 指标名 | 类型 | 标签 | 说明 |+| --- | --- | --- | --- |+| `openviking_executor_max_workers` | Gauge | `pool,process_role,worker` | 默认 executor 最大 worker 数 |+| `openviking_executor_threads` | Gauge | `pool,process_role,worker` | 默认 executor 已创建线程数 |+| `openviking_executor_active_tasks` | Gauge | `pool,process_role,worker` | 当前正在执行的默认 executor 任务数 |+| `openviking_executor_pending_tasks` | Gauge | `pool,process_role,worker` | 当前等待执行的默认 executor 任务数 |+| `openviking_executor_submitted_total` | Counter | `pool,process_role,worker` | 累计提交到默认 executor 的任务数 |+| `openviking_executor_completed_total` | Counter | `pool,process_role,worker` | 累计执行结束的默认 executor 任务数,失败也计入 |+| `openviking_executor_failed_total` | Counter | `pool,process_role,worker` | 默认 executor callable 抛异常的累计次数 |++固定标签:++```text+pool="asyncio_default"+process_role="legacy_server"+worker=multiprocessing.current_process().name+```++`failed_total` 的语义:++- 该指标统计 worker callable 抛出的 `Exception`。+- 如果异常被上层业务捕获并作为正常分支处理,也会计入。+- 它不等同于业务请求失败数。+- `/metrics` scrape 自身会通过 `asyncio.to_thread()` 刷新 collector,因此也会计入 executor 指标。++运行观测:++- monitor 在服务初始化前安装,会统计启动阶段的默认 executor 调用。+- AGFS 初始化中的文件不存在、目录已存在等探测式异常会计入 `failed_total`。+- 当前运行时如果加载旧版 `ragfs_python` binding,`RagfsMetricCollector` 调用+ `service._agfs_client.metrics()` 会失败。+- repo 当前源码 `crates/ragfs-python/src/lib.rs` 已定义 `metrics()`;若运行时对象没有该方法,+ 说明 `.venv` 里的 native binding 产物与源码能力不一致。++### 3.9 多租户支持范围当前代码中,只有进入 `ACCOUNT_DIMENSION_SUPPORTED_METRICS` 支持集的指标族允许注入 `account_id`。支持集如下:diff --git a/docs/en/concepts/12-metrics.md b/docs/en/concepts/12-metrics.mdindex f40bdafa4..07191d558 100644--- a/docs/en/concepts/12-metrics.md+++ b/docs/en/concepts/12-metrics.md@@ -228,6 +228,13 @@ Notes:| `openviking_queue_errors_total` | Counter | `queue` | total error count per queue || `openviking_queue_pending` | Gauge | `queue` | pending queue items || `openviking_queue_in_progress` | Gauge | `queue` | in-progress queue items |+| `openviking_executor_max_workers` | Gauge | `pool, process_role, worker` | maximum workers in the asyncio default executor |+| `openviking_executor_threads` | Gauge | `pool, process_role, worker` | threads created by the asyncio default executor |+| `openviking_executor_active_tasks` | Gauge | `pool, process_role, worker` | default executor tasks currently running |+| `openviking_executor_pending_tasks` | Gauge | `pool, process_role, worker` | default executor tasks waiting to run |+| `openviking_executor_submitted_total` | Counter | `pool, process_role, worker` | total tasks submitted to the default executor |+| `openviking_executor_completed_total` | Counter | `pool, process_role, worker` | default executor tasks that finished, including failed tasks |+| `openviking_executor_failed_total` | Counter | `pool, process_role, worker` | default executor callables that raised exceptions || `openviking_lock_active` | Gauge | none | current published lock leases || `openviking_lock_waiting` | Gauge | none | requests currently waiting for locks || `openviking_lock_stale` | Gauge | none | cumulative count of stale lock tokens removed |@@ -240,6 +247,14 @@ These help answer:- Is there queue backlog?- Is there lock contention or stale locking?+- Is the default executor near its worker limit or building a queue?++Executor metrics only cover `loop.run_in_executor(None, ...)` and+`asyncio.to_thread(...)`. Calls using an explicit custom executor and Rust /+RAGFS internal runtimes or threads are not included. `failed_total` counts+callables that raise exceptions. If the caller catches that exception and treats+it as a normal branch, it still counts here, so this is not the business request+failure count.### RAGFSdiff --git a/docs/zh/concepts/12-metrics.md b/docs/zh/concepts/12-metrics.mdindex 725ac7f96..50c123003 100644--- a/docs/zh/concepts/12-metrics.md+++ b/docs/zh/concepts/12-metrics.md@@ -231,6 +231,13 @@ scrape_configs:| `openviking_queue_errors_total` | Counter | `queue` | 队列累计错误量 || `openviking_queue_pending` | Gauge | `queue` | 队列待处理数 || `openviking_queue_in_progress` | Gauge | `queue` | 队列执行中数量 |+| `openviking_executor_max_workers` | Gauge | `pool, process_role, worker` | asyncio 默认 executor 最大 worker 数 |+| `openviking_executor_threads` | Gauge | `pool, process_role, worker` | asyncio 默认 executor 已创建线程数 |+| `openviking_executor_active_tasks` | Gauge | `pool, process_role, worker` | 默认 executor 当前执行中的任务数 |+| `openviking_executor_pending_tasks` | Gauge | `pool, process_role, worker` | 默认 executor 当前等待执行的任务数 |+| `openviking_executor_submitted_total` | Counter | `pool, process_role, worker` | 默认 executor 累计提交任务数 |+| `openviking_executor_completed_total` | Counter | `pool, process_role, worker` | 默认 executor 累计执行结束任务数,失败也计入 |+| `openviking_executor_failed_total` | Counter | `pool, process_role, worker` | 默认 executor callable 抛异常次数 || `openviking_lock_active` | Gauge | 无 | 当前已发布的锁租约数 || `openviking_lock_waiting` | Gauge | 无 | 当前等待锁的请求数 || `openviking_lock_stale` | Gauge | 无 | 累计已清理的过期锁 token 数 |@@ -243,6 +250,12 @@ scrape_configs:- 是否有队列堆积?- 是否有锁竞争或 stale lock?+- 默认 executor 是否接近线程上限或出现排队?++executor 指标只统计 `loop.run_in_executor(None, ...)` 和 `asyncio.to_thread(...)`。+显式传入自定义 executor 的调用、Rust / RAGFS 内部 runtime 或线程不计入。+`failed_total` 统计 callable 抛异常的次数。如果异常被上层捕获并作为正常分支处理,+也会计入该指标,因此它不等同于业务请求失败数。### RAGFSdiff --git a/openviking/metrics/collectors/async_system_probe.py b/openviking/metrics/collectors/async_system_probe.pyindex 89fcae2c8..02c6f77f0 100644--- a/openviking/metrics/collectors/async_system_probe.py+++ b/openviking/metrics/collectors/async_system_probe.py@@ -15,17 +15,27 @@ from .base import CollectorConfig, ProbeMetricCollector@dataclassclass AsyncSystemProbeCollector(ProbeMetricCollector):"""- Export readiness for individual async-system probe checks.+ Export async-system readiness and Python default executor metrics.- The datasource returns a bounded mapping of probe names to booleans. This collector converts- that mapping into one readiness gauge series per probe so dashboards can distinguish which- async subsystem is currently unhealthy.+ Readiness remains one gauge series per probe. Executor metrics use `pool`,+ `process_role`, and `worker` labels."""DOMAIN: ClassVar[str] = "async_system"# rule: <METRICS_NAMESPACE>_<DOMAIN>_readiness# e.g.: openviking_async_system_readinessREADINESS: ClassVar[str] = MetricCollector.metric_name(DOMAIN, "readiness")+ EXECUTOR_GAUGES: ClassVar[tuple[tuple[str, str], ...]] = (+ (MetricCollector.metric_name("executor", "max_workers"), "max_workers"),+ (MetricCollector.metric_name("executor", "threads"), "threads"),+ (MetricCollector.metric_name("executor", "active_tasks"), "active_tasks"),+ (MetricCollector.metric_name("executor", "pending_tasks"), "pending_tasks"),+ )+ EXECUTOR_COUNTERS: ClassVar[tuple[tuple[str, str], ...]] = (+ (MetricCollector.metric_name("executor", "submitted", unit="total"), "submitted_total"),+ (MetricCollector.metric_name("executor", "completed", unit="total"), "completed_total"),+ (MetricCollector.metric_name("executor", "failed", unit="total"), "failed_total"),+ )data_source: AsyncSystemProbeDataSourceconfig: CollectorConfig = CollectorConfig(timeout_seconds=0.5)@@ -38,36 +48,63 @@ class AsyncSystemProbeCollector(ProbeMetricCollector):self._known_probes = ["queue"]def read_metric_input(self):- """Read the latest async-system probe mapping from the datasource."""- return self.data_source.read_probe_state()+ """Read the latest async-system probe and executor metrics from the datasource."""+ return self.data_source.read_async_system_state()def collect_hook(self, registry, metric_input) -> None:"""- Publish one readiness sample per async-system probe name.+ Publish readiness samples and default-executor runtime samples.- The helper normalizes probe names to strings and writes a fixed 0/1 readiness gauge using- `probe` as the only series label.+ The readiness helper normalizes probe names to strings and writes a fixed 0/1 readiness+ gauge. Executor samples keep a low-cardinality `pool/process_role/worker` label set."""- normalized = {str(k): bool(v) for k, v in metric_input.items()}+ probes = metric_input["probes"]+ probes_valid = bool(metric_input["probes_valid"])+ executor = metric_input["executor"]+ normalized = {str(k): bool(v) for k, v in probes.items()}self._known_probes = list(normalized.keys())- for probe, is_ready in normalized.items():+ self._write_readiness(registry, normalized, valid=probes_valid)+ self._write_executor_metrics(registry, executor)++ def _write_readiness(self, registry, probes: dict[str, bool], *, valid: bool) -> None:+ """Write readiness gauges for async-system probes."""+ for probe, is_ready in probes.items():self.replace_gauge(registry,self.READINESS,- 1.0 if bool(is_ready) else 0.0,+ 1.0 if valid and bool(is_ready) else 0.0,match_labels={"probe": str(probe)},- labels={"probe": str(probe), "valid": "1"},+ labels={"probe": str(probe), "valid": "1" if valid else "0"},label_names=("probe", "valid"),)+ def _write_executor_metrics(self, registry, metrics: dict[str, int | str]) -> None:+ """Write Python default executor metrics using stable labels."""+ labels = {+ "pool": str(metrics["pool"]),+ "process_role": str(metrics["process_role"]),+ "worker": str(metrics["worker"]),+ }+ label_names = ("pool", "process_role", "worker")+ for metric_name, key in self.EXECUTOR_GAUGES:+ registry.set_gauge(+ metric_name,+ float(metrics[key]),+ labels=labels,+ label_names=label_names,+ )+ for metric_name, key in self.EXECUTOR_COUNTERS:+ registry.set_counter(+ metric_name,+ float(metrics[key]),+ labels=labels,+ label_names=label_names,+ )+def collect_stale_hook(self, registry, error: Exception) -> None:"""Emit invalid readiness series for the last known probe set on failure."""- for probe in self._known_probes:- self.replace_gauge(- registry,- self.READINESS,- 0.0,- match_labels={"probe": str(probe)},- labels={"probe": str(probe), "valid": "0"},- label_names=("probe", "valid"),- )+ self._write_readiness(+ registry,+ {str(probe): False for probe in self._known_probes},+ valid=False,+ )diff --git a/openviking/metrics/core/runtime.py b/openviking/metrics/core/runtime.pyindex 31aa07316..08afcc97c 100644--- a/openviking/metrics/core/runtime.py+++ b/openviking/metrics/core/runtime.py@@ -4,7 +4,8 @@"""Runtime utilities for the metrics subsystem.-Currently this module defines the metrics subscriber's event router:+Currently this module defines shared in-process metrics runtime helpers:+- Default executor instrumentation used by async-system metrics.- DataSources publish events to the shared observability event bus.- Metrics subscribes with this router and routes each event to the matching Collector handler.@@ -14,9 +15,185 @@ restricted to Collector implementations.from __future__ import annotations+import asyncio+import multiprocessing+import threadingfrom collections.abc import Callablefrom typing import Any+DEFAULT_EXECUTOR_POOL = "asyncio_default"+DEFAULT_PROCESS_ROLE = "legacy_server"+++class DefaultExecutorMonitor:+ """Monitor tasks submitted to one event loop's default executor."""++ def __init__(+ self,+ *,+ loop: asyncio.AbstractEventLoop,+ process_role: str = DEFAULT_PROCESS_ROLE,+ ) -> None:+ """Create a monitor for a single event loop without installing it."""+ self._loop = loop+ self._process_role = str(process_role)+ self._lock = threading.Lock()+ self._original_run_in_executor: Callable[..., Any] | None = None+ self._active_tasks = 0+ self._pending_tasks = 0+ self._submitted_total = 0+ self._completed_total = 0+ self._failed_total = 0++ def install(self) -> "DefaultExecutorMonitor":+ """Patch the event loop's default executor entrypoint and return this monitor."""+ if self._original_run_in_executor is not None:+ return self+ original_run_in_executor = self._loop.run_in_executor+ self._original_run_in_executor = original_run_in_executor++ def _run_in_executor(executor, func, *args):+ """Wrap default executor submissions and leave custom executors untouched."""+ if executor is not None:+ return original_run_in_executor(executor, func, *args)+ return self._submit_default_executor(func, *args)++ self._loop.run_in_executor = _run_in_executor # type: ignore[method-assign]+ return self++ def uninstall(self) -> None:+ """Restore the original event loop entrypoint when this monitor installed it."""+ global _current_executor_monitor+ if self._original_run_in_executor is not None:+ self._loop.run_in_executor = self._original_run_in_executor # type: ignore[method-assign]+ self._original_run_in_executor = None+ with _executor_monitor_lock:+ if _current_executor_monitor is self:+ _current_executor_monitor = None++ def read_metrics(self) -> dict[str, int | str]:+ """Return the latest default executor counters and thread state."""+ executor = getattr(self._loop, "_default_executor", None)+ max_workers = int(getattr(executor, "_max_workers", 0) or 0)+ threads = len(getattr(executor, "_threads", ()) or ())+ with self._lock:+ return {+ "process_role": self._process_role,+ "worker": multiprocessing.current_process().name,+ "pool": DEFAULT_EXECUTOR_POOL,+ "max_workers": max_workers,+ "threads": threads,+ "active_tasks": self._active_tasks,+ "pending_tasks": self._pending_tasks,+ "submitted_total": self._submitted_total,+ "completed_total": self._completed_total,+ "failed_total": self._failed_total,+ }++ def _submit_default_executor(self, func, *args):+ """Submit one wrapped call to the original default executor and update counters."""+ run_in_executor = self._original_run_in_executor+ if run_in_executor is None:+ raise RuntimeError("executor monitor is not installed")+ started = False+ finished = False+ with self._lock:+ self._submitted_total += 1+ self._pending_tasks += 1++ def _wrapped():+ """Run the original callable while tracking pending, active, and final states."""+ nonlocal started, finished+ with self._lock:+ started = True+ self._pending_tasks = max(0, self._pending_tasks - 1)+ self._active_tasks += 1+ try:+ return func(*args)+ except Exception:+ with self._lock:+ self._failed_total += 1+ raise+ finally:+ with self._lock:+ finished = True+ self._active_tasks = max(0, self._active_tasks - 1)+ self._completed_total += 1++ try:+ future = run_in_executor(None, _wrapped)+ except Exception:+ with self._lock:+ self._pending_tasks = max(0, self._pending_tasks - 1)+ self._failed_total += 1+ raise++ def _on_done(done_future) -> None:+ """Drop pending state when cancellation prevents a queued call from starting."""+ nonlocal finished+ if not done_future.cancelled():+ return+ with self._lock:+ if started or finished:+ return+ finished = True+ self._pending_tasks = max(0, self._pending_tasks - 1)++ future.add_done_callback(_on_done)+ return future+++_executor_monitor_lock = threading.RLock()+_current_executor_monitor: DefaultExecutorMonitor | None = None+++def install_executor_monitor(+ *,+ loop: asyncio.AbstractEventLoop | None = None,+ process_role: str = DEFAULT_PROCESS_ROLE,+) -> DefaultExecutorMonitor:+ """Install the process-wide default executor monitor on the selected event loop."""+ global _current_executor_monitor+ target_loop = loop or asyncio.get_running_loop()+ with _executor_monitor_lock:+ if _current_executor_monitor is not None:+ _current_executor_monitor.uninstall()+ monitor = DefaultExecutorMonitor(+ loop=target_loop,+ process_role=process_role,+ ).install()+ _current_executor_monitor = monitor+ return monitor+++def uninstall_executor_monitor() -> None:+ """Uninstall the current process-wide default executor monitor if one exists."""+ with _executor_monitor_lock:+ if _current_executor_monitor is not None:+ _current_executor_monitor.uninstall()+++def get_executor_monitor() -> DefaultExecutorMonitor | None:+ """Return the current process-wide default executor monitor, if installed."""+ with _executor_monitor_lock:+ return _current_executor_monitor+++def empty_executor_metrics() -> dict[str, int | str]:+ """Return a zero-valued default executor metric payload."""+ return {+ "process_role": DEFAULT_PROCESS_ROLE,+ "worker": multiprocessing.current_process().name,+ "pool": DEFAULT_EXECUTOR_POOL,+ "max_workers": 0,+ "threads": 0,+ "active_tasks": 0,+ "pending_tasks": 0,+ "submitted_total": 0,+ "completed_total": 0,+ "failed_total": 0,+ }+class EventCollectorRouter:"""diff --git a/openviking/metrics/datasources/probes.py b/openviking/metrics/datasources/probes.pyindex 444428b09..972089ac9 100644--- a/openviking/metrics/datasources/probes.py+++ b/openviking/metrics/datasources/probes.py@@ -6,6 +6,7 @@ from __future__ import annotationsfrom typing import Any, Callablefrom openviking.metrics.core.base import ReadEnvelope+from openviking.metrics.core.runtime import empty_executor_metrics, get_executor_monitorfrom openviking.storage.queuefs import get_queue_managerfrom openviking.storage.viking_fs import get_viking_fsfrom openviking_cli.utils import run_async@@ -136,3 +137,19 @@ class AsyncSystemProbeDataSource(ProbeMetricDataSource):def read_probe_state(self) -> ReadEnvelope[dict[str, bool]]:"""Check whether the shared async queue subsystem can be resolved successfully."""return self.safe_bool_probe("queue", lambda: bool(get_queue_manager()))++ def read_async_system_state(self) -> ReadEnvelope[dict[str, Any]]:+ """Read readiness and executor state in one async-system scrape input."""+ probe_state = self.read_probe_state()+ monitor = get_executor_monitor()+ executor_metrics = (+ monitor.read_metrics() if monitor is not None else empty_executor_metrics()+ )+ return ReadEnvelope(+ ok=True,+ value={+ "probes": probe_state.value,+ "probes_valid": probe_state.ok,+ "executor": executor_metrics,+ },+ )diff --git a/openviking/server/app.py b/openviking/server/app.pyindex 835adfdfd..671235bdc 100644--- a/openviking/server/app.py+++ b/openviking/server/app.py@@ -319,6 +319,10 @@ def create_app("""Application lifespan handler."""nonlocal service_configure_default_executor(config)+ if config.observability.metrics.enabled:+ from openviking.metrics.core.runtime import install_executor_monitor++ install_executor_monitor()owns_service = service is Noneif owns_service:service = OpenVikingService()@@ -386,6 +390,10 @@ def create_app(await shutdown_usage_audit(app=app)await shutdown_metrics_async(app=app)+ if config.observability.metrics.enabled:+ from openviking.metrics.core.runtime import uninstall_executor_monitor++ uninstall_executor_monitor()task_tracker.stop_cleanup_loop()auth_plugin_state = getattr(app.state, "auth_plugin", None)if auth_plugin_state is not None:diff --git a/tests/metrics/collectors/test_collector_contracts.py b/tests/metrics/collectors/test_collector_contracts.pyindex dc9e425ef..68f82705e 100644--- a/tests/metrics/collectors/test_collector_contracts.py+++ b/tests/metrics/collectors/test_collector_contracts.py@@ -22,6 +22,7 @@ from openviking.metrics.collectors.service_probe import ServiceProbeCollectorfrom openviking.metrics.collectors.storage_probe import StorageProbeCollectorfrom openviking.metrics.core.base import ReadEnvelopefrom openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.datasources import probes as probe_datasourcesclass _FailingProbeCollector(ProbeMetricCollector):@@ -279,14 +280,20 @@ def test_model_provider_probe_collector_uses_last_provider_on_failure(registry,assert 'openviking_model_provider_readiness{provider="volcengine",valid="0"} 0.0' in text2-def test_async_system_probe_collector_marks_invalid_on_failure(registry, render_prometheus):- ds = _ProbeDataSource(value={"queue": True}, ok=True)+def test_async_system_probe_collector_marks_invalid_on_failure(+ monkeypatch, registry, render_prometheus+):+ monkeypatch.setattr(probe_datasources, "get_queue_manager", lambda: object())+ ds = probe_datasources.AsyncSystemProbeDataSource()c = AsyncSystemProbeCollector(data_source=ds)c.collect(registry)text = render_prometheus(registry)assert 'openviking_async_system_readiness{probe="queue",valid="1"} 1.0' in text- ds.raises = True+ def _raise_queue_manager():+ raise RuntimeError("probe read failed")++ monkeypatch.setattr(probe_datasources, "get_queue_manager", _raise_queue_manager)c.collect(registry)text2 = render_prometheus(registry)assert 'openviking_async_system_readiness{probe="queue",valid="0"} 0.0' in text2diff --git a/tests/metrics/collectors/test_state_collectors.py b/tests/metrics/collectors/test_state_collectors.pyindex 67ef5b2c1..57b5795cd 100644--- a/tests/metrics/collectors/test_state_collectors.py+++ b/tests/metrics/collectors/test_state_collectors.py@@ -3,17 +3,22 @@from __future__ import annotations+import asyncio+import multiprocessingfrom concurrent.futures import ThreadPoolExecutorfrom threading import Eventfrom types import SimpleNamespaceimport pytest+from openviking.metrics.collectors.async_system_probe import AsyncSystemProbeCollectorfrom openviking.metrics.collectors.observer_health import ObserverHealthCollectorfrom openviking.metrics.collectors.queue import QueueCollectorfrom openviking.metrics.collectors.task_tracker import TaskTrackerCollectorfrom openviking.metrics.collectors.vikingdb import VikingDBCollector+from openviking.metrics.core.base import ReadEnvelopefrom openviking.metrics.core.registry import MetricRegistry+from openviking.metrics.core.runtime import install_executor_monitor, uninstall_executor_monitorfrom openviking.metrics.datasources.observer_state import (ObserverStateDataSource,VikingDBStateDataSource,@@ -23,6 +28,107 @@ from openviking.metrics.datasources.task import TaskStateDataSourcefrom openviking.metrics.exporters.prometheus import PrometheusExporter+def test_executor_collector_exports_default_executor_metrics(registry, render_prometheus):+ class DummyDataSource:+ def read_async_system_state(self):+ return ReadEnvelope(+ ok=True,+ value={+ "probes": {"queue": True},+ "probes_valid": True,+ "executor": {+ "process_role": "legacy_server",+ "worker": "MainProcess",+ "pool": "asyncio_default",+ "max_workers": 4,+ "threads": 2,+ "active_tasks": 1,+ "pending_tasks": 3,+ "submitted_total": 5,+ "completed_total": 4,+ "failed_total": 1,+ },+ },+ )++ AsyncSystemProbeCollector(data_source=DummyDataSource()).collect(registry)+ text = render_prometheus(registry)+ labels = 'pool="asyncio_default",process_role="legacy_server",worker="MainProcess"'+ expected = {+ "openviking_executor_max_workers": "4.0",+ "openviking_executor_threads": "2.0",+ "openviking_executor_active_tasks": "1.0",+ "openviking_executor_pending_tasks": "3.0",+ "openviking_executor_submitted_total": "5",+ "openviking_executor_completed_total": "4",+ "openviking_executor_failed_total": "1",+ }+ for name, value in expected.items():+ assert f"{name}{{{labels}}} {value}" in text+++async def test_executor_monitor_tracks_default_executor_only():+ loop = asyncio.get_running_loop()+ previous_default = getattr(loop, "_default_executor", None)+ default_executor = ThreadPoolExecutor(max_workers=1)+ custom_executor = ThreadPoolExecutor(max_workers=1)+ loop.set_default_executor(default_executor)+ monitor = install_executor_monitor(+ loop=loop,+ process_role="legacy_server",+ )+ started, release = Event(), Event()++ def blocking_default():+ started.set()+ release.wait(5)+ return "ok"++ def raising_default():+ raise RuntimeError("boom")++ try:+ first = loop.run_in_executor(None, blocking_default)+ while not started.is_set():+ await asyncio.sleep(0.01)+ second = loop.run_in_executor(None, lambda: "queued")++ metrics = monitor.read_metrics()+ assert metrics["active_tasks"] == 1+ assert metrics["pending_tasks"] == 1+ assert metrics["submitted_total"] == 2++ release.set()+ assert await first == "ok"+ assert await second == "queued"+ with pytest.raises(RuntimeError, match="boom"):+ await loop.run_in_executor(None, raising_default)+ assert await loop.run_in_executor(custom_executor, lambda: "custom") == "custom"++ final = monitor.read_metrics()+ assert final == {+ "process_role": "legacy_server",+ "worker": multiprocessing.current_process().name,+ "pool": "asyncio_default",+ "max_workers": 1,+ "threads": 1,+ "active_tasks": 0,+ "pending_tasks": 0,+ "submitted_total": 3,+ "completed_total": 3,+ "failed_total": 1,+ }+ finally:+ uninstall_executor_monitor()+ if previous_default is not None:+ loop.set_default_executor(previous_default)+ else:+ loop._default_executor = None # type: ignore[attr-defined]+ custom_executor.shutdown(wait=True)+ default_executor.shutdown(wait=True)++def test_queue_collector_maps_status(monkeypatch):class DummyQueueStatus:def __init__(